Skip to content
xflopsPublic

About

A distributed engine for elastic workload, e.g. AI Agent, RL, Quant

Topics

Resources

Contributing

Security policy

Stars

58 stars

Watchers

5 watching

Forks

Latest commit

 

History

1,338 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Flame: A Distributed Engine for Elastic Workloads

E2E license RepoSize Release Rust OpenSSF Best Practices

Flame is a distributed engine for elastic workloads, including AI agents, reinforcement learning, and quantitative computing. It provides a comprehensive suite of core mechanisms for running these workloads at scale. Built on over a decade and a half of experience running diverse high-performance workloads across multiple systems and platforms, Flame incorporates best-of-breed ideas and practices from the open source community.

Motivation

As AI workloads become increasingly adopted for innovation, a common runtime is essential to accelerate these elastic workloads through the following key aspects:

  • Scale: Unlike applications running on a single node, Flame scales workloads across multiple nodes to maximize performance acceleration while ensuring fair resource sharing across multiple tenants and sessions.
  • Performance: Elastic workloads typically involve tens of thousands of short tasks. Flame leverages cutting-edge features to improve roundtrip times and throughput in large-scale environments, while intelligently sharing runtime within sessions to minimize startup time.
  • Security: Flame utilizes microVM as a runtime for enhanced security, with each runtime environment (executor) dedicated to a single session to prevent data leakage. All Flame components communicate using mTLS for secure inter-component communication.
  • Flexibility: Flame defines a comprehensive set of general APIs to support multiple user scenarios. Additionally, Flame supports applications across multiple programming languages through gRPC, including Rust, Go, and Python.

Performance

Flame is designed for high-throughput task execution. Here's a benchmark running 30,000 tasks on a single-node deployment:

root@06383dd94875:/# flmping -p -t 30000
Session <flmping-1N1sIX> was created in <1 ms>, start to run <30,000> tasks in the session:

============================================================
BENCHMARK RESULTS
============================================================
Duration:        3.29s
Succeeded:       30000/30000
Failed:          0
Throughput:      9124.09 tasks/sec
============================================================

root@06383dd94875:/# flmctl list -s
 ID              State   App      Resources             Priority  Pending  Running  Succeed  Failed  Created
 flmping-1N1sIX  Closed  flmping  cpu=1,mem=1Gi,gpu=0  0         0        0        30000    0       18:11:26

Architecture Overview

Flame Architecture

Core Concepts

Session: A Session represents a group of related tasks. The Session Scheduler allocates resources to each session based on scheduling configurations by requesting the resource manager (e.g., Kubernetes) to launch executors. Clients can continuously create tasks until the session is closed.

Task: A task within a Session contains the main algorithm defined by the task's metadata and input/output information (e.g., volume paths).

Executor: The Executor manages the lifecycle of an Application/Service, which contains the user's code for executing tasks. An idle executor may retain its application instance and reuse it across compatible sessions until the executor is released.

Shim: The protocol implementation used by the Executor to manage applications, supporting various protocols such as gRPC, RESTful APIs, stdio, and more.

How It Works

Flame accepts connections from user clients and creates Sessions for jobs. Clients can continuously submit tasks to a session until it's closed, with no predefined replica requirements.

The Session Scheduler allocates resources to each session based on scheduling configurations by requesting the resource manager (e.g., Kubernetes) to launch executors.

Executors connect back to Flame via gRPC to pull tasks from their bound Session. After unbinding, an executor may be reused by another session of the same application; it is released after the application's configured idle delay or when the application is removed.

Services receive notifications when they're bound or unbound to related sessions, allowing them to take appropriate actions (e.g., connecting to databases). Services can then pull tasks from the Session and reuse data to accelerate execution.

Future enhancements to the Session Scheduler will include features to improve performance and usage, such as proportional allocation, delayed release, and min/max constraints.

Quick Start Guide

Option 1: Docker Compose (Recommended for First-Time Users)

This guide uses Docker Compose to start a local Flame cluster. After installing docker compose, you can start a local Flame cluster with the following steps:

$ docker compose up -d

After the Flame cluster is launched, use the following steps to log into the flame-console pod, which serves as a debug tool for both developers and SREs:

$ docker compose exec flame-console /bin/bash

Option 2: Local Installation with the install scripts (Faster for Development)

For development and testing, you can install Flame directly on your machine (building it needs Rust and uv):

# Build the release package from local source, then install a single node from it and start the services
$ make install

# Install the client (the tools and the Python SDK) and put it on your environment
$ target/flame-package/package/sbin/sdk.sh
$ source ~/.flame/sbin/flmenv.sh

For more details, see the install scripts README.

Verify the Installation

After starting Flame (via either option), verify the installation with flmping. Normal mode prints each task result as it completes, so rows can appear out of submission order. Use --perf for a final benchmark summary without per-task output.

$ flmping
Session <flmping-Sf4R2o> was created in <1 ms>, start to run <10> tasks in the session:

 Session         Task  State    Output
 flmping-Sf4R2o  1     Succeed  Completed on <396003ae48dd> in <0> milliseconds with <0> memory
 flmping-Sf4R2o  2     Succeed  Completed on <396003ae48dd> in <0> milliseconds with <0> memory
 flmping-Sf4R2o  3     Succeed  Completed on <396003ae48dd> in <0> milliseconds with <0> memory
 flmping-Sf4R2o  4     Succeed  Completed on <396003ae48dd> in <0> milliseconds with <0> memory
 flmping-Sf4R2o  5     Succeed  Completed on <396003ae48dd> in <0> milliseconds with <0> memory
 flmping-Sf4R2o  6     Succeed  Completed on <396003ae48dd> in <0> milliseconds with <0> memory
 flmping-Sf4R2o  7     Succeed  Completed on <396003ae48dd> in <0> milliseconds with <0> memory
 flmping-Sf4R2o  8     Succeed  Completed on <396003ae48dd> in <0> milliseconds with <0> memory
 flmping-Sf4R2o  9     Succeed  Completed on <396003ae48dd> in <0> milliseconds with <0> memory
 flmping-Sf4R2o  10    Succeed  Completed on <396003ae48dd> in <0> milliseconds with <0> memory


<10> tasks was completed in <153 ms>.

You can check session status using flmctl. Explore more examples here:

$ flmctl list -s
 ID              State   App      Resources             Priority  Pending  Running  Succeed  Failed  Created
 flmping-Sf4R2o  Closed  flmping  cpu=1,mem=1Gi,gpu=0  0         0        0        10       0       13:33:30

CLI Tools

flmctl is the command-line tool for submitting jobs, managing sessions, and querying cluster state.

Installing Flame with the install scripts

For multi-node bare-metal or VM deployments, run setup.sh on each node and sdk.sh on each client:

# On the head node
curl -fsSL https://xflops.io/flame/setup.sh | sudo sh

# On each worker node (the executor manager and the object cache); FLAME_* settings generate conf/flame-cluster.yaml
curl -fsSL https://xflops.io/flame/setup.sh | sudo FLAME_ENDPOINT=http://HEAD:8080 sh -s -- --join

# On a client machine (flmctl, flmping, flmexec, flmbench and the Python SDK); FLAME_ENDPOINT writes ~/.flame/flame.yaml
curl -fsSL https://xflops.io/flame/sdk.sh | FLAME_ENDPOINT=http://HEAD:8080 sh

For more details, see the install scripts README.

Documentation

API Reference

Contributing

We welcome contributions through GitHub issues and pull requests; see CONTRIBUTING.md for the process, the coding standards and the test policy. To report a security vulnerability, see SECURITY.md; docs/security.md describes the security design and how to deploy Flame safely.

License

This project is licensed under the terms specified in the LICENSE file.

Star History

Star History Chart

About

A distributed engine for elastic workload, e.g. AI Agent, RL, Quant

Topics

Resources

Contributing

Security policy

Stars

58 stars

Watchers

5 watching

Forks

Releases

Packages

Used by

Contributors

Languages