This runbook is written for a release agent. Follow it as a checklist and record the exact versions, commit SHAs, registry URLs, and verification results before calling the release complete.
Set these values at the start of every release:
export RELEASE_TAG=v0.6.0-rc1
export CARGO_VERSION=0.6.0-rc1
export PYTHON_VERSION=0.6.0rc1
export STDNG_VERSION=0.1.8
export DOCKER_TAG="${RELEASE_TAG}"
export RELEASE_BRANCH=release-0.6
export IMAGE_REGISTRY=docker.io/xflops
export RELEASE_IMAGE_PLATFORMS=linux/amd64,linux/arm64
export RUST_BUILDER_IMAGE=docker.io/library/rust:1.100.0
export UBUNTU_BASE_IMAGE=docker.io/library/ubuntu:24.04
# Optional override; Makefile auto-detects usable podman first, then docker.
# export CONTAINER_CLI=podman
export RELEASE_NOTES_FILE=/tmp/flame-${RELEASE_TAG}-notes.mdVersion conventions:
- GitHub release tags and Docker tags use the leading
v:v0.6.0-rc1. - Cargo package versions omit the leading
v:0.6.0-rc1. - PyPI versions follow PEP 440:
0.6.0rc1instead of0.6.0-rc1. - Stable releases use
v0.6.0,0.6.0, and0.6.0. - Published Cargo and PyPI versions are immutable. If one artifact has already been published incorrectly, bump the next release candidate and document the mapping instead of overwriting.
Confirm source and package credentials before changing versions:
git fetch upstream --tags --prune
git status --short
gh auth status
cargo owner --list flame-rs
test -f ~/.pypirc || test -n "${UV_PUBLISH_TOKEN:-}" || test -n "${UV_PUBLISH_PASSWORD:-}"Confirm container image credentials with the tool selected for the image publish step:
make release-images-loginRequired permissions:
- Push access to
upstreamand the release branch. - GitHub release/tag permission for
xflops/flame. - crates.io publish permission for
stdng,flame-rs-macros, andflame-rs. - PyPI publish permission for
flamepy. - Docker Hub publish permission for
xflops/flame-*repositories.
The GitHub workflows require these repository or inherited organization credentials:
| Name | Kind | Required access |
|---|---|---|
PYPI_API_TOKEN |
Actions secret | PyPI project-scoped token allowed to publish flamepy |
CARGO_REGISTRY_TOKEN |
Actions secret | crates.io token allowed to publish stdng, flame-rs-macros, and flame-rs |
DOCKER_HUB_PAT |
Actions secret | Docker Hub personal access token with read/write access to the xflops/flame-* repositories |
DOCKER_HUB_USERNAME |
Actions variable | Docker Hub username that owns the PAT and can push to the xflops organization |
Configure them under Settings → Secrets and variables → Actions. The
repository already exposes DOCKER_HUB_PAT; confirm the username variable and
the two package-registry tokens before publishing. GitHub supplies the
read-only GITHUB_TOKEN automatically.
Keep local tasks/ notes out of commits and container build contexts.
.dockerignore should include tasks/.
If a release event was missed or a publish needs an idempotent recovery run, dispatch the workflow against the existing release tag:
gh workflow run release-publish.yaml --ref main \
-f release_tag="${RELEASE_TAG}"Versioned releases use build-images.yaml to build and publish native amd64 and
arm64 images. Main follows the artifact flow below instead.
Flame CI (e2e.yaml) is the single entry point for main pushes, PRs and manual runs.
It first completes the code, script, publication-rule and CodeQL verification jobs.
Image and package builds depend only on those verification jobs. CodeQL also
retains its independent weekly schedule without starting builds.
Architecture and scheduling-policy variants are explicit parallel jobs. YAML step anchors share identical commands, and a composite action shares the CodeQL implementation. There are no matrix or reusable-workflow job wrappers in main CI; the dashboard shows the actual jobs and their direct dependencies.
The build stage creates all eight image variants on native amd64 and arm64
runners, runs their
binary and SDK smoke checks, and saves the images with their source commit,
architecture, image IDs and archive checksum. Packages are built once with the
latest tag on Ubuntu 24.04 for both architectures, preserving the glibc
baseline. The build stage verifies checksums and smoke-tests package installation.
The E2E, system and benchmark jobs consume those build outputs. Full E2E coverage runs on amd64; arm64 receives native build and smoke coverage. Both publication jobs depend only on the two terminal benchmarks, whose dependencies cover every verification, build, E2E and system job. Each stage names only its immediate predecessors. A failed, cancelled or skipped dependency prevents publication. Pull requests never publish.
After success, the image and package publication jobs download artifacts from
that same run and do not rebuild
images, server binaries or SDK wheels. Image publication validates the source
commit, architecture, checksum and image IDs before pushing, then merges the two
native manifests. Both image transfers, manifest creation and promotion run as
steps of one publication job; no native architecture matrix is needed to upload
already-built images. Package publication verifies the complete asset set and its
checksums before changing the release. Mutable latest outputs are updated only
while the run's commit is still the head of main.
To retry, rerun the failed jobs of the original CI run while its artifacts remain
available. To start a new run, dispatch e2e.yaml against main; this runs the full
validation graph before publication. Publication cannot be dispatched independently.
Versioned release publication remains separate.
-
Start from the release branch or create it from the intended base:
git switch "${RELEASE_BRANCH}" git pull --ff-only upstream "${RELEASE_BRANCH}"
-
Confirm all required source PRs have landed on
mainfirst. Release branch changes should usually be cherry-picks from main, not release-only feature work. -
For each backport PR, verify cherry-pick hygiene:
git range-diff <source-sha>^..<source-sha> <backport-sha>^..<backport-sha> git show <source-sha> -- | git patch-id --stable git show <backport-sha> -- | git patch-id --stable
The backport should contain only the intended cherry-picked commits plus the standard cherry-pick trailer. Do not mix release notes or unrelated fixes into a mechanical backport PR.
-
Verify CI is green for the release branch. At minimum, inspect GitHub checks for the final release commit and confirm Kubernetes E2E passed when the release includes Helm or image changes.
Update the release versions in the files that apply to the release:
sdk/python/pyproject.toml:project.version = "<PYTHON_VERSION>"sdk/python/src/flamepy/__init__.py:__version__ = "<PYTHON_VERSION>"sdk/python/README.md: install examples or version references, if presentsdk/rust/macros/Cargo.toml:version = "<CARGO_VERSION>"stdng/Cargo.toml: bump only when publishing stdng changessdk/rust/Cargo.toml:version = "<CARGO_VERSION>"sdk/rust/Cargo.toml: dependency versions forstdngandflame-rs-macroscharts/flame/Chart.yaml:appVersionfor the Flame releaseCargo.lock: refresh after Cargo metadata changes
Publish order matters for Rust:
stdng, if its version changed.flame-rs-macros.flame-rs, after its registry dependencies exist.
Do not publish flame-rs with duplicated helper code if the helpers belong in
stdng; publish the required stdng version first and depend on it from
sdk/rust/Cargo.toml.
Create the release notes before publishing so package and GitHub metadata use the same wording. The notes should include:
- The source comparison range, previous release tag, and target commit.
- User-facing highlights, breaking changes, and upgrade notes.
- Package, crate, Docker image, and Helm chart version values.
- Known gaps, such as local smoke tests that could not be run.
- Links to source PRs. Backport PRs should be cited only when the backport itself has user-facing behavior.
Write the final body to ${RELEASE_NOTES_FILE} and use that same file for the
GitHub release.
Run the focused checks before publishing anything:
cargo fmt --check
cargo check -p stdng
cargo check -p flame-rs --features macros
cargo package --manifest-path stdng/Cargo.toml --allow-dirty
cargo package --manifest-path sdk/rust/macros/Cargo.toml --allow-dirty
cargo package --manifest-path sdk/rust/Cargo.toml --allow-dirty --features macrosIf flame-rs depends on a flame-rs-macros version that has not been
published yet, the final flame-rs package verification will fail while
resolving registry dependencies. In that case, publish and verify
flame-rs-macros first, then rerun the full flame-rs package command before
publishing flame-rs.
Python package verification:
cd sdk/python
uv run -n --extra dev pytest tests/test_app.py -q
uv run -n python -c 'import flamepy; print(flamepy.__version__)'
uv build --out-dir /tmp/flamepy-${PYTHON_VERSION}-dist
cd -Repository checks:
git diff --check
bash -n ci/k8s/e2e.sh
python3 -m json.tool charts/flame/values.schema.jsonThe non-publishing release sanity script wraps the metadata, local, package, and optional artifact checks:
make release-sanityIf helm and a local Kubernetes backend are available, also run:
helm lint charts/flame
helm template flame charts/flame --set global.imageTag="${DOCKER_TAG}"Build artifacts from sdk/python and publish exactly those files:
cd sdk/python
uv build --out-dir /tmp/flamepy-${PYTHON_VERSION}-dist
uv publish /tmp/flamepy-${PYTHON_VERSION}-dist/*
cd -If uv publish cannot find credentials, set UV_PUBLISH_USERNAME and
UV_PUBLISH_PASSWORD from a secure source. Do not print tokens or .pypirc
contents in logs.
Verify PyPI:
curl -fsSL "https://pypi.org/pypi/flamepy/${PYTHON_VERSION}/json" \
| python3 -m json.tool
curl -fsSL "https://pypi.org/simple/flamepy/" | rg "${PYTHON_VERSION}"For release candidates, PyPI may keep the project-level latest version on the latest stable release. That is expected.
Publish crates in dependency order. Skip the stdng publish command if the
required ${STDNG_VERSION} already exists on crates.io and no stdng changes are
part of the release.
cargo publish --manifest-path stdng/Cargo.toml --allow-dirty
cargo publish --manifest-path sdk/rust/macros/Cargo.toml --allow-dirty
cargo publish --manifest-path sdk/rust/Cargo.toml --allow-dirty --features macrosAfter each publish, wait for the registry to expose the version before publishing the dependent crate:
curl -fsSL "https://crates.io/api/v1/crates/stdng/${STDNG_VERSION}" \
| python3 -m json.tool
curl -fsSL "https://crates.io/api/v1/crates/flame-rs-macros/${CARGO_VERSION}" \
| python3 -m json.tool
curl -fsSL "https://crates.io/api/v1/crates/flame-rs/${CARGO_VERSION}" \
| python3 -m json.toolFor flame-rs, also verify the crates.io dependency list includes the expected
published stdng and flame-rs-macros versions.
Release Docker images as multi-arch manifest tags for linux/amd64 and
linux/arm64. The public repositories are:
xflops/flame-session-managerxflops/flame-object-cachexflops/flame-executor-managerxflops/flame-executor-manager:<tag>-devel(the executor manager plus a C toolchain and Python headers for GPU applications that use Triton, such as vLLM)xflops/flame-consolexflops/flame-instance-runtime(CRI application runtime)xflops/flame-instance-runtime:<tag>-devel(the runtime plus the same C toolchain and Python headers)
The release workflow publishes only the versioned release tag. Among automated
workflows, only the main-branch image workflow moves the mutable latest tag.
It promotes the five images after every SHA-tagged image is verified. The
flame-instance-runtime image is only consumed by CRI applications, but it is
also versioned by the release workflow so CRI users can pin a release. It is not
a Flame control-plane service and must not be added to Compose service checks.
Registries cannot atomically update tags across repositories, so rerun a failed
main image workflow to complete a partially interrupted promotion.
Build release images with manifest lists. The Makefile detects a
Docker-compatible CONTAINER_CLI from the host, preferring a usable Docker
daemon and falling back to Podman when Docker is unavailable. Set
CONTAINER_CLI=docker or CONTAINER_CLI=podman when you need to override that
choice. The selected CLI must support build --manifest and manifest inspect/push
for the release image targets.
Set RUST_BUILDER_IMAGE to the Rust builder image used by the release
Dockerfiles. Do not use rust:latest for release validation because it can drift
from the image build inputs.
Container CLI prerequisites:
make release-images-check-cliIf the amd64 Rust smoke test fails under emulation, do not publish a stable
arm64-only tag by default. Use a Podman farm or remote Podman connection with a
native amd64 builder, or document the Docker release as blocked. If the release
owner explicitly narrows the Docker scope to an arm64-first publish, push only
the versioned arm64 tags, leave latest untouched, and record the missing
amd64/multi-arch artifacts as a release gap. Set
RELEASE_IMAGE_PLATFORMS=linux/arm64 before running the image Make targets for
that scoped build.
Build both platforms into local manifests, inspect them, and push the manifest lists:
make release-imagesTo split the image path into smaller steps, run:
make release-images-build
make release-images-inspect
make release-images-pushVerify the registry exposes every platform listed in RELEASE_IMAGE_PLATFORMS
for all five images:
make release-images-verifyAfter the image tags and PyPI package are published, run the Docker Compose
release smoke check. It pulls the target image tag, starts a compose cluster, and
runs the repository-level App E2E check from a clean Python image that installs
flamepy==${PYTHON_VERSION} from PyPI instead of using the SDK preinstalled in
Flame images:
RELEASE_SANITY_LOCAL_CHECKS=0 \
RELEASE_SANITY_PACKAGE_CHECKS=0 \
RELEASE_SANITY_REMOTE_CHECKS=1 \
RELEASE_SANITY_COMPOSE_E2E=1 \
make release-sanitySet RELEASE_SANITY_COMPOSE_DOWN=0 only when you need to inspect the compose
cluster after a failed run.
The compose smoke uses the TLS settings in ci/docker/flame-cluster.yaml and
ci/docker/flame.yaml. If ci/docker/certs does not already contain release-test
certificates, generate them before running the compose sanity check:
ci/docker/generate-certs.sh --output ci/docker/certs \
--san-list localhost,127.0.0.1,flame-session-manager,flame-object-cache \
--ip-range 172.20.0.0/24For an explicitly scoped arm64-first Docker release, keep the compose smoke versioned-tag-only and set the expected platform list:
RELEASE_SANITY_EXPECTED_PLATFORMS=linux/arm64 \
RELEASE_SANITY_LOCAL_CHECKS=0 \
RELEASE_SANITY_PACKAGE_CHECKS=0 \
RELEASE_SANITY_REMOTE_CHECKS=1 \
RELEASE_SANITY_COMPOSE_E2E=1 \
make release-sanityIf Docker Hub times out while pulling base images, retry the base image pull for the affected platform before rebuilding. Use the matching tool for the selected build path:
make release-images-pull-basesThe Helm chart defaults to:
global.imageRegistry: xflopsglobal.imageTag: latest- component repositories matching the Docker Hub names above
For release validation, install with the versioned tag:
helm template flame charts/flame --set global.imageTag="${DOCKER_TAG}"
helm install flame charts/flame \
--namespace flame --create-namespace \
--set global.imageTag="${DOCKER_TAG}"
helm test flame --namespace flameFor Kind-based validation, load or pull the four versioned images and run:
IMAGE_REGISTRY="${IMAGE_REGISTRY}" IMAGE_TAG="${DOCKER_TAG}" ci/k8s/e2e.shCapture the helm test, flmctl list, flmctl list -n, flmping, and Python
Pi example output when reporting the result.
The release workflow's publish-binaries job builds, on ubuntu-24.04 and ubuntu-24.04-arm
runners (so the binaries run on the oldest glibc we support), these assets for amd64 and arm64, each
with a .sha256 file, and uploads them to the GitHub release with gh release upload --clobber, so the
release must exist first and a rerun replaces the assets:
flame-<tag>-linux-<arch>.tar.gz: every binary,uv, the Python SDK wheel, the application manifests, the examples (what git tracks underexamples/), andsbin/setup.shandsbin/sdk.shthat install it, andflmctl-<tag>-linux-<arch>,flmping-<tag>-linux-<arch>,flmexec-<tag>-linux-<arch>andflmbench-<tag>-linux-<arch>: the client tools, also published on their own.
scripts/build.sh --output DIR --tag TAG makes them (it builds everything). The job smoke-tests the
assets the way a user installs them: the checksums, then the hosted setup.sh piped to sh
installing the archive from a local URL (head, then --join), then the package's own sdk.sh, then
flmctl --version and import flamepy. sdk.sh downloads that package archive and installs the client tools, the examples and the flamepy wheel
from it, so only the wheel's dependencies come from PyPI.
setup.sh and sdk.sh are not release assets: the publish-scripts job (after the assets and flamepy)
tests them, inlines the shared code with this release as the default version
(scripts/inline.sh --version "$RELEASE_TAG"), and pushes them to the xflops/xflops.github.io
repository: flame/<tag>/setup.sh and sdk.sh for every release, and flame/setup.sh and sdk.sh
(the latest) for a stable release, not a release candidate. It needs the XFLOPS_SITE_TOKEN secret, a
token with write access to that repository.
Tag only the final verified release commit:
git status --short
git tag -a "${RELEASE_TAG}" -m "Release ${RELEASE_TAG}"
git push upstream "${RELEASE_TAG}"Create the GitHub release after package and image URLs are verified:
gh release create "${RELEASE_TAG}" \
--repo xflops/flame \
--target "${RELEASE_BRANCH}" \
--title "${RELEASE_TAG}" \
--notes-file "${RELEASE_NOTES_FILE}" \
--prereleaseOmit --prerelease for stable releases.
Before declaring the release complete, record:
- Git tag and commit SHA.
- GitHub release URL.
- PyPI
flamepyversion URL and uploaded files. - crates.io URLs for
stdng,flame-rs-macros, andflame-rs. - Docker Hub tag URLs and manifest digests for all five images.
- The release assets: both architectures' package archives and client tools, with their
.sha256files. - CI run URLs for the final release commit.
- Helm/Kubernetes smoke-test output or the reason it was not run locally.
The release is not complete until every intended artifact has a verified remote URL or the missing artifact is explicitly documented as blocked.
- Do not overwrite Cargo or PyPI versions. Publish a new release candidate.
- Do not force-push public release tags without release-owner approval.
- Do not move
latestfrom the release workflow; automated promotion belongs to main-branch image publishing. - Do not reduce E2E coverage to make a release pass.
- If a release branch needs a fix, land the source change on
mainfirst when practical, then create a dedicated cherry-pick PR to the release branch.