Repository navigation
GitLab runner VMs: graceful drain + fleet-aware delete + --recreate (drained recreation as the lifecycle path) #7257
Description
Summary
Make drained recreation the promoted lifecycle path for GitLab runner VMs. Today delete-*-vm.sh hard-kills the VM (oc delete vm --wait=false / gcloud compute instances delete) with no drain, so any job running on that VM is cut off; and recreating requires a two-command dance (delete-*-vm.sh then create-*-vm.sh) with two different credentials. This issue adds a graceful drain, makes delete fleet-aware, and adds --recreate to the create scripts so recreation is a single, credential-consistent, job-safe command.
Background / why
Both runner fleets use the GitLab Runner Manager model: a single permanent runner entry (gitlab.com 56125242, gitlab.cee 68746, both fullsend-shared-runner, tag fullsend-gitlab-runner) backed by a pool of VMs sharing one authentication token, distinguished by system_id. Recreating a VM is therefore invisible at the fleet level (stable runner ID, other VMs keep serving) — except that deleting a VM kills whatever job is running on it at that instant. Recreation is the natural compliance/update path (clean disk; for GCP a fresh VM is born SA-less once #7254 lands), so it should be graceful.
Measured on both fleets (runner-01, 2026-09-12): KillSignal=SIGTERM, systemd TimeoutStopUSec=45s with TimeoutStopFailureMode=abort (Fedora 10-timeout-abort.conf — overrun ⇒ SIGABRT + coredump), unit Restart=always/RestartSec=120, config shutdown_timeout = 0 (⇒ runner default ~30s), gitlab-runner 19.2.1. So there is effectively no usable drain today: a systemctl stop cuts jobs at ~30–45s and can crash the runner. The implementation must not naively rely on systemctl stop.
Scope
1. hack/gitlab-runner-vm/lib.sh: drain_runner_vm helper (+ lib_test.sh tests)
- Gracefully stop the runner on a target VM so it stops accepting new jobs and lets in-flight jobs finish, bounded by a cap (default 10 min).
- Policy: drain-then-proceed — on cap overrun, warn and continue (do not hang forever).
- Runs from the operator side over the scripts' existing SSH plumbing (
virtctl … sshfor OpenShift,gcloud compute ssh --tunnel-through-iapfor GCP); detect idle by watching for per-jobrunner-*containers (podman ps) draining, not just process exit. - Caveats to handle (discovered above): use the graceful signal (SIGQUIT) rather than a bare
systemctl stopthat trips the 45s abort; account forRestart=always;shutdown_timeout=0means the runner's own wait is only ~30s. (For drain-before-delete theRestart=alwaysrestart is moot since the VM is torn down next.)
2. delete-gcp-vm.sh + delete-openshift-vm.sh: fleet-aware, and drain before delete
- Fleet-awareness (both conditions): skip the runner deregistration step when either
RUNNER_TOKENis set or no runner matching theproj/vmdescription is found; and only requireGL_TOKENwhen a deregistration will actually occur. In the fleet model the shared runner (fullsend-shared-runner) never matchesproj/vm, so deregistration is a guaranteed no-op and the PAT should not be required to delete a fleet VM (GitLab prunes the offlinesystem_id). - Drain before delete: call
drain_runner_vmprior to the VM/instance deletion; add a--no-drainescape hatch (and honor the cap → proceed).
3. create-gcp-vm.sh + create-openshift-vm.sh: --recreate flag
- When the target VM already exists (currently a hard error, e.g. create-gcp-vm.sh:335),
--recreateperforms drain → delete → create in one command, reusing the create script's own environment (in fleet/RUNNER_TOKENmode this needs no extra credential, because delete no longer requiresGL_TOKEN— see §2). - Without
--recreate, keep the current refuse-on-existing behavior.
4. Header/usage comments (this issue's scripts)
- Update the top-of-file usage comments in
create-*-vm.shanddelete-*-vm.shto describe what each can do and how (fleet vs individual mode,--recreate,--no-drain, the drain cap, which credentials are required in which mode).
Acceptance criteria
-
drain_runner_vmexists inlib.sh, unit-tested inlib_test.sh; graceful, bounded (10-min default), drain-then-proceed. - Deleting a fleet VM (RUNNER_TOKEN mode or no
proj/vmmatch) succeeds withoutGL_TOKEN, and does not touch the shared fleet runner. -
delete-*-vm.shdrains the VM before deletion;--no-drainskips it. -
create-*-vm.sh --recreatedrains, deletes, and recreates an existing VM using only create's env in fleet mode; absent the flag, existing behavior (refuse) is unchanged. - Recreating a fleet VM leaves the shared runner ID stable and does not cut a job that finishes within the drain cap.
- Usage comments in create-/delete- describe capabilities, flags, modes, and required credentials.
Out of scope
- runner-hub README updates (handled separately, in those repos).
- setup.sh idempotency/hygiene (tracked in the companion issue).
- GCP VM service-account removal (GCP GitLab runners: remove unnecessary over-privileged VM service account (metadata SA-token theft) #7254).
- Concurrency / per-slot gateways (still out of scope; single job per VM assumed).
Companion issue: setup.sh idempotency + hygiene. Related: #7218 (ephemeral per-job gateway, merged), #7254 (GCP VM SA removal).
Activity
ggallen commentedon Sep 12, 2026
Companion issue: #7258 (setup.sh idempotency + hygiene).
fullsend-ai-triage commentedon Sep 12, 2026
🤖 Finished Triage · ✅ Success · Started 1:41 PM UTC · Completed 1:45 PM UTC
Commit:
ce5f805· View workflow run →Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $0.44
fullsend-ai-triage commentedon Sep 12, 2026
This is a thorough, self-contained feature spec — background, measured drain-timing constraints, scope, and acceptance criteria are all already documented in the issue, so this summary focuses only on what's new.
No open PR or duplicate issue was found for this work. The companion issue (#7258, setup.sh idempotency) is explicitly out of scope here and vice versa, so the two remain separate as intended.
The issue references #7254 (GCP VM service-account removal) as background — noting a fresh VM will be SA-less "once #7254 lands" — but that is contextual motivation, not a functional dependency: the drain/fleet-aware-delete/--recreate mechanics described here do not require #7254 to merge first, so this issue is not blocked on it.
Suggested validation coverage (beyond the stated acceptance criteria) is included in the structured summary: unit tests for drain_runner_vm's cap-overrun and SIGQUIT-vs-systemctl-stop behavior, and integration-style checks that delete's GL_TOKEN requirement and create's --recreate composition behave as specified in both fleet and individual modes.
Labels: Matches the established labeling pattern for similar GitLab-runner-VM feature issues (#7196, #6811): a new-capability request scoped to the runner lifecycle, awaiting human prioritization.
Next steps:
/fs-code— agent creates a PR to implement this issue/fs-code <your instruction>— agent implements with your specific guidanceggallen commentedon Sep 12, 2026
/fs-code
fullsend-ai-coder commentedon Sep 12, 2026
🤖 Finished Code · ❌ Failure (post-script /home/runner/work/fullsend/fullsend/.fullsend/.fullsend-cache/resources/sha256/47106004ec3cf7c0276a1222d251ee87ee3a18a76d0f02142777f77912edc34a/scripts/post-code.sh failed: exit status 1) · Started 1:51 PM UTC · Completed 2:54 PM UTC
Commit:
e244970· View workflow run →Runtime: pi · Model: xai/grok-4.6 · Effort: high · Cost: $17.85
fullsend-ai-coder commentedon Sep 12, 2026
The code agent left uncommitted changes in the extracted repo and produced no commit to push. The run was likely killed (timeout) before
git commitexecuted, so this work was discarded and no PR was opened.This is not a successful no-op: staged or untracked files were present when the sandbox was extracted.
Workflow run: https://github.com/fullsend-ai/fullsend/actions/runs/34697524737
Details:
Uncommitted files:
M hack/gitlab-runner-vm/README.md
M hack/gitlab-runner-vm/create-gcp-vm.sh
M hack/gitlab-runner-vm/create-openshift-vm.sh
M hack/gitlab-runner-vm/delete-gcp-vm.sh
M hack/gitlab-runner-vm/delete-openshift-vm.sh
M hack/gitlab-runner-vm/lib.sh
M hack/gitlab-runner-vm/lib_test.sh
Retry with
/fs-codeif appropriate.ggallen commentedon Sep 12, 2026
/fs-code
The design is fully specified in the issue — implement exactly that, don't re-investigate the runner drain settings, fleet model, or config; treat the issue as the spec. Commit each file as you finish it rather than batching one commit at the end. Run pre-commit run --all-files early, and if hook installation fails with a network error (e.g. HTTP 403 on git fetch), retry it — it's transient, not a code problem. Write any shell in Tirith's dialect: test -n "$X" not [ -n "$X" ], and two-step arithmetic (n=$(date +%s); e=$(( n - START ))) rather than$( ) inside $ (( )).
fullsend-ai-coder commentedon Sep 12, 2026
🤖 Finished Code · ✅ Success · Started 4:28 PM UTC · Completed 5:31 PM UTC
Commit:
e244970· View workflow run →Runtime: pi · Model: xai/grok-4.6 · Effort: high · Cost: $4.82
feat(#7257): fleet-aware drain-before-delete for GitLab runner VMs
fix(#7257): insert shutdown_timeout in global table, validate GCP_USE…
fix(#7257): insert shutdown_timeout at start of config.toml, not befo…
feat(#7257): fleet-aware drain-before-delete for GitLab runner VMs
fix(#7257): confirm service stopped before idle, retry drain polls, p…
test(#7257): pin config ownership/mode restore in drain_runner_vm cov…
chore(fullsend-ai#7258): make gitlab-runner setup.sh idempotency expl…
Merge pull request fullsend-ai#7262 from fullsend-ai/agent/7257-gitla…
fix(fullsend-ai#7405): steer create-vm errors to drain-safe delete