Skip to content

Commit f95c6a5

Browse files
mxfactorialclaude
andauthored
fleet deploy: the brute force, in pieces (#57)
scripts/fleet.py: six pieces, each one AWS call shape with tab-separated lines: listzipversions (the snapshot, once), listgerps, listzipfnsconf --gerp, status (the join), update (one UpdateFunctionCode per behind line), push. deploy.sh fleet is the laptop's composition; the yaml's is to be the same pipe. deploy.py loses push --all and image --all. No script learns where it runs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 parent 347155c commit f95c6a5

8 files changed

Lines changed: 450 additions & 68 deletions

File tree

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -146,7 +146,7 @@ for the http API surface, `tests/server/per_customer/` IS the per_customer stack
146146
## commands
147147

148148
- `bash scripts/test.sh` — run every module's local-mode suite; `--module <name>`, `--name <substring>`, `--env integ`, `--logs` available. see `tests/AGENTS.md` for authoring conventions
149-
- `bash scripts/deploy.sh status|push [--gerp <gerp_id>]` — artifact deploys (TF owns shape, S3 owns bytes), **no terraform anywhere in the push path**: the builder is `if package.json → zip the dir; elif .py → zip the dir + its transitively-resolved local imports; else exit 1` — **the import graph IS the bundle manifest** (shared `_helpers`/`*_rules`/vendored trees resolve off the code itself; no recipe files, no `archive_file` — those blocks are deleted). Bespoke needs are `match src_dir:` cases in `build_artifact` (scripts/deploy.py) that *do things first* — prune, stage — then call `build_node`/`build_py`; it's a build script, explicit cases at any count are fine. `status` = the `gerp:src-dir` tag query × live `CodeSha256` × artifact checksums (~12s, direction-aware); `push [--dirs …] [--notes …]` = deterministic zips → versioned artifacts + `provenance`/`release` annotations (operator-profile writes; org read) → `update-function-code` at the pinned version (tenant profile); `--deploy` builds and puts nothing, so the functions and the BFF take the artifacts already in the bucket (what `zip.sh lambda` + `upload.sh lambda` put); `--all` pushes every active gerp through its own `gerp-<id>` profile, whose region picks the bucket, and the BFF once. deployed state is never recorded — it's one `get-function` away. **functions SOURCE code from the bucket** (`data "aws_s3_object"` pins latest per function), so ANY applier — codebuild's bundled snapshot, a stale checkout — deploys bucket truth, never its local tree; a NEW function is push-then-apply (the data source fails the plan until its artifact exists). post-push, a plan shows function updates until the next apply — that's the state's `s3_object_version` pointer re-syncing FORWARD to the version the push already deployed (AWS doesn't expose code source, so state can't refresh it); the apply is a same-bytes no-op
149+
- `bash scripts/deploy.sh status|push [--gerp <gerp_id>]`, `deploy.sh fleet [--report]` (every gerp at once, scripts/fleet.py's pieces) — artifact deploys (TF owns shape, S3 owns bytes), **no terraform anywhere in the push path**: the builder is `if package.json → zip the dir; elif .py → zip the dir + its transitively-resolved local imports; else exit 1` — **the import graph IS the bundle manifest** (shared `_helpers`/`*_rules`/vendored trees resolve off the code itself; no recipe files, no `archive_file` — those blocks are deleted). Bespoke needs are `match src_dir:` cases in `build_artifact` (scripts/deploy.py) that *do things first* — prune, stage — then call `build_node`/`build_py`; it's a build script, explicit cases at any count are fine. `status` = the `gerp:src-dir` tag query × live `CodeSha256` × artifact checksums (~12s, direction-aware); `push [--dirs …] [--notes …]` = deterministic zips → versioned artifacts + `provenance`/`release` annotations (operator-profile writes; org read) → `update-function-code` at the pinned version (tenant profile); `--deploy` builds and puts nothing, so the functions and the BFF take the artifacts already in the bucket (what `zip.sh lambda` + `upload.sh lambda` put); `--all` pushes every active gerp through its own `gerp-<id>` profile, whose region picks the bucket, and the BFF once. deployed state is never recorded — it's one `get-function` away. **functions SOURCE code from the bucket** (`data "aws_s3_object"` pins latest per function), so ANY applier — codebuild's bundled snapshot, a stale checkout — deploys bucket truth, never its local tree; a NEW function is push-then-apply (the data source fails the plan until its artifact exists). post-push, a plan shows function updates until the next apply — that's the state's `s3_object_version` pointer re-syncing FORWARD to the version the push already deployed (AWS doesn't expose code source, so state can't refresh it); the apply is a same-bytes no-op
150150
- `bash scripts/deploy.sh image` — the agent-container deploy: build → push (auto-incremented immutable `vNN` tag) → CLI-update a gerp's runtime to the new digest, from the copy ECR replication makes in the gerp's own region, once it has arrived (FULL config carried — UpdateAgentRuntime replaces wholesale) → wait READY → re-pin the named endpoint (DEFAULT auto-tracks). The gerp is gradienterp by default (the dogfood takes an image first), `--gerp <id>` for another, `--all` for every active gerp — a fleet push is said, never implied; `--no-build` moves the named gerps onto the image already at the top of ECR, or `--no-build --tag vNN` onto that one. No terraform: the runtime tf reads `data.aws_ecr_image` `most_recent`, so a post-deploy plan is ALREADY clean (AgentCore exposes its config, unlike lambda code). No `agent_image_tag` var exists anymore; no fake gateway-target churn — routine image deploys change zero tf
151151
- `bash scripts/docker.sh --build|--run|--stop|--push <ecr-uri>` — the underlying agent-container build/run primitives (`deploy.sh image` drives build/push; use directly for local container smoke)
152152
- `bash scripts/local-dev.sh --install` — once per clone: creates `.venv` (python3.12), installs the tests', the agent container's and the local servers' python deps, and runs `npm ci` in each Node lambda. Every script that runs `.venv/bin/python` (`test.sh`, `deploy.sh`, `investigate.sh`) needs it

‎modules/agent/AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -171,7 +171,7 @@ westwood's runtime moved onto it, one tool-calling turn through `scripts/chat.sh
171171
posted on the pull request; then westwood back onto the latest `vNN` and the `pr-` tag deleted,
172172
since the runtime's terraform pins the newest image in ECR and a `pr-` tag must never be it. The
173173
environment, its secret and the deploy role's subject stay in place for it. Merge, then
174-
`bash scripts/deploy.sh image --all` from main moves the fleet onto the next `vNN`.
174+
`bash scripts/deploy.sh image --gerp <id>` moves one gerp onto the next `vNN`; the fleet at once is deploy.yaml's `runtimes` job (`-f image=build`), until the image × runtime loop joins scripts/fleet.py.
175175

176176
## tearing down
177177

‎modules/terraform/AGENTS.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,3 +101,24 @@ that already exists, and the apply then sets the retention. One gerp at a time i
101101
dir (`prod/per_customer`, `prod/init_customer`): the script's init points the dir's backend at
102102
that gerp's state. A fresh account needs no import. A function that moves from a bare
103103
`aws_lambda_function` onto this module moves by a `moved` block in the calling module.
104+
105+
## the fleet deploy
106+
107+
Every gerp at once is `bash scripts/deploy.sh fleet [--report] [--gerp a,b] [--dirs …]`, a pipe
108+
over `scripts/fleet.py`'s pieces, each one AWS call shape with tab-separated lines on stdout:
109+
`listzipversions` (the bucket's latest version and sha256 per zip, read once: the run's target),
110+
`listgerps` (every active row with an account), `listzipfnsconf --gerp X` (the gerp's functions
111+
by the `gerp:src-dir` tag with their `CodeSha256`, through the `gerp-X` profile), `status
112+
<snapshot>` (the join: `in-sync`, `behind` with the snapshot's version to move to, `left` for a
113+
function with no artifact or a src_dir another gerp carries and this one does not), `update` (one
114+
`UpdateFunctionCode` per `behind` line, a gerp outside us-east-1 from its region's replica),
115+
`push <dir>` (build and put with provenance, the only piece that zips). Ten gerps run at once
116+
through `xargs -P`; a failed move is a line and the exit code, and the others finish; the report
117+
is `.build/fleet.tsv`. A loop is one artifact type paired with one resource type, zip × lambda
118+
function here; another deployable is another loop after it, not a column in these lines.
119+
120+
The workflow yaml runs the same pipe on a runner. **Keep the two compositions identical in
121+
logic**: the same pieces in the same order with the same flags. A runner thing (`parallel` in
122+
the xargs slot, `::group::`, the step summary) is written in the yaml and never in a script, so
123+
no script under scripts/ reads `GITHUB_ACTIONS`. Two copies of a pipe are a hazard for people
124+
and a small one with agents reading both; the test that reads both holds them together.

‎scripts/deploy.py‎

Lines changed: 13 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -13,18 +13,17 @@
1313
states: in-sync / artifact-ahead (pushed, not deployed) / repo-ahead
1414
(local zip differs from artifact — push needed) / no-artifact.
1515
16-
push [--gerp G | --all] [--dirs modules/x/lambdas/y ...] [--notes "..."] [--deploy] [--profile P]
16+
push [--gerp G] [--dirs modules/x/lambdas/y ...] [--notes "..."] [--deploy] [--profile P]
1717
build each function's zip from its src dir (no terraform), then for each
1818
fleet function whose zip sha differs from the artifact's latest version:
1919
put-object (new version, sha256 checksum) + provenance annotation
2020
{src_dir, src_sha256, built_at} + release annotation (--notes; the
2121
agent-readable "what's in this version") + update-function-code from
2222
that exact version, then wait for LastUpdateStatus. `--deploy` builds and
2323
puts nothing: the functions (and the BFF) take the artifacts already in
24-
the bucket. `--all` pushes every active gerp through its gerp-<id> profile,
25-
the BFF once.
24+
the bucket. Every gerp at once is the fleet pipe, `deploy.sh fleet` (scripts/fleet.py).
2625
27-
image [--gerp G | --all] [--no-build [--tag vNN] | --tag pr-<n>-<sha>]
26+
image [--gerp G] [--no-build [--tag vNN] | --tag pr-<n>-<sha>]
2827
build, `upload image`, then each gerp's runtimes onto the digest from its
2928
own region's copy of the image, named endpoints re-pinned.
3029
@@ -76,7 +75,7 @@ def bucket_for(region: str) -> str:
7675
# `build_artifact` routes to `build_webapp`.
7776
WEBAPP_DIR = "prod/gradienterp_cloud/bff"
7877
# tower's own functions: the same bundle and bucket as the fleet's, in the OPERATOR account, so
79-
# they answer to no gerp's tag query. Pushed when named by `--dirs`, never by a bare push or --all
78+
# they answer to no gerp's tag query. Pushed when named by `--dirs`, never by a bare push
8079
TOWER_DIR = "prod/tower/lambdas/"
8180
WEBAPP_ROOT = os.path.dirname(WEBAPP_DIR)
8281
BFF_FN = f"{CONFIG['STACK_PREFIX']}-cloud-bff"
@@ -527,22 +526,13 @@ def _push_webapp(args):
527526

528527

529528
def cmd_push(args):
530-
"""The BFF, once, then the named gerp's fleet — or with `--all`, every active gerp's in turn."""
529+
"""The BFF, once, then the named gerp's fleet. Every gerp at once is the fleet pipe (fleet.py)."""
531530
dirs = list(args.dirs) if args.dirs else None
532-
if args.all and args.profile:
533-
sys.exit("--all reaches each gerp through its own gerp-<id> profile; --profile names one")
534531
tower_dirs = [d for d in (dirs or []) if d.startswith(TOWER_DIR)]
535-
if tower_dirs and args.all:
536-
sys.exit("tower's functions live in the operator account and take no --all; push them by --dirs alone")
537-
# the gerps are checked before anything builds or uploads, the BFF included; a push of the BFF
532+
# the gerp is checked before anything builds or uploads, the BFF included; a push of the BFF
538533
# or of tower alone reaches only the operator account and names no gerp
539534
fleet_dirs = None if dirs is None else [d for d in dirs if d != WEBAPP_DIR and not d.startswith(TOWER_DIR)]
540-
gerps = ([r["gerp_id"] for r in _active_rows(_boto(args.operator_profile))] if args.all else [args.gerp]) \
541-
if fleet_dirs != [] else []
542-
sessions = []
543-
for g in gerps:
544-
args.gerp, args.profile = g, (None if args.all else args.profile)
545-
sessions.append((g, _target_session(args)))
535+
sessions = [(args.gerp, _target_session(args))] if fleet_dirs != [] else []
546536
# The owner web app is outside the fleet enumeration below — `fleet()` is a tag query scoped to
547537
# the session's account and the BFF lives in the operator's, and its bundle needs the web/ files
548538
# `build_py` knows nothing about. So a bare push has to reach it EXPLICITLY: without this it
@@ -552,9 +542,7 @@ def cmd_push(args):
552542
if tower_dirs:
553543
# the same push as a fleet's, in the operator session: build, put, update-function-code
554544
_push_fleet(args, _boto(args.operator_profile), tower_dirs)
555-
for g, session in sessions:
556-
if args.all:
557-
print(f"==> {g}")
545+
for _g, session in sessions:
558546
_push_fleet(args, session, fleet_dirs)
559547

560548

@@ -777,9 +765,9 @@ def cmd_image(args):
777765
of any gerp agrees with what this just did.
778766
779767
Which gerp: the operator's own by default (`--gerp gradienterp`) — an image lands on the
780-
dogfood first, then `--gerp westwood-…` for the staging pair, then `--all` for every active
781-
gerp. A fleet-wide push is said, never implied. `--no-build` skips the build and moves the
782-
named gerps onto the image already at the top of ECR."""
768+
dogfood first, then `--gerp westwood-…` for the staging pair. The fleet at once is the
769+
runner's (deploy.yaml `runtimes`) until the image × runtime loop joins fleet.py. `--no-build`
770+
skips the build and moves the named gerp onto the image already at the top of ECR."""
783771
op = _boto(args.operator_profile)
784772
ecr = op.client("ecr")
785773
if args.tag and not args.no_build and re.fullmatch(r"v\d+", args.tag):
@@ -797,13 +785,13 @@ def cmd_image(args):
797785
print(f"==> {tag} = {digest}")
798786

799787
walked = 0
800-
for gerp_id, account, region, ses in _gerp_sessions(op, only=None if args.all else args.gerp):
788+
for gerp_id, account, region, ses in _gerp_sessions(op, only=args.gerp):
801789
print(f"==> {gerp_id} ({account}, {region})")
802790
if not image_replicated(op, region, digest):
803791
sys.exit(f"{gerp_id}: {tag} has not replicated to {region} yet — run again with --no-build")
804792
_update_runtimes(ses.client("bedrock-agentcore-control"), digest, gerp_id, region)
805793
walked += 1
806-
if not args.all and not walked:
794+
if not walked:
807795
sys.exit(f"no active gerp named {args.gerp} with an account")
808796
print(f"deployed {tag} to {walked} gerp(s) — a warm chat session stays on the old container; fresh sessions get {tag}")
809797

@@ -1054,13 +1042,11 @@ def main():
10541042
ps.add_argument("--deploy", action="store_true",
10551043
help="no build: point the live functions at the bucket's current artifacts (build + put alone is zip.sh then upload.sh)")
10561044
ps.add_argument("--gerp", default="gradienterp", help="the gerp whose functions take the push (default: the operator's own)")
1057-
ps.add_argument("--all", action="store_true", help="every active gerp, each through gerp-<id> — said, never implied")
10581045
ps.add_argument("--profile", default=None, help="the profile reaching it (default: gerp-<gerp>)")
10591046
ps.add_argument("--operator-profile", default="operator-org")
10601047
ps.set_defaults(fn=cmd_push)
10611048
im = sub.add_parser("image")
10621049
im.add_argument("--gerp", default="gradienterp", help="the gerp whose runtime takes the image (default: the operator's own)")
1063-
im.add_argument("--all", action="store_true", help="every active gerp — said, never implied")
10641050
im.add_argument("--no-build", action="store_true", help="no build: move the named gerps onto the image already at the top of ECR")
10651051
im.add_argument("--tag", help="with --no-build: the pushed vNN to move onto, rather than the top of ECR; "
10661052
"with a build: push under this tag instead of the next vNN (a pull request's pr-<n>-<sha>)")

‎scripts/deploy.sh‎

Lines changed: 36 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,51 @@
11
#!/usr/bin/env bash
2-
# Artifact deploys — thin wrapper over scripts/deploy.py (see its docstring).
2+
# Artifact deploys — thin wrapper over scripts/deploy.py (see its docstring), and the fleet pipe.
33
#
44
# bash scripts/deploy.sh status # gradienterp's gerp, through gerp-gradienterp
55
# bash scripts/deploy.sh status --gerp westwood-c40fd8 --all
66
# bash scripts/deploy.sh push --dirs modules/shipping/lambdas/manage_shipments --notes "..."
77
# bash scripts/deploy.sh push --gerp westwood-c40fd8 # whole fleet of one gerp
8+
# bash scripts/deploy.sh push --deploy --dirs <src-dir> # no build: take the artifact upload.sh put
9+
# bash scripts/deploy.sh image [--gerp <id>] [--no-build]
10+
# bash scripts/deploy.sh fleet [--report] [--gerp a,b] [--dirs <src-dir>…]
811
#
912
# status and push refuse a profile whose account isn't the gerp row's; `bash scripts/awsacct.sh --all`
1013
# writes the gerp-<gerp_id> profiles.
11-
# bash scripts/deploy.sh push --all --dirs <src-dir> # every active gerp, the BFF once
12-
# bash scripts/deploy.sh push --deploy --dirs <src-dir> # no build: take the artifact upload.sh put
13-
# bash scripts/deploy.sh image [--gerp <id> | --all] [--no-build]
14+
#
15+
# `fleet` is the laptop's composition of scripts/fleet.py's pieces (issue #57): the snapshot of the
16+
# bucket once, every active gerp's functions ten at a time, the compare, and one update per function
17+
# that differs; `--report` stops after the compare, and the report is .build/fleet.tsv either way.
18+
# The workflow yaml runs the same pipe on a runner (deploy.yaml): keep the two identical in logic —
19+
# the same pieces in the same order with the same flags; a runner thing (`parallel`, `::group::`,
20+
# the step summary) is written in the yaml, never here.
1421
# the zips are zip.sh's and the puts upload.sh's; a gerp's stack, its stop and its start are apply.sh's.
1522
set -euo pipefail
1623
cd "$(dirname "$0")/.."
17-
# the laptop's .venv; python3 where there is none (a GitHub runner with the requirements installed)
1824
PY=python3
1925
[[ -x .venv/bin/python ]] && PY=.venv/bin/python
26+
27+
if [[ "${1:-}" == fleet ]]; then
28+
shift
29+
report=0; only=""; dirs=()
30+
while [[ $# -gt 0 ]]; do
31+
case "$1" in
32+
--report) report=1; shift ;;
33+
--gerp) only="$2"; shift 2 ;;
34+
--dirs) shift; while [[ $# -gt 0 && "$1" != --* ]]; do dirs+=("$1"); shift; done ;;
35+
*) echo "fleet: unknown argument $1" >&2; exit 2 ;;
36+
esac
37+
done
38+
mkdir -p .build
39+
snapshot=.build/fleet-snapshot.tsv
40+
"$PY" scripts/fleet.py listzipversions > "$snapshot" # once, before any account: the run's target
41+
echo "==> snapshot: $(wc -l < "$snapshot" | tr -d ' ') artifacts" >&2
42+
"$PY" scripts/fleet.py listgerps | cut -f1 \
43+
| { if [[ -n "$only" ]]; then grep -Fx -f <(tr ',' '\n' <<< "$only"); else cat; fi; } \
44+
| xargs -P 10 -I{} "$PY" scripts/fleet.py listzipfnsconf --gerp {} \
45+
| "$PY" scripts/fleet.py status "$snapshot" ${dirs[@]:+--dirs "${dirs[@]}"} \
46+
| tee .build/fleet.tsv \
47+
| { if [[ $report -eq 1 ]]; then cat; else "$PY" scripts/fleet.py update; fi; }
48+
exit
49+
fi
50+
2051
exec "$PY" scripts/deploy.py "$@"

0 commit comments

Comments
 (0)