From a57e6f4a56d60adb96494e38cea09fe7fad9d046 Mon Sep 17 00:00:00 2001 From: Maya Wang Date: Mon, 7 Sep 2026 14:53:23 -0700 Subject: [PATCH 1/2] docs: warn that worker node pools must not auto-upgrade or use spot An actor that is awake when its worker pod goes away moves to ACTOR_STATE_CRASHED, which is terminal: resume and suspend are both refused, there is no recover verb, and the snapshot the actor still holds cannot be used to start it. GKE enables node auto-upgrade by default and create cluster does not turn it off, so a cluster built from the documented path can lose actors on the provider's maintenance schedule with nothing in the install docs to warn about it. Add the prerequisite to the Create Cluster section, and state in the upgrade runbook what actually happens to an actor when a serving worker pool is edited or scaled down, rather than only that it should not be. --- docs/upgrade.md | 16 +++++++++++++++- tools/setup-gcp/README.md | 26 ++++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 1 deletion(-) diff --git a/docs/upgrade.md b/docs/upgrade.md index 970a60d8c4..8340c13843 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -27,6 +27,13 @@ a version suffix, and the installed ate-api-server serves install instead, because its DaemonSet selector cannot be changed in place. +Node auto-upgrade is off on every node pool that runs workers, and +none of those nodes is spot or preemptible. The roll paces node moves +one at a time so that no actor loses state; a node the cloud provider +moves on its own schedule ignores that pacing, and any actor awake on +it is destroyed. See the +[Create Cluster warning](../tools/setup-gcp/README.md#2-create-cluster). + Actor snapshots are readable by both the old and the new build. An actor can therefore suspend on one version and resume on the other in either direction, which is what lets the two versions serve side by @@ -48,7 +55,14 @@ Three things break an upgrade. same node, and old workers end up next to the new atelet: exactly the version skew the roll exists to prevent. 2. **Do not edit a serving worker pool.** The controller would roll - the pool's Deployment straight through live actors. + the pool's Deployment straight through live actors, and an actor + that is awake when its worker pod goes away does not survive it. + It moves to `ACTOR_STATE_CRASHED`, which is terminal: `resume` and + `suspend` are both refused, there is no recover verb, and the + snapshot the actor still holds cannot be used to start it. The + actor has to be deleted and recreated, losing its state. The same + applies to scaling a serving pool down, which removes pods without + suspending the actors on them. 3. **(If on GKE) Do not touch the node pool's label until every node is rolled.** A pool label update applies in place to every node in the pool, so the whole fleet flips at once, with no drain and no diff --git a/tools/setup-gcp/README.md b/tools/setup-gcp/README.md index fe761fe966..c1e90a41a0 100644 --- a/tools/setup-gcp/README.md +++ b/tools/setup-gcp/README.md @@ -95,6 +95,32 @@ Filestore CSI driver disabled). > podcertificate ClusterTrustBundles to be ready" and `kubectl get > clustertrustbundles` reports the resource type is not served. +> [!WARNING] +> **Turn node auto-upgrade off on any node pool that runs workers, and do not +> use spot or preemptible nodes for them.** An actor that is awake when its +> worker pod goes away is moved to `ACTOR_STATE_CRASHED` with its worker +> assignment cleared, and `CRASHED` is terminal: `resume` and `suspend` are both +> refused, there is no recover verb, and the snapshot the actor still holds +> cannot be used to start it. The only way out is to delete the actor and create +> a new one, which loses its state. +> +> Auto-upgrade is the trigger to plan for, because GKE enables it by default and +> it fires on Google's maintenance schedule rather than yours. `create cluster` +> does not disable it, so do it yourself on every pool that runs workers: +> +> ```bash +> gcloud container node-pools update "${NODE_POOL}" \ +> --cluster "${CLUSTER_NAME}" --location "${CLUSTER_LOCATION}" \ +> --no-enable-autoupgrade +> ``` +> +> This is a management setting, so it takes effect without recreating nodes and +> is safe to apply to a serving cluster. Node auto-repair, preemption and OOM +> kills reach the same path and cannot be configured away, so treat the setting +> as removing the scheduled risk rather than all of it. Change versions through +> the [rolling upgrade runbook](../../docs/upgrade.md), which has you suspend +> every actor on a node before the node moves. + ```bash go run ./tools/setup-gcp create cluster [flags] ``` From 375040e122becaefc353a9619196f9c72eadcf0c Mon Sep 17 00:00:00 2001 From: Maya Wang Date: Tue, 8 Sep 2026 13:13:49 -0700 Subject: [PATCH 2/2] docs: describe the eviction grace window in the worker pool warnings A deleted worker pod is not an immediate loss: ateom forwards SIGTERM into the actor's containers and keeps serving CheckpointWorkload for the workload grace period, so a suspend that lands inside it saves the actor's state. Only an actor still awake when the window closes is marked CRASHED. Also shortens the runbook's node-drain assumption to point at the Create Cluster warning rather than restate it. --- docs/upgrade.md | 32 ++++++++++++++++++-------------- tools/setup-gcp/README.md | 16 +++++++++------- 2 files changed, 27 insertions(+), 21 deletions(-) diff --git a/docs/upgrade.md b/docs/upgrade.md index 8340c13843..80a7aa7255 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -27,12 +27,11 @@ a version suffix, and the installed ate-api-server serves install instead, because its DaemonSet selector cannot be changed in place. -Node auto-upgrade is off on every node pool that runs workers, and -none of those nodes is spot or preemptible. The roll paces node moves -one at a time so that no actor loses state; a node the cloud provider -moves on its own schedule ignores that pacing, and any actor awake on -it is destroyed. See the -[Create Cluster warning](../tools/setup-gcp/README.md#2-create-cluster). +Nothing drains worker nodes on its own: node auto-upgrade is off on +every pool that runs workers and none of them is spot or preemptible, +as the +[Create Cluster warning](../tools/setup-gcp/README.md#2-create-cluster) +requires. Actor snapshots are readable by both the old and the new build. An actor can therefore suspend on one version and resume on the other in @@ -55,14 +54,19 @@ Three things break an upgrade. same node, and old workers end up next to the new atelet: exactly the version skew the roll exists to prevent. 2. **Do not edit a serving worker pool.** The controller would roll - the pool's Deployment straight through live actors, and an actor - that is awake when its worker pod goes away does not survive it. - It moves to `ACTOR_STATE_CRASHED`, which is terminal: `resume` and - `suspend` are both refused, there is no recover verb, and the - snapshot the actor still holds cannot be used to start it. The - actor has to be deleted and recreated, losing its state. The same - applies to scaling a serving pool down, which removes pods without - suspending the actors on them. + the pool's Deployment straight through live actors. A deleted + worker pod does go through the eviction path: `SIGTERM` is + forwarded into the actor's containers and the control plane keeps + accepting a suspend for about 60 seconds, so an actor suspended + inside that window saves its state and stays resumable. Handling + `SIGTERM` by exiting cleanly is not enough on its own; the suspend + has to reach the control plane and finish. An actor still awake + when the window closes moves to `ACTOR_STATE_CRASHED`, which is + terminal: `resume` and `suspend` are both refused, there is no + recover verb, and the snapshot the actor still holds cannot be + used to start it. It has to be deleted and recreated, losing its + state. The same applies to scaling a serving pool down, which + removes pods without suspending the actors on them. 3. **(If on GKE) Do not touch the node pool's label until every node is rolled.** A pool label update applies in place to every node in the pool, so the whole fleet flips at once, with no drain and no diff --git a/tools/setup-gcp/README.md b/tools/setup-gcp/README.md index c1e90a41a0..4dc9bc3df9 100644 --- a/tools/setup-gcp/README.md +++ b/tools/setup-gcp/README.md @@ -97,12 +97,14 @@ Filestore CSI driver disabled). > [!WARNING] > **Turn node auto-upgrade off on any node pool that runs workers, and do not -> use spot or preemptible nodes for them.** An actor that is awake when its -> worker pod goes away is moved to `ACTOR_STATE_CRASHED` with its worker -> assignment cleared, and `CRASHED` is terminal: `resume` and `suspend` are both -> refused, there is no recover verb, and the snapshot the actor still holds -> cannot be used to start it. The only way out is to delete the actor and create -> a new one, which loses its state. +> use spot or preemptible nodes for them.** When a worker pod is deleted, +> `SIGTERM` is forwarded into the actor's containers and the control plane keeps +> accepting a suspend for about 60 seconds. An actor suspended inside that +> window keeps its state. One still awake when the window closes is moved to +> `ACTOR_STATE_CRASHED` with its worker assignment cleared, and `CRASHED` is +> terminal: `resume` and `suspend` are both refused, there is no recover verb, +> and the snapshot the actor still holds cannot be used to start it. The only +> way out is to delete the actor and create a new one, which loses its state. > > Auto-upgrade is the trigger to plan for, because GKE enables it by default and > it fires on Google's maintenance schedule rather than yours. `create cluster` @@ -119,7 +121,7 @@ Filestore CSI driver disabled). > kills reach the same path and cannot be configured away, so treat the setting > as removing the scheduled risk rather than all of it. Change versions through > the [rolling upgrade runbook](../../docs/upgrade.md), which has you suspend -> every actor on a node before the node moves. +> every actor on a node at your own pace before the node moves. ```bash go run ./tools/setup-gcp create cluster [flags]