What happened?
When launching an actor and runsc fails (e.g. grpc ResumeActor: rpc error: code = Internal desc = while restoring "glutton" application container: while running runsc restore: exit status 128), the actor remains stuck in resuming and assigned to a worker, removing that worker from the pool of available workers.
Expected Behavior
Would expect the actor to go into crashed state or similar and the worker to be freed.
Steps to Reproduce
You can use this patch to force a runsc error whenever resuming an actor:
maxsmythe@6e27c95
shell log of behavior in action:
smythe@smythe63:~/dev/substrate$ kubectl ate create actor probe-2 -a benchmark-workloads --template-ref sleep
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_SUSPENDED <none> 1 0s
smythe@smythe63:~/dev/substrate$ kubectl ate resume actor probe-2 -a benchmark-workloads
Error: failed to resume actor: rpc error: code = Internal desc = while restoring pause container: while running `runsc restore`: exit status 128
smythe@smythe63:~/dev/substrate$ kubectl ate get actor probe-2 -a benchmark-workloads
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-94bfj 10.80.2.28 2 30s
smythe@smythe63:~/dev/substrate$ kubectl ate get actors -a benchmark-workloads
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-1 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-g887j 10.80.7.21 2 4m12s
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-94bfj 10.80.2.28 2 52s
smythe@smythe63:~/dev/substrate$ kubectl ate get workers -n benchmark-workloads
NAMESPACE POOL CLASS POD STATUS ASSIGNED ACTOR
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-94bfj ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-2
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-g887j ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-1
smythe@smythe63:~/dev/substrate$ kubectl ate get actors -a benchmark-workloads
ATESPACE NAME TEMPLATE STATE ATEOM POD ATEOM IP VERSION AGE
benchmark-workloads probe-1 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-g887j 10.80.7.21 2 15m
benchmark-workloads probe-2 benchmark-workloads/sleep ACTOR_STATE_RESUMING benchmark-workloads/benchmark-ateom-674c8979d-94bfj 10.80.2.28 2 12m
smythe@smythe63:~/dev/substrate$ kubectl ate get workers -n benchmark-workloads
NAMESPACE POOL CLASS POD STATUS ASSIGNED ACTOR
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-94bfj ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-2
benchmark-workloads benchmark-ateom gvisor benchmark-ateom-674c8979d-g887j ASSIGNED benchmark-workloads/sleep/benchmark-workloads/probe-1
Sandbox Runtime
gVisor (runsc)
Agent Substrate Version / Commit SHA
patched version of main (to force error)
Kubernetes Version & Environment
No response
Host OS & Architecture
No response
Relevant Logs and Diagnostic Output
Additional Context
No response
Confirmation
What happened?
When launching an actor and runsc fails (e.g.
grpc ResumeActor: rpc error: code = Internal desc = while restoring "glutton" application container: while runningrunsc restore: exit status 128), the actor remains stuck in resuming and assigned to a worker, removing that worker from the pool of available workers.Expected Behavior
Would expect the actor to go into crashed state or similar and the worker to be freed.
Steps to Reproduce
You can use this patch to force a runsc error whenever resuming an actor:
maxsmythe@6e27c95
shell log of behavior in action:
Sandbox Runtime
gVisor (runsc)
Agent Substrate Version / Commit SHA
patched version of main (to force error)
Kubernetes Version & Environment
No response
Host OS & Architecture
No response
Relevant Logs and Diagnostic Output
Additional Context
No response
Confirmation
main.