fix: serve link's health port before waiting for the schema - #33
Merged
Conversation
link waits up to 60s for the tables the worker creates, and only then starts its HTTP server. The liveness probe fires after ~65s, so on a fresh cluster it kills the pod for doing exactly what it should. That happened on the first real deployment: the link pod restarted once before the tables existed, and only came up because the retry landed after the worker had migrated. The server now starts first, so /healthz answers immediately, and /readyz returns 503 until the schema is there -- Kubernetes holds traffic back without restarting anything. `healthz`'s own comment already said a slow database must not get the process killed; the start order contradicted it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Seen in production on the first deployment:
sturnus-linkrestarted once before coming up.linkwaits up to 60 seconds for the tables the worker creates (_wait_for_schema), and starts its HTTP server only afterwards. The liveness probe fires after roughly 65 seconds. On a fresh cluster — where the tables genuinely do not exist yet — the probe kills the pod for doing exactly what it is supposed to do.It recovered because the retry happened to land after the worker had migrated. With a slower database or a longer migration that becomes a loop the pod never escapes.
The comment on
healthzalready stated the intent:The start order contradicted it. Now the server starts first:
/healthzanswers from the first moment, and/readyzreturns 503 until the schema appears, so Kubernetes holds traffic back instead of restarting anything.Two tests cover both directions —
/healthzanswering while the schema is absent, and/readyzflipping from 503 to 200 when it arrives. 363 tests pass.