fix(ci): stop creating MinIO ILM rules from the baked cache helper - #19
Merged
Conversation
The minio-cache helper baked into the keploy-ci-node and keploy-ci-playwright images ran 'mc ilm rule add' on every cache save. The command is not idempotent - each call appends a duplicate lifecycle rule, and the accumulated duplicates hit MinIO's hard cap of 1000 rules on 2026-08-13, breaking retention enforcement bucket-wide. Retention is now owned by the server-side canonical rule set on the MinIO instance, so the helper must not manage lifecycle rules. Signed-off-by: Shubham Jain <shubhamkjain@outlook.com>
slayerjain
force-pushed
the
fix/ci-no-ilm-rules
branch
from
August 13, 2026 15:01
d32f0ae to
befaa7a
Compare
slayerjain
added a commit
that referenced
this pull request
Aug 22, 2026
… old one (#20) keploy-ci-lighthouse builds FROM keploy-ci:node-1.2.17, which predates the fix in #19 and bakes a minio-cache that ran 'mc ilm rule add' on every cache save. Duplicate lifecycle rules accumulate toward MinIO's 1000-rule cap, which silently breaks artifact retention. Copy the current helper into the image the way keploy-ci-playwright already does, rather than bumping the base tag: every node tag from 1.2.24 onward is Node 24, so a base bump would drag an unrelated Node 20 -> 24 jump into the landing Lighthouse build. Add a helper-parity job so the per-context copies can never drift again - divergence between them is exactly how this regression shipped. Signed-off-by: Shubham Jain <shubhamkjain@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The
minio-cachehelper baked into thekeploy-ci-nodeandkeploy-ci-playwrightimages runsmc ilm rule add ... || trueon every cache save. Despite the comment, this is not idempotent — MinIO appends a duplicate lifecycle rule on every call. Accumulated duplicates hit MinIO's hard cap of 1000 rules per bucket on 2026-08-13, breaking retention enforcement and contributing to the CI storage outage. (Companion PRs already merged: enterprise#2375, api-server#1972, enterprise-ui#1684, playwrightDB#31.)What
Remove the
mc ilm rule addcall from both copies of the helper; retention is owned by the server-side canonical rule set on the MinIO instance (applied 2026-08-13).These scripts are baked into the CI base images — the fix only takes effect once the images are rebuilt/pushed and consuming pipelines pick up the new tag. Until then, running pipelines keep re-adding rules (slowly, one prefix).
Verification
grep -rn 'mc ilm'returns nothingbash -npasses on both scripts