Repository navigation
bug: standalone/ADC - an out-of-band secrets resource deadlocks the sync loop (secrets_conf_version 400), blocking all reconciliation #2821
Description
Current Behavior
We run APISIX in API-driven standalone mode driven by the ingress controller
(ADC, provider.type: apisix-standalone). We wanted a route's plugin to reference a
Vault secret via APISIX's native $secret://vault/.... There is no CRD or
GatewayProxy field we could find to declare the backing secrets secret-manager, so
the only way to get one onto the data plane is to register it directly on APISIX's
standalone Admin API (out of band).
Doing that deadlocks the controller. Once the out-of-band manager bumps APISIX's
secrets_conf_version from 0 to 1, every subsequent controller/ADC full-config
push is rejected:
PUT /apisix/admin/configs → 400 Bad Request:
secrets_conf_version must be greater than or equal to (1)
ADC has no secrets resource type, so its push carries secrets_conf_version: 0
while APISIX is now at 1; APISIX's monotonic version check rejects the whole
push. Because each standalone update replaces the full configuration atomically,
nothing reconciles after that: the out-of-band manager is not wiped, the controller
is simply stuck, and even unrelated routes stop applying (they 404 at the gateway).
This looks like a robustness bug independent of the secret use case: once
secrets_conf_version is > 0 by any means (an out-of-band manager, a leftover
from a prior setup), ADC - which always sends 0 - can never sync again, and it takes
all other resources down with it.
For reference, APISIX itself handles $secret:// fine in this mode - we confirmed it
end-to-end with two standalone docker/podman baselines (in the attached bundle), so the
problem is on the controller/ADC side, not APISIX:
compose/api-driven/- in API-driven standalone, APISIX resolves$secret://vault/...
when asecretsmanager is present in the/configspayload.compose/file-driven/- the documentedapisix.yamlsecrets:block resolves, but
that mode disables the Admin API the controller drives (so it can't be used together
with the controller).
Expected Behavior
- The controller/ADC should not deadlock on a resource type it doesn't manage: it
should send a valid (or otherwise tolerated)secrets_conf_versionso a stray or
out-of-bandsecretsresource can't block reconciliation of everything else. - Ideally there would be a supported way to declare an APISIX
secretssecret-manager
through the controller (e.g. asecretsfield onGatewayProxy, or a dedicated
CRD), so$secret://vault/...references can be used. Today we can't find one.
(secretRefreads a Kubernetes Secret and injects plaintext - a different mechanism,
not APISIX's native$secret://manager.)
Error Logs
Controller manager container - this block repeats on every sync attempt after the
out-of-band secrets resource exists:
2026-07-22T09:14:48.222Z INFO provider.client client/client.go:214 syncing all resources
2026-07-22T09:14:48.239Z ERROR provider.executor client/executor.go:420 ADC Server sync failed {"result": {"status":"all_failed","total_resources":0,"success_count":0,"failed_count":1,"success":[],"failed":[],"endpoint_status":[{"server":"http://10.244.0.3:9180","success":false,"reason":"PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}]}, "error": "ADC Server sync failed (standalone mode): http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:48.240Z ERROR provider.executor client/executor.go:153 failed to run http sync for server {"server": "http://10.244.0.3:9180", "error": "ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:48.240Z ERROR provider.client client/client.go:306 failed to execute adc command {"config": {"name":"GatewayProxy/ingress-apisix/apisix-config","serverAddrs":["http://10.244.0.3:9180"],"tlsVerify":false}, "error": "ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]"}
2026-07-22T09:14:48.240Z ERROR provider.client client/client.go:247 failed to sync resources {"name": "GatewayProxy/ingress-apisix/apisix-config", "error": "ADC execution errors: [ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]]"}
2026-07-22T09:14:48.240Z ERROR provider apisix/provider.go:282 failed to sync {"error": "failed to sync 1 configs: GatewayProxy/ingress-apisix/apisix-config"}
2026-07-22T09:14:56.241Z INFO provider.client client/client.go:214 syncing all resources
2026-07-22T09:14:56.254Z ERROR provider.executor client/executor.go:420 ADC Server sync failed {"result": {"status":"all_failed","total_resources":0,"success_count":0,"failed_count":1,"success":[],"failed":[],"endpoint_status":[{"server":"http://10.244.0.3:9180","success":false,"reason":"PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}]}, "error": "ADC Server sync failed (standalone mode): http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:56.254Z ERROR provider.executor client/executor.go:153 failed to run http sync for server {"server": "http://10.244.0.3:9180", "error": "ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:56.254Z ERROR provider.client client/client.go:306 failed to execute adc command {"config": {"name":"GatewayProxy/ingress-apisix/apisix-config","serverAddrs":["http://10.244.0.3:9180"],"tlsVerify":false}, "error": "ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]"}
2026-07-22T09:14:56.255Z ERROR provider.client client/client.go:247 failed to sync resources {"name": "GatewayProxy/ingress-apisix/apisix-config", "error": "ADC execution errors: [ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]]"}
2026-07-22T09:14:56.255Z ERROR provider apisix/provider.go:282 failed to sync {"error": "failed to sync 1 configs: GatewayProxy/ingress-apisix/apisix-config"}
Steps to Reproduce
A self-contained reproduction is attached as apisix-secrets-repro.zip:
k8s/- a kustomization that stands everything up + a Job that automates steps 3–6
below and prints the controller's own rejection log.compose/api-driven/- docker/podman baseline: APISIX resolves$secret://vault/...
in API-driven standalone (shows the gap is controller-side, not APISIX).compose/file-driven/- docker/podman baseline: the documentedapisix.yaml
secrets:block resolves, but that mode has no Admin API for the controller to drive.
To reproduce the deadlock manually against the controller:
- Deploy APISIX in API-driven standalone mode (
deployment.role: traditional,
role_traditional.config_provider: yaml) + APISIX Ingress Controller 2.1.0 — e.g.
theapisixHelm chart 2.16.0 withingress-controller.enabled: true,
ingress-controller.config.provider.type: apisix-standalone, and
ingress-controller.gatewayProxy.createDefault: true. - Apply any
ApisixRoute; confirm the controller syncs it —
GET /apisix/admin/configsshows the route and.secretsis empty (secrets = 0). - Register a
secretsmanager out of band via the standalone Admin API (echo the
existing*_conf_versionvalues back, omitsecrets_conf_version, add the manager):ConfirmPUT /apisix/admin/configs { ...existing *_conf_version..., "secrets": [ { "id": "vault/oidc", "uri": "http://vault.example:8200", "prefix": "kv/apisix", "token": "root" } ] }.secretsnow has 1 entry (secrets = 1); APISIX'ssecrets_conf_version
is now1. - Trigger a controller re-sync (e.g.
kubectl annotate apisixroute <name> poke=$(date +%s) --overwrite). - Watch the controller logs: every sync now fails with
400 ... secrets_conf_version must be greater than or equal to (1). - Observe the blast radius: the out-of-band manager is not removed, and because the
push is atomic, no route reconciles —curlagainst the gateway returns404even
for routes unrelated to secrets.
Environment
APISIX Ingress controller version (apisix-ingress-controller version --long): 2.1.0
Git SHA:
Go Version: go1.24.13
Building OS/Arch: /
Running OS/Arch: linux/amd64
Kubernetes cluster version (kubectl version):
Client Version: v1.33.13
Kustomize Version: v5.6.0
Server Version: v1.35.1
(reproduced on minikube, Podman driver)
APISIX: 3.17.0; installed via the apisix Helm chart 2.16.0;
provider.type: apisix-standalone
Activity
AlinsRan commentedon Jul 29, 2026
Thanks for the detailed reproduction.
This is a constraint of the standalone model, not a bug in the sync path
PUT /apisix/admin/configsreplaces the whole configuration atomically, and the controller always pushes a full snapshot. There is no incremental path, so anything not in that snapshot isn't part of the configuration the controller declares.Out-of-band resources survive today only because ADC echoes back fields it doesn't recognize from the config it fetched. That's an implementation detail, not a contract — and it's the same path that carried a stale
secrets_conf_versioninto the payload and produced your 400.So we won't be making out-of-band registration a supported setup.
Unblocking it today
Restart the ADC container (or the controller pod) — the cached snapshot is rebuilt from the data plane on the next sync and reconciliation resumes. The cache is an LRU with a 1h TTL (
ADC_APISIX_STANDALONE_CACHE_TTL_MS), so it also recovers on its own eventually.What we will fix
The control plane shouldn't be blocked indefinitely by external state. ADC already exposes
bypassCacheon its sync API, but the controller doesn't pass it, and ADC keeps its cached snapshot on a failed sync. Wiring that up turns the deadlock into one failed sync that self-heals — robustness, not support for out-of-band writes.$secret://supportYou're right that there's no way to declare a secrets manager through the controller today.
secretsisn't in the resource model, andsecretRefis a different mechanism (reads a Kubernetes Secret, injects the value). The real answer is bringingsecretsinto the control plane so it becomes part of the snapshot.That's a feature and needs its own design discussion — note that once the controller owns
secrets, anything registered out of band gets removed on the next sync.jens-skribble commentedon Jul 29, 2026
Thank you @AlinsRan for the answer. I went ahead and created a feature request #2831 to track the
$secret://support.juzhiyuan commentedon Sep 18, 2026
Thanks @jens-skribble, #2821 has been fixed already on master - regarding #2831, I will ask for reviewers to check again.