'Useless' validation keys generated by Automatic Key management when running IdentityServer on multiple instances

[roemba]

We are currently migrating our application to use IdentityServer’s Automatic Key management. We have been using a key management service built ourselves to do this before this migration, but to reduce maintenance we opted to switch to the built-in key management of IdentityServer as it is included with our Enterprise license.

Studying and testing the IdentityServer implementation revealed a peculiar behavior that does not break functionality but might not be desired in all scenarios. For context: our application gets a lot of traffic and might be running on 30+ instances or more. 

We noticed that in KeyManager.GetAllKeysInternalAsync, the code loads the keys from the cache but then continues to always check if a rotation is required on every request to load the keys (e.g. on every request to the JWKs endpoint). If we have 10 instances, a continuous stream of requests for the JWKs endpoint and assume the keys need to be rotated at that moment, all 10 instances will attempt to do so, resulting in 10 new keys being added and also served on the JWKs endpoint. This scenario assumes there is some delay between reading the keys from the store and writing them. If the key write or instance is slow for some reason (network, high load on DB, etc.), the chances of this happening become larger. We have observed this scenario occurring when doing a rollout on our Kubernetes Deployments.

Functionally this doesn’t matter much for IdentityServer, as the KeyManager ensures all instances will eventually use the oldest created key for signing. However, our set of validation keys will be cluttered with 9 other keys that will only be used for signing for a very short period (e.g. 1-3 seconds). As our JWKs endpoint is called quite a bit it matters that this increases the request size by a lot.

As we see it, there are three solutions to this problem:

1. Prevention: If the KeyManager were to cache the rotation check as well, then some level of control can be introduced to which instance might be doing the rotating. If the rotation check is cached with a short random delay + a fixed cache duration, the chances that two or more instances decide to rotate at the same time are greatly reduced.  The random delay is required in case multiple instances are started at the same time.
2. Prevention: Some configuration item is introduced in the KeyManager to control if it is allowed to do key rotation. This would allow library users to configure instances that can and cannot rotate keys. This would work well with clusters like Kubernetes, that can periodically run a CronJob. Once every hour or so, an instance can be started that is allowed to rotate the keys, while all other instances cannot. This is very similar to how our own solution currently solves the concurrency problem.
3. Cleanup: The KeyManager will delete / ignore the keys that will only be used for a very short time (e.g. <10s) once rotation is complete (RotationInterval + some delay, to ensure no server is attempting rotation anymore).

With the current version of IdentityServer (7.4.3), the only option we see to fix this validation key clutter is to implement option 3 in our ISingingKeyStore.LoadKeysAsync implementation. As IdentityServer’s implementation is 90% the same as our own, we still think it is a good decision to use it instead of our own key management service. But a good fix for this problem as part of IdentityServer, or at least some extensibility to allow us to build option 1 or 2, would be appreciated and we hope you consider adding it to IdentityServer.

[wcabus]

The automatic key management system is a complex beast, especially in a multi-instance environment. You can try to tweak the InitializationDuration and InitializationSynchronizationDelay settings in the key management options to set different times per instance, but in the end, your instances will still generate keys next to each other before deciding on a single new key to use across every instance.

We are discussing internally how we can improve this, but can't say when we will release changes to the key management system yet. Your best bet if you desire more control over the key renewal process, for now, is to roll your own solution as you have already done. Or, for example, to disable automatic key generation and replace it with a cron job instead.

[rasander]

@wcabus Is there any news on this?