fix: don't set global credential system properties for third-party S3 targets - #83
Merged
Merged
Conversation
… targets For third-party/S3-compatible endpoints (e.g. Ceph RGW), credentials already travel scoped to the filesystem instance via the s3x:// URI's userInfo. The code additionally wrote them to the process-wide aws.accessKeyId/aws.secretAccessKey system properties unconditionally, which is redundant there and unsafe: any other credential resolution in the same JVM that falls back to the default AWS credentials chain (observed: the async read-ahead path, S3ReadAheadByteChannel) would read whatever value this method last wrote, independent of which storage's request is actually in flight. Observed in production against Ceph RGW: intermittent, unpredictable 403s isolated to the async read-ahead path, with the rejected requests carrying no identifiable user at all - consistent with a credential resolution racing against an overwritten system property rather than a real permissions problem (bucket ownership and the access key itself were verified correct on the RGW side). Left the system-property assignment in place for the real-AWS-S3 branch, which has no URI-based credential mechanism and still needs it. This is a hypothesis-driven fix based on production log analysis (RGW access logs, bucket/user state, and code review) - not confirmed against a live repro, since the failure is intermittent. Worth monitoring after merge.
The detailed why (production symptoms, ruled-out causes, the specific async-path theory) belongs in the PR, not as a wall of comments in the code itself.
TheMeinerLP
force-pushed
the
fix/avoid-global-credential-system-properties
branch
from
July 19, 2026 10:33
0a49d4c to
fa927c5
Compare
TheMeinerLP
added a commit
to OneLiteFeatherNET/Kubernetes-FLUX
that referenced
this pull request
Jul 19, 2026
Includes the credential-scoping fix (TheMeinerLP/BlueMapS3Storage#83) for the intermittent Ceph RGW 403s traced earlier - third-party S3 targets no longer leak credentials through global JVM system properties.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
For third-party/S3-compatible endpoints (Ceph RGW, R2, MinIO, ...), credentials already travel scoped to the filesystem instance via the
s3x://URI'suserInfo(accessKeyId:secretAccessKey).S3FileSystemFactory.build()additionally wrote the same credentials to the process-wideaws.accessKeyId/aws.secretAccessKeysystem properties unconditionally, before branching onthirdParty.That's redundant for the third-party path, and unsafe: system properties are global mutable JVM state. Any credential resolution elsewhere in the same JVM that falls back to the default AWS credentials chain — rather than using the URI-scoped credentials this method sets up — reads whatever value was written there most recently, independent of which storage's request is actually in flight.
What prompted this
Running BlueMapS3Storage against Ceph RGW in production, I started seeing an elevated Ceph RGW failed-request-rate alert. Investigation (RGW access logs,
radosgw-admin bucket stats/user info, Kubernetes secret contents) ruled out:bucket statsconfirmsbluemapowns the bucket.What stood out: 99.8% of the failing requests were
GET .../tiles/<...>reads, rejected with no identifiable authenticated user at all (blank, not "wrong user") — and the original crash that kicked off the investigation was a409thrown fromS3ReadAheadByteChannel(the async read-ahead path inaws-java-nio-spi-for-s3), not from the synchronous request path.That pointed at the async path resolving credentials differently than the synchronous one — and
S3FileSystemFactorywriting credentials to global system properties, unconditionally, is the one place in this codebase where that kind of cross-path interference could happen.Fix
Only set
aws.accessKeyId/aws.secretAccessKeyon the real-AWS-S3 branch, which has no URI-based credential mechanism and genuinely needs them. The third-party/S3X branch now relies solely on the URI-embedded credentials it already had.Confidence level
Honest caveat: this is a hypothesis-driven fix, not a confirmed-and-reproduced one — the failure is intermittent (roughly 2-17 failed req/s in bursts, not constant), and I don't have a minimal repro isolating the exact internal code path in
aws-java-nio-spi-for-s3that falls back to the system-property-based default credentials chain. The change itself is low-risk (it only removes a redundant write on the third-party path; the real-AWS-S3 path is untouched), so I think it's worth merging and monitoring rather than blocking on a full repro.Test plan
./gradlew compileJava— builds clean./gradlew spotlessCheck— formatting passes