Stabilize flaky collector and JetStream integration tests

[CAOShurong]

Description of the PR

This removes three sources of nondeterminism from the integration tests tracked in #2930.

1. Git collector test

• Build a two-document Git fixture in a temporary local repository instead of cloning GitHub.
• Assert only the known fixture documents instead of depending on the number of .git implementation files.
• Stop the polling case as soon as both fixture documents arrive, with a 10-second safety timeout.

A recent integration run (32978753396, job 98209847685) timed out after collecting 0 files. On the original base, the polling subtest also failed 20/20 local runs because the checkout contained 10 files while the test hard-coded 9. The updated test passed 50 consecutive runs.

2. OCI collector test (13 vs 14 referrers)

Test_ociCollector_RetrieveArtifacts/get_OCI_referrers intermittently fails with g.RetrieveArtifacts() = 13, want 14 (including run 33735694815 after the first retry change). A referrer descriptor points to a concrete manifest, but registry replicas can briefly return not found while converging. The previous retry implementation immediately accepted ErrNotFound as a permanent absence, so that transient response still dropped one of the 14 expected documents.

The updated fix:

• Retries all manifest-fetch failures, including transient not-found responses, with a bounded 250ms/500ms backoff.
• Treats a persistent ErrNotFound after all attempts as the existing “no attestation” case.
• Surfaces any other persistent error instead of silently dropping the artifact.
• Adds deterministic unit coverage for transient and persistent not-found sequences through a narrow manifest-getter interface.

The earlier change also fixes a latent error-wrapping bug in fetchReferrerArtifacts: the error channel received the outer err value rather than the actual fetch error e.

3. JetStream emitter stream-recreate race

Test_ProcessSubscribe intermittently fails with unexpected error recreating jetstream: failed to create stream: nats: error creating store for stream (for example, main run 33452015674). Each subtest deletes and recreates the DOCUMENTS stream, and the NATS server can still be finalizing deletion of the old stream's store. RecreateStream now retries stream creation five times with a short delay.

Validation

• go test -run TestGetManifestWithRetry -count=10 ./pkg/handler/collector/oci/
• go test ./pkg/handler/collector/oci/...
• go test -tags integration -run 'Test_ociCollector_RetrieveArtifacts/get_OCI_referrers' -count=30 ./pkg/handler/collector/oci/ (live registry, 30 consecutive passes)
• go test -tags integration -run Test_ProcessSubscribe -count=5 ./pkg/handler/processor/process/
• go test -tags integration -count=3 ./pkg/handler/collector/git/
• go test ./pkg/emitter/...
• go vet ./pkg/handler/collector/oci/... ./pkg/emitter/...
• git diff --check

The broader collector suite still has pre-existing Windows-only path-separator failures in file-collector tests; those reproduce unchanged on the base and are outside this PR.

Implementation and investigation were assisted by OpenAI Codex. I reviewed the diff and ran the validation above.

PR Checklist

• All commits have a Developer Certificate of Origin (DCO) -- generated using git commit -s.
• All new changes are covered by tests
• GraphQL schema is unchanged
• OpenAPI spec is unchanged
• ent schema is unchanged
• collectsub protobuf is unchanged
• All CI checks are passing (awaiting CI)
• There are no dependent PRs

[gaganhr94]

Integration tests are still flaky. Looks like this PR would not solve the issue

[gaganhr94]

@CAOShurong can you try some other fix for the integration tests

[CAOShurong]

Thanks for the nudge. I dug into which tests were still failing on top of the git-collector change and found two separate sources of flakiness, both now fixed in a new commit (e635bc8):

1. Test_ociCollector_RetrieveArtifacts/get_OCI_referrers — intermittently failed with g.RetrieveArtifacts() = 13, want 14 (e.g. main run 33508995724, Sep 1). fetchOCIArtifactBlobs swallowed every rc.ManifestGet error and treated it as "no attestation for this digest", so a single transient registry failure (timeout, rate limit, connection reset) silently dropped one of the 14 expected referrer documents. Manifest fetches are now retried with a short backoff (3 attempts, 250ms/500ms); only a true 404 (regclient's errs.ErrNotFound) is treated as "no attestation", and any other persistent error is surfaced to the caller.

2. Test_ProcessSubscribe — intermittently failed with nats: error creating store for stream (e.g. main run 33452015674, Aug 31). Each subtest deletes and recreates the JetStream DOCUMENTS stream, and the NATS server can still be finalizing the deletion of the old stream's store when the replacement is created. RecreateStream now retries the stream creation a few times (5 attempts, 100ms apart) instead of failing on the first transient store error.

Local validation (all passing):

• go test -tags integration -run Test_ociCollector_RetrieveArtifacts -count=3 ./pkg/handler/collector/oci/ (live registry)
• go test -tags integration -run Test_ProcessSubscribe -count=5 ./pkg/handler/processor/process/
• go test -tags integration -count=3 ./pkg/handler/collector/git/
• go vet on the affected packages

I'll leave it to CI to confirm.

[gaganhr94]

@CAOShurong the integration tests are still flaky. Your fix doesn't solve the test flakiness

[gaganhr94]

Integration tests are still flaky even after multiple fixes. I will close this PR. Let's try to understand the root cause in the issue section and then come up with a fix.