Elasticsearch sink integration test flakes when connectors health never becomes ready

[ryerraguntla]

Bug description

Body

Summary

The Elasticsearch sink integration test
(elasticsearch_sink_stores_json_messages) can fail with HealthCheckFailed for iggy-connectors after many retries. Connectors runtime logs stop at Opening Elasticsearch sink... and never reach a successful open. Observed locally against a reused Testcontainers Elasticsearch instance; matches Discord failure reports. [ elastic_connector_files.zip](url)

Root cause is a combination of:

1. Sink open() can hang indefinitely on Elasticsearch HTTP (no client timeout applied), and the connectors HTTP API only binds after plugin open completes.
2. Fixture reuse (ReuseDirective::Always, container name iggy-test-elasticsearch) can attach to a days-old container without re-validating cluster health the same way as a cold start.
3. On macOS, localhost can resolve to IPv6 ::1 first while Docker publishes IPv4 0.0.0.0:HOST→9200, so the elasticsearch-rs client can
   hang while a request health probe still looks fine.
4. Harness readiness polled a generic HTTP path and did not fail fast if iggy-connectors crashed during open.

Stale leftover indices alone do not appear to break the happy path on a healthy host, but accumulated leftovers on a reused container are still worth cleaning so the shared node stays healthy across runs.

Impact

• Flaky CI / local integration runs for Elasticsearch sink (and related harness readiness).
• Hard to diagnose: failure surfaces as connectors health timeout, not as an Elasticsearch client timeout error.

Proposed fix

Branch in progress: fix/elasticsearch-sink-test-flakiness

• Sink: apply timeout_seconds (default 30s) on TransportBuilder so hung ES I/O cannot block FFI open() forever.
• Fixture container: prefer http://127.0.0.1:{port}; always poll /_cluster/health after attach; if the reused container is unusable, docker rm -f iggy-test-elasticsearch once and recreate (named harness container only — does not touch other Elasticsearch containers).
• Fixture container: best-effort sweep of stale test indices (iggy_messages_*, test_documents_*) older than 30 minutes so prior empty/partial indices do not accumulate on the reused node.
• Sink fixture: refresh index before each document-count poll; keep last error in timeout messages.
• Harness: poll connectors /health and fail fast with ProcessCrashed if the child dies during retries.

Acceptance criteria

• timeout_seconds is applied to the Elasticsearch transport client.
• Elasticsearch fixture readiness is re-checked on every setup, including reuse attach.
• Wedged iggy-test-elasticsearch is removed and recreated once; other Docker Elasticsearch containers are never targeted.
• Stale harness test indices older than the age threshold are cleaned best-effort before the test proceeds.
• Connectors runtime readiness uses /health and detects a dead process during retries.
• elasticsearch_sink_stores_json_messages passes locally against reused container.

References

• Discord report / log set (connectors stuck at "Opening Elasticsearch sink...", harness HealthCheckFailed).
• Local analysis notes (commit where flake was investigated): 6f1d5485d06c3c43afd2510e432b2f7c1a3ac41a
• Related discussion context: connectors / integration test stability.

Out of scope

• Changing connectors runtime to bind HTTP before plugin open() (possible follow-up for clearer "starting" health).
• Broader Test containers reuse policy changes for non-Elasticsearch fixtures.

elastic_connector_files.zip

Affected area / component

Connectors

Deployment

Docker (DockerHub image)

Versions

No response

Hardware / environment

No response

Sample code

No response

Logs

No response

Iggy server config

No response

Reproduction

No response

Contribution

• I'm willing to submit a pull request to fix this bug

Good first issue

• I think this could be a good first issue for a new contributor