feat(db): support explicit Postgres DSNs

[ankitgoswami]

Reviewable diff: +248/-36 across 21 files (excludes generated, test, and story files).

Summary

Fleet can now take an explicit PostgreSQL DSN through DB_DSN while preserving the existing DB_USERNAME / DB_PASSWORD / DB_NAME / DB_ADDRESS / DB_SSL_MODE configuration path. This gives on-prem HA deployments a topology-neutral way to point Fleet at a writable PostgreSQL endpoint, including pgx multi-host DSNs that use target_session_attrs=read-write.

The PR also adds reset-on-failover behavior at Fleet-owned database boundaries. When those paths observe a failover-class error, they drop idle connections so the next operation can reconnect through the HA endpoint; they do not replay ambiguous writes or transactions.

Stack: #768 fix(db): require shared transaction helpers -> #759 this PR. This PR's diff is relative to #768; reviewers should take #768's shared-transaction-helper guard as the upstream context and review this PR for explicit DSN support plus HA pool reset behavior.

How it works

At startup, DB_DSN wins when it is set; otherwise Fleet builds the same legacy PostgreSQL URL from the existing DB fields. The final DSN is parsed by pgx before Fleet opens the connection. Malformed explicit DSNs return a generic invalid-DSN error, pgx configs with more than one distinct host:port endpoint require target_session_attrs=read-write, and hostaddr is rejected for now with an explicit "use host" error.

Once Fleet opens the shared *sql.DB, it registers an idle-pool reset callback for that connection. The reset callback is then picked up by the DB paths this PR wires into: RetryDB ExecContext / QueryContext, prepared sqlc complete-operation retries, non-transaction store queries returned by SQLConnectionManager.GetQueries(ctx), and db.WithTransaction(...) begin/action/commit handling. Structured PostgreSQL errors are classified by SQLSTATE first, so non-failover PgErrors do not fall through to any fallback classifier. Unstructured errors classify as failover only when they match typed/sentinel driver or network failures. Existing serialization/deadlock retry behavior remains unchanged; failover-class errors reset the pool and return the original failure to the caller.

Transaction-bound store contexts still use the transaction's *sqlc.Queries; the transaction helper owns failover reset for begin, body, and commit errors. Code that bypasses Fleet's DB setup or constructs raw sqlc handles directly is outside this automatic reset boundary.

Diagrams

flowchart TB
  Env["Environment"] --> Config["DB config"]
  Config --> Legacy["Legacy DB_* fields"]
  Config --> Explicit["DB_DSN override"]
  Legacy --> FinalDSN["Final PostgreSQL DSN"]
  Explicit --> Validate["pgx parse + Fleet guards"]
  Validate --> MultiHost["Multiple endpoints require read-write target"]
  Validate --> Hostaddr["hostaddr rejected; use host"]
  MultiHost --> FinalDSN
  Hostaddr --> ConfigError["Startup validation error"]
  FinalDSN --> SQLDB["Shared sql.DB"]
  SQLDB --> Registry["Idle-pool reset callback"]
  SQLDB --> RetryDB["RetryDB Exec / Query"]
  SQLDB --> StoreQueries["SQLConnectionManager non-tx queries"]
  SQLDB --> Prepared["Prepared sqlc querier"]
  SQLDB --> Tx["db.WithTransaction"]
  RetryDB --> Reset["Reset idle pool on failover-class error"]
  StoreQueries --> Reset
  Prepared --> Reset
  Tx --> Reset

Loading

sequenceDiagram
  participant Path as "Fleet DB path"
  participant PG as "Postgres HA endpoint"
  participant Pool as "sql.DB pool"

  Path->>PG: "Run operation"
  PG-->>Path: "Failover-class error"
  Path->>Pool: "SetMaxIdleConns(0), then restore configured idle limit"
  Path-->>Path: "Return original error"
  Note over Path,PG: "No failover write or transaction replay"

Loading

Areas of the code involved

Area / package / file	What changed	Why it matters for review
server/internal/infrastructure/db/config.go	Adds DB_DSN, DSN selection, pgx parsed-config validation, generic malformed-DSN errors, multi-host read-write targeting, and explicit hostaddr rejection	This is the compatibility and safety boundary for operator-provided connection strings
server/internal/infrastructure/db/database_connection.go, pool_reset.go	Registers an idle-pool reset callback for the shared connection after pool limits are applied	Centralizes the HA reset hook at Fleet's DB construction point
server/internal/infrastructure/db/retry.go	Adds failover classification and resets the pool on failover-class errors while preserving serialization/deadlock retries	Defines the runtime recovery behavior after PostgreSQL role changes or connection loss
server/internal/infrastructure/db/failover_reset_querier.go, prepared_querier.go	Uses the generated sqlc retry decorator to reset after complete-operation errors, including deferred scan errors from :one queries	Covers sqlc operations without asking store authors to call an HA-specific method
server/internal/infrastructure/db/with_transaction.go	Resets on failover-class begin/action/commit errors and wraps begin/commit causes with %w	Covers the normal transaction helper without replaying ambiguous transactional work
server/internal/domain/stores/sqlstores/**, server/internal/domain/authz/reconcile.go	Widens local helper types from *sqlc.Queries to sqlc.Querier where those helpers can receive reset-aware query handles	Lets existing store code keep calling GetQueries(ctx) while non-transaction paths get the decorator
server/docker-compose.base.yaml, deployment-files/docker-compose.yaml, deployment-files/README.md	Passes through and documents DB_DSN	Gives deployment files an opt-in path for HA-aware DSNs without changing default installs
Tests under server/internal/infrastructure/db and server/cmd/fleetd	Adds unit coverage for DSN validation, env parsing, failover classification, reset-on-error behavior, and transaction reset points	Pins the new behavior without requiring local DB-backed integration tests

Key technical decisions & trade-offs

Decision	Trade-off
Keep DB_DSN opt-in and preserve legacy DB fields	Existing installs stay unchanged; HA deployments get a lower-level DSN escape hatch
Let pgx own DSN parsing and inspect only its parsed config	Avoids a Fleet-specific URL/keyword parser; validation is intentionally limited to the HA guardrails Fleet needs
Require target_session_attrs=read-write only when pgx's final config has more than one distinct endpoint	Single-host DSNs remain flexible; multi-host HA DSNs avoid silently selecting a replica
Reject hostaddr for now instead of partially supporting it	Keeps validation and operator guidance simple; deployments that want IP literals should put them in host
Return generic malformed-DSN errors	Avoids leaking credentials or topology from invalid DSNs; logs lose malformed endpoint detail by design
Register reset callbacks by *sql.DB	Avoids threading HA options through store constructors; raw connections created outside Fleet's DB setup do not get automatic reset behavior
Use sqlc's generated retry decorator for reset-only :one coverage	Covers deferred Scan errors without a custom row wrapper; it still relies on stores using the normal SQLConnectionManager path
Trust structured PostgreSQL SQLSTATEs and typed driver/network errors	Avoids pool churn from user-controlled error text in non-failover database or domain errors; plain error-message text no longer drives failover classification
Reset on failover-class errors but do not retry failover writes or transactions	The next operation can reconnect through the HA endpoint; ambiguous mutations are not automatically replayed
Do not prove successful reads are on the writable primary in this PR	Keeps the PR small and topology-neutral; stricter read-write transaction checks for auth/session/readiness paths should be a separate HA hardening PR if needed

Testing & validation

Current stacked validation after simplifying DSN validation:

• source ../bin/activate-hermit && go test ./internal/infrastructure/db -run 'TestConfig|TestDSN|TestIsFailoverPostgresError|TestRetrier|TestRetryDB|TestWithTransaction'
• source ../bin/activate-hermit && go test ./cmd/fleetd -run 'TestFleetdLoadsExplicitDBDSNFromEnv|TestConfig'
• source ../bin/activate-hermit && go test ./internal/domain/stores/sqlstores ./internal/domain/authz -run '^$'

Per local testing policy, DB-backed integration tests were not run locally; CI should cover those surfaces.

[github-actions]

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

• Reviewed pull request diff only (c161fc9a15c30055a30f917a418e9c6354af9baa...e6db4a1fef5a90867eaaa429775fcc4517d03a4a, exact PR three-dot diff)
• Model: gpt-5.5

💡 Click "edited" above to see previous reviews for this PR.

---

Review Summary

Overall Risk: MEDIUM

Findings

[MEDIUM] DB_DSN deployments remain blocked by bundled TimescaleDB health

• Category: Infrastructure
• Location: deployment-files/docker-compose.yaml:16
• Description: The PR adds DB_DSN so fleet-api can connect to an external or multi-host PostgreSQL topology, but the same service still has an unconditional depends_on: timescaledb health gate in this compose file. In an external-DB/HA deployment, an unhealthy bundled TimescaleDB container can prevent fleet-api from starting even though the configured DB_DSN target is healthy and writable.
• Impact: This undermines the reliability goal of the new external database override: a local, unused database container can block API startup or restarts during maintenance/failover, extending outages for operators who moved fleet data to an HA database.
• Recommendation: Provide an external-DB compose overlay/profile or installer-generated override that removes the fleet-api -> timescaledb dependency when DB_DSN is set. If Grafana alerts are also pointed at the external database, remove or override Grafana's local timescaledb dependency in the same flow and document the required deployment mode explicitly.

Notes

No auth bypass, SQL injection, command injection, cryptostealing/pool hijack, or protobuf wire-format issues were found in the scoped diff. Focused Go tests could not be run because this sandbox is read-only and Go could not create module/build cache directories.

---

_{Generated by Codex Security Review |
Triggered by: @ankitgoswami |
Review workflow run}

[Copilot]

Pull request overview

Adds support for providing an explicit PostgreSQL connection string via DB_DSN, while keeping the existing DB_USERNAME/DB_PASSWORD/DB_NAME/DB_ADDRESS/DB_SSL_MODE configuration path. It also expands DB retry behavior to classify failover-like errors, optionally reset idle pools on those errors, and selectively retry read-only QueryContext calls.

Changes:

• Add DB_DSN support to DB config, including DSN parsing/validation and multi-host safety checks.
• Extend RetryDB with failover-class error detection, optional pool reset, and read-only query retry behavior.
• Wire retry/pool-reset options through Fleet’s SQL store constructors and deployment compose/docs to pass through DB_DSN.

Reviewed changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
server/internal/infrastructure/db/config.go	Adds explicit DSN support, DSN validation, and multi-host/read-write targeting checks.
server/internal/infrastructure/db/config_test.go	Adds unit coverage for DSN precedence, validation, multi-host detection, and non-echo invalid DSNs.
server/internal/infrastructure/db/database_connection.go	Validates config before opening and logs/returns connection context using ConnectionTarget().
server/internal/infrastructure/db/retry.go	Adds failover-class error classification, pool reset option, and read-only query detection for failover retries.
server/internal/infrastructure/db/retry_test.go	Adds tests for failover classification, pool reset behavior, and read-vs-write retry semantics.
server/internal/handlers/minerproxy/handler.go	Threads RetryDBOption variadic options through handler construction.
server/internal/domain/stores/sqlstores/sql_connection_manager.go	Propagates RetryDB options into SQLConnectionManager via a type alias.
server/internal/domain/stores/sqlstores/user.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/transactor.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/site.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/session.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/schedule.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/pool.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/notification_history.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/infrastructure_device.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/fleetnodepairing.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/fleetnodeenrollment.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/fleetnodeauth.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/error.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/discovered_device.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/device.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/curtailment.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/collection.go	Adds an options-aware constructor variant to pass RetryDB options while keeping existing signature.
server/internal/domain/stores/sqlstores/building.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/apikey.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/alert_channel.go	Updates constructor to accept optional connection manager options.
server/internal/domain/stores/sqlstores/activity.go	Updates constructor to accept optional connection manager options.
server/internal/domain/miner/service.go	Allows passing RetryDB options into the miner service’s connection manager.
server/cmd/fleetd/main.go	Wires a pool-reset function into store constructors and the miner proxy handler.
server/cmd/fleetd/main_test.go	Adds coverage that DB_DSN is loaded into the embedded DB config.
server/docker-compose.base.yaml	Passes DB_DSN through the base compose environment.
deployment-files/docker-compose.yaml	Passes DB_DSN through the deployment compose environment.
deployment-files/README.md	Documents DB_DSN override behavior and the multi-host read-write targeting requirement.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e66395b913

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5631372a07

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 947dd61bc8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".