[Bug] on_run_end hangs indefinitely on dbt-fusion in dbt platform after a successful insert - `adapter.commit()` never returns (BigQuery)

[aw-stevens]

Describe the bug

On dbt-fusion running in dbt platform, with BigQuery, elementary.on_run_end() hangs forever after an upload's INSERT has already succeeded. No error, no panic, no further log output — the process simply stops making progress until the orchestrator kills it (dbt platform cancels on a 10-minute inactivity timeout).

The hang is in upload_run_results(), isolated by controlled bisect. Crucially it happens after the insert completes: BigQuery's INFORMATION_SCHEMA.JOBS shows the INSERT into dbt_run_results reaching DONE in ~1s, and then the invocation issues no further query, ever. The next statement in the macro is the commit:

{# macros/utils/table_operations/insert_rows.sql:76-78 #}
{% if should_commit %}
    ...
    {% do adapter.commit() %}

BigQuery has no interactive transactions, so on dbt-core this commit is effectively a no-op. In dbt platform on Fusion it appears to block indefinitely.

Important scoping: this does not reproduce on Fusion generally — see the matrix under Additional context. The same commit path completes on Fusion in GitHub Actions and on Fusion locally. dbt platform is the only variable that isolates the failure, so this may be a platform-runner issue rather than an Elementary one; filing here because Elementary's on_run_end is what issues the commit, and because the should_commit behaviour looks avoidable regardless.

To Reproduce

1. Run a dbt command that produces Elementary uploads on dbt-fusion in dbt platform, against BigQuery — we hit it with both dbt source freshness and dbt run.
2. Let on_run_end reach an upload that actually inserts rows (upload_run_results always does).
3. The INSERT succeeds in BigQuery, then the process stops producing output and never exits.

Isolated by adding one disable var at a time to dbt source freshness:

vars	result
(none)	stalls → 10-min inactivity timeout
disable_freshness_results: true	stalls
+ disable_dbt_invocation_autoupload: true	stalls — identical 4-job trail in BigQuery
+ disable_run_results: true	succeeds, 56.65s

That eliminates upload_dbt_invocation() and clean_elementary_temp_tables() (a no-op here — clean_elementary_test_tables is guarded by {% if test_table_relations %} and the cache is empty on a freshness run), and leaves upload_run_results().

Note the bisect does not clear or convict upload_source_freshness(): in every stalling run upload_run_results was also enabled, so it would have stalled regardless. upload_run_results and upload_source_freshness are the only two uploads that pass append=True, so both bypass the diff path and always insert-then-commit — which makes freshness a plausible second instance of the same problem, but we did not isolate it independently.

The artifact uploads ran to completion in the successful run:

Elementary: [dbt_groups] Artifacts did not change.
Elementary: Uploaded dbt artifacts.
============================ Execution Summary ============================
    INFO Invocation ... freshness with 4 warnings for target prod [56.65s]

BigQuery job trail for a stalled invocation (INFORMATION_SCHEMA.JOBS_BY_PROJECT, filtered on the dbt_invocation_id label):

created	secs	state	statement_type	destination
19:42:39	2	DONE	SELECT	(freshness metadata)
19:42:44	0	DONE	SELECT	(our own on-run-end hook)
19:42:47	1	DONE	SELECT	dbt_artifacts_hashes
19:43:02	1	DONE	INSERT	dbt_run_results

Nothing after 19:43:02. The run was killed at the 10-minute mark.

Expected behavior

on_run_end completes, or fails with a clear error. Specifically, adapter.commit() should not be able to block the run indefinitely on a warehouse with no interactive transactions — skipping the commit when the adapter is non-transactional (or bounding it) would avoid this entirely, regardless of where the underlying blocking behaviour originates.

Screenshots

dbt platform run summary showing the step cancelled at exactly 10m after dbt deps and freshness analysis had both succeeded — can attach if useful.

Environment

• Elementary CLI (edr) version: not involved in the failing run (the failure is in the dbt package's on_run_end; our edr report job runs separately on dbt-core and is unaffected)
• Elementary dbt package version: 0.25.1
• dbt version you're using: dbt-fusion 2.0.0-preview.205 in dbt platform (fails). Same project on dbt-core 1.12.0b2 / dbt-bigquery 1.11.3 in dbt platform completes on_run_end in 66s.
• Data warehouse: BigQuery
• Infrastructure details: dbt platform (dbt Cloud) scheduled job, Fusion Stable release track, prod target. Project size: ~530 models, ~2,160 tests, 188 sources, 34 exposures. columns_upload_strategy: none.

Additional context

Where it does and doesn't reproduce. Neither the Fusion build nor the target dataset isolates the failure — only the execution environment does:

Context	Engine	Elementary schema	Insert + commit exercised	Result
dbt platform	Fusion 2.0.0-preview.205	prod	upload_run_results	hangs
dbt platform	dbt-core 1.12.0b2	prod	all uploads	completes, 66s
GitHub Actions	Fusion 2.0.0-preview.202	prod	artifacts delete_and_insert (which also calls adapter.commit(), delete_and_insert.sql:43)	completes
Local (macOS)	Fusion 2.0.0-preview.205	dev	all uploads	completes, 233s (vs 223s on dbt-core — within 5%)

The GitHub Actions row is a full artifacts change path against the production Elementary schema — temp CTAS → INSERTs → DELETE FROM dbt_models → INSERT INTO dbt_models → DROP temps, all DONE — so a Fusion adapter.commit() after a real insert demonstrably completes outside dbt platform. The pure Jinja/upload work is likewise not pathological on Fusion (local row).

• The log stream lags execution, which made this hard to localize: a BigQuery job at 19:43:02 post-dated the final log line at 19:43:00. Anyone debugging this should correlate against INFORMATION_SCHEMA.JOBS on the dbt_invocation_id label rather than trusting the log tail.
• Possibly related: dbt-fusion panics w "Arrow byte array offset overflow" during Elementary on_run_end even after successful model/test runs #2309 (dbt-fusion panics with "Arrow byte array offset overflow" during on_run_end after successful uploads/inserts — same hook, same trigger point, different symptom, older Fusion preview 173 and Elementary 0.22.1). Also process_freshness_result crashes with "Failed to render SQL undefined value" on dbt-fusion #2245 (process_freshness_result on dbt-fusion), which appears already guarded in 0.25.1.
• Minor adjacent papercut: there are five upload disable vars, and disable_freshness_results is easy to miss — it is not covered by the four listed together in the reduce on-run-end time guide, so dbt source freshness keeps uploading when you believe you've disabled everything. Happy to open that separately as a docs issue.

Would you be willing to contribute a fix for this issue?

Yes — happy to test a candidate fix against our project, where this reproduces reliably in dbt platform. Can attempt a PR if you can point at the preferred approach (skip the commit for non-transactional adapters vs. handle it in insert_rows).

[aw-stevens]

Edited the issue body shortly after filing to correct two overclaims, flagging here for anyone who read the original:

1. I originally wrote that the artifact uploads would hang too once artifacts actually change. That's wrong — I've since confirmed a full artifacts change path (temp CTAS → INSERTs → DELETE → INSERT → DROP, via delete_and_insert, which also calls adapter.commit()) completing successfully on Fusion in GitHub Actions against the same production Elementary schema.

2. I originally implied upload_source_freshness() was also demonstrated to hang. It wasn't isolated — upload_run_results was enabled in every stalling run, so it would have stalled regardless. Freshness takes the same append=True insert-then-commit path so it's a plausible second instance, but that's inference, not a result.

Net effect: the failure is scoped to dbt platform specifically. Neither the Fusion build nor the target dataset isolates it — Fusion preview.205 completes the same path locally, and Fusion preview.202 completes it in GitHub Actions. Full matrix is now in the body. That may well make this a platform-runner problem rather than an Elementary one; leaving it open here since on_run_end is what issues the commit, and should_commit on a non-transactional warehouse looks avoidable either way. Happy to move it if you'd rather.