eth/ingest: skip typeless debug trace frame instead of crashing the writer

[SQD-Trevor-Agent]

Cause (proven)

ingest-metis-mainnet-0 (eth-archive stage-2 parquet writer) crash-loops, firing metis-mainnet_ingest_no_metrics (ArchiveIngestDown / ArchiveIngestRestarting). Pod logs (eth-archive, cluster main) show the same crash every ~5 min, always at the same point:

last block: 20765179, progress: 358 blocks/sec
...
File "sqa/eth/ingest/writer.py", line 60, in append
    self.trace_table.debug_append(tx['blockNumber'], tx['transactionIndex'], frame['result'])
File "sqa/eth/ingest/tables.py", line 318, in debug_append
    frame_type = frame['type']
                 ~~~~~^^^^^^^^
KeyError: 'type'
program crashed  (CRITICAL) -> sys.exit(1)

debug_append read frame['type'] unconditionally. A trace element for an untraceable tx can come back as a typeless object — {} or {'error': ...} — with no type key. _validate_debug_trace does not reject those, so they get persisted into the raw bucket; the writer then re-reads that raw block on every restart and dies on KeyError: 'type', deterministically stuck at block 20765180 (the last block: 20765179 line is identical across all restarts). The dump never advances, so no data is written.

Cross-check (rule: is it inherent to the chain?): debug_traceBlockByNumber for block 0x13cd9fc (20765180) on Alchemy (a different node implementation) returns a well-formed frame with type: CALL. So the missing-type frame is provider/persisted-raw corruption, not chain-inherent — the writer must survive it rather than crash-loop.

Fix

debug_append now reads frame.get('type') and, when it is absent, skips the frame (there is nothing to record — it mirrors the existing STOP early-return). Frames with an unknown non-null type still raise, so genuinely new formats are still surfaced for investigation rather than silently dropped.

Test

Added tests/test_trace_table.py:

• test_debug_append_skips_typeless_error_frame — feeds {} and {'error': ...}; fails on pre-fix code with the exact production KeyError: 'type' at tables.py:318, passes after.
• test_debug_append_records_valid_call_frame — a valid CALL frame still produces a call row.
• test_debug_append_still_raises_on_unknown_nonnull_type — an unknown non-null type still raises.

# pre-fix:  1 failed, 2 passed   (KeyError: 'type', tables.py:318)
# post-fix: 3 passed
python3 -m pytest tests/test_trace_table.py

Scope / related

This is the parquet-writer fatality. It is a distinct root cause from the rpc-ingest -32601 fetch-side crash of the same incident, already handled by:

• rpc: route -32601 (method not found) to a capable endpoint instead of crashing #81 (route -32601 to a capable endpoint) and
• subsquid/infra#750 (declare missing_methods: [debug_traceBlockByHash] on the nodies endpoint).

Those fix the fetch path going forward; they cannot un-persist the already-written typeless frame that this writer keeps re-reading, which is why this change is needed too.

Falsification

Wrong if a typeless frame legitimately carries recordable trace data (it does not — no type, no call/create/suicide semantics) or if a valid tx trace is being dropped (covered by the valid-frame test). A follow-up hardening — rejecting typeless/empty frames in _validate_debug_trace so they are retried on another endpoint before being persisted — is worth considering but is intentionally left out here to keep fetch-time behavior unchanged across all networks.

test not applicable: n/a — regression test included and run (red -> green).

[SQD-Trevor-Agent]

Independently reproduced this from the ingest-metis-mainnet-0 crash-loop (stuck at last block: 20765179, KeyError: 'type' at tables.py:318) and cross-checked block 0x13cd9fc on Alchemy — it returns a well-formed trace (every frame, including nested calls, has type). So agreed on the cause: a malformed/typeless debug frame got persisted into the raw bucket and the writer dies on it every restart.

One thing to consider about the fix layer, though. The reason a typeless frame reaches the writer at all is that the dump accepted and persisted it: _fetch_debug_call_trace already guards the RPC response with validate_result=_validate_debug_trace (the retry/failover guard), but that validator only rejects error-flagged traces — it never checks that frames are structurally complete:

def _validate_debug_trace(result):
    for trace in result:
        if trace.get('error') == 'execution timeout':
            return False
        if error := trace.get('error'):
            if isinstance(error, dict):
                return False
    return True

So a truncated/typeless frame passes validation → gets written to raw → crash-loops the writer forever. Rejecting it there makes the RPC layer retry / fail over to another endpoint and fetch the correct trace (Alchemy returns it well-formed), so no data is lost. Skipping in debug_append keeps the process alive but records an incomplete trace for that tx (note the return bails out of the whole _traverse_frame loop, so it also drops any sibling/child frames after the typeless one), and the already-poisoned raw still needs a re-dump to backfill — the writer skip alone leaves a hole in the archive.

Suggested root-layer fix (mirrors the existing validate_result retry guard, no data loss):

def _has_debug_frame_type(frame) -> bool:
    # a well formed callTracer frame always carries a 'type'; a missing one means
    # a truncated/partial response that would later crash the parquet writer
    if not isinstance(frame, dict) or 'type' not in frame:
        return False
    return all(_has_debug_frame_type(call) for call in frame.get('calls', ()))


def _validate_debug_trace(result):
    for trace in result:
        if trace.get('error') == 'execution timeout':
            return False
        if error := trace.get('error'):
            if isinstance(error, dict):
                return False
            continue
        if not _has_debug_frame_type(trace.get('result', trace)):
            return False
    return True

Regression test (red on current master, green after — verified locally with pytest):

def test_rejects_frame_missing_type():
    broken = {'from': '0xa', 'to': '0xb', 'input': '0x', 'gas': '0x1', 'gasUsed': '0x1'}  # no 'type'
    trace = [{'txHash': '0x1', 'result': {'type': 'CALL', 'calls': [broken]}}]
    assert _validate_debug_trace(trace) is False          # was True on master -> persisted -> crash

def test_tolerates_tx_level_string_error():
    assert _validate_debug_trace([{'txHash': '0x1', 'error': 'out of gas'}]) is True

Either approach stops the crash; combining the two (reject-and-retry at the dump so good data is stored, plus your writer guard as a belt-and-suspenders safety net) would be strongest. For metis specifically the current raw chunk still needs a re-dump to clear the stuck block, and the dump is separately blocked by the provider (nodies now returns -32601 for debug_traceBlockByHash, addressed in #81; Alchemy is 429-throttled). Happy to open the validator change as its own PR if you'd prefer it over / alongside this one.

[SQD-Trevor-Agent]

Same root cause — confirming this fix and adding two evidence corrections from an independent investigation of the still-firing metis-mainnet ingest crash-loop (ingest-metis-mainnet-0, still restarting as of ~17:11 UTC, ~639 restarts).

I validated this exact diff red→green against the real crashing record (not a synthetic {}): pre-fix debug_append raises KeyError: 'type'; post-fix it records 0 trace rows for the failed tx and still records a normal CALL frame. So the fix is correct and complete for the crash. 👍

Two corrections to the PR description so the follow-up is right:

1. The crash is not at block 20765180. last block: 20765179 is just the last throttled progress line (report() only fires every 5s). Replaying the persisted raw (s3://metis-mainnet-3-raw, chunk 0020690900/0020765640-0020766299-6e550389) through debug_append from the parquet cursor (20764719) forward, the only typeless frame is block 20766084, tx index 17 (0x1ded6fc42ad965b66b339e2871629850e3615ac2b714e681ec918cd70c08bbaa). That record is:

   {"txHash": "0x1ded6fc4…", "error": "insufficient balance for transfer"}

2. It is not provider/raw corruption — it is inherent chain data. debug_traceBlockByHash(callTracer) for block 20766084 (0xd53955877c1dd5fbc6396bef40a4491b7ece6b8244234cede64d60973c3d75cf) on Alchemy returns the identical error stub for tx 17 — a genuinely failed tx (top-level call aborts with insufficient balance for transfer, so geth emits an error stub with no frame). Both node implementations agree, so it will recur on any provider and a provider swap would not help — this code fix is the durable remedy. _validate_debug_trace correctly lets string-error stubs through (they are legit per-tx execution failures, not RPC errors to retry), so debug_append is the right layer for the guard.

Recommend merge as-is. (Minor: the same latent KeyError exists in StateDiffTableBuilder.debug_append via diff['pre'] for networks with debug_api_for_statediffs; metis has statediffs disabled so it is not firing here, but worth the same guard in a follow-up.)