Record exact label availability when building LMDB datasets

[OutisLi]

Summary

LMDB generation should record exact per-frame label availability in dataset metadata. Data readers should consume that metadata instead of reading or probing frame payloads at training startup to infer label-availability signatures.

Motivation

A training batch exposes one find_* value for each label. Frames with different optional-label availability therefore need either exact homogeneous grouping or conservative reduction at batching time.

Determining those groups online is problematic for large LMDB datasets:

• An exact result requires inspecting every frame before useful training work starts.
• A bounded probe avoids that startup scan for common uniform datasets, but it is inherently probabilistic.
• Rare availability signatures outside the sample may be missed. If they later enter a mixed batch, conservative reduction preserves correctness but discards valid supervision from the other frames in that batch.
• Genuinely heterogeneous datasets may still trigger an expensive full scan.

Label availability is an intrinsic property of the serialized dataset, so it should ideally be computed once when the LMDB is created rather than rediscovered by every consumer.

Proposal

When generating an LMDB dataset:

1. Normalize label names according to the same key mapping used by readers.
2. Compute each frame's effective availability from source presence and any explicit find_* value.
3. Encode distinct availability bitsets as compact signature IDs.
4. Store a schema-versioned signature table and one compact signature ID per frame in LMDB metadata.
5. Store dataset-level uniformity directly when all frames share one signature.

At read time:

1. Select only labels required by the active model and loss.
2. Project the stored signatures onto those labels without decoding frame payloads.
3. Partition tracked optional labels exactly.
4. Keep mandatory-label validation during frame decoding as a corruption and contract check.
5. Fall back to the existing reader-side detection path for legacy LMDBs without this metadata.

Implementation considerations

• Metadata should describe source facts, not model-specific policy. Decisions such as whether a missing label may use a default should remain with the consumer.
• LMDB merge and filtering utilities should preserve or rebuild the signature metadata, including remapping signature IDs where necessary.
• The metadata schema should be versioned so readers can reject malformed or incompatible indexes safely.
• A compact representation should have low overhead: typically one small integer per frame plus a small table of unique signatures.

Expected benefits

• Exact label grouping without probabilistic misses.
• No O(number of frames) payload scan at training startup.
• Reusable availability information for training, statistics, validation, sampling, and inference.
• A clearer separation between immutable dataset facts and consumer policy.

Relationship to reader-side mitigation

A reader-side change remains useful for existing datasets: defer availability checks until label requirements are known, probe only active optional labels, and build an exact compact index only when necessary. That approach should be treated as backward-compatible mitigation for legacy LMDBs; generation-time metadata should be the preferred long-term source of truth.