Skip to content

Batch bounded sequences of small asset writes through Git #110

Description

@flyingrobots

Problem

A Think-sized git-warp v18-to-v19 rehearsal still showed short-lived Git children after commit reads and commit writes were moved onto persistent plumbing. The remaining hot path translates thousands of small patches and attachments through repeated assets.put() calls.

On git-cas v6.5.5, ChunkRepository.storeChunk() calls persistence.writeBlob() once per chunk, and GitPersistenceAdapter.writeBlob() runs one-shot git hash-object -w --stdin. git-cas already has a bounded writeBlobs() path backed by one scoped fast-import session, but ordinary asset ingestion does not consume it across independent small assets. This is distinct from #98, which preserves the one-shot fallback when duplex sessions are unavailable.

Desired outcome

Provide a git-cas-owned, bounded batch boundary for a sequence of independent asset writes so callers such as a migrator can submit many small streaming assets without receiving raw Git/session authority. The implementation may use putMany, an explicit scoped asset batch, or another semantic API, but process count should scale with bounded flush windows rather than one-chunk asset count.

Required invariants

  • Sources remain streaming; batching has explicit item and byte bounds.
  • Results preserve input order and produce the same manifests, trees, handles, and Git OIDs as individual put() calls.
  • Chunk hashing, encryption, compression, deduplication, and integrity metadata remain unchanged.
  • A failed item cannot silently expose partial success as a complete batch.
  • Scoped fast-import checkpoints and closes before returned handles escape.
  • Sessionless Plumbing continues to use the typed one-shot fallback from Fall back to one-shot Git I/O when Plumbing lacks duplex sessions #98.
  • Callers never receive raw Git plumbing or a mutable persistence adapter.

Acceptance criteria

  • A real-Git test stores a bounded sequence of one-chunk assets with O(windows) Git write processes rather than O(assets).
  • A mixed-size test proves byte/item bounds and backpressure.
  • Golden tests prove byte-identical manifests, trees, handles, and OIDs versus repeated put().
  • Failure and cancellation tests prove deterministic cleanup and result semantics.
  • SHA-1 and SHA-256 repositories are covered.
  • A git-warp migration witness records child-process count and wall-time before and after adoption.

Non-goals

  • A persistent repository-lifetime fast-import process.
  • Exposing raw Git sessions publicly.
  • Buffering an entire migration or large attachment in memory.
  • Expanding git-warp PR #825; this is follow-up performance work.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:storageStorage, Git objects, manifests, or persistenceenhancementNew feature or requeststatus:needs-designRequires a design before implementationtype:designDesign doc or proof-plan worktype:sliceTurn-sized proof work under a goalpost

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions