csveee

A very fast, parallel CSV parser.

csveee splits a CSV file into chunks and parses them across all cores. It inverts the usual interface: instead of the parser handing records to your code, you hand your code to the parser. That buys what an iterator cannot — parsing that stays lazy and runs in parallel. Your code runs inside the parse. Speculatively, before a chunk's true record boundaries are known (how it works).

It does all this without giving up on the messy files the real world is full of, and it is quick about it: across 1,000 CSV files csveee is around 10× faster than rust-csv, and on a large server it peaks at 192 GB/s.

The parsing scheme and the fused programming model come from One Pass to Parse Them All: Fused Parallel CSV Processing (VLDB '26); the crate has grown past the paper since.

Features

• Parallel by default. Every chunk is parsed on its own thread and the per-chunk results are folded back together in file order.
• Parsing and processing in one pass. A file larger than the CPU caches passes through them once, not twice — twice would halve throughput.
• Two chunk parsers. SIMD on nightly, a DFA on stable for any configuration.
• Many dialects, not just RFC 4180, real files. Configurable delimiters, terminators, escapes, comments, and three different quote handling modes. Records of varying length, \r\n/\n/\r and mixed newlines, headers, comments, blank lines.
• I/O picked by file size. Per-chunk reads or a ring buffer; mmap opt-in.
• No per-record allocation. Fields are slices into the parser's own buffer. Nothing is copied unless you copy it.

Usage

use csveee::Parser;

let mut parser = Parser::new();

let cities = parser.parse(
    "data.csv",
    Vec::new,                       // init  -> Vec<String>, one per chunk
    |state, [_name, _age, city]| {  // acc   (&mut Vec<String>, [&mut str; 3])
        state.push(city.to_string());
        Ok(())                      //        Err(_) rejects the record
    },
    |states| states.concat(),       // merge (&mut [Vec<String>]) -> Vec<String>
)?;

The three callbacks form a user-defined aggregate: init creates a state, acc fills it, and merge combines the per-chunk states. The [_name, _age, city] pattern declares the record arity — a record with a different number of fields is rejected, just as when the accumulator returns an error. Fields are &mut str into the parser's buffer, valid for the call only. init and acc are Fn and Sync — they run on all the threads in parallel, init once per chunk, in fact once per speculative pass.

The accumulator does double duty: it folds records into the state, and its rejections refute the parse state assumed for a chunk, which is then parsed again under the next one. So acc may see the same bytes twice, cut into different records each time.

merge is FnOnce: it folds the surviving states, in file order, sequentially.

For bytes already in memory there is parse_slice, and for sources without random access parse_stream, a sequential fallback.

Configuration

ParserBuilder sets the dialect and the execution backends:

use csveee::{ParserBuilder, QuoteHandling, RecordTerminator};

let mut parser = ParserBuilder::new()
    .delimiter(b';')
    .quote(Some(b'"'))
    .quote_handling(QuoteHandling::Strict)
    .terminator(RecordTerminator::LF)
    .comment(Some(b'#'))
    .has_headers(false)
    .concurrency(4)
    .build();

• Quote handling: Toggle — every quote toggles quoting, whatever its position; Strict — RFC 4180, where a quote inside an unquoted field is an error; Literal — that same quote is an ordinary character.
• Output modes: fixed-arity arrays by default; flexible() hands the accumulator a slice so records may vary in length — validate in the accumulator to keep speculation cheap, see How it works; bytes() hands out &mut [u8] and skips UTF-8 validation entirely.
• Execution: concurrency, chunk size, parser backend, I/O backend, and a memory cap for the ring buffer.

The examples/ directory has a runnable program for each of these.

Performance

Measured over 1,000 files from Kaggle on an AMD EPYC server, one point per file:

The geometric mean of per-file throughput is 7.7 GB/s — 4.7 GB/s with the DFA parser on stable — against 0.8 GB/s for rust-csv and 0.3 GB/s for DuckDB: a geometric-mean speedup of 10.85× over rust-csv and 25× over DuckDB.

The speedup is a curve. Startup is a fixed cost, so the more bytes there are to parse the smaller its share becomes, and the parsing that remains is split across every available core. Against rust-csv, small files spend most of their time in startup and land nearer 2–4×; from there the margin widens with each extra byte and each extra core, past 200× on the largest files. DuckDB pays a fixed per-query cost of its own, so its curve starts high, bottoms out near 100 MB once that cost is amortized over enough bytes, and widens again as parallelism takes over.

Benchmarks live in benches/throughput.rs. Reproduce them with:

$ pip install -r scripts/requirements.txt
$ python3 scripts/provision_data.py   # downloads the corpora
$ cargo bench --features bench

How it works

A chunk boundary can land anywhere, including in the middle of a quoted field, so a worker cannot know the state its chunk starts in. Rather than scanning the file first to find out, each chunk is parsed speculatively: one pass per possible starting state, each from a fresh initial state, until one holds up. Every pass is kept, not just the one that turns out to be right: on a genuinely broken file they all end in an error, and the merge needs them all to report the real one.

The merge phase then walks the chunks in order, aligning record boundaries: the pass whose first record starts where its predecessor's last record ended is the correct one. The surviving states are folded with your merge callback (see Usage). The cost of the speculation is a constant factor of extra parsing work per chunk, and it buys a parser that never has to look at the file twice.

What kills a wrong pass early is validation. The declared record arity and the errors the accumulator returns (e.g., failed type conversions) form the oracle the speculation is checked against, so the more it rejects, the sooner a wrong pass dies. An oracle with no information — flexible() plus an accumulator that accepts every record — lets a wrong pass run to the end of the chunk, and the merge reparses it sequentially. The result is correct either way; the cost is the wasted work.

There are two chunk parsers, and the same one handles every chunk of a parse. The DFA parser drives a state machine byte by byte; it supports every configuration and builds on stable Rust. The SIMD parser (the simd feature, nightly-only, built on portable_simd) finds delimiters, terminators, and quotes a vector at a time and resolves quoted regions with bit-parallel arithmetic. It covers the common dialects, and ParserBackend::Auto takes it whenever the feature is on and the configuration allows — the DFA otherwise.

Input is read through one of three I/O backends. Per-chunk reads copy a chunk at a time into a per-thread buffer; a ring buffer gives each thread its own bounded window, for large machines or streamed input; memory maps give each thread its own private mapping and let the page cache do the work. Auto takes the first for small files and the second for large ones; mmap is opt-in and Unix-only.

Speculative parsing, the merge phase, the vectorized parser, and the ring buffer were developed and evaluated in full in the paper; the DFA parser, the dialect coverage, and the other I/O backends go beyond it.

Correctness

The test suite parses ~2,500 real-world CSV files — 1,000 sampled from Kaggle, DuckDB's and Postgres's own CSV test corpora, and rust-csv's — and compares csveee's output field for field against rust-csv, across output modes, parser backends, and I/O backends. cargo test runs the checked-in fixtures; the larger corpora are downloaded by scripts/provision_data.py (the Kaggle suite is opt-in and needs credentials).

Feature flags

Feature	Description
simd	SIMD-vectorized chunk parser. Requires nightly.
simdutf8	SIMD-accelerated UTF-8 validation for Text output.
trace	tracing instrumentation.
bench	Required to build the benchmarks.

All are off by default; the crate builds on stable with the DFA parser alone.

Status

Early days — the API is not stable yet and may change between releases. Requires Rust 1.88 or newer (edition 2024). Tested on Linux, macOS, and Windows.

Citation

If you use csveee in academic work, please cite the paper:

Simon Ellmann and Thomas Neumann. One Pass to Parse Them All: Fused Parallel CSV Processing. PVLDB, 19(11): 3579–3591, 2026. doi:10.14778/3836663.3836710

@article{ellmann2026onepass,
  author  = {Simon Ellmann and Thomas Neumann},
  title   = {One Pass to Parse Them All: Fused Parallel {CSV} Processing},
  journal = {Proceedings of the VLDB Endowment},
  volume  = {19},
  number  = {11},
  pages   = {3579--3591},
  year    = {2026},
  doi     = {10.14778/3836663.3836710}
}

The evaluation artifacts are at ackxolotl/csveee-evaluation.

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.