Read remote SQLite databases through a DuckDB FileSystem VFS

[ak2k]

Reading a SQLite database that lives in object storage or at a URL currently means downloading it first. This lets sqlite_scan and ATTACH open it in place:

SELECT * FROM sqlite_scan('https://example.com/chinook.sqlite', 'Artist');

ATTACH 'https://example.com/chinook.sqlite' AS db (TYPE sqlite);
SELECT count(*) FROM db.Album;

A remote path is opened through a SQLite VFS that forwards reads to DuckDB's CachingFileSystem; local paths keep the native SQLite code path. Because it sits at the FileSystem layer, it works for any filesystem DuckDB exposes (HTTP, S3, GCS, Azure, …) and reuses DuckDB's existing file cache. Queries fetch byte ranges on demand rather than downloading the whole file.

Limitations

• Remote databases are read-only. As a consequence, SQLite's temp store stays in memory, so a large sort or hash that would normally spill to disk is kept in RAM.
• The database must be checkpointed and self-contained; reading a remote WAL is not supported. PRAGMA wal_checkpoint(TRUNCATE) or VACUUM INTO produces a servable copy.
• A query treats the remote object as immutable (the VFS reports SQLITE_IOCAP_IMMUTABLE). httpfs raises an error if the object's ETag changes mid-read; on a server that returns no stable ETag, a concurrent change can be read inconsistently.
• Some network-error classification matches on httpfs message text, so it can be imprecise if those messages change.

Tests are hermetic (after setup): a stdlib Python HTTP server under scripts/ serves committed fixtures over localhost, so core CI needs no network. Optional S3 and high-concurrency paths are gated on environment flags and use a pinned rclone binary the CI fetches. The remote tests pull httpfs (pinned to the commit the bundled DuckDB engine coordinates, with its patches) as a test-only dependency; the WASM build skips it — those tests don't run there, and httpfs's OpenSSL dependency didn't build for emscripten in our build. The extension compiles for the WASM targets, where all paths route through the FileSystem; that path is not exercised at runtime.

Notes

• The remote connection opens lazily, per transaction, on first use. A catalog-level open (one per attachment) is also feasible — the file cache is reachable from the database instance — matching the approach suggested in Transaction starts prematurely during "did you mean" suggestion, causing database open failure #183/[fix #183] add lazy start for sqlite transaction #184; happy to switch to that if you'd prefer.
• Block data is served from DuckDB's file cache after the first read; the -wal/-journal sidecar checks are not — each open re-probes them with a network HEAD, so repeated and parallel queries re-pay those round-trips.

Related Issues: #39, #141

[Maxxen]

Hello!

Looks cool, but I wonder why/if this vfs is limited to http, why not just wrap DuckDBs regular filesystem? You can then let DuckDB handle the underlying abstraction and ensure sqlite can access files through all duckdb supported filesystems (e.g. compressed files, or regular files on WASM)

[ak2k]

[Updated]

@Maxxen Thanks for the excellent suggestions! I've re-implemented based on your feedback.

Integrated with DuckDB's filesystem and caching infrastructure

• Now wraps DuckDB's FileSystem API for all remote file access (HTTP/HTTPS via httpfs)
• Uses DuckDB's CachingFileSystem which provides intelligent block caching
• Cache is managed by DuckDB's CachingFileSystem infrastructure

Simplified concurrency model

• Removed per-file mutexes - file operations are lock-free
• Only uses mutex for VFS registry operations (registration/unregistration)
• Relies on CachingFileSystem's internal synchronization for thread safety

Adaptive read-ahead optimization

• Implements dynamic read-ahead (1MB-128MB) to reduce network round trips
• Adjusts read size based on sequential vs random access patterns
• Works on top of CachingFileSystem's block caching

The implementation is much cleaner now - it properly delegates to DuckDB's existing infrastructure rather than reimplementing caching logic. Thanks again for the great foundation and suggestions!

[Alex-Monahan]

This is very cool! Does it work with hosted SQLite solutions like Turso?

[ak2k]

This is very cool! Does it work with hosted SQLite solutions like Turso?

Thank you @Alex-Monahan! I wasn't familiar with Turso, but it seems to implement a custom wire format rather than providing HTTP-like access to the stored file format, so regrettably, it likely wouldn't.

[ak2k]

I've reorganized this PR into 5 logical commits to attempt to make the review process easier. Each commit is self-contained, compiles, and passes all tests:

Commit 1: Add ClientContext parameter to SQLiteDB::Open methods

• Minimal API change to support future extensibility
• Required for subsequent commits that need access to DuckDB's context

Commit 2: Lazy initialization when opening remote SQLite databases

• Implements lazy initialization to prevent deadlocks during remote file access
• Includes necessary validation logic for busy_timeout

Commit 3: Add SQLite VFS implementation for remote file support

• Core VFS implementation that integrates with DuckDB's CachingFileSystem
• Handles HTTP error mapping and adaptive read-ahead optimization
• No user-facing changes yet

Commit 4: Enable HTTP/HTTPS SQLite database access via sqlite_scan

• Activates HTTP/HTTPS support for sqlite_scan() function
• Includes 7 working tests demonstrating basic functionality
• ATTACH support intentionally deferred to next commit

Commit 5: Add HTTP ATTACH support and improve error handling

• Completes the implementation with ATTACH functionality
• Improves error handling throughout the PR
• Adds remaining tests for complex queries

Let me know if you'd like me to adjust the commit structure or if you have any questions about specific changes or feedback.

[ngalluzzo]

Curious if this is blocked by something? Would love to help get this merged if any additional contributions are necessary

[FIGIO55]

Trying to see if there are updates on this matter since it would be very useful for me

[ak2k]

Keeps the existing approach — remote databases open through a SQLite VFS over DuckDB's CachingFileSystem — and revises the implementation:

• drops the adaptive read-ahead layer (CachingFileSystem already caches);
• expands the -wal/-journal fail-closed handling so an un-checkpointed remote database is refused rather than read inconsistently;
• adds a hermetic stdlib HTTP test server in place of network-dependent tests;
• narrows the diff to the feature (the earlier version also modified many existing storage tests).

Restructured into four focused commits and rebased onto current main (the prior base was ~a year old).

[ak2k]

CI surfaced a race in http_sqlite_02_concurrent_scans: under concurrent first-access, CachingFileHandle::GetFileHandle() opened the shared underlying handle without FILE_FLAGS_PARALLEL_ACCESS, so parallel block fetches raced on the non-thread-safe HTTPFS handle and a cache block could be read with zero bytes — surfacing in FileBufferHandleGroup::CopyTo.

That race is fixed upstream in duckdb/duckdb#22979 (merged 2026-06-01). The pinned engine here, a966898d (2026-05-28, the same commit main pins), predates it. This bumps the duckdb submodule to the #22979 merge commit to pick up the fix.

I can split the engine bump into its own PR if you'd rather land it separately.

[staticlibs]

Hi, thanks for the update! It is fine to have the duckdb submodule updated as part of this PR.