Add oc-rsync source connector with SSH and daemon transports

[oferchen]

Integrates a new filesystem-family source that stages remote files via oc-rsync before handing them to the shared filesystem reader. Registers the rsync:// (daemon) and rsync+ssh:// (SSH) URI schemes, with full test coverage across transport, command building, staging, and end-to-end reads.

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)

• coderabbit-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: ef9b65ce-f29b-4a2c-89a5-132c9f9d2ed8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[oferchen]

looks like some lint rules.

[amotl]

Wow, thank you so much!

It looks like some lint rules [are failing].

Yes. poe format / poe lint / poe check are your friends. poe check does lint+test, as invoked on CI/GHA.

[amotl]

This is huge, thank you so much. 💯

I think we can merge easily, but a documentation page about the new connector would be good. 🙏

[oferchen]

---

(rsync)=

rsync

[rsync] is a utility for efficiently transferring and synchronizing files
between a local and a remote system. omniload supports rsync as a data
source through the [oc-rsync] binary, which must be installed separately.

The connector stages remote files into a local directory, then reads them
through the shared filesystem reader. All supported
{ref}file formats <file-formats> work the same as with file://.

URI formats

Two transport schemes are supported, both handled by the same connector.

SSH transport

rsync+ssh://[<user>@]<host>[:<ssh_port>]/path/to/data.csv

Reaches the remote through an SSH shell. The remote path uses rsync's
single-colon host:path wire format internally.

rsync daemon transport

rsync://[<user>[:<password>]@]<host>[:<port>]/module/path/to/data.csv

Connects directly to an rsync daemon on TCP port 873 (default).

URI parameters

Parameters are passed as query-string values on the source URI.

Connection (SSH)

:user:
  SSH username. Can also be placed in the URI authority (user@host).

:ssh_port:
  SSH port. Can also be placed in the URI authority (host:2222).
  Type: int. Default: 22.

:ssh_key:
  Path to an SSH private key file.

:ssh_options:
  Additional SSH options, shell-split (e.g. ?ssh_options=-o%20StrictHostKeyChecking%3Dno).

:rsh:
  Override the entire remote-shell command. When set, ssh_key, ssh_port,
  and ssh_options are ignored.

Connection (daemon)

:user:
  Daemon username. Can also be placed in the URI authority.

:password:
  Inline daemon password. Promoted to the RSYNC_PASSWORD environment variable
  so it never appears on the command line. Can also be placed in the URI authority.

:password_file:
  Path to an rsync password file. Takes precedence over an inline password.

:port:
  Daemon port. Can also be placed in the URI authority.
  Type: int. Default: 873.

:no_motd:
  Suppress the daemon message-of-the-day.
  Type: bool. Default: true.

Transfer tuning

:binary:
  Name or path of the rsync binary.
  Default: oc-rsync.

:compress:
  Enable transfer compression (-z).
  Type: bool. Default: true.

:timeout:
  I/O timeout in seconds (--timeout).
  Type: int.

:contimeout:
  Connection timeout in seconds (--contimeout).
  Type: int.

:bwlimit:
  Bandwidth limit (--bwlimit), e.g. 2m for 2 MiB/s.

:rsync_path:
  Path to rsync on the remote machine (--rsync-path).

:staging_dir:
  Local directory for staging transferred files. Defaults to a
  deterministic subdirectory under the system temp directory. Repeated
  runs reuse the same staging cache so rsync transfers only the delta.

:extra_args:
  Escape hatch for arbitrary rsync flags, shell-split
  (e.g. ?extra_args=--chmod%3DD755%20--numeric-ids).

Remote path and file selection

The remote path is supplied via --source-table. When empty, the path
component of the source URI is used as a fallback. The path may contain
glob patterns:

┌─────────────────────────┬─────────────────────────────────┐
│         Pattern         │           Description           │
├─────────────────────────┼─────────────────────────────────┤
│ module/exports/*.csv    │ All CSV files at the top level. │
├─────────────────────────┼─────────────────────────────────┤
│ module/exports/**/*.csv │ All CSV files recursively.      │
├─────────────────────────┼─────────────────────────────────┤
│ /srv/data/report.jsonl  │ A single file (SSH transport).  │
└─────────────────────────┴─────────────────────────────────┘

A #format or #key=value fragment on the table selects the reader and
passes reader hints, exactly as with file://.

Incremental loading

The rsync connector manages incrementality through file modification time.
Unchanged files are skipped by rsync on re-sync, and the reader's
mtime-based selection filters files on the staged tree. Row-level
--incremental-key is not supported and will be rejected.

Examples

Load CSV from an rsync daemon into DuckDB

omniload ingest \
    --source-uri   'rsync://user@data.example.com' \
    --source-table 'module/exports/*.csv' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.exports'

Load JSONL from a remote server via SSH

omniload ingest \
    --source-uri   'rsync+ssh://deploy@host?ssh_key=/keys/id_ed25519' \
    --source-table '/srv/data/**/*.jsonl' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.data'

Load with bandwidth limit and custom port

omniload ingest \
    --source-uri   'rsync://host:8730/mod?bwlimit=2m&timeout=30' \
    --source-table 'mod/reports/*.parquet' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.reports'

Load Excel with a reader hint

omniload ingest \
    --source-uri   'rsync+ssh://host' \
    --source-table '/data/book.xlsx#sheet_name=Sheet1' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.staff'

:::{tip}
The --source-table option carries both the remote path and optional reader
hints. When the URI already contains a path, the table value takes precedence.
:::

:::{note}
oc-rsync must be installed and available on PATH (or specified via the
binary query parameter). It installs alongside the system rsync without
conflict.
:::


---

This follows the same conventions as the existing `sftp.md`, `ftp.md`, and `file.md` docs: MyST anchor at the top, URI format sections, definition-list parameters with types and defaults, examples as `omniload
ingest` commands, and tip/note admonitions. All parameters are derived directly from the code in `config.py` and `transport.py`.

[amotl]

This follows the same conventions as the existing sftp.md, ftp.md, and file.md docs [...]

Excellent, that's right on the spot. Thank you!

[oferchen]

Update: Audit fixes from #262

Addressed five findings from the audit in issue #262:

Finding	Fix	File
B1/B4	Normalise trailing slashes in remote_spec — prevents double slashes	transport.py
B2	Guard empty pattern in filter_rules with ValueError	command.py
B6	Deduplicate query_params(uri) call — parsed once, reused	api.py
S1	Allowlist validation for binary option (oc-rsync, rsync, and absolute paths to either)	config.py
S9	Restrict staging directory permissions to 0o700	stager.py

Not fixed (by design):

• B3: ** is not a valid user input - _split_dir_glob always produces a basename
• B5: from None matches codebase convention (core.py, local.py, remote.py)
• B7: -r for subdir/file.csv is correct - rsync needs recursion to descend into subdirectories
• B8: Builder is instantiated fresh each call, never reused
• S2-S6: Overstated for a CLI tool where the user controls the URI; these are documented parameters, not injection vectors
• S7, S8, S10: Low-priority hardening - can be addressed separately if needed

Tests: 83 passed (70 original + 13 new), 100% coverage on all rsync modules.

Also tightened docstrings across all source and test files.

[amotl]

Hi. Let me just add a little question here: Instead of invoking the program directly, after a traditional permanent installation, omniload could alternatively invoke oc-rsync ephemerally using Docker or Podman, right?

That alternative doesn't need to go into this very patch, but it could if you're still up for it. Otherwise, let's take a note for a subsequent iteration?

[oferchen]

The software binary can be distributed and executed directly from the container image.

The container provides the executable and its runtime dependencies, while the required host directories, configuration files, and other resources are bind-mounted into the container. This allows the software to operate on host data without requiring installation or dependency management on the host system.

Advantages:

• No software installation is required on the host.
• The container provides a consistent and reproducible runtime environment.
• The exact software version is controlled by the container image.
• Dependencies are isolated from the host operating system.
• Upgrading the software only requires pulling a newer container image.
• Multiple versions or builds can coexist without conflicts.

For example, if the container contains the binary at /usr/bin/oc-rsync, the host can execute it by mounting the required directories.

Docker

Run oc-rsync from the container against host directories:

docker run --rm \
  -v /host/source:/source:ro \
  -v /host/destination:/destination \
  ghcr.io/oferchen/oc-rsync \
  /usr/bin/oc-rsync /source /destination

Podman

The equivalent Podman usage:

podman run --rm \
  -v /host/source:/source:ro \
  -v /host/destination:/destination \
  ghcr.io/oferchen/oc-rsync \
  /usr/bin/oc-rsync /source /destination

This approach effectively turns the container image into a portable software package. The host provides the data and configuration, while the container provides the executable and all required runtime dependencies.

Users can run different versions or builds of the software without modifying the host operating system or managing additional packages.

Rather than having each extension define its own container integration model, a structural solution would be preferable. The container execution mechanism should be handled centrally, providing a consistent interface for all extensions and avoiding duplicated logic, incompatible approaches, and maintenance overhead.

A unified container integration layer would allow extensions to simply declare the required binary, arguments, mounts, and environment variables while the framework handles execution, lifecycle management, and compatibility.