Filter image files by run number in VENUS image cataloging

[darshdinger]

Short description of the changes:

Catalog only the image files that belong to the run being cataloged, identified by the Run_<run_number>
token the VENUS DAQ writes into every image file name. This removes the redundant re-cataloging of earlier
runs' images that was making VENUS image cataloging slow enough to back up the shared autoreducers.

Long description of the changes:

The image file path PV is meant to name an image file, but for several detectors (QHY, Andor) it names the
directory holding a whole series of images, and consecutive runs of a series share that directory. The agent
cataloged every FITS/TIFF file in that directory, so each run re-cataloged every image written by the runs
before it. The cost grows with the length of the series, and since cataloging produces a thumbnail by reading
the full image, it ended up taking longer than writing the data out.

Checking ONCat for already-ingested files was considered and rejected: it becomes wasteful once a series
has hundreds of images, and it makes a job's behavior depend on the order and success of other jobs.
Per the agreement with the DAQ team, this filters by run number instead, which is a local, self-contained
rule.

Verified against production data before implementing, for each detector family:

Run	Detector	PV points at	Files in location	Cataloged now
24828 (IPTS-25778)	QHY411	series directory	881, one per run	1
24436 (IPTS-35825)	Andor iKon-XL	series directory	1	1
17853 (IPTS-25778)	TPX1	run directory	1362	1362
23221 (IPTS-36967)	MCP TPX1	two run directories	5001 + run 23220's	5001

The 881-file QHY directory is the pathological case from the report: 881 images spanning 881 distinct runs,
of which exactly one belongs to run 24828.

The run token is matched wherever it appears in the name, not only as a YYYYMMDD_Run_<run> prefix.
The convention quoted in the discussion holds for 114 of 120 recent runs sampled (2026-03 to 2026-06), but
TPX3 prepends a second, fixed date — 20250428_20260310_Run_15395_no_samp_ugg_test_...tiff — and some 2025
TPX1 data leads with the run token instead (Run_8787_20250516_...fits). Anchoring to the start of the name
would catalog zero images for those runs while still reporting COMPLETE. Checked end-to-end against 143
runs across every IPTS: the anchored form silently skips 19 of them, matching the token by position skips
none. This also matches the request in the discussion to filter on the run number appearing "somewhere in
the full tiff file path".

Two further findings from that check, both of which the filter now handles correctly:

• Some runs' image path points at the previous run's directory (run 7353 -> .../Run_7352_DSet0,
  run 15291 -> ..._Run_15289_...). Those images are no longer cataloged under the wrong run; previously
  6000 and 2075 images respectively were.
• Across the 143 runs, 547,415 candidate images reduce to 326,505 cataloged, so about 40% of the
  cataloging work was redundant. For the QHY series specifically the reduction is 881 -> 1.

Specifics:

• matches_run_number() implements the naming rule: a Run_<run_number> token that starts the name or
  follows an underscore, and is followed by a non-digit. The trailing boundary matters — otherwise run 2482
  would claim the images of runs 24820-24829, and that is not hypothetical: a substring match on Run_2482
  hits 2 files in the QHY directory above, while the bounded match hits 0.
• image_files() now takes the run number and filters every candidate through that rule.
• _image_files_at() resolves a single location, which may be a file or a directory. Both are filtered
  the same way, as agreed — a single file is not assumed to belong to the run naming it. This also fixes a
  latent bug: a PV naming a file was previously dropped by an os.path.isdir() guard, so those runs
  cataloged no images at all.
• A metadata path can report several locations, and MCP TPX1 runs report both this run's directory and the
  previous run's; the previous run's images are now filtered out. Duplicate locations ingest once.
• Logging states how many files were skipped as belonging to other runs, how many are being cataloged, and
  warns when a location yields nothing or does not exist.

Out of scope, noted while surveying the data: .tif files exist under VENUS images (~1500 in the sampled
IPTS directories) but only .fits and .tiff are globbed. That is pre-existing behavior, unchanged here.

Check list for the pull request

• I have read the [CONTRIBUTING]
• I have read the [CODE_OF_CONDUCT]
• I have added tests for my changes
• I have updated the documentation accordingly

Check list for the reviewer

• I have read the [CONTRIBUTING]
• I have verified the proposed changes
• best software practices
  • all internal functions have an underbar, as is python standard
  • clearly named variables (better to be verbose in variable names)
  • code comments explaining the intent of code blocks
• All the tests are passing
• The documentation is up to date
• code comments added when explaining intent

Manual test for the reviewer

Unit tests:

pixi run test-unit

Integration tests (requires Docker):

docker compose -f tests/integration/docker-compose.yml up -d --build
pixi run test-integration
docker compose -f tests/integration/docker-compose.yml down

The VENUS integration fixture directory now holds a series shared by several runs, mirroring production:

• test_oncat_catalog_venus_images — run 12345's PV names the directory. Only its 4 files are batched,
  including a TPX3-style name with a second date prefix; runs 12344, 12347 and 123450 (the prefix-collision
  case) are skipped.
• test_oncat_catalog_venus_single_image_path — run 12347's PV names a single file, which is batched alone.
• test_oncat_catalog_images_disabled — unchanged, confirms the catalog_<INSTRUMENT>.py toggle still gates
  the whole substep.

Against a running agent, the log lines to look for are Skipping N image file(s) in <dir> belonging to other runs and Cataloging N image file(s) for run <run>.

References

EWM 17220

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.33%. Comparing base (c030ccf) to head (d677c4e).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #110      +/-   ##
==========================================
+ Coverage   82.04%   82.33%   +0.29%     
==========================================
  Files          16       16              
  Lines        1303     1325      +22     
==========================================
+ Hits         1069     1091      +22     
  Misses        234      234              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[backmari]

Looks good to me 👌 Good call to compare with real data.

I was going to suggest adding type hints, but I see that type hints are not use anywhere in this repo, so we can do that separately as a small maintenance story.