Spec for streaming inference

[drewoldag]

This PR puts together a spec file for "streaming inference". The gist is that we could load the model into memory once and process batches of data as they arrive with the expectation that there are potentially large gaps between the batches arriving.

Two additional things worth calling out that aren't mentioned in the spec:

1. It probably makes sense for there to be a timeout of some kind that would release the model from memory if a batch hasn't arrived in the last N seconds
2. We could probably turn infer into a special case of infer_streaming such that infer simply calls infer_streaming under the hood and provides all the data to it, like it normally would.

[codecov]

Codecov Report

❌ Patch coverage is 37.50000% with 55 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.49%. Comparing base (61d4977) to head (c218026).

Files with missing lines	Patch %	Lines
src/hyrax/verbs/infer_stream.py	30.00%	49 Missing ⚠️
src/hyrax/pytorch_ignite.py	40.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #930      +/-   ##
==========================================
- Coverage   63.80%   63.49%   -0.31%     
==========================================
  Files          74       75       +1     
  Lines        7689     7775      +86     
==========================================
+ Hits         4906     4937      +31     
- Misses       2783     2838      +55     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB