[Feat] Online Checker

.pre-commit-config.yaml

@@ -23,11 +23,8 @@ repos:
         args: [--exclude, tests]
         # args: [--check]
 
-  - repo: local
+  - repo: https://github.com/pre-commit/mirrors-isort
+    rev: v5.10.1
     hooks:
       - id: isort
-        name: isort
-        entry: isort
-        language: system
-        types: [python]
-        args: [--profile=black, --skip, tests]
+        args: [--profile=black, --skip=tests]

docs/check.md

@@ -2,19 +2,31 @@
 
 `traincheck-check` is the **final stage** of the TrainCheck workflow. It verifies a set of invariants against trace files or streams from target programs, reporting any detected violations—helping you catch silent issues in your ML training pipelines.
 
-## 🔧 Current Status
+## 🔧 Checking Modes
 
-`traincheck-check` is designed to support two modes:
+TrainCheck supports two checking modes:
 
-- **Offline Checking**:  
-   Perform invariant checking on completed trace files after the training job finishes. ✅ *[Fully Supported]*
+- **Post-training Checking (`traincheck-check`)**:  
+   Perform invariant checking on completed trace files after the training job finishes. ✅
 
-- **Online Checking**:  
-   Perform real-time checking while the target training job is running. 🚧 *[In Development]*
+- **On-the-fly Checking (`traincheck-onlinecheck`):**
+   Perform real-time checking while the target training job is running. ✅
 
-At present, only **offline checking** is available. Support for online mode is actively being developed.
+## How to Use: On-the-fly Checking
 
-## How to Use: Offline Checking
+While training is in progress with `traincheck-collect`, run the following command:
+
+```bash
+traincheck-onlinecheck -f <trace_folder> -i <path_to_invariant_file>
+```
+
+- `-f <trace_folder>`: Path to the folder where traces are:
+  - Already collected, or
+  - **Actively being collected** by `traincheck-collect` during the training job.
+
+- `-i <path_to_invariant_file>`: Path to the JSON file containing inferred invariants.
+
+## How to Use: Post-training Checking
 
 Run the following command:
 
@@ -25,11 +37,12 @@ traincheck-check -f <trace_folder> -i <path_to_invariant_file>
 - `-f <trace_folder>`: Path to the folder containing traces collected by `traincheck-collect`.
 - `-i <path_to_invariant_file>`: Path to the JSON file containing inferred invariants.
 
-For details on result format and interpretation, refer to [5. Detection & Diagnosis)](./5-min-tutorial.md#5-detection--diagnosis) in the **5-Minute Tutorial**.
+## Interpreting the Results
 
-## How to Use: Online Checking
+After running either checking mode, TrainCheck will output a summary of detected invariant violations. Each violation entry typically includes:
 
-**🚧 Coming Soon**
-Support for real-time, online checking is under construction. This mode will allow TrainCheck to monitor running training jobs and surface invariant violations as they happen.
+- **Trace file or stream name**: Identifies where the issue was found.
+- **Invariant description**: Details the specific invariant that was violated.
+- **Violation details**: Provides context, such as the step or epoch where the violation occurred.
 
-Stay tuned for updates in future releases.
+Review these results to pinpoint silent errors or unexpected behaviors in your ML training pipeline. For more information on result formats and how to diagnose issues, see [5. Detection & Diagnosis](./5-min-tutorial.md#5-detection--diagnosis) in the **5-Minute Tutorial**.

pyproject.toml

@@ -52,3 +52,4 @@ homepage = "https://github.com/OrderLab/TrainCheck"
 traincheck-collect = "traincheck:collect_trace.main"
 traincheck-infer = "traincheck:infer_engine.main"
 traincheck-check = "traincheck:checker.main"
+traincheck-onlinecheck = "traincheck:checker_online.main"

tests/test_utils.py

@@ -16,6 +16,7 @@ def test_typename_builtin_function():
     assert typename(len) == "len"
     assert typename(print) == "print"
 
+
 def test_typename_tensor_and_parameter():
     t = torch.tensor([1.0])
     assert typename(t) == t.type()

traincheck/checker.py

@@ -42,7 +42,6 @@ def check_engine(
             inv.precondition is not None
         ), "Invariant precondition is None. It should at least be 'Unconditional' or an empty list. Please check the invariant file and the inference process."
         logger.info("=====================================")
-        # logger.debug("Checking invariant %s on trace %s", inv, trace)
         res = inv.check(trace, check_relation_first)
         res.calc_and_set_time_precentage(trace.get_start_time(), trace.get_end_time())
         logger.info("Invariant %s on trace %s: %s", inv, trace, res)
@@ -147,7 +146,7 @@ def main():
     for inv_file in args.invariants:
         os.system(f"cp {inv_file} {args.output_dir}/invariants.json")
 
-    logger.info("Reading invaraints from %s", "\n".join(args.invariants))
+    logger.info("Reading invariants from %s", "\n".join(args.invariants))
     invs = read_inv_file(args.invariants)
 
     traces = []