implement task evaluation framework - #187
implement task evaluation framework#187Kavya Sree Kaitepalli (KavyaSree2610) wants to merge 12 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #187 +/- ##
==========================================
+ Coverage 65.33% 69.72% +4.39%
==========================================
Files 36 43 +7
Lines 2406 2755 +349
==========================================
+ Hits 1572 1921 +349
Misses 834 834
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Kavya Sree Kaitepalli (KavyaSree2610)
changed the title
4dd41ea to
5cbf8ee
Compare
Bala (0xba1a)
left a comment
Bala (0xba1a)
left a comment
There was a problem hiding this comment.
Will continue the review after rebasing
|
|
||
| logger = getLogger(__name__) | ||
|
|
||
| SWE_BENCH_SUITE = "SWE-bench/SWE-bench_Verified" |
There was a problem hiding this comment.
Keep the name as "SWE_BENCH_VERIFIED". Because, there is an extended dataset called swe-bench.
| If no instance with the given ``instance_id`` exists in the | ||
| dataset. | ||
| """ | ||
| rows = load_dataset(dataset_name, split="test") |
There was a problem hiding this comment.
This requires huggingface dataset library.
Can you please make the training part as an optional for microbots library and keep depending libraries under that?
| dataset. | ||
| """ | ||
| rows = load_dataset(dataset_name, split="test") | ||
| for row in rows: |
There was a problem hiding this comment.
Please check whether the load_dataset caches's the values. Otherwise, its better to store the (once loaded) dataset into a global variable.
|
|
||
| Parameters | ||
| ---------- | ||
| repo_path : str |
There was a problem hiding this comment.
Why do we need this repo_path argument for this function?
| ).stdout | ||
|
|
||
| run_id = f"microbots-{uuid.uuid4().hex[:8]}" | ||
| model_name_or_path = "microbots-eval-agent" |
There was a problem hiding this comment.
Please move this hard-coded agent's name as a global variable
| capture_output=True, text=True, | ||
| cwd=report_dir, | ||
| ) | ||
| with open(log_path, "a") as f: |
There was a problem hiding this comment.
Instead of getting a log file as the argument and dumping all logs into that, please create a log directory and store log in a separate file named after the instance_id.
Because a single shared log-file may become corrupted if we run eval on multiple instances in-parallel.
| parser.add_argument("--memory-dir", default="./memory", help="Directory to store the memory file") | ||
| parser.add_argument("--model", required=True, help="Model identifier, e.g. azure-openai/gpt-4o") | ||
| parser.add_argument( | ||
| "--iterations", |
There was a problem hiding this comment.
Similar to this let's have one-more argument that will control the max-iterations for the training-bot will have while creating memory.
…p and add training_iterations parameter
5cbf8ee to
b2b08ff
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Evaluation correctness, repository preparation, optional dependency loading, and memory snapshot handling have unresolved defects.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Introduces an extensible train/evaluate loop that builds persistent repository memory and validates it through SWE-bench tasks.
Changes:
- Adds task orchestration, registration, workdir, and memory lifecycle APIs.
- Implements SWE-bench Verified evaluation and feedback generation.
- Adds optional training dependencies and comprehensive unit tests.
File summaries
| File | Description |
|---|---|
src/microbots/auto_memory/__init__.py |
Exports public framework types. |
src/microbots/auto_memory/cli.py |
Adds train/eval CLI. |
src/microbots/auto_memory/evalTask.py |
Defines evaluation interfaces and results. |
src/microbots/auto_memory/eval/swebenchverified.py |
Implements SWE-bench evaluation. |
src/microbots/auto_memory/orchestrator.py |
Coordinates training and evaluation rounds. |
src/microbots/auto_memory/task_registry.py |
Provides task discovery and registration. |
src/microbots/auto_memory/workdir.py |
Manages run paths and memory snapshots. |
test/auto_memory/test_cli.py |
Tests CLI behavior. |
test/auto_memory/test_orchestrator.py |
Tests orchestration flows. |
test/auto_memory/test_task.py |
Tests evaluation abstractions. |
test/auto_memory/test_task_registry.py |
Tests task registration and discovery. |
test/auto_memory/test_workdir.py |
Tests workdir and memory handling. |
test/auto_memory/eval/test_swebenchverified.py |
Tests SWE-bench integration behavior. |
requirements.txt |
Removes optional training dependencies. |
pyproject.toml |
Adds the training extra. |
.github/workflows/test.yml |
Installs training dependencies in CI. |
Review details
Suppressed comments (1)
src/microbots/auto_memory/workdir.py:303
dirs_exist_ok=Trueonly overlays files; it never removes top-level notes deleted or renamed during training. SinceMemoryToolexplicitly supports deleting stale notes, those deletions are lost and obsolete memory is carried into later rounds. Replace the top-level directory with the round snapshot instead of merging it.
dst.mkdir(parents=True, exist_ok=True)
if src.is_dir():
shutil.copytree(src, dst, dirs_exist_ok=True)
- Files reviewed: 15/16 changed files
- Comments generated: 8
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Summary:
Each round works like this:
Dependencies
datasets and swebench are only needed to run the SWE-bench eval task, so they're declared as an optional training extra in pyproject.toml ( pip install .[training] ) rather than in the base requirements.txt
Removed datasets ' full transitive dependency closure ( dill , filelock , fsspec , hf-xet , huggingface_hub , multiprocess , numpy , pandas , pyarrow , xxhash ) from requirements.txt , since datasets itself moved to the optional training