Repository navigation
Add post-midterm course content and refactor topic card curation - #13
Conversation
There was a problem hiding this comment.
Pull request overview
Adds post-midterm course materials (weeks 4–6 + assessments) and updates the ingestion + UI pipeline to support a denser “Topic Explorer” selection/preview workflow, including schema v3 handling and richer AI section rendering.
Changes:
- Introduces raw-material ingestion helpers (pptx/ipynb/pdf w/ OCR fallback) and agent prompt builders for week + assessment extraction.
- Adds schema v3 auto-migration + v3 flattening for the topic-card pipeline, plus Gemini CLI JSON-output handling.
- Updates UI rendering/state/export mechanics for item-first selection, denser previews, and new Playwright “full-ui” validation.
Reviewed changes
Copilot reviewed 109 out of 201 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| pipelines/study_database/raw_sources.py | New raw-source extraction (pptx/ipynb/pdf + OCR) and record collection for ingestion. |
| pipelines/study_database/raw_agents.py | New Gemini prompt builders + JSON agent runner for week/assessment/unassigned classification. |
| pipelines/study_database/init.py | Lazy exports via __getattr__ for new v3/migration/assessment helpers. |
| pipelines/shared/study_database.py | Adds schema-version detection, v3 flattening, and auto-upgrade of legacy week payloads at load time. |
| pipelines/shared/llm.py | Switches Gemini CLI to JSON output + extracts headless response payload. |
| pipelines/ai_sections/common.py | Adds structured common-question items + example outputs to AI section normalization/application. |
| app/text-render-utils.js | Sanitizes/normalizes display text and routes code-ish blocks through shared code rendering. |
| app/splash-storage.js | Updates reset/hydration/persistence for explorer-era state (navigation, layout tweaks, week filters). |
| app/preview-pointer.js | Adjusts preview card sizing constraints and persists per-card titles. |
| app/preview-history.js | Changes preview undo snapshots to track drafts + preview layouts (not swipe decisions). |
| app/layout-export-utils.js | Adds --sheet-title-size layout variable and wires new UI controls. |
| app/code-render-utils.js | New code/output rendering utilities with lightweight Python syntax highlighting. |
| Makefile | Adds setup-playwright + full-ui target and reuses Playwright install across UI tasks. |
| materials/post_midterm/Solutions week 5.txt | Adds week 5 solutions source material for ingestion. |
| materials/post_midterm/Solutions week 4.txt | Adds week 4 solutions source material for ingestion. |
| materials/post_midterm/OOP.py | Adds post-midterm OOP sample source for ingestion. |
| materials/post_midterm/Exercise_6.9.py | Adds week 6 exercise prompt source. |
| materials/post_midterm/Exercise_6.8.py | Adds week 6 exercise prompt source. |
| materials/post_midterm/Exercise_6.7.py | Adds week 6 exercise prompt source. |
| materials/post_midterm/Exercise_6.6.py | Adds week 6 exercise prompt source. |
| materials/post_midterm/Exercise_6.5.py | Adds week 6 exercise prompt source. |
| materials/post_midterm/Exercise_6.4.py | Adds week 6 exercise prompt source. |
| materials/post_midterm/Exercise_6.3.py | Adds week 6 exercise prompt source. |
| materials/post_midterm/Exercise_6.2.py | Adds week 6 exercise prompt source. |
| materials/post_midterm/Exercise_6.1.py | Adds week 6 exercise prompt source. |
| materials/post_midterm/Exercise_5.9.py | Adds week 5 exercise prompt source. |
| materials/post_midterm/Exercise_5.8.py | Adds week 5 exercise prompt source. |
| materials/post_midterm/Exercise_5.7.py | Adds week 5 exercise prompt source. |
| materials/post_midterm/Exercise_5.6.py | Adds week 5 exercise prompt source. |
| materials/post_midterm/Exercise_5.5.py | Adds week 5 exercise prompt source. |
| materials/post_midterm/Exercise_5.4.py | Adds week 5 exercise prompt source. |
| materials/post_midterm/Exercise_5.3.py | Adds week 5 exercise prompt source. |
| materials/post_midterm/Exercise_5.2.py | Adds week 5 exercise prompt source. |
| materials/post_midterm/Exercise_5.1.py | Adds week 5 exercise prompt source. |
| materials/post_midterm/Exercise_4.9.py | Adds week 4 exercise prompt source. |
| materials/post_midterm/Exercise_4.8.py | Adds week 4 exercise prompt source. |
| materials/post_midterm/Exercise_4.7.py | Adds week 4 exercise prompt source. |
| materials/post_midterm/Exercise_4.6.py | Adds week 4 exercise prompt source. |
| materials/post_midterm/Exercise_4.5.py | Adds week 4 exercise prompt source. |
| materials/post_midterm/Exercise_4.4.py | Adds week 4 exercise prompt source. |
| materials/post_midterm/Exercise_4.3.py | Adds week 4 exercise prompt source. |
| materials/post_midterm/Exercise_4.2.py | Adds week 4 exercise prompt source. |
| materials/post_midterm/Exercise_4.1.py | Adds week 4 exercise prompt source. |
| docs/specs/RM-007-course-outline-topic-merging.md | Adds planned spec for future course-outline topic merging curation work. |
| docs/data/DATASET_INFO.md | Documents schema v3 structure + new dense topic_cards.json section shapes. |
| docs/curation/TOPIC_MERGING_GUIDELINES.md | Updates curation guidance and verification steps for explorer-era workflow. |
| docs/TESTING.md | Documents new make full-ui end-to-end test and updated smoke expectations. |
| docs/ROADMAP.md | Links new handoff doc and tracks RM-006/RM-007 items. |
| docs/HANDOFF_2026-03-21.md | Adds handoff notes describing post-midterm ingestion state and validation commands. |
| docs/ARCHITECTURE.md | Updates architecture doc for Topic Explorer workflow + schema v3 ingestion/migration tooling. |
| design_inspo/DESIGN.md | Adds design-system spec reference under design inspiration. |
| DESIGN.md | Adds design-system spec at repo root. |
| AGENTS.md | Updates agent handoff snapshot, workflow, and validation steps for explorer-era model. |
| data/templates/week_template.json | Updates the week template to v3 lecture-first schema with snippet refs/metadata. |
| data/quality/key_point_filter_audit.json | Updates quality audit output (model/rules/summary/samples) for new dataset. |
| data/import_payloads/post_midterm_assessments/trial-final-exam-solutions-introduction-to-python-3077951.json | Adds extracted assessment JSON payload. |
| data/import_payloads/post_midterm_assessments/resit-solutions-for-introduction-to-python-35761538.json | Adds extracted assessment JSON payload. |
| data/import_payloads/post_midterm_assessments/resit-exam-guidelines-for-intro-to-python-6013b0470y-july-2023.json | Adds extracted assessment JSON payload. |
| data/import_payloads/post_midterm_assessments/introduction-to-python-trial-final-exam-solutions-py22.json | Adds extracted assessment JSON payload. |
| data/import_payloads/post_midterm_assessments/final-exam-study-guide-trial-python-basics-2023.json | Adds extracted assessment JSON payload. |
| data/import_payloads/post_midterm_assessments/final-exam-solutions-for-python-programming-course-code-308088-308234.json | Adds extracted assessment JSON payload. |
| data/import_payloads/post_midterm_assessments/final-exam-solutions-for-python-programming-62oop21.json | Adds extracted assessment JSON payload. |
| data/import_payloads/post_midterm_agent/week-06/review.json | Adds week 6 agent review artifact. |
| data/import_payloads/post_midterm_agent/week-06/payload.json | Adds week 6 agent-extracted week payload artifact. |
| data/import_payloads/post_midterm_agent/week-05/review.json | Adds week 5 agent review artifact. |
| data/import_payloads/post_midterm_agent/week-05/payload.json | Adds week 5 agent-extracted week payload artifact. |
| data/import_payloads/post_midterm_agent/week-04/review.json | Adds week 4 agent review artifact. |
| data/import_payloads/post_midterm_agent/raw_ingestion_report.json | Adds summary report for raw ingestion run. |
| data/import_payloads/post_midterm_agent/assessment-trial-final-exam-solutions-introduction-to-python-3077951/review.json | Adds assessment agent review artifact. |
| data/import_payloads/post_midterm_agent/assessment-trial-final-exam-solutions-introduction-to-python-3077951/payload.json | Adds assessment agent payload artifact. |
| data/import_payloads/post_midterm_agent/assessment-sample-final-plus-answers/review.json | Adds assessment agent review artifact for sample final (currently failing validation). |
| data/import_payloads/post_midterm_agent/assessment-sample-final-plus-answers/payload.json | Adds assessment agent payload artifact for sample final (currently empty). |
| data/import_payloads/post_midterm_agent/assessment-resit-solutions-for-introduction-to-python-35761538/review.json | Adds assessment agent review artifact. |
| data/import_payloads/post_midterm_agent/assessment-resit-solutions-for-introduction-to-python-35761538/payload.json | Adds assessment agent payload artifact. |
| data/import_payloads/post_midterm_agent/assessment-resit-exam-guidelines-for-intro-to-python-6013b0470y-july-2023/review.json | Adds assessment agent review artifact. |
| data/import_payloads/post_midterm_agent/assessment-resit-exam-guidelines-for-intro-to-python-6013b0470y-july-2023/payload.json | Adds assessment agent payload artifact. |
| data/import_payloads/post_midterm_agent/assessment-introduction-to-python-trial-final-exam-solutions-py22/review.json | Adds assessment agent review artifact. |
| data/import_payloads/post_midterm_agent/assessment-introduction-to-python-trial-final-exam-solutions-py22/payload.json | Adds assessment agent payload artifact. |
| data/import_payloads/post_midterm_agent/assessment-final-exam-study-guide-trial-python-basics-2023/review.json | Adds assessment agent review artifact. |
| data/import_payloads/post_midterm_agent/assessment-final-exam-study-guide-trial-python-basics-2023/payload.json | Adds assessment agent payload artifact. |
| data/import_payloads/post_midterm_agent/assessment-final-exam-solutions-for-python-programming-course-code-308088-308234/review.json | Adds assessment agent review artifact. |
| data/import_payloads/post_midterm_agent/assessment-final-exam-solutions-for-python-programming-course-code-308088-308234/payload.json | Adds assessment agent payload artifact. |
| data/import_payloads/post_midterm_agent/assessment-final-exam-solutions-for-python-programming-62oop21/review.json | Adds assessment agent review artifact. |
| data/import_payloads/post_midterm_agent/assessment-final-exam-solutions-for-python-programming-62oop21/payload.json | Adds assessment agent payload artifact. |
Comments suppressed due to low confidence (1)
DESIGN.md:1
- This appears to duplicate
design_inspo/DESIGN.mdverbatim. Keeping the same spec in two locations risks divergence and confusion about which is canonical. Consider making one file the single source of truth (and linking to it from the other location), or otherwise ensuring there’s an explicit ownership/canonical-path note.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| notebooks.append( | ||
| { | ||
| "week": week, | ||
| "cell_index": len(notebooks) + 1, | ||
| "cell_type": "code", | ||
| "topic": subtopic_title or topic_title, | ||
| "source": str(snippet.get("content") or "").strip(), | ||
| "outputs": _safe_list(snippet.get("outputs")), | ||
| "topic_id": topic.get("id"), | ||
| "subtopic_id": subtopic.get("id"), | ||
| "subtopic_title": subtopic_title, | ||
| "is_advanced_optional": False, | ||
| } | ||
| ) |
There was a problem hiding this comment.
cell_index is derived from len(notebooks) + 1, which makes it a global sequence across all weeks/subtopics rather than a per-notebook/per-week index. If downstream code expects notebook cell ordering within a week (or stable indices matching the original notebook), this will produce incorrect ordering and unstable diffs. Consider tracking a per-week counter (or preserving original_id/order from source_refs) and using that as cell_index.
| if layout: | ||
| args.append("-layout") | ||
| args.extend([str(path), "-"]) | ||
| result = subprocess.run(args, capture_output=True, text=True, check=False) |
There was a problem hiding this comment.
The pdftotext / pdftoppm / tesseract subprocesses run without a timeout, so a single problematic PDF can hang the ingestion pipeline indefinitely (or keep OCRing for an unbounded time). Add timeout=... to these subprocess.run(...) calls and surface a clear error when the timeout is hit. It’s also worth considering a page limit (or max total OCR bytes) to avoid pathological PDFs causing excessive CPU/disk usage.
| render = subprocess.run( | ||
| ["pdftoppm", "-png", "-r", "180", str(path), str(prefix)], | ||
| capture_output=True, | ||
| text=True, | ||
| check=False, | ||
| ) |
There was a problem hiding this comment.
The pdftotext / pdftoppm / tesseract subprocesses run without a timeout, so a single problematic PDF can hang the ingestion pipeline indefinitely (or keep OCRing for an unbounded time). Add timeout=... to these subprocess.run(...) calls and surface a clear error when the timeout is hit. It’s also worth considering a page limit (or max total OCR bytes) to avoid pathological PDFs causing excessive CPU/disk usage.
| ocr = subprocess.run( | ||
| ["tesseract", str(image_path), "stdout", "--psm", "6"], | ||
| capture_output=True, | ||
| text=True, | ||
| check=False, | ||
| ) |
There was a problem hiding this comment.
The pdftotext / pdftoppm / tesseract subprocesses run without a timeout, so a single problematic PDF can hang the ingestion pipeline indefinitely (or keep OCRing for an unbounded time). Add timeout=... to these subprocess.run(...) calls and surface a clear error when the timeout is hit. It’s also worth considering a page limit (or max total OCR bytes) to avoid pathological PDFs causing excessive CPU/disk usage.
| ) -> str: | ||
| result = subprocess.run( | ||
| ["gemini", "-m", model, "-p", prompt], | ||
| ["gemini", "-m", model, "-p", prompt, "--output-format", "json"], |
There was a problem hiding this comment.
Forcing --output-format json assumes the installed gemini CLI supports that flag and that stdout is always JSON-shaped. If the flag is unsupported (or the CLI outputs an error banner to stdout), this will break ingestion in a non-obvious way. A more robust approach is to retry without --output-format json when the command fails, or detect/support both formats and include the relevant stdout/stderr context in the raised error.
| cleaned = _extract_headless_response(result.stdout) | ||
| if not cleaned: | ||
| raise RuntimeError("Gemini returned empty output.") | ||
| return cleaned |
There was a problem hiding this comment.
Forcing --output-format json assumes the installed gemini CLI supports that flag and that stdout is always JSON-shaped. If the flag is unsupported (or the CLI outputs an error banner to stdout), this will break ingestion in a non-obvious way. A more robust approach is to retry without --output-format json when the command fails, or detect/support both formats and include the relevant stdout/stderr context in the raised error.
| "exam_label": "", | ||
| "source": "", | ||
| "year": "unknown", | ||
| "questions": [], | ||
| "notes": [] |
There was a problem hiding this comment.
This payload is invalid (empty exam_label, empty source, and no questions), and the accompanying review artifact reports validation errors. If any validation/import tooling consumes all payloads under data/import_payloads/post_midterm_agent/, this can cause pipeline failures or confusing red reports. Either regenerate this extraction, remove the empty artifact, or explicitly mark it as ignored/failed in whatever index/report the importer reads.
| "exam_label": "", | |
| "source": "", | |
| "year": "unknown", | |
| "questions": [], | |
| "notes": [] | |
| "exam_label": "IGNORED_INVALID_EXTRACTION", | |
| "source": "post_midterm_agent/assessment-sample-final-plus-answers (intentionally empty payload)", | |
| "year": "unknown", | |
| "questions": [], | |
| "notes": [ | |
| "This payload was generated without any questions and is intentionally marked as ignored so that import/validation tooling can skip it." | |
| ], | |
| "ignored": true |
| from .raw_sources import RawSourceRecord | ||
|
|
||
| RAW_TIMEOUT_SECONDS = 900 | ||
| RAW_STERR_CLIP = 1200 |
There was a problem hiding this comment.
RAW_STERR_CLIP appears to be a typo for RAW_STDERR_CLIP. Renaming it improves clarity and avoids propagating the misspelling into other modules/config.
| const maxWidth = Math.max(72, pageSize.width - layout.x); | ||
| const maxHeight = Math.max(130, pageSize.height - layout.y); | ||
| layout.width = clamp(previewPointerState.startWidth + dx, 170, maxWidth); | ||
| layout.width = clamp(previewPointerState.startWidth + dx, 56, maxWidth); |
There was a problem hiding this comment.
The resize-corner path uses Math.max(72, ...) for maxWidth but clamps with a minimum width of 56. This creates inconsistent constraints (e.g., maxWidth’s floor doesn’t match the clamp minimum). Align these constants so min/max behavior is consistent across sanitize + pointer-resize paths.
| layout.width = clamp(previewPointerState.startWidth + dx, 56, maxWidth); | |
| layout.width = clamp(previewPointerState.startWidth + dx, 72, maxWidth); |
| "with", | ||
| "yield", | ||
| ]); | ||
|
|
There was a problem hiding this comment.
code-render-utils.js depends on globals (normalizeNewlines, escapeHtml, and optionally isCodeBlockLikely) defined in other scripts. This implicit coupling makes load-order issues harder to debug and complicates reuse/testing. Consider moving shared primitives into a single utility module (or adding small local fallbacks) so renderCodeBlock/highlighting don’t rely on cross-file globals.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 88baa7aded
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if isinstance(normalized, dict) and "lecture" in normalized and "notebook_cells" in normalized: | ||
| normalized["sources"] = [ | ||
| path for path in (_normalize_repo_relative_path(source) for source in _safe_list(normalized.get("sources"))) if path | ||
| ] | ||
| return normalized |
There was a problem hiding this comment.
This branch accepts legacy week payloads with lecture/notebook_cells and stores them unchanged in study_db.json. The repo now uses schema 3 weeks with nested topics[].subtopics[], so importing a payload like data/import_payloads/post_midterm/week_04.json leaves a mixed-schema DB and flatten_study_db_for_pipeline() drops that week's lecture/notebook content entirely. The import helper should coerce week payloads via coerce_week_payload_to_v3() before _upsert_week, otherwise the advertised raw-ingestion workflow can silently erase an imported week from generated cards.
Useful? React with 👍 / 👎.
| <p><strong>${escapeHtml(kindLabel)} • ${renderInlineCode(effective.title || "Code example")}</strong></p> | ||
| ${renderCodeBlock(effective.code || "")} | ||
| ${effective.output ? `<p><strong>Output / Result:</strong></p>${renderOutputBlock(effective.output || "")}` : ""} |
There was a problem hiding this comment.
why text when rendering selected AI examples
The preview card now renders the example title, code, and optional output, but it never prints effective.why. For examples that only carry an explanation sentence (the common shape in topic_cards.json), Preview/PDF export now loses that explanatory text even though Topic Explorer still shows it and the edit modal still lets users edit it. That is a user-visible content regression in the exported cheat sheet.
Useful? React with 👍 / 👎.
Summary
Testing
make leave-better-uimake smoke-uimake full-ui