MLIR lit Testing Extension for VS Code

A VS Code extension that supports LLVM lit tests (.mlir + RUN: / FileCheck) in the native Testing sidebar by implementing a TestController.

The discovery design is infer, don't configure: the extension finds lit.cfg.py and lit.site.cfg.py files in the workspace, locates lit, and asks lit what the tests are. There is no .mlir filesystem glob, no testRoots setting, no fileSuffixes setting. It works build-system agnostically (CMake, Bazel, anything that produces a lit.site.cfg.py — or even a bare hand-written lit.cfg.py with no build configured yet).

Thanks to Jakub Jalowiec and River Riddle for unknowingly supplying some design wisdom that informed this project. We don't use any of the code, but the comments in the PR were useful nonetheless. github PR link

Caveat emptor: this extension is heavily vibe-coded. I wanted a practical solution now, not a long-term maintainable project. So don't go and rely on this for anything important.

---

Build

You'll need Node 20+ and npm. vsce is fetched on demand via npx.

npm install
npm run build         # compile TypeScript to dist/
npm run package       # produce lit-test-extension.vsix

The resulting lit-test-extension.vsix is what you install into VS Code.

---

Install

The manifest declares extensionKind: ["workspace"], which means the extension is required to run on the workspace side — locally when you open a folder directly, or container-side when you "Reopen in Container."

Plain (no devcontainer)

code --install-extension lit-test-extension.vsix

Inside a devcontainer

The easiest path for a devcontainer is just to put the .vsix alongside the devcontainer.json and reference it via a relative path in customizations.vscode.extensions. The Dev Containers extension supports local file paths there in addition to marketplace IDs:

// .devcontainer/devcontainer.json
{
  "customizations": {
    "vscode": {
      "extensions": [
        "ms-python.python",
        "./lit-test-extension.vsix"
      ]
    }
  }
}

Copy lit-test-extension.vsix into the .devcontainer/ directory (or wherever your devcontainer.json lives — paths are resolved relative to it) and rebuild the container. The extension installs automatically on container build.

---

Quick start

1. Open a workspace that contains either a hand-written lit.cfg.py or a build-generated lit.site.cfg.py somewhere under the root.
2. Make sure lit is on PATH (lit --version should work in a terminal). pip install lit is enough.
3. Open the Testing sidebar (the flask icon). Tests should appear under one or more suite nodes.
4. Click the gutter run icon on a test, or the play button next to any node in the Testing sidebar.

If lit cannot be found, a dismissable warning appears and the tree stays empty. Set litTest.litPath or install lit on PATH, then trigger Lit Test: Refresh Tests from the command palette.

---

Settings (all optional; overrides only)

Setting	Default	Description
litTest.litPath	""	Override path to the lit binary. Empty = infer (PATH → python -m lit).
litTest.pythonPath	""	Override Python interpreter. Empty = Python extension's selected interpreter, then python3.
litTest.suiteConfigPaths	[]	Override list of lit anchors (lit.cfg.py and/or lit.site.cfg.py, files or dirs containing them). Empty = auto-search the workspace. Use absolute paths for builds outside the workspace folder.
litTest.litArgs	["-v"]	Extra args for run invocations. -v is required for failure diffs.
litTest.env	{}	Extra env vars merged into lit's env (e.g. LD_LIBRARY_PATH, FILECHECK_OPTS). Merge, not replace.
litTest.cwd	""	Working directory. Empty = workspace root.
litTest.maxParallelism	0	Passed to lit as -j. 0 = lit decides.
litTest.excludeDirs	["third_party", "node_modules", ".git", ".venv", "venv", "__pycache__", ".tox", ".mypy_cache", ".pytest_cache", "dist", "out"]	Directory names skipped when searching for anchor files. Dotted dirs are always skipped.
litTest.searchMaxDepth	8	Max walk depth for the anchor search.

There is deliberately no testRoots or fileSuffixes setting — both are inferred from lit.

Out-of-workspace builds

If your build tree lives outside the workspace folder (common in devcontainer setups where the build cache is on a separate mount), the auto-search won't find your lit.site.cfg.py. Set litTest.suiteConfigPaths once:

// .vscode/settings.json
{
  "litTest.suiteConfigPaths": [
    "/absolute/path/to/your/build/test/lit.site.cfg.py"
  ]
}

You can pass either the file directly or the directory holding it.

Environment for integration tests

MLIR integration tests typically need LD_LIBRARY_PATH set so the JIT can find runtime support libs (libmlir_c_runner_utils.so, etc.). Merge it through litTest.env:

{
  "litTest.env": {
    "LD_LIBRARY_PATH": "/absolute/path/to/build/lib"
  }
}

VS Code does not expand ${env:...} inside object values; provide a literal path.

---

Commands

Command	Description
Lit Test: Refresh Tests	Re-run the inference chain and rebuild the tree.
Lit Test: Show Output	Open the extension's output channel.

Auto-refresh on filesystem changes:

• Editing or creating a lit.cfg.py / lit.site.cfg.py triggers a full re-enumeration.
• Creating a new .mlir file triggers a full re-enumeration (lit decides whether it's actually a test).
• Deleting a .mlir file drops the corresponding TestItem in place — no lit invocation.
• Saving a .mlir file is a no-op. Content edits don't change a test's identity in the tree, so we don't pay enumeration cost for every save.

---

Result mapping

lit status	VS Code outcome
PASS, XFAIL, FLAKYPASS	passed
FAIL, XPASS	failed
UNSUPPORTED, SKIPPED	skipped
UNRESOLVED, TIMEOUT	errored

On failure, the test item carries a TestMessage containing lit's -v output verbatim (the script that ran, exit code, stdout, and the FileCheck diff if any).

---

Scope

Supported now:

• Discovery via lit --show-suites --show-tests.
• File-granularity tree (one node per .mlir).
• Batched run invocation.
• Pass / fail / skip / error reporting with verbatim diff on failure.
• Watcher-driven refresh on filesystem changes.
• Devcontainer support via extensionKind: ["workspace"].

Not supported (yet):

• Per-chunk granularity under -split-input-file. lit doesn't model chunks as tests; a 20-chunk file shows as one node.
• Debug profile.
• Coverage profile.
• Line-level CHECK: failure locations.
• Lazy per-suite enumeration. Initial discovery enumerates everything; fine for small/medium projects, slow on the full LLVM monorepo.

Hard non-goals:

• Reimplementing lit. We shell out and trust its discovery.

---

How it works

On activation, the extension walks the workspace for lit.cfg.py and lit.site.cfg.py files (with sensible excludes), locates lit (litTest.litPath → PATH → python -m lit), runs lit --show-suites --show-tests <anchor dirs> to enumerate, and builds the TestController tree from that output.

Both anchor kinds are supported: lit.site.cfg.py is the build-generated file and is preferred when present (it carries the build's resolved tool paths). A bare lit.cfg.py is sufficient when no build is configured. Overlapping anchors that resolve to the same suite (e.g. a site config and the lit.cfg.py it loads) are collapsed in the enumeration parser — lit itself does not dedupe its inputs, so we dedup the output by (suite name, source root) and by test id.

Running a test (or a directory, or everything) is a single batched lit invocation passing absolute source paths; the output is parsed for per-test status lines and ******** TEST '...' FAILED ******** detail blocks, which map onto TestRun.passed/failed/skipped/errored and TestMessage.

All format-fragile knowledge lives in src/litQueryParse.ts (enumeration parsing) and src/litOutput.ts (run output parsing). When lit ships a new format variant, those are the two files to touch.

---

Development

lit must be on PATH to build and test (the test suite drives a real lit against a tiny fixture in test/fixtures/example-suite/). pip install lit is sufficient — FileCheck is not required by the tests.

npm install
npm run build            # compile to dist/
npm run watch            # rebuild on save
npm run test:unit        # run parser tests against real lit
npm run package          # produce .vsix

Press F5 in VS Code to launch the Extension Development Host against this repo for manual exercise.

Pre-commit hook

A pre-commit config runs npm run test:unit before every commit. Set up once:

pip install pre-commit
pre-commit install