feat(cli): add --batch directory conversion (#77) - #89
Conversation
Supports recursive convert-and-passthrough of folder trees for issue firecrawl#77 without racing the Rust CLI work in firecrawl#29.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
6 issues found across 4 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="node/test.mjs">
<violation number="1" location="node/test.mjs:152">
P2: The batch test covers only `.txt` passthrough files, so regressions in the documented `.json`, `.md`, or `.py` passthrough behavior will go undetected. Add fixtures and byte-for-byte output assertions for all four extensions.</violation>
</file>
<file name="node/cli.js">
<violation number="1" location="node/cli.js:191">
P2: When `-o` points into (or equals) the input directory tree — e.g. `anydoc --batch ./docs -o ./docs/out` — the output directory is created before walking, so `walkFiles` descends into it and re-processes its contents. Freshly written `*.md` files and copied passthrough files get detected, converted/copied again, and passthrough copies can land back inside the tree; on repeated runs the prior output is re-consumed as input. There is no guard rejecting or warning when the output root is inside the input root. Refuse (or at least warn) when `relative(inputRoot, outputRoot)` is not strictly outside the input tree.</violation>
<violation number="2" location="node/cli.js:197">
P2: When a nested directory cannot be read, `walkFiles` rejects outside the per-file error handling and aborts the batch. Catch traversal errors per directory so sibling files still run and the command reports exit 1 with a concise diagnostic.</violation>
<violation number="3" location="node/cli.js:200">
P3: This path-escape guard is unreachable. `absPath` is always produced by `join(inputRoot, entry.name)` inside `walkFiles`, and `inputRoot` is `resolve()`d, so `relative(inputRoot, absPath)` can never be `..` or start with `../` — its first path segment is always a child of the input root. The `continue` is dead code that can mislead future readers into thinking escaping paths are possible. Consider dropping it (and the now-misleading comment) or replacing it with a comment explaining why it cannot occur.</violation>
<violation number="4" location="node/cli.js:214">
P2: Batch skips valid documents when their filenames lack a recognized extension. `toMarkdown` supports content-based detection before extension fallback; use content detection before treating a file as unsupported, while still silently skipping genuinely unsupported content.</violation>
<violation number="5" location="node/cli.js:219">
P2: When a converted file and a passthrough `.md` file share a target name, batch silently overwrites one source's output. Detect duplicate output paths and report the collision instead of allowing data loss.</violation>
</file>
Tip: instead of fixing issues one by one fix them all with cubic
Re-trigger cubic
| await mkdir(join(input, 'nested'), { recursive: true }) | ||
| await copyFile(OUTLINE, join(input, 'nested', 'handbook.docx')) | ||
| await copyFile(CSV, join(input, 'sheet.csv')) | ||
| await writeFile(join(input, 'notes.txt'), 'plain text\n') |
There was a problem hiding this comment.
P2: The batch test covers only .txt passthrough files, so regressions in the documented .json, .md, or .py passthrough behavior will go undetected. Add fixtures and byte-for-byte output assertions for all four extensions.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node/test.mjs, line 152:
<comment>The batch test covers only `.txt` passthrough files, so regressions in the documented `.json`, `.md`, or `.py` passthrough behavior will go undetected. Add fixtures and byte-for-byte output assertions for all four extensions.</comment>
<file context>
@@ -136,6 +136,64 @@ test('cli help and version go to stdout and exit 0', async () => {
+ await mkdir(join(input, 'nested'), { recursive: true })
+ await copyFile(OUTLINE, join(input, 'nested', 'handbook.docx'))
+ await copyFile(CSV, join(input, 'sheet.csv'))
+ await writeFile(join(input, 'notes.txt'), 'plain text\n')
+ await writeFile(join(input, 'nested', 'skip.png'), Buffer.from([0x89, 0x50, 0x4e, 0x47]))
+
</file context>
| continue | ||
| } | ||
|
|
||
| const outPath = join(outputRoot, `${rel}.md`) |
There was a problem hiding this comment.
P2: When a converted file and a passthrough .md file share a target name, batch silently overwrites one source's output. Detect duplicate output paths and report the collision instead of allowing data loss.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node/cli.js, line 219:
<comment>When a converted file and a passthrough `.md` file share a target name, batch silently overwrites one source's output. Detect duplicate output paths and report the collision instead of allowing data loss.</comment>
<file context>
@@ -113,16 +146,93 @@ async function readStdin() {
+ continue
+ }
+
+ const outPath = join(outputRoot, `${rel}.md`)
+ try {
+ const markdown = await toMarkdown(absPath)
</file context>
| } | ||
|
|
||
| let failed = 0 | ||
| for await (const absPath of walkFiles(inputRoot)) { |
There was a problem hiding this comment.
P2: When a nested directory cannot be read, walkFiles rejects outside the per-file error handling and aborts the batch. Catch traversal errors per directory so sibling files still run and the command reports exit 1 with a concise diagnostic.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node/cli.js, line 197:
<comment>When a nested directory cannot be read, `walkFiles` rejects outside the per-file error handling and aborts the batch. Catch traversal errors per directory so sibling files still run and the command reports exit 1 with a concise diagnostic.</comment>
<file context>
@@ -113,16 +146,93 @@ async function readStdin() {
+ }
+
+ let failed = 0
+ for await (const absPath of walkFiles(inputRoot)) {
+ const rel = relative(inputRoot, absPath)
+ // Guard against path escape on odd relative() results.
</file context>
| continue | ||
| } | ||
|
|
||
| if (formatFromPath(absPath) === null) { |
There was a problem hiding this comment.
P2: Batch skips valid documents when their filenames lack a recognized extension. toMarkdown supports content-based detection before extension fallback; use content detection before treating a file as unsupported, while still silently skipping genuinely unsupported content.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node/cli.js, line 214:
<comment>Batch skips valid documents when their filenames lack a recognized extension. `toMarkdown` supports content-based detection before extension fallback; use content detection before treating a file as unsupported, while still silently skipping genuinely unsupported content.</comment>
<file context>
@@ -113,16 +146,93 @@ async function readStdin() {
+ continue
+ }
+
+ if (formatFromPath(absPath) === null) {
+ // Unsupported (images, videos, archives, unknown extensions): skip.
+ continue
</file context>
| const outputRoot = resolve(args.output) | ||
|
|
||
| try { | ||
| await mkdir(outputRoot, { recursive: true }) |
There was a problem hiding this comment.
P2: When -o points into (or equals) the input directory tree — e.g. anydoc --batch ./docs -o ./docs/out — the output directory is created before walking, so walkFiles descends into it and re-processes its contents. Freshly written *.md files and copied passthrough files get detected, converted/copied again, and passthrough copies can land back inside the tree; on repeated runs the prior output is re-consumed as input. There is no guard rejecting or warning when the output root is inside the input root. Refuse (or at least warn) when relative(inputRoot, outputRoot) is not strictly outside the input tree.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node/cli.js, line 191:
<comment>When `-o` points into (or equals) the input directory tree — e.g. `anydoc --batch ./docs -o ./docs/out` — the output directory is created before walking, so `walkFiles` descends into it and re-processes its contents. Freshly written `*.md` files and copied passthrough files get detected, converted/copied again, and passthrough copies can land back inside the tree; on repeated runs the prior output is re-consumed as input. There is no guard rejecting or warning when the output root is inside the input root. Refuse (or at least warn) when `relative(inputRoot, outputRoot)` is not strictly outside the input tree.</comment>
<file context>
@@ -113,16 +146,93 @@ async function readStdin() {
+ const outputRoot = resolve(args.output)
+
+ try {
+ await mkdir(outputRoot, { recursive: true })
+ } catch (error) {
+ fail(CONVERSION_ERROR, error.message)
</file context>
| for await (const absPath of walkFiles(inputRoot)) { | ||
| const rel = relative(inputRoot, absPath) | ||
| // Guard against path escape on odd relative() results. | ||
| if (rel.startsWith(`..${sep}`) || rel === '..') continue |
There was a problem hiding this comment.
P3: This path-escape guard is unreachable. absPath is always produced by join(inputRoot, entry.name) inside walkFiles, and inputRoot is resolve()d, so relative(inputRoot, absPath) can never be .. or start with ../ — its first path segment is always a child of the input root. The continue is dead code that can mislead future readers into thinking escaping paths are possible. Consider dropping it (and the now-misleading comment) or replacing it with a comment explaining why it cannot occur.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node/cli.js, line 200:
<comment>This path-escape guard is unreachable. `absPath` is always produced by `join(inputRoot, entry.name)` inside `walkFiles`, and `inputRoot` is `resolve()`d, so `relative(inputRoot, absPath)` can never be `..` or start with `../` — its first path segment is always a child of the input root. The `continue` is dead code that can mislead future readers into thinking escaping paths are possible. Consider dropping it (and the now-misleading comment) or replacing it with a comment explaining why it cannot occur.</comment>
<file context>
@@ -113,16 +146,93 @@ async function readStdin() {
+ for await (const absPath of walkFiles(inputRoot)) {
+ const rel = relative(inputRoot, absPath)
+ // Guard against path escape on odd relative() results.
+ if (rel.startsWith(`..${sep}`) || rel === '..') continue
+ if (isPassthrough(absPath)) {
</file context>
1 participant
Summary
anydoc --batch <dir> -o <dir>to the Node CLI for recursive directory conversionoriginal.ext.md; copy.txt/.json/.md/.pyas-is; skip unsupported files silently--helpCloses #77
Notes
src/bin/anydoc.rson main).Test plan
cd node && node --test(16 tests, including 3 batch cases)anydoc --batch ./sample -o ./outand confirm tree /*.ext.md/ passthrough / skipsSummary by cubic
Adds
--batchdirectory conversion to the Node CLI to recursively convert trees and copy selected plaintext files as-is. Previously the CLI converted one file per run; now it supports--batch <dir> -o <dir>with preserved relative paths and safer error handling, enabling bulk migrations (addresses #77)..txt/.json/.md/.pyunchanged; skips unsupported files silently; overwrites existing outputs;--batchdoes not read stdin.-o <dir>with--batch; rejects--formatin batch mode; accepts--batch <dir>or--batchfollowed by a positional dir.runBatch,walkFiles, and passthrough handling; deferred binding load remains. Tests add batch success/failure/usage cases; README and Node README document the new flag.Rollout/Migration
anydoc --batch <inputDir> -o <outputDir>; do not pass--formatin batch mode.Written for commit fa2cfee. Summary will update on new commits.