docs(nav): keep live v2 pages indexed under the hidden v1 nav

[claude]

Requested by Hiba Fathima, Rak Ramprakash · Slack thread

Before: A reader opens https://docs.firecrawl.dev/introduction. The page renders correctly. The page source carries <meta name="robots" content="noindex">. The page is absent from sitemap.xml. Google drops it. The same is true for 36 more live v2 pages, including every features/*, every sdks/*, and every webhooks/* page. Search for "firecrawl scrape docs" and the official page does not come back.

After: The same pages render the same way. The robots meta tag is gone. All 37 pages return to sitemap.xml, to search, to the docs AI assistant, and to llms.txt. The 20 genuinely-v1 pages stay noindexed.

This change adds "searchable": true beside the existing "hidden": true on the v1 Documentation and SDKs tabs, in the en block and in all five localized blocks.

How. docs.json declares two versions per language. v2 is visible. v1 carries hidden: true. The v1 Documentation and SDKs tabs re-list the same page paths that the visible v2 nav uses. They do not point at v1-specific copies. Mintlify resolves indexing per page path, so the hidden v1 listing wins over the visible v2 one. Per the docs.json schema, searchable on a hidden node keeps descendant pages indexed for search, sitemap, AI assistant, and llms.txt while they stay out of the rendered navigation. The flag has no effect unless hidden is set on the same node, which it already is. Nothing that renders changes.

Regression history. Please do not remove this flag again.

• 570395e (2026-05-07) added hidden: true to the v1 version. This started the problem.
• PR #1253, commit 4bcbc3d (2026-08-12) fixed it. It added searchable: true beside hidden: true on the v1 Documentation and SDKs tabs in the en block.
• PR #1346, commit df1028c (2026-09-01) deleted both English searchable lines. It stated the intent "English v1 stays hidden and unsearchable".
• PR #1347, commit ebdca99 deleted the localized lines. The Locadex bot owns the localized page directories and mirrors English, so it followed Fix init skill-family copy and remove English migrate-to-v2 #1346.
• At main before this PR, docs.json held zero searchable keys. The bug is live.

The intent in #1346 and the goal of this PR are compatible. This PR keeps every genuinely-v1 page unsearchable. It restores only the page paths that v1 shares with the visible v2 nav. The v1 Integrations and API Reference tabs are untouched, so their 20 v1-only pages stay hidden and unsearchable. Removing the flag again re-breaks 37 live v2 pages. docs.json is strict JSON and takes no comment, so this note has to live here in the PR body.

Note on the localized blocks: the Locadex bot owns the es/ fr/ ja/ pt-BR/ zh/ page directories and mirrors English. This PR edits only the localized navigation blocks inside docs.json, which is what PR #1253 edited by hand. No locale .mdx file is touched.

Verification. Every one of the 37 paths that this change makes searchable also appears under the visible v2 navigation. Zero v1-only paths are affected. The count is 37 per language, and 222 path entries across the 6 languages. docs.json parses as JSON. The three repo checks in .github/workflows/checks.yml produce byte-identical output with and without this change.

The 37 restored paths

advanced-scraping-guide
agents/fire-1
billing
dashboard
enterprise
introduction
mcp-server
partner-credits
rate-limits
features/batch-scrape
features/change-tracking
features/crawl
features/enhanced-mode
features/fast-scraping
features/ip-restrictions
features/key-restrictions
features/llm-extract
features/map
features/proxies
features/scrape
features/search
features/siem
features/threat-protection
sdks/dotnet
sdks/elixir
sdks/go
sdks/java
sdks/node
sdks/overview
sdks/php
sdks/python
sdks/ruby
sdks/rust
webhooks/events
webhooks/overview
webhooks/security
webhooks/testing

Each locale carries the same 37 paths under its own prefix, for example es/introduction.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CEeUdmLgiSQtgd8qrKjKSU

---

Generated by Claude Code

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
firecrawl	🟢 Ready	View Preview	Sep 9, 2026, 5:15 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

[firecrawl-spring]

Why the red check is not from this PR

The failing check is "Locale literals and extraction-hostile markdown", run 34314030564.

It is pre-existing on main. I ran the exact CI script on the base commit and on this branch:

sh scripts/check-locale-api-literals.sh

Base d705404: exit 1, 14 findings.
This branch 77217de: exit 1, 14 findings.
diff of the two outputs: 0 lines differ. The output is byte-identical.

Every finding sits in a locale .mdx file under es/, fr/, or pt-BR/. One example:

translated API literal: es/features/change-tracking.mdx uses `seguimientoDeCambios`, which no English page uses

I did not re-run the failed check. The byte-identical base reproduction is stronger evidence than a re-run. It isolates the change instead of retrying the same job.

This diff cannot affect the check. The script reads its whole input set from git ls-files '*.mdx'. It never opens docs.json. This diff touches docs.json only, and zero .mdx files.

The failure is invisible on main. .github/workflows/checks.yml triggers on pull_request only. It never runs on a push to main. So main shows no red check.

No fix is ported here. Fixing it means editing Locadex-generated locale .mdx files. The repo CLAUDE.md forbids hand edits to those files. The Locadex owner needs to make that change separately.

Mintlify Validation link-rot

That check is neutral and non-blocking. It reports 208 broken links. All of them are pre-existing. None fall in the 37 restored paths, and this diff adds none.

The preview cannot verify this fix

The preview host serves robots: noindex, nofollow on every page. Here is the control. Preview /api-reference/introduction is a v2 page that the bug never touched, and it also returns noindex, nofollow. Production /api-reference/introduction has no robots tag at all. So the preview proves nothing either way.

Verification needs a production scrape after merge. Production /introduction and /features/scrape both serve robots: noindex today, which matches the diagnosis.

---

Generated by Claude Code