Search every issue of The FLUX Review newsletter. Hybrid lexical + semantic search across all issues of The FLUX Review (published weekly since 2021), deployed as a single Cloudflare Worker.
Live: flux-search.adewale-883.workers.dev
- Hybrid search -- D1 FTS5 (BM25) and Vectorize (cosine similarity) fused via weighted reciprocal rank fusion
- Search operators --
before:2024,year:2023,issue:198,section:lens,"exact phrase" - Section-aware results -- each issue is parsed into its intrinsic structure (lead essay, signposts, lens, book, postcard) with section filtering and faceted results
- Confidence tiers -- results ranked high/medium/low with visual weight reflecting match quality
- Auto-sync -- weekly cron discovers and ingests new issues from the Substack sitemap
| Operator | Example | Effect |
|---|---|---|
"..." |
"institutional trust" |
Exact phrase match |
before: |
before:2024-01-01 |
Issues published before date |
after: |
after:2023-06-01 |
Issues published after date |
year: |
year:2023 |
Issues from a specific year |
issue: |
issue:198 |
Jump to a specific issue |
section: |
section:lead_essay |
Filter by section type |
topic: |
topic:"institutional trust" |
Filter to issues tagged with an extracted topic |
Section types: lead_essay, signposts, lens, book, postcard, worth_your_time, fluxers, other.
Requires Node.js 22+, a Cloudflare account, and
npm install -g wrangler
git clone https://github.com/adewale/flux-search.git
cd flux-search
npm installCreate a D1 database and Vectorize index, then update wrangler.jsonc with your IDs:
wrangler d1 create flux-search-db
wrangler vectorize create flux-search-chunks --dimensions 768 --metric cosine
npm run vectorize:metadata-indexesThe metadata-index step enables pre-topK Vectorize filtering for date, section, and topic-constrained semantic searches.
Run migrations and start the dev server:
npm run db:migrate:local
npm run devOpen http://localhost:8787. The search is empty until you bootstrap content.
Set an ADMIN_TOKEN in .dev.vars, then bootstrap from the Substack sitemap:
curl -X POST http://localhost:8787/admin/bootstrap \
-H "Authorization: Bearer YOUR_TOKEN"This fetches all issues, normalizes content, chunks text, generates embeddings, and indexes everything. Run it multiple times if needed -- it's idempotent.
For faster iteration on content normalization without re-fetching from Substack:
npm run corpus:fetch # download raw HTML once
npm run corpus:process # normalize locally (no network)
npm run corpus:validate # validates all corpus recordsnpm run dev # local dev server (port 8787)
npm test # run unit + corpus tests (offline)
npm run test:live # checks against a deployed Worker; requires SEARCH_URL
npm run typecheck # generate Worker types and run TypeScript checks
npm run test:watch # watch modeTests include property-based testing with fast-check for regex-heavy parsers and mathematical invariants.
npm run db:migrate:remote # apply migrations to production D1
npm run deploy # deploy to Cloudflare WorkersBrowser --> Cloudflare Worker (Hono)
|
+---------+----------+-----------+
| | | |
Static D1 (FTS5) Vectorize Workers AI
Assets (lexical) (semantic) (embeddings)
A search query is parsed into free text, quoted phrases, and filter operators. FTS5 and Vectorize run in parallel, and results are fused via reciprocal rank fusion with deterministic reranking boosts for phrase matches, title overlap, and cross-signal agreement.
See docs/architecture.md for the full module map, database schema, and configuration reference, and docs/search.md for a detailed explanation of the hybrid search design and how to tune it for a different corpus.
- Cloudflare Workers -- compute
- D1 -- SQLite database with FTS5
- Vectorize -- vector search index
- Workers AI --
bge-base-en-v1.5embeddings - Hono -- web framework
- Vitest + fast-check -- testing