A self-hosted dashboard plus a composite stress index per country, fed by four independent open-data domains (market signals, geopolitical events, hazards, wildfire load). A personal infrastructure project meant to run for years on a Raspberry Pi.
The project is built around one specific claim: a composite of three heterogeneous OSINT signal domains discriminates later instability events better than the best single-domain baseline. The dashboard, the Pi, the maps — they are the system that lets that claim be measured.
Status, plainly: the claim has been tested five times and failed every time. The composite does not discriminate better than the dumb baselines — see §2.3. Separately, the live composite currently returns the constant
0.5for every country: retention holds 30 days while the rolling z-score needs at least three monthly observations, so every domain z-scores to zero and the index rescales to its neutral midpoint (#586). The historical panel is unaffected — it is built byapp/composite/backfill.py, which fetches history directly rather than reading the rolling events table. Ingestion, stories, corroboration, coverage bias and the brain are working and are what this system currently does well.
- Chapter 1 — Switch it on
- Chapter 2 — Can we trust the data?
- Chapter 3 — How the data gets collected
- Chapter 4 — The brain
- Chapter 5 — How to read the dashboard
- Reference shelf — deep architecture material
Everything runs locally. All persistent data lives in one folder —
$OSINT_DATA_DIR (default ./data, gitignored).
# prerequisites: Docker Desktop, Python 3.14, Node + pnpm
python3 -m venv .venv && .venv/bin/pip install -e .
cp env.example .env # set POSTGRES_PASSWORD (+ any API keys)
docker compose up -d # stores: Postgres + Redis
.venv/bin/alembic upgrade head # create the schema
cd osint-frontend && pnpm install && cd ..There are only three you need. Everything else in the Makefile is either an alias kept for muscle memory or a one-shot analysis task.
| Command | What it does |
|---|---|
make up |
Start everything: Docker stores, worker, beat, API, dashboard, Ollama |
make down |
Stop everything, keep all data |
make clear |
Remove regenerable junk: build caches, __pycache__, logs, Docker cruft |
Restart after a code change is make down && make up. make clear never
touches your data — only things that rebuild themselves.
Older names still work (make stop, make clean-dev, make docker-prune),
and make off additionally quits Docker Desktop on macOS. make logs tails
the background logs without stopping anything.
Dashboard: http://localhost:3000 · API health: curl localhost:8000/health → {"status":"ok"}.
Local dev keeps raw API pulls deliberately capped so the same data is not held
at full size by Postgres, FastAPI, Next dev, browser state, and the map at once.
Tune API_MAX_LIMIT plus the NEXT_PUBLIC_*_LIMIT values in .env when moving
between laptop and Pi-class hardware.
make up is self-healing: it opens Docker if needed, skips what already runs,
and if the Docker daemon has corrupted the compose project (the "No such
container" disease) it automatically switches to a fresh project name — your
data never moves, it lives on bind mounts.
Each one records itself in the top-bar activity monitor — a chip per job,
green pulsing while working (with live progress), red while idle, red
failed with the error in the tooltip if something breaks.
| Command | What it does |
|---|---|
make labels |
rebuild ground-truth labels from the ACLED xlsx drop-folder |
make panel |
build the country-month panel (labels + composite) → data/exports/panel.parquet |
make baselines |
score the composite against the no-skill baselines → the head-to-head report |
make coverage |
measure who gets covered and who gets ignored → the attention-bias report |
make journal |
run the forward-prediction journal once (emit + grade + scoreboard) |
make briefing |
generate the weekly briefing — the newsletter-ready one-pager (#401) |
make stories |
cluster the rolling news window into stories |
make stories-audit |
emit the cluster hand-check sheet (threshold audit) |
make sensor-checks |
check story claims against physical sensors → verdict board |
make disagreement |
score cross-country telling divergence → most contested stories |
make indicator-ranking |
rank every dashboard indicator by measured predictive value |
make onset-eval |
run the pre-registered onset evaluation (calm-window months only) |
make within-eval |
run the pre-registered within-country evaluation — a country's own onset months vs its own calm months (#582) |
make validator |
local-LLM claim extraction over window stories (needs Ollama) |
make validator-audit |
emit the human-check sheet that gates validator use |
make validator-agreement |
publish the model-vs-human agreement rate from the filled sheet |
make severity-grade |
grade stored news severity with the local model — reports; --apply writes (#591) |
make severity-audit |
emit the human-check sheet that gates LLM severity use (#593) |
make severity-agreement |
publish the model-vs-human severity agreement from the filled sheet (#593) |
make brain-qa-eval |
compare Q&A candidate models locally and write the Phase C report |
make backfill-signals |
rebuild 2015-2024 composite history (market + GDELT + hazard); resumes via checkpoints |
python scripts/data_audit.py |
audit every source against its declared expectation — does severity parse, vary, and reach anything downstream (#580) |
make data-size # what is using disk
make data-prune # run retention now
make data-reset # ⚠️ delete ALL local data
uv run python scripts/db_snapshot.py # exact row counts, for docsRetention keeps the events table small — ~30 days for news, hazard and
GDELT raw events (RETENTION_NEWS_DAYS, RETENTION_HAZARD_DAYS,
RETENTION_GDELT_DAYS in .env). Market and macro history is never deleted
because it cannot be re-fetched, and everything analytical — scores, labels,
stories, gists, journal, the GDELT daily-volume aggregate — is derived and
kept long-term.
Manual mode — one process per terminal
docker compose up -d
.venv/bin/celery -A app.celery_app worker -l info
.venv/bin/celery -A app.celery_app beat -l info
.venv/bin/uvicorn app.api:app --host 0.0.0.0 --port 8000
cd osint-frontend && pnpm devThis is the project's main scientific problem, explained from zero.
The dashboard eats the world's open data all day: news feeds, earthquake sensors, satellites, market prices, conflict databases. The obvious question — the one an analyst or a customer asks first — is:
"How do you know any of this is true, complete, and not just one loud country's version of events?"
Honest answer: you can never fully know. Anyone who says their data is "unbiased" is selling something. What you can do is measure three things and publish all of them. That is this chapter.
sensors (don't lie) text (can lie)
┌──────────────────────┐ ┌──────────────────────┐
│ USGS quakes │ │ 44 news feeds (RSS) │
│ NASA FIRMS fires │ │ GDELT world events │
│ GDACS disasters │ └──────────┬───────────┘
│ market prices │ │
└──────────┬───────────┘ │
└────────────┬────────────────┘
▼
┌─────────────────────┐
│ events table │ ← every row has a fingerprint
│ (Postgres) │ (source, source_event_id):
└─────────┬───────────┘ the SAME quake fetched 100×
│ is still ONE row. The DB
▼ enforces it, not good manners.
scores · labels · stories · journal
(all versioned, all append-only or
overwrite-in-place — never duplicated)
Corruption guards, in one breath: files are written atomically (a crash leaves no half-file), a failed download raises instead of silently writing a partial month, and every methodology change gets a new version stamp instead of editing history.
We can't compute "truth", so we compute three honest proxies per story:
CORROBORATION (WS-C)
"how many INDEPENDENT
tellers + does machine
data confirm it?"
▲
╱ ╲ 10 copies of one Reuters wire = 1 source.
╱ ╲ Story says "earthquake" → is there a
╱ ╲ matching USGS row? Machines don't spin.
╱ trust ╲
╱─────────╲
DISAGREEMENT COVERAGE BIAS (WS-D ✅ live)
(WS-B) "who gets talked about
"how differently do at all?" — top-5 countries
countries tell the eat ~30% of all event
same story?" volume. Measured monthly,
shown on /coverage.
- Corroboration (WS-C, done) — every story cluster shows how many
independent owners tell it, not just feeds: wire copies and co-owned
outlets collapse into one teller (BBC's two feeds are one owner; RT + TASS
are one state controller — #356). Claims are checked against sensors
(#361): an earthquake story either has a matching USGS row or it doesn't —
wildfire→FIRMS, disaster→GDACS, market crash→drawdown; verdicts keep their
evidence snapshot after retention deletes the sensor row. Both fold into
one fixed, versioned number (
corroboration-v1.0, #363): each extra owner halves the remaining doubt, a sensor confirmation halves it once more — shown with its full evidence trail on the /stories card (#365). The clustering threshold under all of this was hand-audited first —docs/audits/stories-threshold-audit.md. - Disagreement (WS-B, done) — same story, different countries' outlets:
how far apart are the tellings? Every cross-country story carries a
disagreement-v1.0divergence (#370), rolled up per (country-pair, month) (#372) — first live board: RU|US at 0.832 mean over 11 co-told stories. Whether divergence predicts instability is under a pre-registered forward evaluation (#374): divergence exposures enter the prediction journal daily and get graded like every other forecast — protocol frozen indocs/disagreement-exam.mdbefore the first prediction was issued. - Coverage bias (WS-D, done) — the dashboard publishes its own blind spots instead of hiding them.
- Local AI checker (WS-G, running) — a local Ollama model
(
qwen3.5:4b-q4_K_M, 3.4 GB, nothing leaves the machine) extracts structured claims per story nightly — countries, event type, casualty count (#378). Treated as another fallible annotator, never a judge: nothing consumes its output until its agreement with a human-checked sample (make validator-audit) is measured and published.
pre-registered rules ──► evaluation runs ──► number published
(methodology.md, (baselines vs (win OR lose —
written BEFORE composite on the scoreboard
any result exists) 2015-2022 only) shows it in red)
2023-2024 = TEST WINDOW — locked, untouched, for the final held-out evaluation.
- Ground truth comes from ACLED (human-validated conflict data), versioned like code: labels-v1.0 → v1.1, never edited in place.
- Forecasts are server-stamped before outcomes are known, graded once, and can never be back-filled. The track record is earned or it is nothing.
- The current headline: with all three domains live, the composite scores
AUROC 0.502 against a per-country base rate of 0.929. A coin flip —
published, not hidden. The generous reading was that our index measures
"is this country behaving unusually vs its own past?" while the evaluation asks
"which countries have conflict at all?" — so a second, pre-registered
test restricted scoring to onset months (calm-before-the-storm cases,
docs/onset-eval.md, #380). Verdict: coin flip there too (0.496–0.526 vs a 0.744 base rate) — even among countries calm for a full year, long-run relapse history dominates and the composite adds nothing measurable yet. - The objection that both exams were the wrong instrument — tested, and it
does not rescue the composite. Both scored a single pooled AUROC, while
the composite z-scores within country and so carries no cross-country level
by construction. With 133 of 238 panel countries never labelled and 10
labelled in ≥90% of months, a pooled metric is largely rewarded for telling
Norway from Syria — which is where the ~0.93 base rate comes from. So a
third evaluation was pre-registered (
docs/within-country-eval.md, #582) asking whether the composite ranks a country's own onset months above its own calm months. Verdict: negative — best 0.531 at k=6, below the declared 0.55 threshold, 95% CI [0.474, 0.582] containing 0.5. The stratification demonstrably worked: the base rate collapsed from 0.93 pooled to 0.30 within country. The composite simply had nothing hidden underneath. This settles it for the composite as constructed. - All negatives are published. What it does not settle is whether the
inputs carry anything:
scripts/data_audit.py(#580) shows severity is a two- or three-level categorical across nearly every source, and #579 that the FIRMS value is detection confidence rather than fire intensity — and non-monotonic against it. "Bad construction" and "bad inputs" remain indistinguishable from these results alone. News severity in particular is no longer a keyword substring match — every score now records why it was assigned (#592), and the local-model regrade behind it is gated on a measured model-vs-human agreement rate (make severity-audit/severity-agreement, #593) before anything downstream trusts it.
| Workstream | What | State |
|---|---|---|
| WS-A story clustering | one row per real-world story | ✅ live, threshold audited |
| WS-D coverage bias | attention-bias table | ✅ live on /coverage |
| WS-E prediction journal | forward track record | ✅ live on /scoreboard |
| GDELT backfill | third composite domain, 2014-2024 | ✅ done — fair test ran |
| Onset evaluation | the composite's second pre-registered test | ✅ ran — coin flip again, honestly published (#380) |
| Within-country evaluation | the composite's third, on the axis it was built for | ✅ ran — negative, 0.531 vs a declared 0.55 (#582) |
| Source data audit | does each source's data mean what it claims | ✅ live — 50 findings across 47 sources (#580) |
| WS-C corroboration | independent-owner counts + sensor cross-checks | ✅ live — corroboration-v1.0 on /stories (#365) |
| WS-B disagreement index | cross-country telling divergence | ✅ live — index + pre-registered forward evaluation (#374) |
| WS-F indicator ranking | which dashboard number predicts best | ✅ ranked — |
| WS-G local AI checker | Ollama claim extraction w/ measured error rate | 🔨 machinery done (#386) — awaiting The operator's filled audit sheet |
The living log of all of this is pinned issue #282.
Every source, how it is downloaded, and how the disk is managed. No scraping anywhere: official APIs, published bulk files and RSS only — if a future source needs real scraping, its terms of service get read first or it stays out.
Once make up runs, the scheduler fires these forever:
| Source | What | How it is pulled | Cadence |
|---|---|---|---|
| yfinance | country-ETF prices → market severity | Yahoo Finance API | 5 min |
| GDELT v2 | world geopolitical events | lastupdate.txt pointer → newest 15-min export zip |
15 min |
| USGS | earthquakes | FDSN query API (geojson) | 15 min |
| GDACS | multi-hazard alerts | official geojson feed | 15 min |
| NASA FIRMS | active fires | official CSV endpoint | hourly |
| EONET | natural events | NASA API | 30 min |
| 44 news feeds | world/regional news | RSS (published by the outlets) | hourly, staggered |
| EM-DAT / FRED | disasters / macro series | official APIs | daily |
| ACLED | conflict events | manual drop-folder + opt-in API (see 3.3) | hourly check |
| OpenSky | aircraft states | REST API | 2 min |
| abuse.ch / Polymarket / UK Police | cyber / prediction markets / crime | official APIs | 15 min – daily |
| named-place enrichment | verified RSS buildings/streets/sites | Wikidata API + persistent cache | 30 min, bounded |
Every fetched row gets a fingerprint (source, source_event_id) — the
database rejects the same real-world record twice no matter how often it is
re-fetched (Chapter 2.1). Per-source successes/failures land in
ingest_health, which is what the dataset chips in the top bar read.
58 collectors in total: 14 named fetchers plus 44 RSS feeds loaded from
app/sources/rss_feeds.json.
Snapshot of 2026-07-24 — 939,854 events across 53 active sources, 1,077 MB.
Regenerate with uv run python scripts/db_snapshot.py rather than quoting
these forever: retention prunes continuously, so every figure below moves.
| source | rows | span held |
|---|---|---|
| nasa-firms | 764,814 | rolling ~2 weeks |
| gdelt | 71,474 | rolling ~30 days |
| opensky-adsb | 61,313 | rolling, hourly country rollups |
| abuse-ch-urlhaus | 15,391 | rolling ~30 days |
| 44 RSS feeds | ~20,000 | rolling ~2 weeks |
Derived on top: 19,140 stories · 26,985 story members · 10,949 embeddings · 6,742 gists · 3,791 claims · 17,004 corroboration rows · 1,444 disagreement rows · 61,306 scores · 582 journal predictions · 15,600 rows of GDELT daily volume across 93 ingested archive days.
scripts/data_audit.py (#580) is the companion to this table: it reports not
how many rows a source has but whether they mean anything — whether severity
parses, varies, and reaches anything downstream, and whether country fields
required by navigation remain populated. RSS country coverage is required
because the map uses it to retrieve news by country (#717); unresolved rows are
reported as findings rather than hidden behind an optional declaration. Its
first run returned 50 findings across 47 sources.
"Span held" is not "span covered." The events table is a rolling window,
not an archive. GDELT's eleven years of history exists as monthly aggregate
JSONs in data/gdelt/ (§3.2) and the daily-volume table — never as raw rows,
which retention would delete anyway. A source can vanish from this table
entirely: uk-police published one month, and once that month aged past
retention its rows went with it.
Every number here comes from
scripts/db_snapshot.py, which counts rows. An earlier version of this section was built frompg_stat_user_tables.n_live_tup— an autovacuum estimate that reported 18 rows inpredictionsagainst an actual 582. Estimates are for query planners, not documentation.
The shape is lopsided on purpose. Fire pixels and aircraft states are cheap and high-volume; they are sensors. News is low-volume and expensive to reason about; it is narrative. The whole analytical question is whether the first moves before the second.
GDELT publishes a whole family of data products (Event Database 1.0/2.0, Global Knowledge Graph, BigQuery mirrors, an Analysis Service, ngrams, quotation/entity/frontpage graphs...). We use exactly two of them, both plain static files off their CDN, and skip the rest deliberately:
| GDELT product | Use it? | Why |
|---|---|---|
| Event DB 2.0 (15-min zips) | ✅ live | freshest event stream; lastupdate.txt always points at the newest file — trivially pollable, no key |
| Event DB 1.0 (daily zips) | ✅ history | the only product covering our whole 2014-2024 window at daily grain with stable format |
| Event DB 1.0 "reduced" | ❌ | ends Feb 2014 — misses the window; its one-event-per-day collapse also destroys volume information |
| Global Knowledge Graph 1.0/2.0 | ❌ (for now) | themes/emotions/counts at ~2.5 TB per year — far beyond what the Goldstein signal needs; a candidate for the WS-B disagreement work later, as its own decision |
| 2.0 "mentions" table | ❌ (for now) | per-article mention records — interesting for corroboration counts someday, unnecessary for a monthly country mean |
| BigQuery mirror | ❌ | needs a Google Cloud account + billing; this project is local-first with zero cloud dependencies |
| Analysis Service | ❌ | browser/email tool for humans, GDELT 1.0 only, not automatable |
| Normalization files | ❌ | they correct for news-volume growth over time — our signal (mean Goldstein per country-month) is already volume-independent, and the rolling z-score handles drift |
| Frontpage / quotation / entity graphs, ngrams, TV/visual datasets | ❌ | different research questions entirely |
The two we use, side by side:
LIVE (every 15 min) HISTORY (one-time, 2014-2024)
GDELT v2 GDELT v1 daily archives
───────────────── ──────────────────────────
lastupdate.txt ── points to ──► ~4,000 files, one per day:
newest export.CSV.zip YYYYMMDD.export.CSV.zip
│ unzip in memory │ download → unzip in memory
▼ │ keep 3 columns only
events table │ (date, GoldsteinScale, country)
(retention ~30 days) ▼
per-country monthly sums
→ one small JSON checkpoint per month
in data/gdelt/ — raw file DISCARDED
The history run downloads ~40 GB over its lifetime but keeps under 1 MB: only the monthly aggregates survive. Checkpoints are written atomically (temp file → rename), so a killed run resumes at the next month for free and a finished cache makes re-runs instant. Missing days (GDELT has known gaps) are recorded in the checkpoint, not papered over; transient download errors retry three times, then fail the whole month loudly rather than write a partial one.
The analysis side needs a different thing from the two paths above: how much
coverage a country got on a day, as a time series. The DOC API answers that
directly but reaches back only about three months and is aggressively rate
limited, so scripts/gdelt_archive.py walks the same 15-minute export grid and
aggregates it instead:
uv run python scripts/gdelt_archive.py --start 2026-04-20 --end 2026-07-19Roughly 4 seconds per day of history — 91 days took under 5 minutes.
Two rules make it work. Counts, not rows: raw GDELT events are pruned at 30
days, so a multi-year raw backfill would delete itself; gdelt_daily_volume
holds one row per country-day and sits outside retention. And rows are bucketed
by the file's timestamp, not their own Day column — a single export from
2026-04-15 carries rows dated 2025-04-15, because Day is when the event
happened and the file stamp is when GDELT saw it reported. Coverage timing is
the thing being measured, so the file stamp is the correct clock.
A second table records which days were actually walked. Without it, a country with no coverage on a day is indistinguishable from a day nobody downloaded, and the analysis cannot tell a quiet narrative from a missing one.
What this path is not good for: topic-scoped questions. GDELT's Events
export codes political actions, so there is no earthquake event type to filter
on — the series is whole-country volume. See docs/backtest/ for where that
limitation bites.
There are precisely two GDELT URLs in the codebase:
| Path | URL | In code |
|---|---|---|
| live pointer | http://data.gdeltproject.org/gdeltv2/lastupdate.txt |
app/sources/gdelt_fetcher.py |
| history pattern | http://data.gdeltproject.org/events/YYYYMMDD.export.CSV.zip |
app/composite/gdelt.py |
The pointer file, fetched every 15 minutes, returns three lines — size, MD5 hash, URL — for the newest batch:
54849 e2077439... http://data.gdeltproject.org/gdeltv2/20260709103000.export.CSV.zip
71626 33a5f6da... http://data.gdeltproject.org/gdeltv2/20260709103000.mentions.CSV.zip
4517072 d04d0311... http://data.gdeltproject.org/gdeltv2/20260709103000.gkg.csv.zip
We take the export line only. (GDELT publishes those MD5 hashes; we do not
verify them yet — a cheap integrity upgrade on the list.)
"Pulling" is a plain HTTP GET via httpx with an honest User-Agent
(OSINT-project-project (academic)), a 120 s timeout, and 3 retries with
backoff on the history path — no browser, no scraping, no auth. The zip is
unzipped in memory, rows are split on tabs, three columns are kept, the
rest is discarded.
Check it yourself: paste
http://data.gdeltproject.org/events/20220301.export.CSV.zip into a browser.
That downloads the exact file the backfill parsed for 1 March 2022 — open it
and you are looking at the same raw rows we aggregated.
ACLED is registered-access and its data API rejects many valid academic
accounts, so the ground-truth labels are built from manually downloaded
aggregate xlsx files dropped into the data directory; make labels
rebuilds from whatever is there. A helper (scripts/acled_browser_sync.py)
can capture downloads from your own logged-in myACLED browser session — it
automates your clicks and bypasses nothing.
Manual is a feature, not a gap: ground truth must never silently drift
under a running evaluation. Label changes only happen as versioned,
pre-registered amendments (labels-v1.0 → v1.1 lives in docs/methodology.md).
make backfill-signals rebuilds the composite's 2015-2024 history from three
sources in one run — market (full ETF price history via yfinance), hazard
(all M≥4.5 quakes via USGS, yearly chunks under their 20k-row cap) and
geopolitical (the GDELT v1 path above). Everything flows through the same
code path as live data, rows carry backfill: true for provenance, and the
whole thing is idempotent — run it twice, get the same database.
data/ (one folder, bind-mounted, survives anything)
├── postgres/ 1.5 GB the database — 940k events, 1077 MB of it
├── private/ 104 MB ACLED drop-folder (gitignored)
├── redis/ 48 MB queue state, ephemeral
├── gdelt/ 3.1 MB monthly checkpoint JSONs (11 years of GDELT)
├── exports/ 2.9 MB panel.parquet + all reports the dashboard serves
└── backtest_cache/ 216 KB cached GDELT DOC responses + rate-limit state
Retention (03:00 UTC daily + make data-prune): ~30 days for news, hazard
and GDELT raw events, overridable per source in .env. The events table stays
small because everything analytical is derived and kept forever (scores,
labels, stories, gists, journal, job runs, the GDELT daily-volume aggregate),
and because market/macro history is exempt — it cannot be re-fetched. Backups: make data-size to inspect, snapshot script in
scripts/, and the whole folder is portable — point OSINT_DATA_DIR at an
external disk and move it.
Hazard rows arrive as a single point; the map wants the real shape. A few
minutes after each USGS/GDACS fetch, enrich_footprints pulls the actual
geometry — USGS ShakeMap contours, GDACS alert polygons, and EONET's reported
extent for wildfires and sea ice (#205, #612/#614). Stored geometry is capped
at 50 kB per event so one sprawling polygon cannot bloat the row (#615),
and footprints survive a later refresh of the same event instead of being
overwritten back to a bare point (#604/#611/#618/#621). A watchdog alerts if
footprint coverage suddenly collapses — a silent upstream format change should
not quietly empty the map (#617/#620).
RSS feeds rarely supply coordinates. The fast resolver uses bundled city and
region data; enrich_news_places then checks explicit named buildings,
streets, and sites against Wikidata and upgrades markers only after exact name
and country matches (#745, #747, #748). For a single-place story, the entity
must also match its city anchor and sit within 75 km. Several
candidates use country-only gates because one row-level city cannot govern all
of them. Accented Latin, Cyrillic, Arabic, and Devanagari kind words select a
bounded local-language Wikidata search; an exact local label or alias is still
required, so transliteration alone never proves identity (#749). Search rank
never decides identity. Distinct verified places are
stored on one story row and rendered as separate markers; aliases resolving to
one Wikidata ID collapse. Partial proof draws only proven points. Positive and
negative results are cached in Postgres, and each verified marker carries its
place label, Wikidata ID, precision, source, check time, and resolver version.
Every place-backed news/GDELT coordinate in the active client event window
enters one MapLibre GeoJSON source; worker-side clustering controls visual
density without sampling, moving, or discarding those points. Hazards retain
their independent marker and footprint layer. A refreshed database row replaces
the client-buffer copy even when its event ID is unchanged, so later exact-place
enrichment reaches an already-open map (#762).
The system has a small local brain: a light model (qwen2.5:1.5b-instruct-q4_K_M,
~1 GB) that runs only when the box has headroom and narrates what is going on —
both the world signal and the pipeline itself.
Production is an 8 GB Raspberry Pi. A model pinned in RAM 24/7 would fight scraping
and the analytical batch and OOM the Pi. So the brain uses adaptive keep-alive:
warm during idle windows, evicted the instant a heavy job starts, reloaded when
the box goes quiet again. The eviction is wired into job_run() — every heavy job
passes through it, so the model always steps aside before the pandas parse grabs
memory. A resource gate (app/brain/gate.py) also refuses to load unless there is
enough free RAM and no heavy job is already running.
Every ~15 minutes, when the gate allows, the brain reads a compact snapshot (top stories, job outcomes, ingest freshness) and writes a short JSON narrative:
- headline — the single most important thing right now
- world — 2-4 sentences on the story signal
- system — 1-2 sentences on pipeline health
- watch — a few things to keep an eye on
It describes only the numbers it is given — same no-fabrication discipline as the validator. It never invents facts.
Expected shape (GET /brain/narrative/latest):
{
"present": true,
"model": "qwen2.5:1.5b-instruct-q4_K_M",
"created_at": "2026-07-12T12:00:00+00:00",
"payload": {
"headline": "Border-clash coverage is the loudest signal; pipeline healthy.",
"world": "Seven outlets are carrying a border-clashes story across twelve members. No other cluster is close in reach.",
"system": "All six analytical jobs completed in the last hour; ingest last checked two minutes ago.",
"watch": ["Whether the border story keeps gaining outlets", "The failed composite job from earlier"]
}
}When the box is busy, the brain steps aside; GET /brain/narrative/latest simply
returns the last narrative and the Situation card renders "brain resting" so the
backoff is visible.
make up starts Ollama for you: if it's installed and not already running, the
start-up brings up ollama serve, waits for it, and pulls the light brain model on a
fresh box — so one command runs the whole app with its brain, and make down/make off stops the Ollama it started (a hand-started ollama serve is left alone). It's
best-effort: if Ollama isn't installed the app still comes up and the brain stays
dormant. Skip the autostart with OLLAMA_AUTOSTART=0.
Run one pass by hand:
make brain # run one narration nowThe nightly validator keeps its own 4b model; the brain is separate and lighter.
Turn the brain off entirely with BRAIN_ENABLED=false in .env.
Beyond narrating on its own schedule, the brain answers questions on demand.
POST /brain/ask with {"question": "..."} returns {"answer": "...", "context_digest": "...", "sources": [...]}. The answer is grounded in the same live
snapshot the narrative uses, plus question-retrieved recent stories carrying source
outlets, corroboration, contested flags, sensor verdicts, and gists. It answers only
from that context and says "I don't have data on that" otherwise — it never invents.
Because a question is user-initiated (you're waiting for the answer), Q&A does not back off on every running job the way the scheduled narrative does — it refuses only when free RAM is below the floor, returning a short "brain busy" answer so the Pi never OOMs. If Ollama is down it answers "The brain is offline right now." Nothing is persisted; the ask box lives at the bottom of the Situation card and clears on reload.
Answers reflow into short readable paragraphs rather than one wall of text (#599), and an elaborate chip under the latest answer re-runs it in an ELI10 mode that spells out each inference step with its own label (#601/#603) — for when the terse answer moved too fast.
Model policy (#413/#433): the 1.5b brain_model stays warm for the scheduled
narrative and story enrichment above; every user ask (/brain/ask and
/brain/ask/stream) runs the heavier 4b model per-call with keep_alive=0, so
the 4b never lingers after an ask. The nightly validator reuses the same 4b
model but keeps it warm (keep_alive=5m) across its own batch. Q&A refuses
politely below qa_min_free_mb (3800 MB free RAM).
Example:
POST /brain/ask {"question": "what is the most contested story?"}
→ {"answer": "The most contested story is the border-clashes report, with a
divergence of 0.83 across the outlets telling it [1].",
"context_digest": "sha256:…",
"sources": [{"n": 1, "outlets": ["Reuters", "BBC"], "contested": true, ...}]}
Phase C evaluated whether the validator's heavier 4b model was worth using for Q&A;
the answer was yes — Q&A has run the 4b in production since #433. make brain-qa-eval
remains the before/after measurement tool, comparing it with the 1.5b on identical
retrieved contexts and writing data/exports/brain-qa-model-eval.md and .json.
The nightly validator gives stories full claims once a night with the heavy 4b model.
The brain adds a faster, lighter first-look: every ~20 minutes, on idle windows, the
1.5b model gives each new story a one-line gist plus two tags — a category
(conflict, economy, disaster, politics, other) and an escalating flag
(yes, no, unclear). It reads only the story's own headlines and invents nothing;
anything a small model returns off-vocabulary is coerced to other / unclear, so the
tags stay clean and filterable.
The pass is idle-gated (same RAM + no-heavy-job gate as the narrative) and batch-capped
(~20 stories per run), so a burst of new stories clears within an hour or two without
straining the Pi. Gists land on /stories/top and show as a line under each story on
the Stories card, with a small category chip (↑ marks an escalating story). Run one pass
by hand with make enrich. Stored 30 days, then pruned.
Every number on the analytical cards, in plain language: what it is, what good and bad look like, and what to actually do with it. No statistics degree required.
The right pane is a deck of five cards (swipe, or click for fullscreen): situation, briefing (the landing card, explained below), console (the raw event feed as it arrives), and the three analytical cards — stories, coverage, scoreboard. Geography lives in the map on the left; the 3D globe was removed in #494 because its WebGL context was the tab's largest memory holder.
The deck opens with the answer instead of a hunt (a pattern carried over from the news-intelligence-platform, this project's predecessor):
- World stress level in one plain word — calm / elevated / high stress — computed from the average of every country's latest composite score. Hover it for the honest caveat: the word describes measured stress today, not a validated forecast.
- Most trustworthy story right now — the highest-confidence story of the last 24 hours, with its owners badge and sensor ✓.
- Most contested telling — the story whose country blocs word it most differently, with who disagrees with whom.
- Biggest stress movers — countries whose index moved most since last month, ▲ rising / ▼ easing.
Everything on it is a doorway: the detail always lives on one of the three
cards below. The same idea as a weekly artifact: make briefing writes a
newsletter-ready one-pager (also generated automatically every Monday
morning) — the first step on the productization path tracked in issue #400.
Every row is one real-world story, no matter how many outlets wrote it up. The machine groups similar headlines every 30 minutes, so "Earthquake strikes Tokyo" and "Dozens injured as quake hits Tokyo" are one row, not two.
The badge on the left — 7 src — is the number that matters. It counts
independent tellers, not feeds: ten copies of one Reuters wire story
count as one, the BBC's two feeds count as one, RT and TASS (both
controlled by the Russian state) count as one. So 7 src means seven
genuinely separate organisations chose to tell this story.
The badge's colour is a confidence score (0 to 1, shown in the tooltip):
| colour | score | plain meaning |
|---|---|---|
| dim grey | 0 | one unverified teller — this is a rumour until proven otherwise |
| grey | up to 0.5 | a second independent teller exists — worth a look |
| green | 0.5–0.75 | several independent tellers — probably real |
| cyan | 0.75+ | many independent tellers, often machine-confirmed — take seriously |
The formula behind the colour is one sentence: each additional independent teller halves the remaining doubt; a physical-sensor confirmation halves it once more.
The ✓ earthquake / ✓ disaster chips are the machine check: the story
claimed a physical event, and a sensor that cannot spin a narrative (USGS
seismometers, NASA fire satellites, GDACS disaster feeds, market data)
confirmed something physical actually happened there, then. A story with
a ✓ chip is corroborated by hardware, not just by other journalists. No chip
on a physical claim means no matching sensor row was found — often because
the story is about a past event, sometimes because it's inflated.
What to do with it: scanning for what's real, sort by instinct — cyan
badges with ✓ chips first, dim single-teller rows last. The
min owners filter (set it to 2+) hides everything only one organisation
has said. The header line — "X stories · Y told by 2+ independent owners ·
Z sensor-confirmed" — is the day's honesty summary: how much of the news
flow is corroborated versus single-sourced.
Framing and the deep read. Each story carries a deterministic framing breakdown — which angle each bloc leads with — replacing the old raw keyword dump (#606). For a contested story you can pull an on-demand deep read (#608): the 4b model explains, in plain prose, why the tellings diverge, returned as readable text rather than truncated JSON (#610). Both are opt-in per story, so the Pi only spends the tokens when you ask.
This card is the dashboard admitting its own blind spots. Media attention is wildly uneven: the top five countries absorb roughly 30 % of all recorded event volume. If you don't measure that, every other number quietly inherits the bias.
| column | plain meaning |
|---|---|
| months | how long we've had data for this country — short history = shaky baselines |
| events / events/mo | raw attention: how much gets recorded about this country at all |
| share | this country's slice of global recorded events — the loudness ranking |
| fatal/event | fatalities per recorded event — the quiet-country warning: a high number means events only get recorded there when people die |
The one comparison worth internalising: the US logs ~0.01 fatalities per event (everything gets reported, however minor), Afghanistan ~3 (only catastrophe makes the record). Same planet, hundred-fold difference in what "an event" means.
What to do with it: before believing any cross-country comparison on this dashboard — including ours — check both countries here. A spike in a high-share country is probably just loudness; a small spike in a country with high fatal/event may be a big deal that barely made the record. Loud countries are judged against their own past for exactly this reason.
The honesty engine. Two sections:
Forward track record. Every month the system writes down its forecasts — "country X will/won't see instability in the next 1 / 3 / 6 months" — server-stamped before the outcome is knowable, impossible to edit afterwards, graded automatically once reality catches up. The columns:
| column | plain meaning |
|---|---|
| source | which instrument made the forecast — composite (the stress index) or disagreement (do countries telling the same story differently predict trouble?) |
| k | horizon: predicting 1, 3 or 6 months ahead |
| issued / graded / pending | how many forecasts made, how many reality has already marked, how many still waiting |
| pos rate | how often the bad outcome actually happened in the graded set |
| mean score | how worried the instrument claimed to be, on average |
| Brier | the grade: 0 = clairvoyant, 0.25 = coin flip, 1 = perfectly wrong. Lower is better. This one number is the difference between a forecasting system and a mood ring |
Baselines. The composite index versus deliberately dumb rivals —
"predict randomly", "predict yesterday's weather", "predict each country's
long-run average". The published result, stated plainly: the composite has
not yet beaten the dumb rivals (AUROC ≈ 0.5 = coin flip, on both the
ordinary evaluation and the pre-registered onset evaluation). That negative is published
on purpose — it is what makes every other number here credible, and the
per-indicator decomposition (make indicator-ranking) shows where the
recoverable signal lives for the next version.
The lead-time gate. A second published negative, and a sharper one. The
premise underneath this whole project is that physical sensors move before the
news does. That was tested directly: for M6.0+ earthquakes, sensor spikes do
not precede narrative spikes at a rate distinguishable from chance
(p ≥ 0.50), and the result holds across every threshold from 1.0 to 2.5 — so it
is not an artefact of where the line was drawn. Full working:
docs/backtest/f74fb156-report.md and docs/backtest/threshold-sensitivity.md.
Earthquakes may simply be the wrong anchor — a major quake is reported within minutes, so there may be no lead to find. Slow-onset hazards (drought, flood, sustained unrest) are where a sensor could plausibly lead coverage by days, and that is the open question rather than a settled failure.
What to do with it: treat the dashboard as monitoring, not prophecy. The stories card tells you what's happening and how corroborated it is — that part works today. The predictive claim is on trial in public: watch the Brier column accumulate; if the instruments are worth anything, it sinks below 0.25 and stays there. Until then, nobody here will pretend otherwise.
Stories = what happened, weighted by independent confirmation, machine-checked against sensors. Coverage = which countries this whole apparatus is blind to. Scoreboard = whether the forecasting ambition is earning its keep, graded in public, currently honest about not winning yet. Read them in that order.
The deep material — architecture, domains, pipeline, evaluation protocol.
The diagrams below describe the target production shape, not what runs today. The live system is a single box, Postgres-only: the Raspberry Pi 5 + RAID1, the Parquet cold-archive tier, Pushover alerting, Caddy/Tailscale are all planned, not yet wired. Retention prunes raw events at ~30 days and every derivation stays in Postgres — there is no hot/cold split yet.
Essentials only — the files you actually open. Two apps (Python backend + Next.js frontend) over local Postgres/Redis; all data sits in one folder.
OSINT/
├── app/ ← PYTHON BACKEND (ingest · score · serve)
│ ├── api.py FastAPI read-API: /events /scores /ingest-health /stream(SSE)
│ ├── celery_app.py Celery app instance (broker = Redis)
│ ├── tasks.py Celery tasks + beat schedule (cadence + 03:00 prune)
│ ├── fetcher_registry.py maps source name → fetcher
│ ├── persistence.py upsert events into Postgres (+ Redis "new rows" tick)
│ ├── events_bus.py Redis pub/sub channel powering the live SSE stream
│ ├── housekeeping.py retention policy (GDELT/news/hazard 30d, per RETENTION_* in .env)
│ ├── db.py / db_models.py SQLAlchemy engine/session + table definitions
│ ├── settings.py ALL config (reads .env): POSTGRES_*, OSINT_DATA_DIR, RETENTION_*
│ ├── models.py canonical Event/Score pydantic shapes
│ ├── watchdog.py ingest health monitor
│ ├── sources/ one fetcher per feed (gdelt, gdacs, nasa_firms, fred, abuse_ch…)
│ ├── cii/ Country Instability Index scoring
│ ├── composite/ composite-score aggregation/normalisation
│ └── enrichment/ country/city geocode · NER · sentiment (+ enrichment/data/ polygons)
│
├── osint-frontend/ ← NEXT.JS DASHBOARD (reads app/api.py)
│ ├── app/ routes: page.tsx (dashboard), layout.tsx, providers.tsx, api/
│ ├── lib/
│ │ ├── apiClient.ts ★ all backend calls (fetchEvents/Scores/IngestHealth, SSE url)
│ │ ├── queries.ts data hooks (windowing, filters, latest scores)
│ │ ├── realtime.ts EventSource SSE buffer + reconnect/poll fallback
│ │ └── types.ts EventRow / ScoreRow / IngestHealthRow types
│ ├── components/ panes: MapPane, DashboardSection, FilterRail, ui/
│ ├── stores/ zustand filter store
│ └── public/ static assets
│
├── data/ ← ALL LOCAL STORAGE (= $OSINT_DATA_DIR, gitignored)
│ ├── postgres/ Postgres data files (the actual DB)
│ └── redis/ Redis append-only file
├── backups/ ← snapshot.py dumps (gzipped CSV per table, gitignored)
│
├── migrations/ ← Alembic schema migrations (versions/ = each change)
├── scripts/ ← one-off tools: snapshot.py (backup) · prune_now.py · backfill_*.py · enrich_*.py
├── tests/ ← pytest suite (backend); frontend tests live in osint-frontend/__tests__ + lib/*.test.mts
│
├── docs/ ← architecture-spec.md · methodology.md · data-coverage.md · severity-grading.md · frontend/ · superpowers/(specs+plans)
│
├── docker-compose.yml ← Postgres + Redis services (bind-mount → $OSINT_DATA_DIR)
├── Makefile ← make data-size / data-prune / data-reset
├── alembic.ini ← migration config
├── pyproject.toml ← Python deps + build (requirements.txt mirrors runtime deps)
├── env.example ← copy → .env, then fill secrets
└── .env ← YOUR live config + secrets (gitignored — never commit)
Quick "where is…?"
- My config / secrets →
.env(template:env.example); read in code viaapp/settings.py. - The database itself →
data/postgres/(change location withOSINT_DATA_DIR). - What the dashboard fetches →
osint-frontend/lib/apiClient.ts↔ served byapp/api.py. - Add/adjust a data source →
app/sources/+ register inapp/fetcher_registry.py. - How long data is kept →
app/housekeeping.py(+RETENTION_*in.env). - A backup of old data →
backups/<timestamp>/.
The ASCII tree above shows the shape; this shows the route data takes. Every path is a link — click to open the file on GitHub.
Follow one row, source → screen (a GDELT event's whole life):
| # | Stage | File | What happens |
|---|---|---|---|
| 1 | fetch | app/sources/gdelt_fetcher.py |
HTTP GET the newest 15-min export |
| 2 | register | app/fetcher_registry.py |
name → fetcher lookup |
| 3 | schedule | app/tasks.py |
Celery beat fires it every 15 min |
| 4 | dedup + store | app/persistence.py |
upsert on (source, source_event_id) → events |
| 5 | normalise | app/composite/normalization.py |
rolling within-country z-score |
| 6 | score | app/composite/scoring.py |
weighted-z → sigmoid → scores |
| 7 | serve | app/api.py |
/scores, /events, SSE stream |
| 8 | fetch (UI) | osint-frontend/lib/apiClient.ts |
all backend calls in one file |
| 9 | render | osint-frontend/components/ |
map + cards |
Where each source lands:
| Source | Fetcher | Table / output |
|---|---|---|
| GDELT live | app/sources/gdelt_fetcher.py |
events (rolling ~30d) |
| GDELT history | app/composite/gdelt.py |
data/gdelt/ monthly JSON + gdelt_daily_volume |
| USGS / GDACS / FIRMS / EONET | app/sources/ |
events → footprint enrich in app/tasks.py |
| yfinance / FRED | app/sources/ |
events (market/macro, never pruned) |
| ACLED (labels) | scripts/ drop-folder |
labels table (ground truth, kept separate) |
| RSS ×44 | app/sources/rss_feeds.json |
events → stories |
Analytical subsystems — one folder each, formula inside:
| Concern | Folder | Formula file |
|---|---|---|
| Composite index | app/composite/ |
normalization.py · scoring.py |
| Corroboration | app/corroboration/ |
score.py |
| Divergence / disagreement | app/divergence/ |
config.py |
| CII | app/cii/ |
scoring.py |
| The brain (LLM) | app/brain/ |
gate.py · client.py |
| Prediction journal | app/journal/ |
emit.py |
| Retention | app/housekeeping.py |
— |
flowchart LR
subgraph SRC["Open data sources (free)"]
S1[yfinance + FRED]
S2[GDELT events]
S3[USGS + GDACS + FIRMS]
end
subgraph PI["Raspberry Pi 5 + 2x4TB btrfs RAID1"]
W[Celery workers<br/>fetch · dedup · normalise]
DB[(Postgres<br/>events + scores)]
AR[(Parquet archive<br/>cold storage)]
COMP[Composite worker<br/>JRC handbook<br/>method_version v1.0]
end
subgraph OUT["Outputs"]
DASH[Next.js dashboard<br/>MapLibre GL]
PHONE[Pushover<br/>phone alert]
EVAL[Evaluation report<br/>AUROC / AUPR / Brier]
end
S1 & S2 & S3 --> W
W --> DB
W --> AR
DB --> COMP
COMP --> DB
DB --> DASH
COMP --> PHONE
AR --> EVAL
Three sources in. One pipeline. Three outputs: a live dashboard you can pull up on your phone, an alert when a country crosses a threshold, and an evaluation report at the end.
| Question | Answer |
|---|---|
| What is it? | A small early-warning dashboard. It watches four kinds of open data — markets, geopolitical news events, natural hazards, and satellite-detected fire load — and combines them into a single number per country that goes up when things look stressed. |
| Why these three? | The design brief: "must not depend on a single data source." Three independent domains keep the score honest: if only one domain spikes, the composite stays calm. If multiple domains spike together, the composite goes red. |
| What is it for? | (a) The claim — test whether this multi-modal composite is better at flagging real instability events than watching one domain alone. (b) Personal — a self-hosted situational-awareness tool that keeps running indefinitely. |
| What is NOT it? | Not a prediction system. Not Palantir. Not Shadowbroker. Not finance-only. Does not claim to predict specific events. Does not use private intelligence feeds. |
The system tests a composite over three domains, not finance alone, not GDELT alone.
flowchart LR
subgraph A["Module A — Market signals"]
A1[yfinance<br/>equities, FX, vol]
A2[FRED<br/>CPI, unemployment, yields]
A3[FinBERT-on-news<br/>auxiliary signal]
end
subgraph B["Module B — Geopolitical events"]
B1[GDELT events + GKG<br/>deduplicated<br/>CAMEO-filtered<br/>Goldstein-weighted]
end
subgraph C["Module C — Hazards / disaster"]
C1[USGS Quake]
C2[GDACS multi-hazard alerts]
C3[NASA FIRMS fires]
end
subgraph D["Module D — Composite stress index"]
D1[JRC 10-step methodology<br/>normalise · weight · aggregate<br/>method_version v1.0]
end
A --> D
B --> D
C --> D
D --> E[Score per country, per month]
| Module | Domain | What goes in | Where it lives |
|---|---|---|---|
| A | Market / macro | yfinance, FRED, optional FinBERT-on-news | docs/architecture/01-overview.md |
| B | Geopolitical | GDELT v2 events + GKG | same |
| C | Hazard / earth | USGS Quake, GDACS, NASA FIRMS | same |
| D | Composite | JRC handbook 10-step methodology | docs/methodology.md |
| E | Evaluation | Pre-registered AUROC / AUPR / Brier vs ground truth | docs/methodology.md |
Layer 3 feeds (satellites, news RSS, aviation, maritime, weather, mesh) sit on the dashboard for situational awareness only. They do not enter the composite or the evaluation. See the feed taxonomy for the full list.
sequenceDiagram
autonumber
participant API as Source APIs<br/>(GDELT, yfinance, USGS, ...)
participant W as Celery worker<br/>(fast or slow queue)
participant R as Redis<br/>(queue + rate-limit bucket)
participant PG as Postgres<br/>events / scores / labels
participant PQ as Parquet archive<br/>cold storage
participant COMP as Composite worker
participant API2 as FastAPI read API
participant UI as Next.js + MapLibre dashboard
participant PUSH as Pushover
Note over W,R: Beat scheduler triggers fetcher
W->>R: take rate-limit token
W->>API: HTTP fetch
API-->>W: raw response
W->>PQ: write raw + parsed Parquet
W->>PG: INSERT INTO events ON CONFLICT DO NOTHING
Note over COMP: Every 1 hour
COMP->>PG: SELECT recent events per country
COMP->>COMP: normalise (z-score, rolling window)
COMP->>COMP: weight + aggregate (JRC handbook)
COMP->>PG: INSERT INTO scores (method_version='v1.0')
COMP->>PUSH: if score > threshold, alert (dedup'd)
UI->>API2: GET /api/scores?country=...
API2->>PG: SELECT FROM scores
API2-->>UI: JSON
UI->>UI: render MapLibre layer + time series
Plain version:
- A scheduler wakes up a worker (every 5 minutes for fast feeds, every 15 minutes for slow ones).
- Worker takes a rate-limit token from Redis so we never burn the daily allowance.
- Worker fetches from the source and writes the parsed events to Postgres. Duplicates are filtered by
(source, source_event_id). - Once an hour, the composite worker reads the last 90 days of events per country, normalises and weights them per the JRC handbook, and writes a score row with a
method_versiontag. - If the score crosses a threshold, Pushover gets called and your phone lights up.
- The dashboard pulls from Postgres via FastAPI and renders the country map plus per-country time series.
Retention prunes raw events at ~30 days overnight so the database stays small; every derivation (scores, stories, journal) is kept. The Parquet hot/cold split drawn above is planned, not yet built — today it is Postgres only.
The system is multi-modal, so the answer key is too. Five label codes, three domains:
| Code | Domain | What it means | Source |
|---|---|---|---|
| P1 | Geopolitical | Armed conflict onset | ACLED battle events with ≥10 fatalities |
| P2 | Geopolitical | Mass protest escalation | ACLED protest events with violent escalation in 7-day window |
| P3 | Geopolitical | State-based violence intensification | Month-over-month doubling of ACLED state-based fatalities |
| P4 | Market | Country-level market crisis | NBER recession; IMF currency-crisis entry; sovereign yield spike > 200bps; equity drawdown > 20%; VIX > 30 sustained |
| P5 | Hazard | Hazard-induced societal disruption | EM-DAT disaster with ≥100 deaths or ≥100k affected, or GDACS red-alert, with sustained composite stress in following 30 days |
The primary classification target is any-positive across P1-P5. Per-domain subtasks are reported as secondary. Full ground-truth definition: docs/methodology.md.
The labels live in their own database table, kept strictly separate from input events so the answer key is never accidentally treated as a feature.
flowchart TD
A[Pi runs ingestion<br/>2025-2026 live] -.demo only.-> Z[Dashboard]
B[Cloud backfill<br/>2015-2024 historical] --> C[Parquet archive]
L[Label backfill<br/>ACLED + NBER + IMF + EM-DAT] --> C
C --> SPLIT{Train / Val / Test split}
SPLIT -->|2015-2021| TRAIN[Train composite weights]
SPLIT -->|2022| VAL[Tune hyperparameters]
SPLIT -->|2023-2024| TEST[Held-out final evaluation]
TRAIN --> LOCK[Lock methodology v1.0]
LOCK --> RUN[Run 9 baselines<br/>B0..B8]
VAL --> RUN
RUN --> METRICS[AUROC · AUPR · Brier · lead-time]
TEST --> METRICS
METRICS --> REPORT[Evaluation report]
Nine baselines compete:
| ID | Baseline | What it is |
|---|---|---|
| B0 | Random | Sanity check, AUROC ≈ 0.5 |
| B1 | Persistence | "Same as last month" |
| B2 | Base rate | Country's historical positive rate |
| B3 | Geo only | Module B score alone |
| B4 | Market only | Module A score alone |
| B5 | Hazard only | Module C score alone |
| B6 | Composite (equal weights) | The headline claim |
| B7 | Composite (PCA weights) | Alternative weighting |
| B8 | Composite (geometric mean) | Less-compensatory aggregation |
For the composite to land its primary claim, B6 (or B7, or B8) must beat each of B3, B4, B5 on both AUROC and AUPR on the held-out test set. If it doesn't, the system says so honestly — pre-registered protocols make negative results respectable.
| Layer | Choice | Why |
|---|---|---|
| Hardware | Raspberry Pi 5 (8 GB) + 2x4TB USB3 HDDs in btrfs RAID1 | Low power, runs 24/7, RAID1 survives single-disk fail |
| OS | Raspberry Pi OS Lite 64-bit | Standard, well-supported |
| Reverse proxy / TLS | Caddy | Auto-TLS, simple config |
| VPN access | Tailscale | Reach the Pi from anywhere with no port-forwarding |
| Queue | Celery + Redis | Worker isolation, retry, rate limiting per source |
| Hot store | Postgres 16 | Indexed queries for dashboard + composite |
| Cold archive | (planned) Parquet on btrfs (Hive-partitioned) | Not built yet — today raw events are pruned by retention and all derivations stay in Postgres |
| Backup | restic → Backblaze B2 or Cloudflare R2 | Encrypted off-site |
| API | FastAPI | Async Python, fits the worker stack |
| Frontend | Next.js + MapLibre GL | Vector map tiles, off-Pi build |
| Alerting | Pushover REST | Cheap, reliable, phone-native |
| Schema migrations | Alembic | Standard for SQLAlchemy / Postgres |
Full reasoning: docs/architecture/ sections 01-07.
Sits on the dashboard for situational awareness only — not in the composite, not in the evaluation.
Live as of the latest source-expansion batch — 58 collectors (14 named fetchers + 44 RSS feeds; see §3.1 for the authoritative cadence table):
- News (RSS, 44 feeds) — BBC World, BBC UK, Reuters/Yahoo, Dawn, Guardian, Geo English, Al Jazeera, CNN, NYT, France 24, DW, NHK, RT, TASS, Times of India, The Hindu, Tribune PK, CBC, ABC AU, RNZ, Straits Times, Jerusalem Post, Haaretz, Arab News, Kyiv Independent. JSON-registry driven (#158).
- Aviation — OpenSky public ADS-B (#161). 2 min cadence, every aircraft broadcasting ADS-B in the last 10 s.
- Cyber-threat — abuse.ch URLhaus malware URLs + Feodo Tracker botnet C2 IPs (#163). 15 min cadence each.
- Prediction markets — Polymarket public Gamma API (#165). 30 min cadence. Severity reads as "tail-event awareness" (peaks at p = 0.5).
- Crime — UK Police data.police.uk monthly snapshots.
- Hazard / geo / market (Layer 1+2) — yfinance, FRED, GDELT, USGS, GDACS, FIRMS, EONET.
Every news row gets the following stamped on payload at fetch time. See docs/architecture/ENRICHMENT-METHODOLOGY.md:
- VADER sentiment v1.0 (
compound ∈ [-1, 1]+ label). - spaCy NER v1.0 (optional dep) —
entities = [{text, label}, …]. - News-scope classifier (
local | world | unknown) — distinguishes a Dawn-published US story from a Karachi street-level event. - Offline city pinpoint (Natural Earth 10m, 7,484 cities) — drives map lat/lon.
- Bounded named-place verification — exact Wikidata point for one unambiguous building, street, or site; persistent positive and negative cache.
- Image URL (media:thumbnail / media:content / enclosure / first
<img>fallback). - News-scope-aware impact ranking (NIP §3 formula) —
0.30 |sentiment| + 0.25 cluster + 0.25 sourceWeight + 0.20 recency.
CII v1.1 country-instability scoring runs hourly across the 31 Tier-1 countries. Methodology in docs/architecture/CII-METHODOLOGY.md.
docs/storage.md— local storage & data:OSINT_DATA_DIR, where the live DB lives vs backups vs the config pointer, retention, move/back-up/restore/wipedocs/methodology.md- Part A — pre-registered evaluation protocol (ground truth, splits, baselines, metrics, sensitivity, reporting checklist)
- Part B — literature baseline (citations, reading priority, BibTeX snippets)
docs/project-direction.md— what the project is, who it serves, why it matters, and the long-term product / research pathdocs/analytical-agenda.md— the WS-A…G workstreams: what we actually do with the data (quantify, validate, predict)- The three pre-registered evaluations —
docs/onset-eval.md,docs/within-country-eval.md,docs/disagreement-exam.md, each frozen before its first result docs/data-coverage.md·docs/acled-non-api-collection.md·docs/security.mddocs/backtest/— lead-time gate reports ·docs/audits/— clustering-threshold hand-check ·docs/frontend/— dashboard design notesdocs/architecture/— seven-section build spec, all sections drafted:- 01 overview · 02 storage · 03 ingestion · 04 schema · 05 originality · 06 validation · 07 risks
- CII methodology — per-country baseline + 4-component event blend (cii.v1.1, 31 Tier-1 countries)
- Enrichment methodology — VADER sentiment + spaCy NER + city + news-scope classifier + impact formula
- Architectural inspiration only: Shadowbroker, WorldMonitor
- Methodology lineage (cited): OECD/JRC Composite Indicator Handbook (Nardo et al., 2008), ViEWS (Hegre et al., 2019), CEWS field review (Davies et al., 2023), FSI methodology (Fund for Peace), GDELT validity critiques (Wang 2025, Wallace 2014, Öberg & Yilmaz 2025), FinBERT honesty (Yang et al., 2024). Full list with reading priority in
docs/methodology.md.
Every formula below is the one actually in the code, with its file. The first six are the ones that matter; the rest are standard definitions.
1. Within-country rolling z-score — app/composite/normalization.py
Each domain's raw severity is standardised against that country's own trailing history:
A minimum history (≥3 monthly points) is required; cold starts and zero-variance windows emit
2. Composite aggregation — app/composite/scoring.py
Weighted sum of domain z-scores, squashed to [0,1] by a logistic:
Missing domains contribute
3. Goldstein → severity — app/composite/gdelt.py
GDELT's conflict–cooperation scale [−10, +10] inverted to a [0,1] severity (1 = maximal conflict), zero tuning knobs:
4. Corroboration confidence — app/corroboration/score.py
Each additional independent owner halves the remaining doubt; a sensor confirmation halves it once more:
5. Divergence spike + lead — app/divergence/config.py (div.v4)
A physical and a narrative daily series are each z-scored over a 28-day rolling window; a side "spikes" when it crosses
6. CII v1.1 — app/cii/scoring.py
-
Brier
$=\frac{1}{N}\sum_i (p_i - o_i)^2$ — 0 clairvoyant, 0.25 coin flip, 1 perfectly wrong. The journal's headline grade. - AUROC — P(model ranks a random positive above a random negative); 0.5 = chance. A pooled AUROC is rewarded for separating high- from low-base-rate countries, which is where the illusory 0.93 came from.
- AUPR — area under precision–recall; the honest metric under class imbalance (few positive months).
- Base rate — share of positive months; the score a "predict the mean" baseline earns.
-
Impact ranking (NIP §3) —
$0.30,|\text{sentiment}| + 0.25,\text{cluster} + 0.25,\text{sourceWeight} + 0.20,\text{recency}$ .
A candid read of where the project stands — logged here so it stays with the code.
| Helpful | Harmful | |
|---|---|---|
| Internal | Strengths (below) | Weaknesses (below) |
| External | Opportunities (below) | Threats (below) |
Strengths
- A pre-registration + server-stamp honesty machine that structurally cannot lie to itself — five negatives published, none buried.
- Reproducible, local-first, zero-cloud; idempotent backfills; every method version-stamped and never edited in place.
- Genuine multi-modal ingestion (58 collectors) with structural dedup and published coverage bias.
Weaknesses
- The headline claim failed: the composite does not beat the dumb baselines on any of the three pre-registered exams.
- Single developer, heavily AI-assisted → depth-of-understanding of the internals is the real risk (this document and §Core equations are the mitigation).
- The live composite is currently degenerate (0.5) from the retention-vs-z-score mismatch (#586); ACLED ground-truth labels are manually maintained.
Opportunities
- Slow-onset hazards (drought, flood, sustained unrest) are the untested anchor where a sensor could plausibly lead coverage — the open question, not a settled failure.
- Productization path (newsletter → API → app, #400); the corroboration + coverage engine works today and has standalone value independent of the composite claim.
- If any signal is recoverable it is in the hazard domain — the strongest single indicator (|hazard z| ≈ 0.59 full-panel), which even beats the composite that contains it (#573). The open task is making it survive the fair onset evaluation, where it currently fades to ~0.53.
Threats
- The sharpest challenges to answer: the within-country construction, the 0.5 degeneracy, and "how much was built by hand vs the AI."
- Upstream drift: GDELT gaps, ACLED API rejection, RSS format changes, FIRMS confidence ≠ intensity.
- The retention window vs evaluation needs (the #586 class of bug) can silently flatten a signal before it is measured.
Rendered assets live in docs/supplementary/ as SVG (scalable, editable text) and PNG (2× print resolution). Regenerate the PNGs with rsvg-convert -z 2 <file>.svg -o <file>.png.
The result in one picture — the composite is a coin flip in all three exams; the impressive 0.93 was only the base rate.
Why "signal before narrative" is the whole bet — sensors are cheap and high-volume, narrative is scarce and expensive; the question is whether the first moves before the second.
Why it runs on a Raspberry Pi — raw events are pruned at ~30 days, the small permanent derivations are kept forever.
PolyForm Noncommercial 1.0.0 — the full text is in LICENSE.
This is source-available, not open source. The distinction is not pedantry: every OSI-approved licence permits commercial use, so calling this open source would tell you that you have a right you do not have.
You may use it, run it, study it, fork it, modify it and share your changes, for any noncommercial purpose — personal projects, study and research, and use by charities, educational institutions, public research bodies and government. You may not sell it or use it commercially.
Two reasons, and the second is the real one:
- It is under development. Its outputs have been wrong before. The composite is a coin flip in all three pre-registered exams, and that is written down a few sections above rather than hidden. Nothing here is fit to sell.
- It ingests third-party feeds. Several are free for noncommercial or research use and require a separate agreement for anything else. Those terms are not the maintainer's to hand on, so they are not handed on.
NOTICE.md lists every feed and where its terms live. If you
deploy this, you fetch that data under your own credentials and your own
agreements with those providers. Read them.
The bundled gazetteer files under app/enrichment/data/ are Natural Earth,
which is public domain.