Ratatoskr Threads

ratatoskr-threads is the Threads account and capture bounded context for Ratatoskr. It combines official user-authorized account capabilities with explicit capture of public posts, official public representations, and versioned Data Export imports.

Status: implementation plan items 1 through 9 are complete. Item 9 adds explicit media byte-retention decisions, complete owner-deletion enumeration and downstream removal facts, finite public re-resolution guards, and parser-version reprocessing with read-only dry-run fidelity.

Important

Ratatoskr is in development. No database holds data that has to survive a schema change. While this status holds, these two rules replace what the documents below plan:

• the API and the database keep their first version. There is no v2 and no later major version.
• the database has no migrations. One schema definition exists, and a schema change edits it in place.

Only the repository owner changes this status.

Role in Ratatoskr

Threads does not currently provide Ratatoskr with an authoritative API for enumerating a personal user's complete native Saved list. This repository therefore keeps official account data and explicit Ratatoskr captures as separate ingestion lanes.

Official account lane

Where supported by the current Threads API and granted permissions, the connector may handle:

• Threads user identity;
• the user's own posts and replies;
• officially available interaction and insights data;
• publishing only through separately granted write authority;
• token expiry, refresh, reauthorization, and audit.

Explicit capture lane

A public Threads post is archived through a deliberate action:

Threads
  -> Share
  -> Ratatoskr mobile Share Extension
  -> canonical permalink
  -> ratatoskr-threads
  -> official public representation / oEmbed
  -> normalized SocialSource

Desktop capture follows the same semantics through ratatoskr-browser-extension.

Core responsibilities

• official Threads account connection and capability detection;
• encrypted token ownership;
• synchronization of officially available own-account content;
• explicit public-post captures;
• canonical URL and short-link resolution;
• public oEmbed normalization;
• user notes and capture provenance;
• versioned Threads Data Export imports;
• media and attachment references;
• upstream availability observations;
• normalized social-source events;
• preservation of raw provider records.

The service does not use user passwords, server-side browser cookies, hidden bookmark APIs, or stealth automation.

Authority and provenance

Every source records how it was acquired and what it proves:

acquisition = OfficialApi | ShareExtension | BrowserExtension | TelegramCapture | PublicResolution | DataExport | LegacyImport
saved_authority = ExplicitUserCapture | AuthoritativePlatformState | ExportObservation | LegacyObservation

The saved-authority vocabulary equals the published ratatoskr-social-contracts grammar value for value; TelegramCapture is the documented Threads extension (see docs/CAPABILITY_MATRIX.md). What each acquisition mode may prove is fixed by its authority ceiling in the capability matrix.

A typical captured post is represented as:

platform = Threads
acquisition = ShareExtension
saved_authority = ExplicitUserCapture
native_saved_state = Unknown

This means Ratatoskr can reliably state that the user captured the post locally at a known time. It cannot claim that the post remains in the native Threads Saved list unless an official authoritative surface later provides that evidence.

Owned data model

The service owns a threads_archive.* PostgreSQL schema, created from schema.sql and applied at startup. The first version declares:

accounts
credentials
posts
post_revisions
post_relations
media
captures
capture_resolutions
export_runs
export_records
raw_objects
tombstones
outbox_events
inbox_events
social_sources
social_source_revisions
social_analysis_links
deletion_operations
deletion_effects
local_source_removals
blob_deletion_tasks
reresolution_runs
reresolution_items
export_reprocessing_runs
export_reprocessing_items

Rows are written by implemented and later capabilities; public-resolution rows retain immutable raw oEmbed bytes before parser-versioned post revisions and first-class directed relation rows. The schema vocabulary is already enforced: acquisition methods (official_api | share_extension | browser_extension | telegram_capture | public_resolution | data_export | legacy_import) and saved authorities (explicit_user_capture | export_observation | authoritative_platform_state | legacy_observation) are closed CHECK constraints aligned with the published social-contract grammar, so a capture cannot be stored as native Saved-list state. Large export archives, media, raw API/oEmbed responses, and unknown provider records are stored in the content-addressed BlobStore.

Lifecycle operations

Provider-media bytes remain metadata-only unless acquisition, rights, MIME/kind, URL lifetime, object size, owner storage, and explicit-action guards all admit an immutable fetch lease. Expiry or owner deletion schedules digest-verified BlobStore cleanup only after a database-wide reference check; failure stays durable and retryable.

Capture and connection deletion are owner-bound, replay-safe operations. Preview and apply share a complete table/blob classification and content-free counts. The final local holding publishes one social.source.removed.v1; Knowledge owns deletion of derived analysis and late completion cannot resurrect the local source.

reprocess-export dry-run|apply reinterprets a retained export under an explicitly registered parser version. Dry-run is read-only and produces the same ordered report and fingerprints as apply. This is operational parser reprocessing, not database migration tooling; development databases still initialize only from the current schema.sql. Normal tests use synthetic/redacted exports, so compatibility with a real protected export remains unverified until an authorized fixture is exercised outside CI.

Capture flow

Clients submit a capture through Platform with an idempotency key. Threads then:

1. validates and canonicalizes the URL;
2. resolves supported short-link forms;
3. deduplicates the explicit capture;
4. retrieves the supported public representation;
5. stores raw evidence and normalized metadata;
6. records warnings or unavailable state without losing the user's note;
7. transactionally appends a contract-conformant social.source.captured.v1 or social.source.updated.v1 fact;
8. lets Knowledge perform optional analysis asynchronously.

A replay of the same capture converges on the same local source record while preserving new notes or collection links according to their owning context.

Browser captures routed by Platform

A browser-extension capture arrives as a social.capture.requested.v1 command on cmd.threads.capture.requested.v1. The report shapes, the retry schedule and the exactly-once terminal report are defined once for X, Instagram and Threads in the workspace changeset XR-021 (CONTRACTS.md section S10, CD1, CD2, CD5, CD7); this is the Threads side of it.

1. Intake commits the inbox marker, the capture (with the Platform operation id, the command id, zero resolution attempts and next_resolution_at = now()) and a capture_queued operation report in one transaction, then acknowledges the command.
2. The capture resolver, its own task beside the NATS worker, claims due captures with FOR UPDATE SKIP LOCKED under a 120 second lease and resolves each through the credential-free oEmbed client.
3. A deleted post (HTTP 404 or 410) writes a tombstone, marks the capture unavailable and reports failed with social.source.deleted, not retryable. A forbidden post (403) or an unusable response reports failed with social.source.unavailable, not retryable.
4. A transient failure (429, any 5xx, a network error, an oversized response) is stored as a resolver_failed resolution row and retried after 30 seconds, 2 minutes, 8 minutes and 30 minutes. The fifth failure reports failed with social.source.unavailable, retryable, and leaves the capture accepted: missing output is never deletion evidence.
5. A resolved post appends social.source.captured.v1 (or updated when its digest changed) and reports succeeded with one social.post result naming the stored social_source:<id>.
6. The report and the operation_reported_at guard that makes it exactly-once commit in one transaction, so a crash between recording an observation and reporting it only reports on the next pass; a capture that is already terminal is never fetched again.
7. A Threads command that names its operation and owner but that this service cannot map (a permalink outside the Threads host policy, widened acquisition or saved authority, an unusable capture time or digest) is answered, not dropped: the inbox marker (rejected) and one terminal failed social.source.unavailable report, not retryable, commit in one transaction filed under the operation (outbox_events.aggregate_type = 'operation'), then the command is acknowledged and no capture exists. Input nobody is waiting on (undecodable, another provider, no tenant) is terminated at the broker. A failed inbox write is a negative acknowledgement with a 2 second delay and the worker keeps running.
8. A resolved capture whose source has since vanished (an owner deletion) is reported failed social.source.unavailable, not retryable, instead of stopping the resolver.

The outbox relay publishes platform.operation.reported.v1, social.source.captured.v1, social.source.updated.v1 and social.source.removed.v1 to evt. plus the event type. outbox_events.event_type carries a CHECK listing exactly those four, a failed publication records attempt_count, last_error and a backoff next_attempt_at so one failing row never starves the rows behind it, and an acknowledgement timeout is indistinguishable from a permission denial: check the NATS server log for a Publish Violation.

When the NATS worker, the consumer of Knowledge's deletion acknowledgements or the resolver returns before an orderly shutdown the process reports /health/ready as 503 (bus, deletion_acks or resolver check) and exits non-zero.

Configuration, all under RATATOSKR__PUBLIC_RESOLUTION__: RAW_OBJECT_ROOT (required, absolute directory), OEMBED_ENDPOINT (default https://graph.threads.com/oembed, https only), MAX_ATTEMPTS (5), BATCH_SIZE (8) and POLL_INTERVAL_MS (2000). The oEmbed client carries no credential. If Meta requires an app token for the Threads oEmbed endpoint, the resolver reports every capture as unavailable until the token support that this change leaves out is added.

Normalized post representation

The common social projection may include:

• local source ID and provider external ID;
• canonical URL;
• author identity and handle;
• post text;
• publication and edit timestamps;
• reply, quote, and repost relationships where exposed;
• media metadata and blob references where permitted;
• capture time and client source;
• user note and local linkage metadata;
• raw provider record reference;
• upstream availability state.

Unknown fields remain available in raw records so parser upgrades do not require reacquiring the post.

Unavailable and private content

Ratatoskr does not bypass account privacy, login requirements, regional restrictions, removal, or other provider controls.

When a post cannot be resolved, the service may retain:

• canonical URL;
• capture timestamp and source client;
• user note and local collection references;
• selected text explicitly supplied by the user;
• an optional user-uploaded screenshot or file with separate provenance;
• the observed unavailability reason and time.

A user upload is never relabeled as an official provider response.

Data Export imports

Threads exports are immutable, versioned observations rather than one permanent schema. An authenticated caller supplies its user_ref; receipt streaming hashes and stores the exact bytes before any inspection, bounded to 64 MiB. The run is unique per owner and archive digest, so a retry returns the existing receipt rather than replacing raw bytes or creating a second import.

The current parser accepts the synthetic/redacted threads-export-v1 layout: a safe ZIP with threads_export.json, sorted posts and directed relations. Inspection rejects path traversal, absolute/backslash paths, more than 1,000 entries, path depth over 16, cumulative compressed bytes over 64 MiB, declared decompressed bytes over 256 MiB, and a ratio above 100:1. Optional extraction uses a caller-owned isolated directory and never writes an archive-supplied path outside it.

Each import reads the retained digest-verified blob, persists parser/version/run evidence, creates normalized posts and relations transactionally, and publishes Data Export observations with export_observation authority. Unknown ZIP sections remain linked to the immutable archive in export_records and make the run completed_with_warnings; they are never guessed as a newer schema or native Saved state.

The report counts distinct export identities, captured identities that match, export-only identities, capture-only comparable identities, and captures lacking a stable identity. It is owner-scoped evidence of overlap, not a deletion signal or a claim that the archive is a complete native Saved list. A refused archive remains retained and its run is marked failed without a normalized projection.

Official account connection

Credential requirements:

• OAuth state and callback-user binding;
• minimum required scopes;
• encrypted token storage and rotation;
• expiry and refresh tracking;
• explicit capability detection;
• separate consent for publishing or other provider mutations;
• no tokens in events, logs, traces, or public client responses.

Account synchronization and explicit captures are related by provider identity but remain distinct sources of authority.

Linked content

External URLs from a post are delegated to ratatoskr-extractor. ratatoskr-knowledge may then create a composite analysis that separates:

• what the Threads post states;
• what the linked document contains;
• how the sources relate;
• which claims map to which provenance.

The Threads service itself does not run LLM analysis or own the search index.

Commands and events

Expected contracts include:

threads.account.connected.v1
threads.account.reauth_required.v1
threads.account.sync_requested.v1
threads.post.upserted.v1
threads.capture.requested.v1
threads.capture.resolved.v1
threads.capture.unavailable.v1
threads.export.ingest_requested.v1
threads.export.ingested.v1
social.source.captured.v1
social.source.updated.v1
social.source.removed.v1 # reserved for a future local-library deletion, never a provider tombstone
knowledge.analysis.completed.v1

Handlers are idempotent under at-least-once delivery. Capture, import, and account-sync results retain separate provenance.

Platform browser-capture consumer

Threads requires a NATS endpoint at startup and owns the durable pull consumer threads_browser_capture on ratatoskr_commands, filtered to cmd.threads.capture.requested.v1. Platform preprovisions that consumer; the Threads identity can only open it and cannot create broker topology. It accepts only a canonical SocialCaptureRequested envelope whose provider is threads, acquisition is browser_extension, and saved authority is explicit_user_capture. The consumer writes its inbox marker, capture record, and a queued platform.operation.reported.v1 outbox fact in one transaction; a duplicate command ID has no additional effect.

The consumer does not manufacture terminal unavailable, deleted, or linked- article partial outcomes. Those require a supported public-resolution or Extractor observation and remain the responsibility of those authoritative pipelines.

Knowledge's acknowledgement of a removal

The ratatoskr_events stream drops its oldest message after 7 days or at 1 GiB, so a social.source.removed.v1 fact that Knowledge has not consumed in time would be lost. Threads therefore keeps every removal until Knowledge answers with knowledge.deletion.acknowledged.v1 and publishes the stored bytes again, with the same event_id as Nats-Msg-Id, while it is silent: the first copy follows RATATOSKR__TOMBSTONE__RESEND_AFTER_SECONDS (default 600, accepted 30 to 86400) after the first publication, each later wait doubles up to RATATOSKR__TOMBSTONE__RESEND_CAP_SECONDS (default 21600, at least the delay, at most 604800) and the resend never stops by count. The events stream collapses a repeated Nats-Msg-Id inside its 120 second duplicate window, which is why the default is 600 and the minimum of 30 exists for tests; a native smoke that wants to see a second copy in the stream must set at least 150.

The service owns the Edge-provisioned durable ratatoskr_threads_deletion_acks on ratatoskr_events (filter evt.knowledge.deletion.acknowledged.v1, explicit acknowledgements, 30 second ack wait). It verifies the durable at startup and exits non-zero when it is missing or differs, and never creates one. A valid acknowledgement from ratatoskr-knowledge of family social sets outbox_events.acknowledged_at and ack_outcome on the removal with that event_id and tenant, which ends the resend. refused also ends it, raises an ERROR log line naming ids and the refusal class only and increments threads_tombstone_refused_total: the data stays in Knowledge and an operator must look. An acknowledgement that names no row of this service, another tenant's row or another family is acknowledged and ignored; a message that is not a valid acknowledgement from Knowledge is terminated.

threads_unacknowledged_tombstones and threads_oldest_unacknowledged_tombstone_age_seconds are on the operator listener. Readiness does not depend on them: a Knowledge outage must not make Threads unready. A removal that is not acknowledged is never deleted by privacy deletion.

Deployment

The host runs one PostgreSQL database, ratatoskr, and every service connects to it as its own role. Provision this service in this order: Platform's 01-database-and-roles.sql, the first Edge start, Platform's 02-grants.sql, then deploy/postgres/role.sql of this repository as a PostgreSQL administrator (it creates ratatoskr_threads without a password, with connect and create on ratatoskr; set the password with \password ratatoskr_threads). The service creates its threads_archive schema from schema.sql at start. Raw evidence is durable state, so its directory lives on the NVMe volume rather than the boot device: create it with install -d -o ratatoskr-threads -m 0750 /mnt/nvme/ratatoskr/blobs/ratatoskr-threads before the first start. Then place the nkey seed at /etc/ratatoskr/threads.nkey, /etc/ratatoskr/threads.conf from deploy/systemd/threads.conf.example, and the public key of the Threads stanza (deploy/nats/identity.conf is this service's copy of it) in the NATS authorization.users list. The stanza lets the identity subscribe only to _INBOX_threads.>, so NatsConnection sets the inbox prefix _INBOX_threads on both of its constructors; a build without the prefix cannot complete a JetStream request under that ACL, so reload NATS and deploy the services of one snapshot together. The development compose.yaml keeps its own database, threads, as a laptop convenience.

Security invariants

1. No Threads password or user browser cookie is collected.
2. Official OAuth and supported public representations are the primary provider surfaces.
3. A capture requires an explicit user action or user-provided export.
4. Private-content controls are never bypassed.
5. Write scopes require a distinct consent flow.
6. Unknown export records are retained losslessly.
7. Absence from an export does not delete a local capture.
8. User-uploaded artifacts have separate provenance.
9. Other services never receive Threads credentials.

Observability

Core metrics include:

threads_capture_duration
threads_capture_resolved
threads_capture_unavailable
threads_oembed_failures
threads_account_sync_duration
threads_rate_limit_waits
threads_export_import_duration
threads_export_unknown_records
threads_export_completeness
threads_reauth_required

Every operation records acquisition method, authority, resolver/parser version, warnings, and correlation identifiers.

Non-goals

• Automatic authoritative mirroring of native Saved items without an official API.
• Server-side login or stealth browser scraping.
• Bypassing private or unavailable content restrictions.
• LLM analysis or search ownership.
• Treating local captures as provider-native saved state.
• Writing Ratatoskr tags and collections back to Threads.
• Claiming export completeness without validated evidence.

Initial milestones

1. Define account, post, capture, export, and provenance schemas.
2. Implement URL recognition and canonicalization.
3. Add explicit capture and public oEmbed resolution.
4. Publish normalized social-source events.
5. Integrate mobile Share Extensions and browser extension.
6. Add encrypted official OAuth and capability discovery.
7. Add official OAuth and own-account synchronization.
8. Add safe versioned Data Export imports. ✓
9. Integrate linked documents with Extractor and analysis with Knowledge.

Workspace integration

ratatoskr-workspace pins this service (workspace.toml entry threads in the social profile, with the commit in workspace.lock) and its harness checks it against the rest of the fleet: the operator listener 9084 in the port allocation table, the Threads stanza and the Edge-provisioned durable threads_browser_capture in the bus ACL and durable checks. No composed Compose profile starts this service, and no cross-service run covering it has been recorded: the social-capture flow of the native smoke of changeset XR-021 is not reported as run, so the hops from the browser extension through Platform to this service, and from its facts to Knowledge, rest on this repository's own broker and database tests. Those include the authorized-broker test crates/threads-archive/tests/nats_inbox_prefix.rs, which runs the shipped deploy/nats/identity.conf stanza on a private broker. The official OAuth, own-account synchronization, Data Export import and publishing capabilities are covered only by recorded provider fixtures, mock servers and synthetic archives, and the public oEmbed success path is live-only because the endpoint is allow-listed to Meta hosts.

Project status

Implementation plan items 1 through 8 exist: the service owns schema/provenance, explicit capture and public resolution, social-source publication, encrypted official OAuth capability discovery, checkpointed official own-post/reply observations, and safe raw-first Data Export reconciliation. Media policy and provider writes remain planned.