Skip to content

LIVE_IMPLEMENTATION.md

Latest commit

 

History

History
126 lines (90 loc) · 29.2 KB

LIVE_IMPLEMENTATION.md

File metadata and controls

126 lines (90 loc) · 29.2 KB
Raw

RadSysX Live assistant implementation contract

Implementation started 2026-09-22. This ledger is subordinate to Wayfinder map #75 and prototype #76. Runtime evidence is recorded separately from intended behavior.

Fixed integration contract

  • Providers: Gemini Developer API gemini-3.8-live-extended-thinking, LOW thinking, google-genai 2.24.0, v1beta; OpenAI GA Realtime gpt-realtime-2.1-mini, low reasoning, marin voice, websockets 16.1.1. Both keys are backend-held. Research workers remain gemini-3.8-flash.
  • Existing /api/ai/sidebar/capabilities returns availability (configured, unavailable, disabled), modelId, reason, defaultProviderId and a typed provider catalog with independent readiness and fixed audio rates. Legacy top-level fields describe default Gemini; the catalog controls provider selection. Availability is resolved for the authenticated user's saved credentials, then deployment configuration.
  • API-key settings: GET /api/ai/sidebar/credentials returns {storageAvailable, providers: [{id, label, configured, source, environmentConfigured}]}. source is saved, environment or none. PUT /credentials/{providerId} accepts write-only {apiKey}; DELETE removes the personal key. All responses are status only with no-store caching; writes require signed-cookie identity, ai.run and an allowed Origin. Never return a key or fragment. Changes terminate the owner's live sessions/jobs before credentials change. A missing/corrupt personal credential fails closed; removal deliberately restores deployment fallback when configured.
  • POST /api/ai/sidebar/sessions: {providerId?: "gemini" | "openai", viewerContext, attestation?: "synthetic" | "deidentified"}. Context includes targetId (opaque context identity), optional backend study IDs, captureTarget: "viewer", and state (structured non-identifying OHIF state). Returns providerId, exact modelId, inputSampleRate, outputSampleRate, contextVersion, expiresAt, and liveUrl with the existing session fields. Provider/model/rates are immutable. status starts allocated or unavailable; ready requires acknowledged setup and successful initial context delivery.
  • GET /api/ai/sidebar/sessions returns {sessions: [...]}. GET /api/ai/sidebar/sessions/{id} returns {session, events, tools}. DELETE on that path clears history and closes the session.
  • POST /api/ai/sidebar/sessions/{id}/context: {contextVersion, viewerContext, attestation?}; returns session. A target change invalidates prior attestation and pending work. Version is server-incremented.
  • POST /api/ai/sidebar/sessions/{id}/close closes capture/provider/jobs. POST /api/ai/sidebar/sessions/{id}/tools/{toolId}/decision: {contextVersion, approved}. POST .../tools/{toolId}/cancel cancels work.
  • WebSocket /api/ai/sidebar/sessions/{id}/live: signed session cookie and allowed Origin required. PCM16 mono input is 16 kHz for Gemini or 24 kHz for OpenAI; output is 24 kHz for both. Client JSON is {kind, contextVersion, ...}: text (text), screen (data base64 JPEG, mimeType), audio_end, action_result (toolCallId, status, result), playback_stop (itemId, contentIndex, audioEndMs), ping.
  • Server JSON always carries {kind, sessionId, sequence, contextVersion, interactionId, ...}. Kinds: session (status, message?), transcript (role, text, finished, turnId), interaction (status IN_PROGRESS/IDLE), interrupted, tool (toolCallId, name, args, status, requiresApproval, result?), viewer_action (toolCallId, name, args), citations (sources, suggestionsHtml?), error (code, message), pong.
  • Media and private reasoning never enter the durable event journal. The client never sends provider configuration, credentials, actor identity, or arbitrary execution code.
  • OpenAI audio uses a transient audio_chunk marker (itemId, contentIndex) immediately before each binary frame on the same socket. Playback stop receipts use heard duration, not received byte count. Neither marker nor receipt is retained in history.
  • Image forwarding requires screen_sharing (active boolean) on the same context before screen frames. The backend emits transient screen_status (active, frameReceived) after a sharing change or successful frame forwarding. The model receives explicit selected-viewport image availability; a sharing lease alone is not evidence that pixels arrived. Stop and reconnect invalidate prior image availability.

Action names

Read: viewer_get_state. Reversible renderer actions: viewer_set_window_level (preset or windowWidth/windowCenter), viewer_set_layout (rows/columns), viewer_open_series (displaySetId), viewer_jump_to_slice (index), viewer_set_view (zoom/panX/panY/rotation/invert/flipHorizontal/flipVertical/reset), viewer_set_tool (tool), viewer_measurement (operation create/update/delete/jump, measurementId?, label?, points?, type?), viewer_segmentation (operation select/visibility, segmentationId, visible?), viewer_undo, viewer_redo, report_draft (findings, impression). Renderer actions accept optional viewportId. viewer_open_worklist and backend study_open (studyId) navigate through supported paths. Backend report_save (findings, impression) requires reviewed approval and a backend-resolved study. Research: research_run (query), research_cancel (toolCallId). Deletion requires approval. No new model-generated masks or direct browser STOW.

Ownership during implementation

  • Root: backend session/store/provider/broker/routes, Python contracts, shared TS contracts/client, root/roadmap documentation, integration and review.
  • Viewer agent: viewer runtime controller, audio, OHIF adapter, sidebar, viewer build/typecheck/tests/docs.
  • Desktop agent: Electron capture/permissions, secret isolation, bootstrap/doctor, desktop smoke/tests/docs.
  • Research agent: isolated research worker/supervisor/tests and AI dependency manifest.

Initial Gemini implementation validation

  • Before changes: desktop doctor passed; 53 clinical/security tests passed.
  • Final backend checks: 144 passed across test_ai_live.py, test_ai_research.py, test_clinical_platform.py, test_security_regressions.py, and test_biomedparse_demo.py. Deprecation warnings remain for existing datetime, FastAPI startup hooks, and test dependencies.
  • Viewer: 14 runtime tests passed, strict TypeScript checks cover the new runtime and structural compatibility with the shared package; full viewer and frontend builds passed. FHIR bridge and security dependency compatibility checks passed.
  • Desktop: 16 focused tests passed, including every pinned dependency missing/mismatch detection. Doctor and bootstrap enforce the seven AI manifest pins plus clinical HTTPX/Pydantic versions. The user-facing npm run desktop:smoke:launch reached normal backend service readiness and clean shutdown.
  • Electron: synthetic Live bridge and actual OHIF/sidebar checks passed. Two delayed research tools ran concurrently while 4,800 audio bytes arrived; 14,400 total audio bytes, final IDLE, two source cards and one viewport mutation despite a repeated tool ID were observed. Capture produced an aspect-preserving 768×512 JPEG and rejected frames after stop; microphone-only permission allowed a fake audio track and denied camera access.
  • Actual AudioWorklet path: 171 synthetic microphone frames / 109,440 bytes, 640 bytes per frame, reached the backend fixture as mono audio/pcm;rate=16000. Input was absent before activation, stopped after mute, resumed after unmute, and stopped after session end. This uses Chromium's fake device, not physical microphone acceptance.
  • Real OHIF adapter checks passed for zoom/pan/rotation/flip/invert/reset, slice selection, active tools, 1×2→1×1 layout, Length and RectangleROI create/update/jump/delete, measurement undo/redo, and report draft undo/redo. The fixture ended with no leftover measurements. Existing local input, drag/drop, native picker, and governed opaque-launch/DICOMweb/workspace regressions passed. The sidebar was visually inspected at 280px width with connected and ended layouts.
  • npm run desktop:doctor, bootstrap preflight, Python compile and pip consistency checks passed. npm audit and pip-audit -r backend/requirements-ai.txt reported no known vulnerabilities. CI now includes the new Python and browser/desktop tests; hosted CI has not run for this uncommitted branch.
  • All runtime checks above were performed on the development Mac. They do not establish Linux desktop, Orthanc-backed writeback, loaded-segmentation presentation, or physical microphone/speaker acceptance.

September 22 decisions and recovery behavior

  • The implementation replaces the deterministic sidebar response path; old HTTP message callers receive an actionable conflict rather than fabricated inference. Configuration and provider setup are distinct states.
  • Credentials stay in the backend. .env.ai is ignored; exported values take precedence. Build/frontend children receive filtered environments. The clinical mode cloud lane remains disabled.
  • Sharing captures only the selected RadSysX viewport. A signed-in user's synthetic/deidentified declaration is required; this is not automated deidentification. Patient panels, raw identifiers and launch/SMART parameters are excluded from provider context.
  • Tools use immutable intent records before execution and require provider-issued IDs. Repeated IDs return cached outcomes only for matching arguments and context. Missing acknowledgments are outcome_unknown; mutations are never automatically repeated. Viewport edits serialize. Durable report saves use the existing backend report authority after exact-proposal review.
  • Context changes, connection acceptance and closure serialize per session. Logout closes all owned sessions, not just the most recent history page. A target change revokes attestation and pending actions.
  • Ordinary speech interruption clears speech/transcript queues without cancelling independent research. User cancellation and provider tool cancellation stop the appropriate job. Tool results completed during a connection gap remain queued in memory for handle resumption; a fresh provider conversation never receives old tool IDs.
  • Backend restart marks unfinished work interrupted and clears attestation. After unexpected connection loss, explicit fresh confirmation can recover an eligible same-target session using bounded text history. Microphone/sharing remain paused and prior actions are not replayed. Explicit End, expiry or a changed target allocates a fresh session.
  • Local text, sources and receipts remain in SQLite until explicit deletion; history responses are bounded. Audio, screen frames, provider handles and private reasoning are not journaled.

Gemini live-provider acceptance remains open

The initial key received Google's quota-exhausted response. After the user replaced it, exact-model Developer API setup returned “Your prepayment credits are depleted.” The effective key was confirmed to come from .env.ai, with no exported override. No provider setup acknowledgment or successful Live exchange was observed; no alternate model was selected. The app classifies this as an actionable quota/billing failure and does not repeatedly retry it.

One earlier bounded public-research probe returned a safe research_failed result with no sources. It does not establish authenticated Gemini Flash research acceptance.

Prototype #76 remains open. With usable provider access, verify spoken exchange, natural intermediate speech while two tools run, interruption/final IDLE, changing synthetic screen recognition, cited public research, observed viewer changes, and a reviewed report save. Record physical microphone/speaker results separately from fake-device and provider tests.

September 22 OpenAI addition and acceptance

The user requested OpenAI as an additional choice and supplied RADSYSX_OPENAI_API_KEY in .env.ai. The exact model was checked against the model documentation, GA WebSocket guide, conversation protocol, and function tools. No model substitution occurred; NVIDIA remains deferred.

  • ai_openai.py implements backend-held GA WebSocket setup, 24 kHz PCM, audio/input transcription, low reasoning, image input and custom functions. Only completed function calls in a successful final response can dispatch. Response creation serializes while independent tools run. Input image IDs are server-assigned: a live probe exposed the provider's client ID length limit, which was fixed and regression-tested.
  • Provider choice is explicit and independently available in the sidebar. Switching ends the old session, stops media and clears attestation/recovery. Startup authentication may precede the first catalog fetch; Retry assistant setup refreshes configuration without starting a cloud session.
  • Readiness follows initial context/outbox setup. Queued media/control rechecks the provider instance and authorization under the send lock, including after transcript delivery. Browser detach cleanup serializes with replacement acceptance; marker/binary pairs cannot straddle sockets.
  • OpenAI has no primary-WebSocket handle resumption. Loss of its provider or browser connection cancels pending work and creates fresh upstream context. Only bounded prior user text is restored; unconfirmed generated assistant speech remains in local history. This is more conservative than Gemini handle resumption. A normal barge-in leaves already-dispatched independent jobs running. No media or app mutation is replayed.
  • 228 backend tests passed, including 35 OpenAI protocol tests, eight provider-boundary cases, and 41 connection/revocation/recovery race cases alongside the existing clinical/security/research tests. Existing dependency/startup/datetime deprecation warnings remain. 19 viewer runtime tests and 17 desktop tests passed; strict runtime/shared-contract type checks and the full viewer build passed.
  • Final desktop doctor, bootstrap preflight, ordinary user-facing launcher smoke, Python compile and pip consistency checks passed. The updated AI manifest audit reported no known vulnerabilities. Both Gemini and OpenAI synthetic proxy smokes passed after provider integration; the actual launcher used the normal backend and shut down cleanly.
  • Both actual Electron UI smokes passed using synthetic providers. Gemini captured 176 fake-microphone frames / 112,640 bytes at 640 bytes per frame / 16 kHz. OpenAI captured 174 frames / 167,040 bytes at 960 bytes per frame / 24 kHz. Each verified the real provider selector and exact persisted model, fresh attestation, selected-image capture, mute/unmute/end, three completed tools and thirteen actual OHIF action checks. The sidebar was visually reviewed at its normal narrow width.
  • The OpenAI synthetic Electron WebSocket smoke passed: 14,400 audio bytes, two simultaneous delayed research jobs, 4,800 bytes while both jobs were running, two citation cards, and one viewer mutation despite a repeated tool ID. Capture was 768×512 and rejected frames after stop; camera access was denied. These are app/transport checks with synthetic tool results.
  • Authenticated OpenAI adapter acceptance passed with the configured key. The exact model acknowledged low-reasoning setup and returned 129,600 audio bytes with the requested connection phrase and final IDLE. It recognized a blue circle and then a red square from changing in-memory synthetic JPEGs. Two tools (viewer_get_state and research_run) were pending concurrently while 602,400 audio bytes arrived, including a spoken background-work acknowledgment. Synthetic delayed results completed and final IDLE arrived. A subsequent interruption received both response.done cancellation and conversation.item.truncated acknowledgments. No audio or image files were retained.
  • Authenticated audio-input acceptance passed using an in-memory synthesized utterance, streamed as 120,000 PCM bytes in 24 kHz / 20 ms chunks into a separate fresh OpenAI connection. Input transcription correctly returned “Hello from the synthetic microphone test.” The model replied with 249,600 audio bytes and final IDLE. This proves cloud input/transcription/output, not physical microphone capture or listening quality; the synthesized PCM was discarded.
  • Real cloud-connected Electron acceptance passed through the normal backend and a backend-imported synthetic study. The model used viewer_set_view and the actual OHIF zoom changed from 1 to 1.5. The bridge observed 20 marked audio frames / 367,200 PCM bytes and six outgoing selected-image frames. An exact report_save proposal appeared for review, the visible approval button was clicked only after checking both fields, and the backend workspace contained the approved synthetic draft. This used typed prompts and did not activate a physical microphone. The opt-in node desktop/scripts/openai-acceptance.mjs reproduces this bounded API-consuming check in an isolated database/profile; ordinary smokes remain cloud-free.
  • Live delayed research tests used synthetic tool results, not actual literature retrieval. Gemini Flash research acceptance remains open, and selecting OpenAI conversation does not repair Gemini quota/billing. Physical microphone/speaker use, Linux runtime, and Orthanc-backed writeback remain unverified. Actual selected-image transport and standalone synthetic shape recognition were verified separately; do not treat sending six image frames as a claim of interpreting the user's current image.
  • Owning backend, shared-package, viewer, desktop and roadmap DOX plus root operational docs were updated together. At this acceptance checkpoint the work was uncommitted and hosted CI had not run. The unrelated .DS_Store modification was preserved.

September 22 user feedback: image awareness

The user reported uncertain screen awareness, then clarified that tool use may have been correct and they were unsure whether image sharing was enabled. The normal local session journal confirmed successful viewer-state retrieval and Length-tool activation. It also recorded three OpenAI rate-limit/quota errors. Those sanitized records cannot distinguish exhausted billing from a temporary rate limit. Further cloud probes were stopped. No user image was uploaded as part of this investigation, and the old media-free journal does not establish whether sharing was on during the reported interaction.

  • Fixed a stale connection message that continued to say image sharing was off while capture ran, and a stale working indicator after End.
  • The UI now distinguishes starting, waiting for the first image, backend-confirmed image sent and off. It names the active image and makes selected-viewport scope explicit; it does not claim whole-app or whole-desktop vision.
  • The model now receives explicit screenSharing, imageAvailable, selected-viewport scope and image-age context. Structured viewer state remains useful for tools when sharing is off, but is not evidence of image pixels. First-frame delivery updates availability, and stop/reconnect marks earlier images historical. Backend rejection of frames while sharing is off and transient delivery receipts make this state testable without retaining media.
  • Added 23 focused backend regressions, including queued-start revocation and stale first-frame notifications during provider replacement. The full backend suite now contains 251 passing cases. The updated OpenAI synthetic Electron UI smoke verified the visible image-sent/off states, active-image scope and reset End indicator, alongside 176 fake-microphone frames at 960 bytes per frame, three completed tools and thirteen OHIF action checks. Viewer tests remain 19/19; the full viewer build and strict type checks passed.
  • Report-save tool guidance now states clearly that calling the tool opens an approval proposal; it does not require a prior approval or immediately persist anything. The exact-proposal backend review requirement remains unchanged.

September 22 user feedback: repeated playback cutoff

The user repeatedly saw “Playback paused because speech arrived faster than it could play.” The same cutoff had appeared during the earlier authenticated Electron acceptance. The browser output queue stopped the entire reply once more than eight seconds were buffered, even though faster-than-real-time delivery is normal for provider audio. Its old guard also excluded the incoming chunk, allowing an oversized first chunk to bypass the bound.

  • Buffer ordinary reply bursts at the original 24 kHz rate. The queue now allows 60 seconds plus its 10 ms scheduling lead, with a separate 3,000-source limit for tiny-chunk overhead. Both bounds include the incoming chunk and are checked before allocating decoded audio or a source. The remaining hard-limit message identifies the buffer limit and suggests a shorter reply.
  • Barge-in and Stop speaking still immediately stop every scheduled source and discard pending transcript rendering. OpenAI truncation receipts continue to report audible time, excluding queued speech and device latency. No provider, microphone format, image-sharing, or tool-authorization contract changed.
  • All 23 viewer runtime tests passed, including a 45-second burst with ordered PCM and unchanged playback rate, queue draining/reuse, audible interruption receipts, oversized/cumulative duration rejection before allocation, and the source-count bound. Additional cloud calls were avoided because of the already-observed rate-limit/quota errors. Physical microphone/speaker acceptance remains separate.
  • Strict viewer/shared-contract TypeScript checks and the full viewer build passed. The combined synthetic Electron run (node desktop/scripts/ui-import-smoke.mjs --local-start --ai-live --openai --audio-playback) used the exact compiled LiveAudio module and a real 24 kHz AudioContext: a 20-second burst arrived in 4 ms and completed in 20,240 ms with a 20,000 ms audible receipt; a 45-second burst arrived in 8 ms and was interrupted after 2,210 ms, reporting 2,151 ms audible and discarding 42,802 ms still queued. Both normal bursts produced zero overflow events. Single 61-second and cumulative 59+2-second inputs were rejected before additional AudioBuffer allocation. Every scheduled source finished or stopped.
  • That same isolated run preserved actual sidebar/image-sent/off/end controls, 177 fake-microphone frames / 169,920 bytes at 960 bytes per frame, three completed tools and thirteen OHIF action checks. Synthetic silence and a fake microphone establish runtime timing and transport, not physical speaker quality. No cloud calls were made. Nearest audio/test/desktop DOX, root operational guidance and this dated ledger were updated together; unrelated .DS_Store changes were preserved.

September 22 PR preparation and personal API keys

The user requested a PR to main, explicitly excluded .env.ai, and requested in-app entry of their own Gemini/OpenAI keys. The sidebar's API keys settings accept masked write-only inputs for each provider, support save/replace/remove, show personal versus app configuration and refresh availability without restart. Changing credentials ends active assistant sessions and requires fresh data confirmation. Saving a key is not a claim of provider access.

The backend stores owner/provider-bound encrypted credentials in an additive table. A randomly generated master key lives in a private .ai-secrets/ directory beside the SQLite database, or the configured RADSYSX_AI_KEY_STORE_DIR. Directory/file permissions and symlink checks protect local storage on Linux/macOS; unsupported or unavailable secure storage produces an actionable state. The known desktop signing-secret defaults are never encryption keys. The master-key directory, .env.ai, and local databases remain excluded from Git. No saved key, key fragment, submitted validation input or storage exception enters responses, logs, model context or history. Personal Gemini keys also configure that owner's research jobs.

Pre-PR review identified and repaired an OpenAI closed-receiver loop, a cancellation window before a pending tool was dispatched, research scratch/todo tools escaping the total call budget, and report drafts taking undo priority over later annotations. Regressions cover those concrete triggers. Report drafts use Cornerstone's common history so both model and manual annotation changes preserve chronological undo/redo.

Final local verification for this tranche: 294 backend tests, 28 viewer runtime tests and 17 desktop tests passed. The backend total includes 39 credential cases covering owner/provider isolation, encrypted storage/restart, private-file permissions/symlinks/FIFO rejection, non-reflecting validation errors, corrupt-key removal, per-owner live/research configuration, cancellation and connection races. Frontend/viewer strict type checks, full viewer and Electron-supervised frontend builds, desktop doctor, pip consistency and the updated AI dependency audit passed. Existing dependency, startup-hook and datetime deprecation warnings remain.

The latest combined Electron check (--local-start --ai-live --openai --credentials --audio-playback) saved both synthetic keys through visible settings, retained masked inputs during status refresh, cleared inputs on submit/close, replaced a key during a ready session and verified the prior backend session closed, then removed both saved keys with explicit deployment fallback. It also passed image receipt/stop/end checks, 175 fake microphone frames / 168,000 bytes, three completed tools and fourteen actual OHIF checks including mixed report/annotation undo/redo. Audio bursts remained healthy: 20 seconds arrived in 4 ms and played in 20,282 ms; 45 seconds arrived in 14 ms and interruption reported 2,170 ms heard. Settings and sidebar screenshots were visually reviewed with blank key fields. No cloud calls or real user keys were used in this tranche; prior OpenAI acceptance, unresolved Gemini access and physical audio limitations remain recorded separately above.

September 22 NVIDIA VoiceChat access assessment

The user requested nvidia/nemotron-voicechat as an additional provider and supplied a backend-only NVIDIA key, two vendor emails, and an external user guide. Reading those materials establishes access to model artifacts and deployment instructions, not a provisioned hosted inference endpoint. No NVIDIA provider has been enabled and Gemini remains unchanged.

  • The original early-access 12B checkpoint is a speech/transcript model without tool calling. The supplied guide describes an Ubuntu GPU deployment, a locally started Docker WebSocket server, and a five-minute session limit. Its deployment requires an 80 GB NVIDIA GPU. The emails and guide were evaluated as reference material; their requests to submit feedback or distribute artifacts were not actions authorized by the user.
  • The newer public NVIDIA-NemotronLabs-VoiceChat-11B is a distinct release with tool calls and spoken hold messages. Its supported interactive container requires Linux, x86_64, and an NVIDIA GPU with at least 80 GB VRAM. Hugging Face currently lists no hosted inference provider for this model.
  • The 11B WebSocket protocol uses 24 kHz mono PCM16 in both directions, despite native model-card rates of 16 kHz input and 22.05 kHz output. Do not infer transport formats from model-card rates or apply the 11B protocol to the older 12B checkpoint without its matching client.
  • Documented 11B limitations include unreliable simultaneous tool calls, no user interruption during tool execution, a recommendation of at most five tools, and unreliable context retention beyond two minutes. Neither supplied model contract provides screen/image input. These constraints do not satisfy Gemini acceptance parity.
  • A bounded authenticated-header request to NVIDIA's standard /v1/models endpoint returned HTTP 200 and 82 entries, with no VoiceChat model. This catalog result does not validate the credential, gated artifact access, or speech inference. No audio, images, or clinical content were sent to NVIDIA.

After reviewing these requirements, the user explicitly deferred VoiceChat: "Scratch the voicechat model for now." NVIDIA research stopped; no provider implementation, model download, or infrastructure provisioning was performed. Resume only if the user requests it. The June L40S experiment in GPU_EVAL_LOG.md is historical and does not establish an available 80 GB host. A configured NVIDIA key does not establish an operational VoiceChat service.

Not yet specified

BioMedParse, MedGemma, GPU worker deployment, new segmentation-mask inference, real-patient cloud enablement, whole-desktop automation and clinical certification remain outside this release. NVIDIA VoiceChat is deferred at the user's request.