Fix release crashes, TTS bugs, and add fully-offline bundling + conversation-swap mode#8
Open
7MS8 wants to merge 21 commits into
Open
Conversation
added 21 commits
July 14, 2026 12:54
R8 was obfuscating/removing ONNX Runtime's Java classes, which its native
JNI layer (libonnxruntime4j_jni.so) looks up by name via reflection. This
caused a JNI abort ("java_class == null" in GetMethodID) inside
OrtSession.run, crashing the NLLB offline translation path in release
builds.
Without this, TextToSpeech cannot see any installed TTS engine on Android 11+ and initialization silently fails with status ERROR.
TtsEngine.initialize() had two divergent code paths: the cold-init path checked setLanguage()'s result and fell back to English on missing data, but the warm re-init path (hit every time the target language changes after the first call) discarded the result entirely with no fallback or logging. Some TTS engines also only fully honor the modern Voice API, so setLanguage() alone can report success without actually switching the active synthesis voice. Both paths now go through a shared applyLocale() that checks the setLanguage() result, explicitly selects a matching offline Voice, logs the outcome, and surfaces a warning via TranslationUiState when the requested language falls back to English.
Gitignore app/src/main/assets/models/ (fetched locally, not committed — multi-GB ASR/NLLB model binaries) and configure the release build to store them uncompressed (required for AssetManager.openFd()-based extraction progress sizing) with a larger Gradle heap to package them. No model files or extraction logic yet — this only prepares the build config ahead of that.
Adds AssetModelProvisioner, which extracts model files from app/src/main/assets/models/ (if bundled) into internal storage — the native ASR/ONNX code needs real filesystem paths, not zipped asset streams. ModelDownloader and NllbModelManager both try this fast local copy first, falling back to the existing HTTP download unchanged when a language's assets aren't bundled (e.g. a language added later without a matching assets rebuild). NllbModelManager.ensureModelAvailable() also gains an optional warmup callback so status-transition/error-handling boilerplate around initializing an NllbTranslator is shared rather than duplicated per call site. No model files are committed — app/src/main/assets/models/ is gitignored and fetched independently per build.
Previously, offline-mode pre-loading only touched the NLLB model if it was already downloaded, silently skipping provisioning otherwise and requiring a manual trip to Settings. With bundled assets making provisioning a fast local copy rather than a multi-hundred-MB network download, it's now safe to call ensureModelAvailable() unconditionally whenever offline mode is enabled. SettingsViewModel's manual "download" button now goes through the same shared warmup-callback path added to NllbModelManager.
Only affects fresh installs / cleared app data. Paired with bundled model assets, this makes the app fully offline-capable out of the box with no user setup required.
Previously muteMicDuringTts defaulted to false, so on the microphone source the mic kept picking up the phone's own TTS playback acoustically, which got transcribed as gibberish and re-translated — a voice feedback loop. isSpeaking now also stays true for a short cooldown after each utterance ends, covering the speaker/room echo tail that would otherwise still leak into the next recognition window.
Adds ConversationRecognizerPool, holding up to 2 SherpaOnnxRecognizer instances keyed by language (LRU-evicted beyond that). Switching the source language previously always released and reinitialized the ASR recognizer (a multi-second reload); now TranslationService and MainViewModel acquire from this pool instead, so once both directions of a conversation have been loaded once, switching between them is a cache hit. SherpaOnnxRecognizer is no longer Hilt-managed (the pool creates plain instances directly, since @singleton is incompatible with holding more than one loaded language at a time) — the now-unused bindSpeechRecognizer binding is removed from AppModule. MainViewModel.preloadPipelineComponents() also pre-warms the target language's ASR model at launch (when it has one), not just the source, so both directions are ready before a swap is ever requested.
Atomically flips source/target language in one DataStore edit, instead of two separate update calls, to avoid a torn intermediate state.
Eagerly pre-warming the target language's ASR model at every app
launch (on top of the already-loaded source model and, in offline
mode, NLLB's ~1GB) pushed a loaded device over its memory watermark
and got the whole process killed by the Android low-memory killer
("min watermark is breached and swap is low") — reproduced on a
Pixel 7a mid-conversation.
The warm pool now only builds up lazily: the source language loads as
before, and the target only gets pre-warmed the first time
swapLanguages() is actually used, not on every launch regardless of
whether the user ever swaps.
As a second line of defense, ConversationRecognizerPool.trimToMostRecentlyUsed()
releases the least-recently-used warm recognizer in response to
Application.onTrimMemory(), so once both directions do get warmed
during a long conversation, the pool shrinks itself back to one under
memory pressure instead of the system killing the whole process.
Icon button between the source/target language labels, flipping the conversation direction via MainViewModel.swapLanguages(). Disabled when the current target language has no ASR model, since there'd be nothing to recognize a reply in that language.
Two distinct issues surfaced testing conversation swaps on a memory-constrained device: 1. onTrimMemory reacted at TRIM_MEMORY_RUNNING_LOW, which fires very often under system-wide (not just this app's own) memory pressure — discarding the other direction's warm ASR recognizer right after it loaded and forcing a reload on essentially every swap. Now only reacts at TRIM_MEMORY_RUNNING_CRITICAL. 2. The external TTS engine app itself can be killed by the system under memory pressure (observed directly: "Kill 'org.woheller69. ttsengine' ... min watermark is breached"). After Android auto-restarts and rebinds it, there's a window where it hasn't finished reloading its voice data yet, so setLanguage() reports the requested language as missing even though it's actually installed — TtsEngine now retries once after a short delay before permanently falling back to English.
Offline mode (NLLB) plus the ASR conversation-swap cache and a memory-hungry third-party neural TTS engine were observed getting killed by the Android low-memory killer on an 8GB Pixel 7a under normal background app load. Document a 6-8GB RAM recommendation for offline mode so users aren't surprised by this on constrained devices.
…uages acquire() had no synchronization: a language swap's pre-warm call and the subsequent pipeline restart's own acquire() could race, each seeing the pool empty and building a duplicate SherpaOnnxRecognizer for the same language, with the second write silently orphaning the first. Observed crashing the pipeline (JobCancellationException followed by a colliding AudioRecord IllegalStateException). Added a Mutex around acquire/reconcile. Also add reconcile(desiredLanguages), releasing any warm recognizer left over from a since-changed source/target setting -- previously a stale entry from before a settings change stayed resident indefinitely alongside the new one.
Stop (and now pause) tore down the recognizer's collection loop via plain cancellation, discarding whatever was still being transcribed if the user pressed Stop/Pause before sustained silence or Sherpa's own endpoint detection had a chance to finalize a segment -- the most common case, since users typically stop right after finishing a sentence rather than waiting through an extra pause first. The recognition coroutine's finally block now recomputes and flushes whatever's buffered, wrapped in withContext(NonCancellable) so the (already-cancelling) coroutine can still emit it. Also add pauseAndFlush(), returning the flushed text directly rather than via recognizedSegments -- that flow's collector is gated on "not paused", and a normally-emitted segment would arrive there right as isPaused turns true, silently dropped by the same guard it needs to get past.
Plumbing only -- TranslationService will set this once pause becomes a distinct action from stop.
Several related fixes to how starting/stopping/pausing tear down (or don't) in-flight work: - Reconcile the ASR pool to the current source/target before acquiring a recognizer, so a language changed via Settings or a swap doesn't leave its old recognizer orphaned but resident (see ConversationRecognizerPool commit) -- this is what was causing an OOM kill shortly after launch. - Translation coroutines now run on serviceScope directly instead of as children of the per-session pipelineJob, tracked in pendingTranslationJobs. stopPipeline() used to cancel pipelineJob immediately, tearing down a translation already in flight for the segment recognized right before Stop was pressed -- it now gives those jobs a bounded grace period to reach TTS before finalizing the stop. - pausePipeline/resumePipeline rewritten: pause now flushes and translates whatever the user was mid-sentence saying (via SpeechRecognizer.pauseAndFlush(), see previous commit) instead of silently dropping it, and resume re-opens audio capture from scratch (extracted into beginListening()) since pausing tears down the underlying AudioRecord too.
Checked level >= TRIM_MEMORY_RUNNING_CRITICAL, but the docstring's stated intent was "only RUNNING_CRITICAL" -- >= also matched TRIM_MEMORY_UI_HIDDEN (20) and everything above it, which fire on any simple app switch or screen lock with no memory pressure implied, unlike RUNNING_CRITICAL (15) specifically. The pool was being trimmed and reloaded on ordinary backgrounding instead of only under real pressure.
…re reload - pauseTranslation()/resumeTranslation() send the existing ACTION_PAUSE/ ACTION_RESUME intents, exposing isPaused as a StateFlow for the UI. - Punctuation model download+extraction is no longer awaited during preload: it's optional (English sentence-casing only), but the pipeline wouldn't reach ModelStatus.Ready until it finished, and its download has been observed stalling for minutes under memory/CPU pressure from the ASR+NLLB loads happening alongside it, making the app appear hung. Now fire-and-forget: it wires into the recognizer whenever/if it lands. - releaseWarmRecognizersThen() releases every warm ASR recognizer before running a start or swap, called unconditionally rather than only under low memory: loading a new ASR model on top of an already-warm one was observed reliably tipping memory-constrained devices into a full system-wide low-memory-killer cascade. swapLanguages() calls this at the point it's safe to (after stopping and waiting out any running session), trading the pool's instant swap-back for a full reload every time (a few seconds) -- confirmed an acceptable default on a memory-constrained device.
The button previously toggled start/stop on every tap. It's now a Surface with combinedClickable (FloatingActionButton's own onClick fought with an overlaid combinedClickable, silently swallowing every tap when tried first): short tap starts, or pauses/resumes while running; long-press fully stops. Ending the session is otherwise only triggered by a language swap, which needs a real restart anyway. Icon reflects current state, not the next action: Mic while actively recording, Pause once actually paused (this was backwards in an earlier iteration -- tapping to start showed the Pause icon immediately). The "Listening..." status line likewise switches to a static "Paused" label and icon instead of a misleading spinner.
Author
Follow-up fixes from live device testing (2026-07-15)Testing this PR's changes on a memory-constrained device (Pixel 7a, GrapheneOS,
All still small, independently revertable commits, same as the rest of this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
OrtSession.run(crashed NLLB offline translation). Fixed with a ProGuard keep rule.<queries>manifest declaration forandroid.intent.action.TTS_SERVICEmeant TextToSpeech couldn't see any installed TTS engine on API 30+ (package visibility), so TTS silently failed to initialize.setLanguage()'s result with no fallback or logging, and the app never used theVoiceAPI (some engines only fully honorsetVoice()). Now both paths share oneapplyLocale()that checks the result, explicitly matches aVoice, and surfaces a UI warning on fallback.muteMicDuringTtsdefaulted off, so the mic picked up the phone's own TTS playback acoustically and re-transcribed/re-translated it. Now on by default, plus a short cooldown after each utterance to cover the acoustic tail.All commits are kept small and independently reviewable/revertable — happy to split into separate PRs if that's preferred.
Test plan
assembleRelease) installs and runs without crashing on a Pixel 7a (GrapheneOS)Hi, all this code was produced by claude code. I just did testing and architecture. Thank you for your work and effort, just wanted to give back. Kind regards, Michael