fix(bot): apply configuration changes without restarting the process - #35
Conversation
`_configure_guild` ran exactly once, at `on_ready`. A guild that was unconfigured at that moment — which is every guild at first install — got a log line and no recording pipeline, and nothing ever revisited that decision. `/config set voice_channel_id ...` wrote the value, `/config show` reported it, and the bot went on watching nothing until the pod was restarted. Replace the one-shot build with an idempotent reconcile pass, driven by three triggers: the ten-second tick loop (the authority — a direct `UPDATE`, a write from another replica, or a command whose hook raised must not be able to leave the process stale), the `/config` and `/setup` cogs (for latency, and so their reply can state what took effect), and a new `on_guild_join`. The mid-session rule is the load-bearing decision: never replace or mutate a `RecordingService` or a `VoiceReceiveAdapter` that is recording. Doing so would drop open `AudioWriter`s (unflushed buffers and a plaintext WAV on the PVC no job points at, invisible until the next restart's orphan scan), strand a live voice connection in the channel, and leave the open `sessions` row naming a channel the audio never came from. So the keys are split: * Tunables — `empty_grace_seconds`, `idle_timeout_minutes`, `max_session_hours`, `audio_retention_days` — are read at exactly one point each (`_due_reason` on the next tick, `close()` when the session is filed) and are applied in place immediately, mid-session included. * Identity — `voice_channel_id`, `consent_role_id` — is deferred while a session runs and applied in `_tick_guild` right after `reset()`, the one instant at which `close()` has provably finished encrypting, uploading, enqueuing and unlinking. The wait is bounded by `max_session_hours` and `/config apply force:true` shortens it by ending the session through the ordinary close path. A recording is never discarded to make a configuration change land sooner. `consent_role_id` was worse than stale: `VoiceReceiveAdapter.join` re-read it per join while the client's consent headcount used a value frozen at startup, so the per-packet filter and the headcount could disagree about who counts. It now moves atomically with the channel. This also fixes a second, latent instance of the same defect: discord.py re-fires `on_ready` after a failed RESUME, and `_configure_guild` unconditionally overwrote `self._guilds[guild.id]` — so a routine gateway blip during a recording destroyed the live session by exactly the mechanism above. The path is now idempotent. The decision itself is a pure function (`application/reconfigure.py`, mirroring the `setup_plan.py` precedent), so every row of its table is tested without a gateway. `ConfigStore.snapshot` replaces five `get()` round-trips with one query, since this now runs per guild per tick on the same task as the readiness heartbeat; an unparseable value there is caught, logged once, and the last known-good configuration is kept — falling back to defaults would silently un-configure a working guild over a typo in a shell. Reporting is part of the fix, not a follow-up: a reply reading "`voice_channel_id` set to `123`" while the bot records the old channel is the same defect one layer up. `/config set`, `/config clear` and `/setup` now say whether a key is in force, waiting behind a recording, or — for `publish_poll_seconds`, which the publish sweep reads once at process start — genuinely needs a pod restart. `/config show` gains a running-configuration line, and `/config apply` re-syncs after a direct database edit.
`_end_session_now` closed the session and then called `reset()`, and `RecordingService.close()` does not move the `SessionMachine` out of RECORDING -- only `tick()` ever did. So `SessionMachine.reset`'s guard fired, the reconcile raised out of `/config apply`, and the guild was left holding a service that was closed but never reset: `is_recording` false, `voice_packet` dropping every packet, `participants_changed` unable to open a row. That guild then recorded *nothing* until some later timeout happened to fire -- roughly fifteen minutes if a stale `_last_audio_at` was there to trip the idle timeout, otherwise until the pod restarted. The command that reaches this is the one the bot itself recommends: `render_apply_result` tells an administrator to run `/config apply force:true` whenever a channel change is waiting, and promises the recording "is still uploaded and transcribed". Following that advice mid-session broke recording for the guild, silently. `SessionMachine.end_now` is the missing edge into CLOSING for a reason the machine can never observe for itself -- a SIGTERM, or an administrator ending the recording deliberately -- and `RecordingService.end_now` pairs it with the ordinary close path, so a forced end is a complete recording (encrypt, upload, enqueue, close the row) and leaves the pipeline as ready for the next session as a timed-out one. `graceful_shutdown` uses it too, so there is one route out rather than two. Three further defects on the same path: * Nothing ever cleared `pending_teardown` or `pending`. An administrator who cleared `voice_channel_id` mid-session and set it again had the teardown announced by `/config show` for the rest of the session and then actually carried out -- the pipeline destroyed the instant the recording ended and rebuilt by the next reconcile ten seconds later, for a change the database had not asked for since. `_forget_stale_deferrals` retracts a deferral as soon as the plan stops asking for it. * Both deferral branches logged at INFO on every pass. A guild reconciles every ten seconds, so one deferred channel change repeated the same sentence some 1400 times over a four-hour session. The transition is news; the state persisting is not. * `/config set` and `/config clear` awaited a write plus a full reconcile before their first response, against Discord's three-second initial-response deadline -- and a miss shows the user "The application did not respond" over a value that was in fact written. They now defer first and answer through `followup`, as `/setup` and `/config apply` already did. Every fix is covered by a test that reproduces the reported sequence and fails without it; the forced-end test asserts both halves of the promise -- that the end succeeds, and that the session's audio still reaches the job queue.
…igure Three ways a guild could end up configured but capturing nothing. A `close()` that raised -- an unreachable object store is enough -- left the `SessionMachine` parked in CLOSING with nothing to call `reset()`, on both the tick sweep and the `/config apply force:true` path. Nothing in the process ever leaves that state, so the guild recorded nothing until a restart, with no complaint anywhere. The return to a recordable state is now keyed on the machine's own state rather than on the close having finished, and the failure is logged loudly since it may mean a recording was not uploaded. `graceful_ shutdown` isolates each guild for the same reason: SIGTERM gives it one pass, and one failing upload must not cost every guild after it. A pipeline that was just built or retargeted never counted the consenting members already sitting in the target channel -- the headcount only ever came from later voice-state updates. An administrator fixing the configuration while people waited in the channel got a bot that reported itself live and did nothing until somebody left and rejoined. Both paths now read the channel's current membership through the same helper the voice-state handler uses. `on_voice_state_update` ran outside the per-guild reconfigure lock, so a join landing inside the `await voice.leave()` in `_teardown` stranded both a session row nothing was left to close and a voice connection nothing was left to disconnect; on `_retarget` it tripped the mid-session assertion and left the reconfigure half-applied. The handler now takes the lock and re-reads everything it decides on afterwards. At most one update per guild waits on that lock -- the handler reads current membership rather than replaying a delta, so one waiter answers for a whole burst.
295bb4a to
af6c390
Compare
|
Rebased onto So
|
Two restructurings of `SturnusClient` met here: main's reconfiguration engine (#35) and this branch's capture pipeline. Main's structure is the trunk; this branch's behaviour is re-expressed inside it. The one decision worth naming: the rejoin guard moved out of `on_voice_state_update` and into `_sync_participants`. On this branch the event handler was the only way a session could start, so the guard was correct there. Under main's engine it no longer is -- `_build`, `_retarget` and `_apply_pending` reach `_sync_participants` too, so a deferred channel change landing on the same tick as a capture failure would have smuggled a session past a guard sitting in the handler. One check at the funnel covers every path. The cooldown is armed between `_return_to_idle` and `_apply_pending`, not after, for the same reason. It lapses in `_tick_guild` after `_reconcile`, so the recount uses the configuration the reconcile just settled on. Also merged: `RecordingService.retarget()` now takes the channel name alongside the id (#34 + #35), and the capture-pipeline fakes follow the `open_session` signature that gained `channel_name`.
The defect that started this: the bot started before any configuration existed, an administrator then ran
/setupand/config setfor every key,/config showreported "All required keys are set" — and the bot still did nothing._configure_guildran only fromon_ready, soself._guildsstayed empty andon_voice_state_updatefound nothing to act on. It only worked after a manual pod restart.The same held for every later change: pointing
voice_channel_idat a different channel had no effect until someone restarted the process, while/config showkept insisting the value was set. That gap — between what the command reports and what the running process uses — is the actual bug.What it does now
Configuration changes reconcile into the running bot. The hard part was never re-reading the config; it was doing so without losing a recording in progress. A change arriving mid-session is deferred and announced rather than forced, and
/config settells the administrator what actually took effect and what waits for the session to end.Three defects review found afterwards, all reproduced
These were found by adversarial review of the first implementation, and each has a test that fails without its fix.
A failed upload could wedge a guild into recording nothing.
SessionMachinemoves to CLOSING beforeclose()does its I/O, so a raise anywhere in encrypt/upload/enqueue left it parked there with nothing to callreset(). Both entry paths were affected. Recovery is now keyed on aneeds_resetproperty —close()'s own precondition, asked rather than asserted — and runs in afinally. The error still propagates so the admin is not told a change took effect when the upload died, but it propagates out of a guild that can record again.Same class, found adjacent and included:
graceful_shutdownhad one guild's failing upload cost every guild after it in the dict its still-open session. SIGTERM gives only one pass.People already in the channel were never counted. A pipeline built or retargeted took its headcount only from subsequent voice-state updates. So an admin fixing the configuration while three people waited in the channel got nothing until somebody left and rejoined — the exact complaint this branch exists to fix, reappearing one layer down. One helper now turns current channel membership into a session, and every path calls it.
Reconfiguration raced with joins.
_teardownawaitsvoice.leave()and pops_guildsonly afterwards. A join inside that window found the pipeline, opened a session row against the channel being abandoned, and reconnected the voice client that had just been disconnected — leaving a row nothing would ever close and a connection nothing held.on_voice_state_updatenow takes the per-guild reconfigure lock and re-reads everything after acquiring it. Waiters are bounded to one per guild: the handler reads current membership rather than replaying a delta, so a later read is at least as fresh as anything a dropped update carried.Verification
434 tests. Each of the six new ones was confirmed to fail against the unfixed client and for the right reason — checked by reverting only
client.py.Two fixtures carry most of the weight:
ExplodingStore, whoseputfails on demand, which is how you get a close to raise after the machine is in CLOSING; andBlockingVoiceReceiver, whoseleave()suspends on demand, holding a reconcile open at exactly the window the race used.The tests assert outcomes, not calls: that a second session opens and its job is enqueued after a failed upload; that people already present get a real recording with a packet landing and a job queued; that a join during teardown opens no row and reconnects nothing.
Residual
Carried knowingly rather than fixed, since neither is worse than today's behaviour:
/config showstill answers directly after its database round-trips and could exceed Discord's three-second deadline on a slow database, and a stored value that will not parse produces a NOTHING plan that reports the current config rather than the broken one.