Skip to content

fix(bot): apply configuration changes without restarting the process - #35

Merged
TheMeinerLP merged 3 commits into
mainfrom
fix/config-reload
Aug 20, 2026
Merged

fix(bot): apply configuration changes without restarting the process#35
TheMeinerLP merged 3 commits into
mainfrom
fix/config-reload

Conversation

@TheMeinerLP

Copy link
Copy Markdown
Contributor

The defect that started this: the bot started before any configuration existed, an administrator then ran /setup and /config set for every key, /config show reported "All required keys are set" — and the bot still did nothing. _configure_guild ran only from on_ready, so self._guilds stayed empty and on_voice_state_update found nothing to act on. It only worked after a manual pod restart.

The same held for every later change: pointing voice_channel_id at a different channel had no effect until someone restarted the process, while /config show kept insisting the value was set. That gap — between what the command reports and what the running process uses — is the actual bug.

What it does now

Configuration changes reconcile into the running bot. The hard part was never re-reading the config; it was doing so without losing a recording in progress. A change arriving mid-session is deferred and announced rather than forced, and /config set tells the administrator what actually took effect and what waits for the session to end.

Three defects review found afterwards, all reproduced

These were found by adversarial review of the first implementation, and each has a test that fails without its fix.

A failed upload could wedge a guild into recording nothing. SessionMachine moves to CLOSING before close() does its I/O, so a raise anywhere in encrypt/upload/enqueue left it parked there with nothing to call reset(). Both entry paths were affected. Recovery is now keyed on a needs_reset property — close()'s own precondition, asked rather than asserted — and runs in a finally. The error still propagates so the admin is not told a change took effect when the upload died, but it propagates out of a guild that can record again.

Same class, found adjacent and included: graceful_shutdown had one guild's failing upload cost every guild after it in the dict its still-open session. SIGTERM gives only one pass.

People already in the channel were never counted. A pipeline built or retargeted took its headcount only from subsequent voice-state updates. So an admin fixing the configuration while three people waited in the channel got nothing until somebody left and rejoined — the exact complaint this branch exists to fix, reappearing one layer down. One helper now turns current channel membership into a session, and every path calls it.

Reconfiguration raced with joins. _teardown awaits voice.leave() and pops _guilds only afterwards. A join inside that window found the pipeline, opened a session row against the channel being abandoned, and reconnected the voice client that had just been disconnected — leaving a row nothing would ever close and a connection nothing held. on_voice_state_update now takes the per-guild reconfigure lock and re-reads everything after acquiring it. Waiters are bounded to one per guild: the handler reads current membership rather than replaying a delta, so a later read is at least as fresh as anything a dropped update carried.

Verification

434 tests. Each of the six new ones was confirmed to fail against the unfixed client and for the right reason — checked by reverting only client.py.

Two fixtures carry most of the weight: ExplodingStore, whose put fails on demand, which is how you get a close to raise after the machine is in CLOSING; and BlockingVoiceReceiver, whose leave() suspends on demand, holding a reconcile open at exactly the window the race used.

The tests assert outcomes, not calls: that a second session opens and its job is enqueued after a failed upload; that people already present get a real recording with a packet landing and a job queued; that a join during teardown opens no row and reconnects nothing.

Residual

Carried knowingly rather than fixed, since neither is worse than today's behaviour: /config show still answers directly after its database round-trips and could exceed Discord's three-second deadline on a slow database, and a stored value that will not parse produces a NOTHING plan that reports the current config rather than the broken one.

`_configure_guild` ran exactly once, at `on_ready`. A guild that was
unconfigured at that moment — which is every guild at first install — got
a log line and no recording pipeline, and nothing ever revisited that
decision. `/config set voice_channel_id ...` wrote the value, `/config
show` reported it, and the bot went on watching nothing until the pod was
restarted.

Replace the one-shot build with an idempotent reconcile pass, driven by
three triggers: the ten-second tick loop (the authority — a direct
`UPDATE`, a write from another replica, or a command whose hook raised
must not be able to leave the process stale), the `/config` and `/setup`
cogs (for latency, and so their reply can state what took effect), and a
new `on_guild_join`.

The mid-session rule is the load-bearing decision: never replace or mutate
a `RecordingService` or a `VoiceReceiveAdapter` that is recording. Doing
so would drop open `AudioWriter`s (unflushed buffers and a plaintext WAV
on the PVC no job points at, invisible until the next restart's orphan
scan), strand a live voice connection in the channel, and leave the open
`sessions` row naming a channel the audio never came from. So the keys are
split:

* Tunables — `empty_grace_seconds`, `idle_timeout_minutes`,
  `max_session_hours`, `audio_retention_days` — are read at exactly one
  point each (`_due_reason` on the next tick, `close()` when the session
  is filed) and are applied in place immediately, mid-session included.
* Identity — `voice_channel_id`, `consent_role_id` — is deferred while a
  session runs and applied in `_tick_guild` right after `reset()`, the one
  instant at which `close()` has provably finished encrypting, uploading,
  enqueuing and unlinking. The wait is bounded by `max_session_hours` and
  `/config apply force:true` shortens it by ending the session through the
  ordinary close path. A recording is never discarded to make a
  configuration change land sooner.

`consent_role_id` was worse than stale: `VoiceReceiveAdapter.join`
re-read it per join while the client's consent headcount used a value
frozen at startup, so the per-packet filter and the headcount could
disagree about who counts. It now moves atomically with the channel.

This also fixes a second, latent instance of the same defect: discord.py
re-fires `on_ready` after a failed RESUME, and `_configure_guild`
unconditionally overwrote `self._guilds[guild.id]` — so a routine gateway
blip during a recording destroyed the live session by exactly the
mechanism above. The path is now idempotent.

The decision itself is a pure function (`application/reconfigure.py`,
mirroring the `setup_plan.py` precedent), so every row of its table is
tested without a gateway. `ConfigStore.snapshot` replaces five `get()`
round-trips with one query, since this now runs per guild per tick on the
same task as the readiness heartbeat; an unparseable value there is
caught, logged once, and the last known-good configuration is kept —
falling back to defaults would silently un-configure a working guild over
a typo in a shell.

Reporting is part of the fix, not a follow-up: a reply reading
"`voice_channel_id` set to `123`" while the bot records the old channel
is the same defect one layer up. `/config set`, `/config clear` and
`/setup` now say whether a key is in force, waiting behind a recording,
or — for `publish_poll_seconds`, which the publish sweep reads once at
process start — genuinely needs a pod restart. `/config show` gains a
running-configuration line, and `/config apply` re-syncs after a direct
database edit.
`_end_session_now` closed the session and then called `reset()`, and
`RecordingService.close()` does not move the `SessionMachine` out of
RECORDING -- only `tick()` ever did. So `SessionMachine.reset`'s guard
fired, the reconcile raised out of `/config apply`, and the guild was left
holding a service that was closed but never reset: `is_recording` false,
`voice_packet` dropping every packet, `participants_changed` unable to open
a row. That guild then recorded *nothing* until some later timeout happened
to fire -- roughly fifteen minutes if a stale `_last_audio_at` was there to
trip the idle timeout, otherwise until the pod restarted.

The command that reaches this is the one the bot itself recommends:
`render_apply_result` tells an administrator to run `/config apply
force:true` whenever a channel change is waiting, and promises the
recording "is still uploaded and transcribed". Following that advice
mid-session broke recording for the guild, silently.

`SessionMachine.end_now` is the missing edge into CLOSING for a reason the
machine can never observe for itself -- a SIGTERM, or an administrator
ending the recording deliberately -- and `RecordingService.end_now` pairs
it with the ordinary close path, so a forced end is a complete recording
(encrypt, upload, enqueue, close the row) and leaves the pipeline as ready
for the next session as a timed-out one. `graceful_shutdown` uses it too,
so there is one route out rather than two.

Three further defects on the same path:

* Nothing ever cleared `pending_teardown` or `pending`. An administrator
  who cleared `voice_channel_id` mid-session and set it again had the
  teardown announced by `/config show` for the rest of the session and then
  actually carried out -- the pipeline destroyed the instant the recording
  ended and rebuilt by the next reconcile ten seconds later, for a change
  the database had not asked for since. `_forget_stale_deferrals` retracts
  a deferral as soon as the plan stops asking for it.
* Both deferral branches logged at INFO on every pass. A guild reconciles
  every ten seconds, so one deferred channel change repeated the same
  sentence some 1400 times over a four-hour session. The transition is
  news; the state persisting is not.
* `/config set` and `/config clear` awaited a write plus a full reconcile
  before their first response, against Discord's three-second
  initial-response deadline -- and a miss shows the user "The application
  did not respond" over a value that was in fact written. They now defer
  first and answer through `followup`, as `/setup` and `/config apply`
  already did.

Every fix is covered by a test that reproduces the reported sequence and
fails without it; the forced-end test asserts both halves of the promise --
that the end succeeds, and that the session's audio still reaches the job
queue.
…igure

Three ways a guild could end up configured but capturing nothing.

A `close()` that raised -- an unreachable object store is enough --
left the `SessionMachine` parked in CLOSING with nothing to call
`reset()`, on both the tick sweep and the `/config apply force:true`
path. Nothing in the process ever leaves that state, so the guild
recorded nothing until a restart, with no complaint anywhere. The
return to a recordable state is now keyed on the machine's own state
rather than on the close having finished, and the failure is logged
loudly since it may mean a recording was not uploaded. `graceful_
shutdown` isolates each guild for the same reason: SIGTERM gives it
one pass, and one failing upload must not cost every guild after it.

A pipeline that was just built or retargeted never counted the
consenting members already sitting in the target channel -- the
headcount only ever came from later voice-state updates. An
administrator fixing the configuration while people waited in the
channel got a bot that reported itself live and did nothing until
somebody left and rejoined. Both paths now read the channel's current
membership through the same helper the voice-state handler uses.

`on_voice_state_update` ran outside the per-guild reconfigure lock, so
a join landing inside the `await voice.leave()` in `_teardown`
stranded both a session row nothing was left to close and a voice
connection nothing was left to disconnect; on `_retarget` it tripped
the mid-session assertion and left the reconfigure half-applied. The
handler now takes the lock and re-reads everything it decides on
afterwards. At most one update per guild waits on that lock -- the
handler reads current membership rather than replaying a delta, so one
waiter answers for a whole burst.
@TheMeinerLP

Copy link
Copy Markdown
Contributor Author

Rebased onto main after #34 landed. The two features collided in a way worth naming: #34 records the voice channel's name onto the session row, and this branch moves an idle service to a new channel in place via retarget(). Left as they were, a retarget would have carried the new channel id and the old channel name — the next protocol headed with a room the recording did not come from, which is worse than the None fallback to a bare link.

So retarget() now takes channel_name as a required argument rather than a defaulted one (a caller that forgets it should not compile past review), resolving it through a new SturnusClient._channel_name seam — the client is the only party holding a Discord connection; the worker that writes the header sees nothing but the id.

test_retarget_carries_the_channel_name_to_the_next_session covers it. Counter-checked: breaking only the assignment in retarget() fails that test and nothing else.

@TheMeinerLP
TheMeinerLP merged commit a943f87 into main Aug 20, 2026
6 checks passed
TheMeinerLP added a commit that referenced this pull request Aug 20, 2026
Two restructurings of `SturnusClient` met here: main's reconfiguration
engine (#35) and this branch's capture pipeline. Main's structure is the
trunk; this branch's behaviour is re-expressed inside it.

The one decision worth naming: the rejoin guard moved out of
`on_voice_state_update` and into `_sync_participants`. On this branch the
event handler was the only way a session could start, so the guard was
correct there. Under main's engine it no longer is -- `_build`,
`_retarget` and `_apply_pending` reach `_sync_participants` too, so a
deferred channel change landing on the same tick as a capture failure
would have smuggled a session past a guard sitting in the handler. One
check at the funnel covers every path.

The cooldown is armed between `_return_to_idle` and `_apply_pending`, not
after, for the same reason. It lapses in `_tick_guild` after `_reconcile`,
so the recount uses the configuration the reconcile just settled on.

Also merged: `RecordingService.retarget()` now takes the channel name
alongside the id (#34 + #35), and the capture-pipeline fakes follow the
`open_session` signature that gained `channel_name`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant