Only send modern transcriptions to client sdks which can auto back convert modern -> legacy transcriptions internally - #7240
Open
1egoman wants to merge 6 commits into
Open
Only send modern transcriptions to client sdks which can auto back convert modern -> legacy transcriptions internally#72401egoman wants to merge 6 commits into
1egoman wants to merge 6 commits into
Conversation
1egoman
force-pushed
the
transcription-deduplication
branch
from
September 11, 2026 20:33
c767c0f to
ecdbb87
Compare
The agent must know if a client can rebuild transcriptions from data streams. A client tells the other participants about this ability with a client protocol number. Client protocol 3 means that the client rebuilds transcription events from the `lk.transcription` text streams. Such a client ignores the deprecated `rtc.Transcription` data packet. This commit adds the `CLIENT_PROTOCOL_TRANSCRIPTION_STREAMS` constant. The client SDKs set the value of this constant. This repository does not set it. This commit also adds the `_client_protocol` function. The function reads the number from a participant. The function returns 0 if the number is not available. A value of 0 means a legacy client. The function reads the private `_info` field on purpose. The client protocol is an internal signal between participants. It is not part of the public participant API.
The agent sends each transcription two times. It sends a deprecated `rtc.Transcription` data packet. It also sends an `lk.transcription` text stream. The two copies fill the reliable data channel. Other reliable traffic becomes slow. This commit stops the legacy packet when no client needs it. Before each legacy publish, the agent examines the remote participants. If all applicable participants have client protocol 3 or higher, the agent does not send the packet. The agent always sends the text stream. Only STANDARD participants are applicable. These are the client SDK instances that users create. SIP, INGRESS, AGENT, CONNECTOR and BRIDGE participants do not show legacy transcripts. EGRESS participants are hidden and do not appear in the participant list. The agent also ignores its own avatar worker. The gate is in `_publish_transcription`. This is the only function that sends the packet. Both `capture_text` and `flush` use this function. Do not move the gate into `capture_text`. Each legacy packet contains the full segment text with a stable identifier. The accumulated text must stay correct. If a legacy client joins during a segment, the next packet gives it the full segment. The agent calculates the result again for each publish. Do not put the result in a cache. The SDK changes the participant data without a Python event. A cache can keep an incorrect result for the full session. The `_legacy_status_logged` field does not control the gate. It holds only the last status written to the log. The code writes a new log message when the status changes. This prevents a log message for each transcription chunk.
This commit adds three test helpers. It adds no tests. `_fake_remote` makes a stand-in for a remote participant. The `attributes` field must be a true dictionary, because the gate calls `get` on it. The `client_protocol` value is on the `_info` field. This is the same shape as a livekit-rtc participant. `_make_legacy_output` makes a legacy output and sets its fields directly. Do not use `set_participant` here. That function reads `track_publications`, and the fakes do not have this attribute. `_capture_and_flush` sends text through an output and then waits for the flush task.
This commit adds three tests for the client protocol comparison. The first test shows that the agent sends no legacy packet when all clients have client protocol 3. The second test shows that the agent sends the legacy packets when one client is older. The test also examines the packets. The partial packet and the final packet must have the same segment identifier and the full text. The third test shows that an unknown client protocol counts as a legacy client. The agent then sends the packet. This is the safe result.
This commit adds six tests. These tests hold the rules about applicable participants. Change these rules only with care. Two tests show that the agent sends no legacy packet when the room has no STANDARD participant. A room with only a SIP participant gives the same result, because a SIP participant does not show transcripts. One test shows that SIP, INGRESS, AGENT and CONNECTOR participants do not control the result. A second test shows that a STANDARD participant with an old client protocol does control the result. Two tests show the avatar worker rules. The agent ignores its own avatar worker. The agent does not ignore the avatar worker of a different agent. The last test is a guard. The exclusion must not use `_is_local_proxy_participant`. That function also agrees with the participant of the output. For the user output, that participant is the user. The agent must not stop the transcriptions of the user.
This commit adds two tests. The first test shows that the gate obeys changes in the room. All clients are modern at the start, and the agent sends no packet for the first text. A legacy client then joins. The agent sends one packet for the second text. This packet contains the full text of both parts, not only the second part. The final packet has the same segment identifier. This test fails if a person moves the gate into `capture_text`. The second test shows that the `lk.transcription` text stream does not change. The stream always goes to all participants.
1egoman
force-pushed
the
transcription-deduplication
branch
from
September 11, 2026 20:45
ecdbb87 to
66ee8de
Compare
1egoman
marked this pull request as ready for review
September 11, 2026 20:50
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Comment on lines
+412
to
+416
| logger.debug( | ||
| "legacy transcription publishing %s", | ||
| "enabled" if needed else "disabled", | ||
| extra={"participant": self._participant_identity}, | ||
| ) |
Contributor
There was a problem hiding this comment.
🟨 Participant identity bypasses log redaction
When _should_publish changes state, it logs the participant identity under participant. The unmarked key prevents configured PII redaction.
Was this helpful? React with 👍 or 👎 to provide feedback.
Contributor
Author
There was a problem hiding this comment.
Ah, interesting - should I get rid of this logging behavior all together? Or if not - I'm assuming that in other places participant ids are logged within the framework's logs. Are there any templates which I can follow on how best to handle this?
xianshijing-lk
left a comment
xianshijing-lk
left a comment
There was a problem hiding this comment.
one question, lgtm if you address it.
| reach ``remote_participants``. | ||
| """ | ||
| local_identity = room.local_participant.identity | ||
| for p in room.remote_participants.values(): |
There was a problem hiding this comment.
how often this _legacy_transcription_needed will be called ?
If it is called frequently, can we improve the code to reduce the overhead ?
| capability signal used for feature detection between participants, not part of the | ||
| public participant API. | ||
| """ | ||
| value = getattr(getattr(participant, "_info", None), "client_protocol", 0) |
There was a problem hiding this comment.
any chance that we can add some tests to make sure _info and proto filed never gets renamed or changed ?
otherwise the value will default to 0 and fail silently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The agents part of livekit/client-sdk-js#2093. A high level summary of the transcription deduplication project:
In livekit/client-sdk-js#2093, I added a new client protocol of 3. Any client which advertises this new client protocol version ignores all legacy transcriptions, and back-converts modern transcriptions into in memory legacy transcriptions.
This pull request implements the converse of this behavior on the agent end: if any connected client sdk advertises support for the client protocol of 3, then the agents sdk now will skip sending legacy transcriptions to all participants.
A few important caveats:
destination_identitieswouldn't only deliver the packet to the listed participants.