feat(datasource): add DingTalk knowledge base connector (#1707) - #46
Draft
lyingbug wants to merge 7 commits into
Draft
feat(datasource): add DingTalk knowledge base connector (#1707)#46lyingbug wants to merge 7 commits into
lyingbug wants to merge 7 commits into
Conversation
… RAG wait row The wait row only appeared once a completed knowledge_search step existed, so attachment-only turns (attachment_parsing / image_analysis with no KB hit) still showed nothing between the last step and the first answer token. Model waiting is now a three-state kind: 'model' after retrieval finished, the neutral 'preparing' row otherwise. A dropped SSE connection never sets is_completed (the stream layer only raises a toast), so the row used to promise an answer forever. It now stops claiming progress after RAG_WAIT_STALL_DELAY_MS and drops the shimmer. Also share RAG_RETRIEVAL_TOOL_NAMES instead of repeating the knowledge_search / search_knowledge alias check, name the reveal delay, and move the reveal/stall timers into a controller with an injectable scheduler so the timing is covered by tests instead of by source-matching alone. Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
role="status" lived on the wait row itself, so the live region was inserted together with its own text and screen readers had nothing to announce. Move the announcement to an sr-only region that stays mounted for the whole turn, which also covers the pre-pipeline wait row that had no announcement at all. Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
Integrate DingTalk knowledge bases through the official OAuth, wiki node, and document block APIs. Add lazy resource selection, protected incremental sync, Markdown rendering, localized frontend configuration, tests, and setup documentation. Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
The block→Markdown path lost content in three ways that only show up against
the shapes DingTalk actually returns:
- Every list item arrives as its own top-level block, so an ordered run was
numbered "1." on each item and never terminated. A following table or
paragraph was absorbed into the list as a lazy continuation. Track the run
(listId + level) to number items sequentially and close it with a blank line.
- heading.level is documented as an integer, but the string forms ("2",
"heading-2") appear in practice and made json.Unmarshal fail for the whole
envelope, dropping the heading text along with its level.
- A type mismatch anywhere in a block's payload discarded the entire block.
Fall back to re-reading just the envelope so an unexpected field costs the
block's formatting rather than its text.
The blocks endpoint is also read more defensively: an omitted `success` field is
no longer read as an explicit failure (the transport already rejects non-2xx),
and a server that ignores startIndex/endIndex now ends pagination instead of
re-reading the same blocks up to the page cap and duplicating the document.
Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
The picker resolves both strings through computed keys
(`datasource.connector.${def.type}`), so the static locale audit never sees
them and the connector card rendered the raw keys instead of "钉钉文档" and its
description. Add the missing entries to all four bundles, plus a test that
checks both bags against the connector list in DataSourceEditorDialog.vue so a
future connector cannot ship with an unlocalized card.
Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes Tencent#1707.
This is @mingri31164's connector from #2263, rebased onto current
main(its own branch is ~140 commits behind and conflicts in four locale bundles), with three follow-up fixes on top. The original commit is preserved with its author, so attribution stays intact.Why this implementation
I reviewed the 19 competing DingTalk submissions and compared their block→Markdown converters by feeding each the payload shapes from DingTalk's official 块元素数据结构 reference. Tencent#2263 was the only one that reproduced every element (headings, bold runs, link and image URLs, blockquote, callout, both list kinds, tables, columns) with no losses. It also uses the documented endpoints —
POST /v1.0/oauth2/accessToken,GET /v2.0/wiki/workspaces,GET /v2.0/wiki/nodes,GET /v1.0/doc/suites/documents/{docKey}/blocks— with thex-acs-dingtalk-access-tokenheader and the mandatoryoperatorId(unionId) on every call, goes throughdatasource.NewConnectorHTTPClient/ValidateConnectorBaseURLfor SSRF safety, and suppresses deletion detection when a traversal is incomplete so a transient listing failure cannot wipe a knowledge base.What the rebase changed
Only the four locale bundles. Upstream regenerated them in 25f3751 after this branch forked, so the positional hunks no longer applied; the same keys were re-inserted by name. The pruned
datasource.syncErrorbag is recreated for the DingTalk reason codes, which the sync-log view reads dynamically.Improvements on top
Block rendering. DingTalk emits every list item as its own top-level block, so an ordered run was numbered
1.on each item and never terminated — a following table or paragraph was absorbed into the list as a lazy continuation. The renderer now tracks the run bylistId+ level, numbers items sequentially, and closes it with a blank line.heading.levelis documented as an integer but the string forms ("2","heading-2") occur in practice and madejson.Unmarshalfail for the whole envelope, dropping the heading text along with its level; it is now decoded tolerantly. More generally, a type mismatch anywhere in a block's payload used to discard the entire block, so an unexpected field now costs the block's formatting rather than its text.Block-range paging. An omitted
successfield is no longer read as an explicit failure, since the transport already rejects non-2xx. And a server that ignoresstartIndex/endIndexnow ends pagination instead of re-reading the same blocks up to the 1000-page cap and duplicating the document.Connector localization. The picker resolves its label and description through computed keys (
datasource.connector.${def.type}), which the static locale audit cannot see, so the card rendered raw key strings in the UI while every check stayed green. Found by clicking through the running app:Connector card rendering the raw i18n key datasource.connector.dingtalk
Both entries are added to all four bundles, with a test that checks the two bags against the connector list in
DataSourceEditorDialog.vueso a future connector cannot ship an unlocalized card:Connector card correctly rendering 钉钉文档 with its description
Verification
Backend
go build ./internal/...,go vet, andgo test ./internal/datasource/... ./internal/container/...all pass, including the DingTalk package. Frontendnpm testpasses 308/308 (locale audit included) andnpm run type-checkis clean. Nine new tests cover sequential numbering, per-group restart, list-run termination, non-integer heading levels, salvaged text from undecodable blocks, the three block-range paging behaviours, and connector-name locale coverage.End-to-end through the UI on a locally built Lite instance (
make build-lite, SQLite):dingtalk_datasource_connector_walkthrough.mp4
The credentials step asks for Client ID (AppKey), Client Secret (AppSecret) and 操作人 UnionID, with the first-run setup guide spelling out the three DingTalk Open Platform steps:
DingTalk credentials form with the setup guide expanded
Known gaps (follow-ups, not regressions)
Non-ALIDOC nodes (locally uploaded files, sheets) are skipped rather than downloaded through the 钉盘 storage APIs, and the connector does not implement the optional
FetchStream, so a very large knowledge base restarts from the last cursor instead of resuming mid-traversal.To show artifacts inline, enable in settings.