diff --git a/.env.example b/.env.example index 8da0ac0aa5..55d641da97 100644 --- a/.env.example +++ b/.env.example @@ -581,6 +581,14 @@ SYSTEM_AES_KEY=weknora-system-aes-key-32bytes!! # 自定义 Skills 目录(挂载后指定,免重建镜像)。 # WEKNORA_SKILLS_DIR= +# 插件组合:profile YAML(默认 config/plugin_profile.yaml)、额外 overlay、逗号分隔启用项。 +# WEKNORA_PLUGIN_PROFILE=config/plugin_profile.yaml +# WEKNORA_PLUGIN_PATCH= +# WEKNORA_PLUGINS=websearch.echo +# 运行时插件目录(默认 plugins.d,: 或系统分隔符可写多个;none 关闭扫描)。 +# 目录里每个子文件夹放 plugin.yaml;语言插件用 runtime: stdio(stdin/stdout JSON-RPC), +# 轻脚本用 runtime: js。不要为了写插件去起 HTTP sidecar。 +# WEKNORA_PLUGIN_DIR=plugins.d # 智能体大模型调用默认超时(秒,默认 120;复杂推理调大如 300/600)。 # 注:全局默认;单个智能体在数据库配独立 llm_call_timeout 时以其为准。 # WEKNORA_AGENT_LLM_TIMEOUT=300 diff --git a/config/plugin_profile.yaml b/config/plugin_profile.yaml new file mode 100644 index 0000000000..fa65d827ad --- /dev/null +++ b/config/plugin_profile.yaml @@ -0,0 +1,22 @@ +# WeKnora plugin composition (Cordis-style profile). +# +# Layers applied onto an empty entry list: +# 1) each named bundle, in order (code-defined; "base" = in-tree web search) +# 2) this file's patch +# 3) WEKNORA_PLUGIN_PATCH overlay (optional extra YAML) +# 4) WEKNORA_PLUGIN_DIR (scan plugins.d/*/plugin.yaml → bundle "external") +# 5) WEKNORA_PLUGINS=id1,id2 (insert-if-missing convenience) +# +# A patch targets a row by id and replaces listed fields, or inserts a new +# row when insert: true. See docs/dev/plugin-architecture.md. +name: standard +bundles: + - base +# patch: +# - id: websearch.exa +# disabled: true +# - id: websearch.echo +# plugin: websearch.echo +# insert: true +# config: +# title: echo diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 3a935ae4dc..29f1647792 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -31,6 +31,8 @@ - [x] 支持与企微、飞书等 IM 系统集成,在 IM 内使用 WeKnora 能力 ## 组件与扩展 +- [x] 引入 Cordis 风格插件核(`internal/plugin`),联网搜索改为 profile/bundle 组合,不再写进 `container.go` +- [ ] 将其余扩展缝(数据源 / IM / 存储 / 模型 / 检索 / 工具 / 分块 / 解析)迁到同一 Host - [ ] 鼓励社区维护各厂商的模型服务、网络搜索服务等组件 - [ ] 鼓励社区提供更多与知识库相关的 Skills diff --git a/docs/dev/plugin-architecture.md b/docs/dev/plugin-architecture.md new file mode 100644 index 0000000000..0a522d4573 --- /dev/null +++ b/docs/dev/plugin-architecture.md @@ -0,0 +1,285 @@ +# WeKnora 插件化:从 DeepSeek Harness 学什么,怎么改 + +本文回答两件事: + +1. DeepSeek Harness(`dsh`)的 **Everything is a Plugin** 到底在解决什么问题; +2. WeKnora 现在的「扩展点都写进主仓库、注册写进 `container.go`」该如何按同一思路重构,而又不把 Go 单体硬套成 Node 动态加载器。 + +配套代码:`internal/plugin`(内核)、`internal/plugin/websearch`(第一条缝)、`plugins/websearch-echo`(树外插件样板)。 + +--- + +## 1. dsh 在说什么 + +dsh 的运行时不是「一个核心 + 一堆插件」,而是 **整棵产品都是插件树**。模型适配器、工具注册表、session 日志、agent loop、沙箱、UI 都是可替换插件。文档原话是:*there is no privileged core to patch*——扩展方式是在旁边再挂一个插件,而不是去改核心源码。 + +底座是 [Cordis](https://github.com/cordiverse/cordis)。作者需要先建立五个概念: + +| 概念 | 含义 | +| --- | --- | +| Plugin | 实现 `Service` 的对象:函数插件(`inject` + `apply(ctx)`)或 Service 子类 | +| Context | 服务仓库。插件把能力挂到稳定的 `ctx.tools` / `ctx.llm` / `ctx.sessions`,别人按 **key** 找,不 import 具体实现 | +| inject | 声明依赖的服务 key。没有这些服务就不挂载,启动顺序由依赖表达,而不是手写 boot 序列 | +| Typed events | `emit` / `waterfall` / `parallel` / `serial`。拦截和策略走事件,直接能力调用走 service 方法 | +| Reversible effects | 注册(prompt 段、工具 schema、adapter、listener)都带 disposer。卸载/热重载按反序撤销 | + +组合不是「扫目录乱加载」,而是有分层: + +```text +空 entry 列表 + → profile 列出的 bundle(如 dsh-base, dsh-web-app) + → profile 自己的 cordis.patch.yml + → home 级 overlay + → --patch +``` + +一行 patch 按 **id** 整行替换或插入。`dsh --dump-config` 打出的每一行都可以被用户覆盖。 + +还有两个和「扩展点接口」不一样的设计: + +**Capability seam(能力缝)** 必须三角齐全: + +- Service Definition:接口 + `ctx` key +- Service Provider:实现 +- Consumer:通常是模型可见的 tool + +只写一个实现、没有定义、没有消费者,不叫缝。所以换一个 `ctx.fs` / `ctx.subprocess` provider,Bash、PTY、LSP 会一起走新的执行世界,而不用给每个 tool 分叉。 + +**Session log 是模型上下文的唯一事实来源。** 能进模型请求的东西必须能从 append-only 日志重建(*Model-visible means logged*)。这是 agent harness 的约束,不是所有产品都要抄,但对 WeKnora 的对话时间轴 / 评测回放有直接借鉴。 + +dsh 用 TypeScript 工作区包(`packages//`)把 Definition / Provider / Consumer 拆开。加一个包有清单、约束脚本、README 合同(含 Model Experience)。这是「社区能在核心仓库外演进」的物理条件。 + +--- + +## 2. WeKnora 现在是什么形态 + +WeKnora **已经有扩展点,但没有插件系统**。 + +九条缝都有接口 + 注册表(见 `website-docs/06-development/03-extension-points.md`): + +| 缝 | 接口 | 今天的注册点 | +| --- | --- | --- | +| 文档解析 | `BaseParser` | `docreader/parser/registry.py` 写死 `_build_default_registry()` | +| 分块 | 包级函数变量 + `runTier` switch | `internal/infrastructure/chunker/strategy.go` | +| 检索引擎 | `RetrieveEngineRepository` | `container.go` `initRetrieveEngineRegistry()` + `RETRIEVE_DRIVER` | +| 模型 | `Provider` / `providerAdapter` | `provider.Register` + chat 适配表 | +| 联网搜索 | `WebSearchProvider` | 曾是 `container.go` `registerWebSearchProviders()`,现已迁到 plugin host | +| 数据源 | `Connector` | `container.go` `initConnectorRegistry()` + 元数据 map | +| IM | `Adapter` | `container.go` `registerIMService()` | +| Agent 工具 | `types.Tool` | `definitions.go` + 会话装配 | +| 对象存储 | `FileService` | `file/factory.go` 的 `switch` | + +另外还有一条 **问答流水线插件**:`chat_pipeline.Plugin` + `EventManager` 责任链(`next()`),语义上已经是 Cordis waterfall,但插件本身仍在 `container.Invoke(chatpipeline.NewPluginXxx)` 里写死,且和「检索引擎 / IM」不是同一套生命周期。 + +`ResourceCleaner` 已经按反序执行析构,接近 reversible effect,但只用于进程退出,不能按插件卸载。 + +这套模式的症状: + +1. **加一个 Brave Search / 一个 GitHub 连接器 = 改主仓库 + 改 `container.go`。** ROADMAP 里「鼓励社区维护各厂商组件」在物理上做不到——社区 PR 必须打进单体。 +2. **九套注册表,九套约定。** 有的 first-wins,有的 switch,有的 `init()` 覆盖函数变量,有的环境变量门控。没有统一的 dump、disable、overlay。 +3. **`container.go` 是隐藏的核心。** 它同时做 DI、条件装配、副作用启动。dsh 要消灭的正是这种「必须改核心才能扩展」的特权点。 +4. **实现和定义住在一起。** `internal/im/wecom`、`internal/infrastructure/web_search/bing.go` 都是主模块的一部分,Lite 二进制也会链上全套 SDK。 +5. **前端表单、类型常量、工厂注册经常不同步。** 扩展指南要改 4~6 个文件才算「加完」。 + +WeKnora 比 dsh 重的地方:它是带租户、迁移、RBAC、异步任务的知识库产品,不是纯 agent harness。不能把 `dig` 整棵树拆掉重写;要做的是 **在扩展缝上套一层组合核,用绞杀式(strangler)把注册中枢从 `container.go` 挪走**。 + +--- + +## 3. 对得上的映射(抄思路,不抄运行时) + +| dsh / Cordis | WeKnora 落地 | +| --- | --- | +| Plugin / `apply(ctx)` | `plugin.Plugin`(`Name` / `Inject` / `Apply`) | +| `ctx.tools` 等 key | `plugin.ServiceWebSearch` 等稳定字符串;现有 `*web_search.Registry` 作为 service | +| inject | `Plugin.Inject()`;Host 按依赖轮转挂载 | +| emit / waterfall / parallel / serial | `plugin.EventBus` 四种模式。`chat_pipeline` 的 `next()` 就是 waterfall,后续可迁到 `chat/*` | +| `ctx.effect()` | `Context.Effect` + 每插件一个 Isolate;`Registry.Unregister` 是 disposer | +| profile / bundle / patch | `config/plugin_profile.yaml` + 代码里的 `base` bundle + `WEKNORA_PLUGINS` | +| `--dump-config` | `Host.Dump()`(启动 debug 日志会打印整棵树) | +| capability seam | 继续用 `internal/types/interfaces` 当 Definition;插件只做 Provider | +| TS 动态 `import()` | **不能进 Go 进程。** 语言插件走同一套 ABI(JSON-RPC),绑定 `runtime: stdio`;轻脚本用 `runtime: js`(goja) | +| `go plugin` `.so` | **不用。** 构建标签、libc、无法跨版本,社区插件会碎 | + +Go 没有 Node 那种「`pnpm add` 完同一进程就能 `import`」。接近 TS 手感、且符合业界惯例的是: + +1. **`runtime: stdio`(推荐给任意语言)**:Host `exec` 你的进程,在 stdin/stdout 上讲 JSON-RPC 2.0。作者实现 `websearch.search`,**不要自己开端口**。MCP、LSP、Dify 本地插件、HashiCorp 系的「进程外插件」都是这条路。 +2. **`runtime: js`**:丢 `search.js`,goja 进程内执行。适合几行适配逻辑;出网走宿主 `httpRequest`(SSRF 白名单)。 +3. **Go `plugin.Register` + blank import**:只有要链进主二进制的实现才走这条。 +4. **`runtime: http`(fallback)**:对方**已经是**一个远程服务时才写 endpoint。不要为了写插件去起 sidecar。 + +热重载 / `!!js` 配置表达式仍未做:改磁盘插件后需要重启进程。 + +### 3.1 业界对照:为什么不把 HTTP 当主路径 + +「让插件作者开一个 HTTP 服务」看起来语言无关,实际多了端口、健康检查、生命周期和「谁先起来」四个问题。业界把 **协议** 和 **传输绑定** 拆开,本地插件几乎都选字节流,而不是让作者当服务器: + +| 方案 | 代表 | 插件作者写什么 | 结论 | +| --- | --- | --- | --- | +| **stdin/stdout + JSON-RPC** | [MCP stdio](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports)、LSP、[Dify 本地 runtime](https://github.com/langgenius/dify-plugin-daemon) | 读 stdin、写 stdout | **语言插件的标准。** 无端口、无 CORS、无鉴权;Host 拉起并回收进程 | +| **子进程 + gRPC / 握手** | [HashiCorp go-plugin](https://github.com/hashicorp/go-plugin)、Terraform Provider、Grafana backend plugin | `plugin.Serve`;握手行打在 stdout,再听 unix/tcp | 适合重 SDK、强类型、双向 RPC。对「一个 search 函数」过重,且作者仍在听套接字 | +| **进程内脚本** | Traefik Yaegi、Kong Lua/PDK、本仓库 goja | 一个函数 | 配置/轻逻辑最快;完整 SDK、原生依赖不合适 | +| **进程内 WASM** | Envoy proxy-wasm、Kong/APISIX WASM | 编成 `.wasm`,同一 ABI | 沙箱最好;要自己养 WASI/host ABI。下一步可以让 WASM 走**同一套** JSON-RPC(WASI stdio),而不是新协议 | +| **`.so` / `plugin.Open`** | Tyk 旧路径、Go plugin | 按本机编译器编共享库 | 编译器、libc、Go 版本钉死,社区插件会碎 | +| **HTTP sidecar** | 早期 MCP HTTP+SSE(已弃用)、自研「插件=微服务」 | 自己 listen | 多一个服务器。MCP 2025-03 已弃用 HTTP+SSE;远程场景改走 Streamable HTTP,**本地场景仍是 stdio** | + +同领域的 Dify 也拆过这件事:API 和 daemon 之间可以是 HTTP,但 **daemon 拉起的本地插件走 STDIN/STDOUT**;HTTP 留给 serverless / 已经在跑的远程运行时。WeKnora 对齐的是「作者这一侧」,不是「再让每个人写一个小网站」。 + +所以主路径是 **一份 ABI,多种绑定**: + +```text +JSON-RPC 2.0 (方法:websearch.search / shutdown;一行一条) + ├─ stdio Host exec,stdin/stdout ← 任意语言(推荐) + ├─ js goja 进程内 ← 轻脚本 + ├─ http 已有远程服务 ← fallback + └─ wasm (未做)WASI 上同一套 RPC +``` + +TS 作者的表面是 `serve({ search })`,见 `plugins/sdk-ts/websearch`。样板:`plugins.d/websearch-stdio-echo`(Python)、`plugins.d/websearch-node-echo`(Node)。 + +--- + +## 4. 目标结构 + +```text +internal/plugin/ 内核(无业务类型) +internal/plugin/websearch/ 内置联网搜索 bundle +internal/plugin/protocol/ 插件 ABI(JSON-RPC 2.0,与传输无关) +internal/plugin/runtime/ stdio / js / http 绑定 +internal/plugin/boot/ 进程 Host +plugins.d//plugin.yaml 运行时插件(默认扫描) +plugins/sdk-ts/websearch/ TS:serve() 读 stdin,不要开 HTTP +config/plugin_profile.yaml 用户可改的组成 +``` + +启动: + +```text +dig 装配 *web_search.Registry + → boot.NewHost + Provide(ServiceWebSearch, registry) + Discover(WEKNORA_PLUGIN_DIR) → RegisterManifests + Compose(base + external + profile patch + WEKNORA_PLUGINS) + → Effect: registry.Register + 动态写入 GetWebSearchProviderTypes() + → ResourceCleaner 在退出时 Host.Unload() +``` + +加一个**内置**引擎:`builtins()` 加一行。 +加一个**免编译**引擎:在 `plugins.d/mysearch/` 放 `plugin.yaml` + 可执行入口(`runtime: stdio`)或 `search.js`,重启。类型会出现在 `/web-search-providers/types`。 +**不再改 `container.go`,也不再改 `internal/plugin/boot`。** + +--- + +## 5. 分阶段重构(由外到内) + +不要一次把九条缝和 chat pipeline 全搬迁。每条缝的完成标准是: + +- Definition 仍在 `types/interfaces`(或 Python 基类) +- Provider 只通过 `plugin.Register` 出现 +- `container.go` 不再出现该缝的实现 import +- profile 能 disable / 替换这一行 +- 卸载会撤销注册 + +建议顺序(按「改 container 的收益 / 行为风险」): + +| 阶段 | 缝 | 为什么先做 | +| --- | --- | --- | +| **0(已做)** | 内核 + 联网搜索 | 工厂表最干净,租户参数运行时实例化,正好当样板 | +| **1** | 数据源连接器、IM 适配器 | 已经是 `Register(factory)`,和搜索同构;`container.go` 里 import 最多 | +| **2** | 对象存储 `factory.go` switch、模型 Provider | switch / 全局 map 改成 seam registry | +| **3** | 检索引擎 | 注意 `RETRIEVE_DRIVER`、`EngineFactory`、租户 `vector_stores` 运行时建连 | +| **4** | Agent 工具 + chat_pipeline | 工具已有 Registry;pipeline 已是 waterfall,改成 `chat/*` 事件即可与内核合流 | +| **5** | 分块策略、docreader 解析器 | 分块是函数变量;解析器在 Python 进程,需要独立 plugin 清单或 gRPC 能力协商 | +| **6** | 前端缝 | 搜索/连接器/IM 的表单按插件元数据渲染,避免再改 Vue 才能「加完」 | +| **7** | 包边界 | 低频实现迁到独立 Go module;Lite 用 build tag 或 profile 去掉重 SDK | +| **8(搜索已做)** | 进程外 / 磁盘加载 | 主路径 `runtime: stdio`(JSON-RPC);`js` 轻脚本;`http` 仅 fallback | + +每阶段保持绞杀:旧接口不变,Host 先 Provide 现有 Registry,再让插件往上注册。 + +### 不要做的事 + +- 不要用 `plugin.Open`(Go `.so`)当社区分发手段。 +- 不要把 dig 换成自研 DI。dig 继续管 Handler/Service/DB;plugin Host 只管 **可替换能力**。 +- 不要把业务规则(RBAC、配额、SSRF 白名单)做成「可卸载插件」。策略可以 listen 事件,但不能让社区插件关掉安全底线。 +- 不要在第一阶段追求热重载。磁盘插件改完重启即可;HMR 是 Cordis 的 TS 特权。 + +--- + +## 6. 今天怎么加一个联网搜索插件 + +内置(仍在本仓库,但不再碰 container): + +1. `internal/infrastructure/web_search/brave.go` 实现接口(API URL 硬编码)。 +2. `internal/types/web_search_provider.go` 加类型常量与 `GetWebSearchProviderTypes()` 元数据(前端下拉仍读这里,阶段 6 再改成插件清单)。 +3. `internal/plugin/websearch/plugins.go` 的 `builtins()` 加 `{"brave", web_search.NewBraveProvider}`。 + +免编译(推荐给社区 / TS): + +```bash +# 默认扫描 plugins.d/。 +# 任意语言:plugins.d/websearch-stdio-echo/(Python)或 sdk-ts 的 serve() +# 轻脚本:plugins.d/websearch-js-echo/ +``` + +Go 样板(仍要编进二进制时): + +```bash +WEKNORA_PLUGINS=websearch.echo +``` + +或在 `config/plugin_profile.yaml`: + +```yaml +patch: + - id: websearch.echo + plugin: websearch.echo + insert: true + config: + title: echo +``` + +关掉某个内置引擎: + +```yaml +patch: + - id: websearch.exa + disabled: true +``` + +环境变量: + +| 变量 | 作用 | +| --- | --- | +| `WEKNORA_PLUGIN_PROFILE` | profile 路径,默认 `config/plugin_profile.yaml` | +| `WEKNORA_PLUGIN_PATCH` | 额外 overlay YAML(只读其中的 `patch`) | +| `WEKNORA_PLUGINS` | 逗号分隔 factory id,insert-if-missing | +| `WEKNORA_PLUGIN_DIR` | 运行时扫描目录,默认 `plugins.d`;`none` 关闭 | + +--- + +## 7. 和 chat_pipeline「插件」的关系 + +`internal/application/service/chat_pipeline` 的 `Plugin.OnEvent(..., next)` 已经是 waterfall。它解决的是 **一次问答的阶段编排**,不是 **进程级能力组合**。两者要合并,而不是互相替代: + +- 进程级:谁提供搜索 / 检索 / IM(本文的 Host) +- 请求级:`QUERY_UNDERSTAND` → `CHUNK_SEARCH` → …(现有 EventManager) + +阶段 4 可以把每个 `NewPluginXxx` 改成一个 `plugin.Plugin`,在 `Apply` 里 `ctx.Events().On("chat/chunk_search", ...)`,`EventManager.Trigger` 改成 `Waterfall`。在此之前不要动问答语义。 + +--- + +## 8. 验收 + +内核与第一条缝的自动化测试: + +```bash +go test ./internal/plugin/... ./internal/plugin/websearch/... ./internal/plugin/boot/... \ + ./plugins/websearch-echo/... ./internal/infrastructure/web_search/ +``` + +期望: + +- Context Provide / 覆盖 / 卸载还原 +- waterfall 可短路,effect 反序撤销 +- profile patch 能 disable `websearch.exa` +- `WEKNORA_PLUGINS=websearch.echo` 能挂上 `echo` 而不改 container +- `Host.Unload` 之后 `Registry.Has("duckduckgo") == false` diff --git "a/docs/wiki/\351\233\206\346\210\220\346\211\251\345\261\225/\346\267\273\345\212\240\347\275\221\347\273\234\346\220\234\347\264\242\345\274\225\346\223\216.md" "b/docs/wiki/\351\233\206\346\210\220\346\211\251\345\261\225/\346\267\273\345\212\240\347\275\221\347\273\234\346\220\234\347\264\242\345\274\225\346\223\216.md" index 50f3b0d79b..2dfdf23cc2 100644 --- "a/docs/wiki/\351\233\206\346\210\220\346\211\251\345\261\225/\346\267\273\345\212\240\347\275\221\347\273\234\346\220\234\347\264\242\345\274\225\346\223\216.md" +++ "b/docs/wiki/\351\233\206\346\210\220\346\211\251\345\261\225/\346\267\273\345\212\240\347\275\221\347\273\234\346\220\234\347\264\242\345\274\225\346\223\216.md" @@ -16,7 +16,7 @@ source: 添加新的网络搜索引擎.md ``` internal/types/web_search_provider.go # 实体定义 + Provider 类型元数据 internal/infrastructure/web_search/ # Provider 实现(bing/google/duckduckgo/tavily) -internal/container/container.go # DI 注册 +internal/plugin/websearch/plugins.go # 插件 Host 注册 internal/types/interfaces/web_search.go # WebSearchProvider 接口 ``` @@ -58,9 +58,9 @@ WebSearchProviderTypeBrave WebSearchProviderType = "brave" 在 `isValidProviderType()` 中添加新类型。 -### 5. DI 注册 +### 5. 插件注册 -在 `registerWebSearchProviders` 中注册。 +在 `internal/plugin/websearch/plugins.go` 的 `builtins()` 中加一行。不要改 `container.go`。树外插件见 `docs/dev/plugin-architecture.md`。 ### 6. 验证 @@ -84,7 +84,7 @@ curl http://localhost:8080/api/v1/web-search-providers/types | `internal/types/web_search_provider.go` | 添加常量 + 类型元数据 | | `internal/infrastructure/web_search/brave.go` | **新建** Provider 实现 | | `internal/application/service/web_search_provider.go` | `isValidProviderType` 加新类型 | -| `internal/container/container.go` | `registerWebSearchProviders` 注册 | +| `internal/plugin/websearch/plugins.go` | `builtins()` 注册 | ## 相关主题 diff --git "a/docs/wiki/\351\241\271\347\233\256\346\246\202\350\277\260/\347\211\210\346\234\254\350\267\257\347\272\277\345\233\276.md" "b/docs/wiki/\351\241\271\347\233\256\346\246\202\350\277\260/\347\211\210\346\234\254\350\267\257\347\272\277\345\233\276.md" index 9d542effb8..79e5748180 100644 --- "a/docs/wiki/\351\241\271\347\233\256\346\246\202\350\277\260/\347\211\210\346\234\254\350\267\257\347\272\277\345\233\276.md" +++ "b/docs/wiki/\351\241\271\347\233\256\346\246\202\350\277\260/\347\211\210\346\234\254\350\267\257\347\272\277\345\233\276.md" @@ -55,6 +55,8 @@ source: ROADMAP.md ## 组件与扩展 +- [x] 引入 Cordis 风格插件核(`internal/plugin`),联网搜索改为 profile/bundle 组合,不再写进 `container.go` +- [ ] 将其余扩展缝(数据源 / IM / 存储 / 模型 / 检索 / 工具 / 分块 / 解析)迁到同一 Host - [ ] 鼓励社区维护各厂商的模型服务、网络搜索服务等组件 - [ ] 鼓励社区提供更多与知识库相关的 Skills diff --git "a/docs/\346\267\273\345\212\240\346\226\260\347\232\204\347\275\221\347\273\234\346\220\234\347\264\242\345\274\225\346\223\216.md" "b/docs/\346\267\273\345\212\240\346\226\260\347\232\204\347\275\221\347\273\234\346\220\234\347\264\242\345\274\225\346\223\216.md" index 83e76abcce..c6eec090d3 100644 --- "a/docs/\346\267\273\345\212\240\346\226\260\347\232\204\347\275\221\347\273\234\346\220\234\347\264\242\345\274\225\346\223\216.md" +++ "b/docs/\346\267\273\345\212\240\346\226\260\347\232\204\347\275\221\347\273\234\346\220\234\347\264\242\345\274\225\346\223\216.md" @@ -15,8 +15,8 @@ internal/ │ ├── google.go # Google 实现 │ ├── duckduckgo.go # DuckDuckGo 实现 │ └── tavily.go # Tavily 实现 -├── container/ -│ └── container.go # DI 注册(registerWebSearchProviders) +├── plugin/ +│ └── websearch/plugins.go # 进程级注册(builtins + plugin.Host) └── types/interfaces/ └── web_search.go # WebSearchProvider 接口 ``` @@ -187,19 +187,21 @@ func isValidProviderType(provider types.WebSearchProviderType) bool { } ``` -### 5. 在 DI 容器中注册 +### 5. 在插件 bundle 中注册 -编辑 `internal/container/container.go` 的 `registerWebSearchProviders` 函数: +编辑 `internal/plugin/websearch/plugins.go` 的 `builtins()`(不要改 `container.go`): ```go -func registerWebSearchProviders(registry *infra_web_search.Registry) { - // ... 已有注册 ... - - // Register Brave provider type - registry.Register(infra_web_search.BraveProviderTypeInfo(), infra_web_search.NewBraveProvider) +func builtins() []spec { + return []spec{ + // ... 已有引擎 ... + {"brave", web_search.NewBraveProvider}, + } } ``` +树外插件用 `plugin.Register` + profile / `WEKNORA_PLUGINS`,见 `docs/dev/plugin-architecture.md`。 + ### 6. 验证 ```bash @@ -264,4 +266,4 @@ type WebSearchProviderParameters struct { | `internal/types/web_search_provider.go` | 添加常量 + 类型元数据 | | `internal/infrastructure/web_search/brave.go` | **新建** Provider 实现 | | `internal/application/service/web_search_provider.go` | `isValidProviderType` 加新类型 | -| `internal/container/container.go` | `registerWebSearchProviders` 注册 | +| `internal/plugin/websearch/plugins.go` | `builtins()` 注册 | diff --git a/go.mod b/go.mod index 150a3fb35b..3536b8eea6 100644 --- a/go.mod +++ b/go.mod @@ -92,6 +92,13 @@ require ( gorm.io/gorm v1.31.1 ) +require ( + github.com/dlclark/regexp2/v2 v2.5.2 // indirect + github.com/dop251/goja v0.0.0-20260806115107-493f22071ef6 // indirect + github.com/go-sourcemap/sourcemap v2.1.3+incompatible // indirect + github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db // indirect +) + require ( cloud.google.com/go/auth v0.20.0 // indirect cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect diff --git a/go.sum b/go.sum index c17f12be65..de4b69d24c 100644 --- a/go.sum +++ b/go.sum @@ -1602,6 +1602,8 @@ github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5Qvfr github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E= github.com/dlclark/regexp2 v1.11.5 h1:Q/sSnsKerHeCkc/jSTNq1oCm7KiVgUMZRDUoRu0JQZQ= github.com/dlclark/regexp2 v1.11.5/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8= +github.com/dlclark/regexp2/v2 v2.5.2 h1:HAsucWRhsqcDzl6Ua9aR8JwYOTzrZyPrF0/FNxJVAI0= +github.com/dlclark/regexp2/v2 v2.5.2/go.mod h1:avUrQvPaLz2DrFNHJF0taWAFFX2C1GMSSoeiqFjcBmU= github.com/docker/docker v28.5.2+incompatible h1:DBX0Y0zAjZbSrm1uzOkdr1onVghKaftjlSWt4AFexzM= github.com/docker/docker v28.5.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= github.com/docker/go-connections v0.7.0 h1:6SsRfJddP22WMrCkj19x9WKjEDTB+ahsdiGYf0mN39c= @@ -1609,6 +1611,8 @@ github.com/docker/go-connections v0.7.0/go.mod h1:no1qkHdjq7kLMGUXYAduOhYPSJxxvg github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4= github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE= +github.com/dop251/goja v0.0.0-20260806115107-493f22071ef6 h1:Oh2rRG1un7tLlC3/NJDzKppZ4CeZGkVFJCUOTRwLpfw= +github.com/dop251/goja v0.0.0-20260806115107-493f22071ef6/go.mod h1:LiIEzozrcvNXorsG/3+ypGqdTUAqZryhzSsqi0oU/Qg= github.com/duckdb/duckdb-go-bindings v0.10502.0 h1:Uhg/dfvPLQv4cH35lMD48hqUcdOh2Z7bcuykjr4qnOA= github.com/duckdb/duckdb-go-bindings v0.10502.0/go.mod h1:8KF3oEKrmYdSbZnQ1BPTdxAZDHRaM1LEv+oBvL2nSLk= github.com/duckdb/duckdb-go-bindings/lib/darwin-amd64 v0.10502.0 h1:1GxSHSI1ef3sCdDVrJ9l8s6aTd7P1K788os9lHrs43g= @@ -1819,6 +1823,8 @@ github.com/go-playground/validator/v10 v10.30.1 h1:f3zDSN/zOma+w6+1Wswgd9fLkdwy0 github.com/go-playground/validator/v10 v10.30.1/go.mod h1:oSuBIQzuJxL//3MelwSLD5hc2Tu889bF0Idm9Dg26cM= github.com/go-shiori/dom v0.0.0-20230515143342-73569d674e1c h1:wpkoddUomPfHiOziHZixGO5ZBS73cKqVzZipfrLmO1w= github.com/go-shiori/dom v0.0.0-20230515143342-73569d674e1c/go.mod h1:oVDCh3qjJMLVUSILBRwrm+Bc6RNXGZYtoh9xdvf1ffM= +github.com/go-sourcemap/sourcemap v2.1.3+incompatible h1:W1iEw64niKVGogNgBN3ePyLFfuisuzeidWPMPWmECqU= +github.com/go-sourcemap/sourcemap v2.1.3+incompatible/go.mod h1:F8jJfvm2KbVjc5NqelyYJmf/v5J0dwNLS2mL4sNA1Jg= github.com/go-sql-driver/mysql v1.10.0 h1:Q+1LV8DkHJvSYAdR83XzuhDaTykuDx0l6fkXxoWCWfw= github.com/go-sql-driver/mysql v1.10.0/go.mod h1:M+cqaI7+xxXGG9swrdeUIoPG3Y3KCkF0pZej+SK+nWk= github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY= @@ -1973,6 +1979,8 @@ github.com/google/pprof v0.0.0-20210601050228-01bbb1931b22/go.mod h1:kpwsk12EmLe github.com/google/pprof v0.0.0-20210609004039-a478d1d731e9/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/pprof v0.0.0-20221118152302-e6195bd50e26/go.mod h1:dDKJzRmX4S37WGHujM7tX//fmj1uioxKzKxz3lo4HJo= +github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db h1:097atOisP2aRj7vFgYQBbFN4U4JNXUNYpxael3UzMyo= +github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db/go.mod h1:vavhavw2zAxS5dIdcRluK6cSGGPlZynqzFM8NdvU144= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= github.com/google/s2a-go v0.1.0/go.mod h1:OJpEgntRZo8ugHpF9hkoLJbS5dSI20XZeXJ9JVywLlM= github.com/google/s2a-go v0.1.3/go.mod h1:Ej+mSEMGRnqRzjc7VtF+jdBwYG5fuJfiZ8ELkjEwM0A= diff --git a/internal/application/service/web_search_provider.go b/internal/application/service/web_search_provider.go index cb472f4dd9..c96a4132d1 100644 --- a/internal/application/service/web_search_provider.go +++ b/internal/application/service/web_search_provider.go @@ -128,22 +128,7 @@ func (s *webSearchProviderService) DeleteProvider(ctx context.Context, tenantID // isValidProviderType checks if the given provider type is supported func isValidProviderType(provider types.WebSearchProviderType) bool { - switch provider { - case types.WebSearchProviderTypeBing, - types.WebSearchProviderTypeGoogle, - types.WebSearchProviderTypeDuckDuckGo, - types.WebSearchProviderTypeTavily, - types.WebSearchProviderTypeOllama, - types.WebSearchProviderTypeBaidu, - types.WebSearchProviderTypeSearxng, - types.WebSearchProviderTypeKeenable, - types.WebSearchProviderTypeMetaso, - types.WebSearchProviderTypeZhipu, - types.WebSearchProviderTypeExa: - return true - default: - return false - } + return types.IsKnownWebSearchProviderType(string(provider)) } // validateProviderParameters validates required parameters for each provider type @@ -192,6 +177,11 @@ func validateProviderParameters(provider types.WebSearchProviderType, params typ if err := infra_web_search.ValidateSearxngBaseURL(params.BaseURL); err != nil { return err } + default: + if info, ok := types.LookupWebSearchProviderType(string(provider)); ok && + info.RequiresAPIKey && params.APIKey == "" { + return fmt.Errorf("API key is required for %s provider", provider) + } } if err := validateOptionalProxyURL(params.ProxyURL); err != nil { return err diff --git a/internal/container/container.go b/internal/container/container.go index 52954a17ab..9c949d52c4 100644 --- a/internal/container/container.go +++ b/internal/container/container.go @@ -84,6 +84,7 @@ import ( "github.com/Tencent/WeKnora/internal/models/embedding" "github.com/Tencent/WeKnora/internal/models/limiter" "github.com/Tencent/WeKnora/internal/models/utils/ollama" + pluginboot "github.com/Tencent/WeKnora/internal/plugin/boot" "github.com/Tencent/WeKnora/internal/router" "github.com/Tencent/WeKnora/internal/storageallowlist" "github.com/Tencent/WeKnora/internal/stream" @@ -249,7 +250,8 @@ func BuildContainer(container *dig.Container) *dig.Container { // Web search service (needed by AgentService) logger.Debugf(ctx, "[Container] Registering web search registry and providers...") must(container.Provide(infra_web_search.NewRegistry)) - must(container.Invoke(registerWebSearchProviders)) + must(container.Provide(pluginboot.NewHost)) + must(container.Invoke(pluginboot.Start)) must(container.Provide(repository.NewWebSearchProviderRepository)) must(container.Provide(repository.NewVectorStoreRepository)) must(container.Provide(repository.NewStorageBackendRepository)) @@ -1590,23 +1592,6 @@ func NewDuckDB() (*sql.DB, error) { return sqlDB, nil } -// registerWebSearchProviders registers all web search provider types to the registry. -// Each provider type is registered with its factory function that accepts parameters. -// Provider instances are created on-demand when tenants configure them. -func registerWebSearchProviders(registry *infra_web_search.Registry) { - registry.Register("duckduckgo", infra_web_search.NewDuckDuckGoProvider) - registry.Register("google", infra_web_search.NewGoogleProvider) - registry.Register("bing", infra_web_search.NewBingProvider) - registry.Register("tavily", infra_web_search.NewTavilyProvider) - registry.Register("ollama", infra_web_search.NewOllamaProvider) - registry.Register("baidu", infra_web_search.NewBaiduProvider) - registry.Register("searxng", infra_web_search.NewSearxngProvider) - registry.Register("keenable", infra_web_search.NewKeenableProvider) - registry.Register("zhipu", infra_web_search.NewZhipuProvider) - registry.Register("exa", infra_web_search.NewExaProvider) - registry.Register("metaso", infra_web_search.NewMetasoProvider) -} - // registerIMService registers adapter factories, loads enabled channels, and // wires the process-lifetime shutdown hook. Each platform's factory lives in // its own subpackage to keep this file focused on wiring. diff --git a/internal/infrastructure/web_search/registry.go b/internal/infrastructure/web_search/registry.go index 16e2739595..bad2699019 100644 --- a/internal/infrastructure/web_search/registry.go +++ b/internal/infrastructure/web_search/registry.go @@ -33,6 +33,33 @@ func (r *Registry) Register(id string, factory ProviderFactory) { r.factories[id] = factory } +// Unregister removes a provider type factory. It is the reverse of Register +// and is used when a plugin isolate unloads. +func (r *Registry) Unregister(id string) { + r.mu.Lock() + defer r.mu.Unlock() + delete(r.factories, id) +} + +// Has reports whether a provider type is registered. +func (r *Registry) Has(id string) bool { + r.mu.RLock() + defer r.mu.RUnlock() + _, ok := r.factories[id] + return ok +} + +// List returns registered provider type IDs in unspecified order. +func (r *Registry) List() []string { + r.mu.RLock() + defer r.mu.RUnlock() + out := make([]string, 0, len(r.factories)) + for id := range r.factories { + out = append(out, id) + } + return out +} + // CreateProvider creates a provider instance by type with the given parameters. func (r *Registry) CreateProvider(providerType string, params types.WebSearchProviderParameters) (interfaces.WebSearchProvider, error) { r.mu.RLock() diff --git a/internal/infrastructure/web_search/registry_test.go b/internal/infrastructure/web_search/registry_test.go new file mode 100644 index 0000000000..005b691f62 --- /dev/null +++ b/internal/infrastructure/web_search/registry_test.go @@ -0,0 +1,40 @@ +package web_search + +import ( + "context" + "testing" + + "github.com/Tencent/WeKnora/internal/types" + "github.com/Tencent/WeKnora/internal/types/interfaces" +) + +type stubProvider struct{ name string } + +func (s stubProvider) Name() string { return s.name } +func (s stubProvider) Search(context.Context, string, int, bool) ([]*types.WebSearchResult, error) { + return nil, nil +} + +func TestRegistryRegisterHasListUnregister(t *testing.T) { + r := NewRegistry() + r.Register("echo", func(types.WebSearchProviderParameters) (interfaces.WebSearchProvider, error) { + return stubProvider{name: "echo"}, nil + }) + if !r.Has("echo") { + t.Fatal("expected echo to be registered") + } + if got := r.List(); len(got) != 1 || got[0] != "echo" { + t.Fatalf("list = %v", got) + } + p, err := r.CreateProvider("echo", types.WebSearchProviderParameters{}) + if err != nil || p.Name() != "echo" { + t.Fatalf("create = %v, %v", p, err) + } + r.Unregister("echo") + if r.Has("echo") { + t.Fatal("echo should be gone") + } + if _, err := r.CreateProvider("echo", types.WebSearchProviderParameters{}); err == nil { + t.Fatal("expected missing provider error") + } +} diff --git a/internal/plugin/boot/host.go b/internal/plugin/boot/host.go new file mode 100644 index 0000000000..2ee69596cb --- /dev/null +++ b/internal/plugin/boot/host.go @@ -0,0 +1,167 @@ +// Package boot wires the plugin Host into WeKnora: it publishes existing +// registries as Context services, stacks the base bundle, then applies +// disk plugins from WEKNORA_PLUGIN_DIR, config/plugin_profile.yaml and +// WEKNORA_PLUGINS overlays. +package boot + +import ( + "context" + "fmt" + "os" + "strings" + + infra_web_search "github.com/Tencent/WeKnora/internal/infrastructure/web_search" + "github.com/Tencent/WeKnora/internal/logger" + "github.com/Tencent/WeKnora/internal/plugin" + pluginruntime "github.com/Tencent/WeKnora/internal/plugin/runtime" + "github.com/Tencent/WeKnora/internal/plugin/websearch" + "github.com/Tencent/WeKnora/internal/types/interfaces" + + _ "github.com/Tencent/WeKnora/plugins/websearch-echo" +) + +const ( + envProfile = "WEKNORA_PLUGIN_PROFILE" + envPlugins = "WEKNORA_PLUGINS" + envPatch = "WEKNORA_PLUGIN_PATCH" + envPluginDir = "WEKNORA_PLUGIN_DIR" +) + +// NewHost constructs and composes the process plugin tree. +func NewHost(registry *infra_web_search.Registry, cleaner interfaces.ResourceCleaner) (*plugin.Host, error) { + host := plugin.NewHost() + host.Context().Provide(plugin.ServiceWebSearch, registry) + ctxLog := context.Background() + + disk, err := plugin.Discover(plugin.ParsePluginDirs(os.Getenv(envPluginDir))) + if err != nil { + return nil, err + } + if err := pluginruntime.RegisterManifests(disk); err != nil { + return nil, err + } + + profile, err := loadProfile() + if err != nil { + return nil, err + } + bundles := websearch.Bundles() + ext := plugin.BundleFromManifests(disk) + ext.Entries = dropKnownIDs(ext.Entries, bundles[websearch.BundleName], ctxLog) + bundles[plugin.ExternalBundle] = ext + if !containsString(profile.Bundles, plugin.ExternalBundle) { + profile.Bundles = append(profile.Bundles, plugin.ExternalBundle) + } + + extra, err := extraPatches() + if err != nil { + return nil, err + } + if err := host.Compose(profile, bundles, extra); err != nil { + return nil, err + } + + ctx := context.Background() + enabled := 0 + for _, m := range host.Mounted() { + if !m.Disabled { + enabled++ + } + } + logger.Infof(ctx, "[Plugin] mounted %d plugins (%d rows, %d disk)", enabled, len(host.Mounted()), len(disk)) + for _, m := range host.Mounted() { + if m.Disabled { + logger.Debugf(ctx, "[Plugin] disabled %s", m.ID) + continue + } + logger.Debugf(ctx, "[Plugin] enabled %s", m.ID) + } + logger.Debugf(ctx, "[Plugin] dump:\n%s", host.Dump()) + + if cleaner != nil { + cleaner.RegisterWithName("PluginHost", func() error { + host.Unload() + return nil + }) + } + return host, nil +} + +// Start is a dig Invoke hook so the host is constructed at process boot. +func Start(h *plugin.Host) { + if h == nil { + logger.Warnf(context.Background(), "[Plugin] host was not constructed") + } +} + +func loadProfile() (plugin.Profile, error) { + fallback := websearch.DefaultProfile() + path := strings.TrimSpace(os.Getenv(envProfile)) + if path == "" { + path = "config/plugin_profile.yaml" + } + loaded, err := plugin.LoadProfile(path, true) + if err != nil { + return plugin.Profile{}, err + } + if loaded == nil { + return fallback, nil + } + if len(loaded.Bundles) == 0 { + loaded.Bundles = fallback.Bundles + } + if loaded.Name == "" || loaded.Name == "unnamed" { + loaded.Name = fallback.Name + } + return *loaded, nil +} + +func extraPatches() ([]plugin.Patch, error) { + var out []plugin.Patch + if raw := strings.TrimSpace(os.Getenv(envPlugins)); raw != "" { + for _, name := range strings.Split(raw, ",") { + name = strings.TrimSpace(name) + if name == "" { + continue + } + out = append(out, plugin.Patch{ID: name, Plugin: name, Insert: true}) + } + } + path := strings.TrimSpace(os.Getenv(envPatch)) + if path == "" { + return out, nil + } + overlay, err := plugin.LoadProfile(path, false) + if err != nil { + return nil, fmt.Errorf("plugin overlay: %w", err) + } + if overlay != nil { + out = append(out, overlay.Patch...) + } + return out, nil +} + +func dropKnownIDs(entries []plugin.Entry, base plugin.Bundle, ctx context.Context) []plugin.Entry { + known := make(map[string]struct{}, len(base.Entries)) + for _, e := range base.Entries { + known[e.ID] = struct{}{} + } + var out []plugin.Entry + for _, e := range entries { + if _, ok := known[e.ID]; ok { + logger.Warnf(ctx, "[Plugin] skip disk plugin %s: id already in bundle %s", e.ID, base.Name) + continue + } + out = append(out, e) + } + return out +} + +func containsString(list []string, want string) bool { + for _, s := range list { + if s == want { + return true + } + } + return false +} diff --git a/internal/plugin/boot/host_test.go b/internal/plugin/boot/host_test.go new file mode 100644 index 0000000000..0e64ce0da1 --- /dev/null +++ b/internal/plugin/boot/host_test.go @@ -0,0 +1,153 @@ +package boot + +import ( + "context" + "os" + "path/filepath" + "testing" + + infra_web_search "github.com/Tencent/WeKnora/internal/infrastructure/web_search" + "github.com/Tencent/WeKnora/internal/types" + "github.com/Tencent/WeKnora/internal/types/interfaces" +) + +type nopCleaner struct { + n int +} + +func (c *nopCleaner) Register(cleanup types.CleanupFunc) { + if cleanup != nil { + c.n++ + } +} + +func (c *nopCleaner) RegisterWithName(_ string, cleanup types.CleanupFunc) { + c.Register(cleanup) +} + +func (c *nopCleaner) Cleanup(context.Context) []error { return nil } + +var _ interfaces.ResourceCleaner = (*nopCleaner)(nil) + +func isolatePlugins(t *testing.T) { + t.Helper() + t.Setenv(envPluginDir, "none") + t.Setenv(envProfile, filepath.Join(t.TempDir(), "missing.yaml")) + t.Setenv(envPlugins, "") + t.Setenv(envPatch, "") +} + +func TestNewHostMountsBuiltinSearch(t *testing.T) { + isolatePlugins(t) + + reg := infra_web_search.NewRegistry() + cleaner := &nopCleaner{} + host, err := NewHost(reg, cleaner) + if err != nil { + t.Fatal(err) + } + if !reg.Has("duckduckgo") || !reg.Has("metaso") { + t.Fatalf("builtins = %v", reg.List()) + } + if reg.Has("echo") { + t.Fatal("echo should stay off unless WEKNORA_PLUGINS enables it") + } + if cleaner.n != 1 { + t.Fatalf("cleaner registrations = %d", cleaner.n) + } + Start(host) + host.Unload() + if reg.Has("duckduckgo") { + t.Fatal("unload should remove builtins") + } +} + +func TestNewHostEnablesEchoViaEnv(t *testing.T) { + isolatePlugins(t) + t.Setenv(envPlugins, "websearch.echo") + + reg := infra_web_search.NewRegistry() + host, err := NewHost(reg, nil) + if err != nil { + t.Fatal(err) + } + if !reg.Has("echo") { + t.Fatalf("echo not mounted, list=%v", reg.List()) + } + p, err := reg.CreateProvider("echo", types.WebSearchProviderParameters{}) + if err != nil { + t.Fatal(err) + } + if p.Name() != "echo" { + t.Fatalf("name = %s", p.Name()) + } + host.Unload() +} + +func TestLoadProfileUsesYAML(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "profile.yaml") + data := []byte("name: lab\nbundles: [base]\npatch:\n - id: websearch.exa\n disabled: true\n") + if err := os.WriteFile(path, data, 0o644); err != nil { + t.Fatal(err) + } + t.Setenv(envPluginDir, "none") + t.Setenv(envProfile, path) + t.Setenv(envPlugins, "") + t.Setenv(envPatch, "") + + reg := infra_web_search.NewRegistry() + host, err := NewHost(reg, nil) + if err != nil { + t.Fatal(err) + } + if reg.Has("exa") { + t.Fatal("yaml patch should disable exa") + } + if !reg.Has("bing") { + t.Fatal("bing should remain") + } + host.Unload() +} + +func TestNewHostLoadsDiskJS(t *testing.T) { + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, "plugin.yaml"), []byte(` +id: websearch.bootjs +name: Boot JS +seam: web_search +runtime: js +entry: search.js +`), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "search.js"), []byte(` +function search(query) { + return [{ title: "boot", url: "https://weknora.local/boot", snippet: query, source: "bootjs" }]; +} +`), 0o644); err != nil { + t.Fatal(err) + } + t.Setenv(envPluginDir, dir) + t.Setenv(envProfile, filepath.Join(t.TempDir(), "missing.yaml")) + t.Setenv(envPlugins, "") + t.Setenv(envPatch, "") + + reg := infra_web_search.NewRegistry() + host, err := NewHost(reg, nil) + if err != nil { + t.Fatal(err) + } + if !reg.Has("bootjs") { + t.Fatalf("disk plugin missing, list=%v dump=\n%s", reg.List(), host.Dump()) + } + p, err := reg.CreateProvider("bootjs", types.WebSearchProviderParameters{}) + if err != nil { + t.Fatal(err) + } + got, err := p.Search(context.Background(), "disk", 1, false) + if err != nil || len(got) != 1 || got[0].Source != "bootjs" { + t.Fatalf("search = %+v, %v", got, err) + } + host.Unload() +} diff --git a/internal/plugin/catalog.go b/internal/plugin/catalog.go new file mode 100644 index 0000000000..bd446ca08b --- /dev/null +++ b/internal/plugin/catalog.go @@ -0,0 +1,54 @@ +package plugin + +import ( + "fmt" + "sync" +) + +var ( + catalogMu sync.RWMutex + catalog = map[string]Factory{} +) + +// Register adds a factory to the process-wide catalog. Duplicate names +// keep the first registration (first-wins) so a later import cannot hijack +// a built-in plugin id. +func Register(name string, factory Factory) { + if name == "" || factory == nil { + return + } + catalogMu.Lock() + defer catalogMu.Unlock() + if _, exists := catalog[name]; exists { + return + } + catalog[name] = factory +} + +// LookupFactory returns a catalog factory by name. +func LookupFactory(name string) (Factory, bool) { + catalogMu.RLock() + defer catalogMu.RUnlock() + f, ok := catalog[name] + return f, ok +} + +// CatalogNames returns registered factory names in unspecified order. +func CatalogNames() []string { + catalogMu.RLock() + defer catalogMu.RUnlock() + out := make([]string, 0, len(catalog)) + for name := range catalog { + out = append(out, name) + } + return out +} + +// MustLookupFactory returns a factory or an error naming the missing id. +func MustLookupFactory(name string) (Factory, error) { + f, ok := LookupFactory(name) + if !ok { + return nil, fmt.Errorf("plugin: factory %q is not registered", name) + } + return f, nil +} diff --git a/internal/plugin/compose.go b/internal/plugin/compose.go new file mode 100644 index 0000000000..a260cfe094 --- /dev/null +++ b/internal/plugin/compose.go @@ -0,0 +1,137 @@ +package plugin + +import ( + "fmt" + "os" + + "gopkg.in/yaml.v3" +) + +// Entry is one row in a composed plugin tree. +type Entry struct { + ID string `yaml:"id"` + Plugin string `yaml:"plugin,omitempty"` + Config Config `yaml:"config,omitempty"` + Disabled bool `yaml:"disabled,omitempty"` + Isolate bool `yaml:"isolate,omitempty"` +} + +// FactoryName is the catalog key. Empty Plugin falls back to ID. +func (e Entry) FactoryName() string { + if e.Plugin != "" { + return e.Plugin + } + return e.ID +} + +// Bundle is a named list of entries that a profile can stack. +type Bundle struct { + Name string `yaml:"name"` + Entries []Entry `yaml:"entries"` +} + +// Patch replaces or inserts one entry by id. +type Patch struct { + ID string `yaml:"id"` + Plugin string `yaml:"plugin,omitempty"` + Config Config `yaml:"config,omitempty"` + Disabled *bool `yaml:"disabled,omitempty"` + Isolate *bool `yaml:"isolate,omitempty"` + Insert bool `yaml:"insert,omitempty"` +} + +// Profile is a named composition: stacked bundles plus optional patches. +type Profile struct { + Name string `yaml:"name"` + Bundles []string `yaml:"bundles"` + Patch []Patch `yaml:"patch,omitempty"` +} + +// ApplyPatches overlays patches onto entries. A patch with Insert appends +// when the id is missing; otherwise the matching row is replaced field-wise. +func ApplyPatches(entries []Entry, patches []Patch) []Entry { + index := make(map[string]int, len(entries)) + for i, e := range entries { + index[e.ID] = i + } + out := append([]Entry(nil), entries...) + for _, p := range patches { + if p.ID == "" { + continue + } + i, ok := index[p.ID] + if !ok { + if !p.Insert { + continue + } + e := Entry{ID: p.ID, Plugin: p.Plugin, Config: p.Config} + if p.Disabled != nil { + e.Disabled = *p.Disabled + } + if p.Isolate != nil { + e.Isolate = *p.Isolate + } + index[p.ID] = len(out) + out = append(out, e) + continue + } + e := out[i] + if p.Plugin != "" { + e.Plugin = p.Plugin + } + if p.Config != nil { + e.Config = p.Config + } + if p.Disabled != nil { + e.Disabled = *p.Disabled + } + if p.Isolate != nil { + e.Isolate = *p.Isolate + } + out[i] = e + } + return out +} + +// StackBundles concatenates named bundles in order. Unknown names error. +func StackBundles(order []string, bundles map[string]Bundle) ([]Entry, error) { + var out []Entry + seen := map[string]int{} + for _, name := range order { + b, ok := bundles[name] + if !ok { + return nil, fmt.Errorf("plugin: unknown bundle %q", name) + } + for _, e := range b.Entries { + if e.ID == "" { + return nil, fmt.Errorf("plugin: bundle %q has an entry with empty id", name) + } + if prev, dup := seen[e.ID]; dup { + return nil, fmt.Errorf("plugin: duplicate entry id %q (bundle %q and earlier row %d)", e.ID, name, prev) + } + seen[e.ID] = len(out) + out = append(out, e) + } + } + return out, nil +} + +// LoadProfile reads a YAML profile from path. A missing file is not an error +// when allowMissing is true; the caller should then use a default profile. +func LoadProfile(path string, allowMissing bool) (*Profile, error) { + data, err := os.ReadFile(path) + if err != nil { + if allowMissing && os.IsNotExist(err) { + return nil, nil + } + return nil, fmt.Errorf("plugin: read profile %s: %w", path, err) + } + var p Profile + if err := yaml.Unmarshal(data, &p); err != nil { + return nil, fmt.Errorf("plugin: parse profile %s: %w", path, err) + } + if p.Name == "" { + p.Name = "unnamed" + } + return &p, nil +} diff --git a/internal/plugin/compose_test.go b/internal/plugin/compose_test.go new file mode 100644 index 0000000000..7a20c415c8 --- /dev/null +++ b/internal/plugin/compose_test.go @@ -0,0 +1,78 @@ +package plugin + +import ( + "os" + "path/filepath" + "testing" +) + +func TestApplyPatchesReplaceAndInsert(t *testing.T) { + entries := []Entry{ + {ID: "websearch.duckduckgo", Plugin: "websearch.duckduckgo"}, + {ID: "websearch.exa", Plugin: "websearch.exa"}, + } + off := true + patched := ApplyPatches(entries, []Patch{ + {ID: "websearch.exa", Disabled: &off}, + {ID: "websearch.echo", Plugin: "websearch.echo", Insert: true}, + {ID: "missing", Plugin: "nope"}, + }) + if len(patched) != 3 { + t.Fatalf("len = %d", len(patched)) + } + if !patched[1].Disabled { + t.Fatal("exa should be disabled") + } + if patched[2].ID != "websearch.echo" { + t.Fatalf("insert = %+v", patched[2]) + } +} + +func TestStackBundlesDuplicateAndUnknown(t *testing.T) { + bundles := map[string]Bundle{ + "base": {Name: "base", Entries: []Entry{{ID: "a"}}}, + "extra": {Name: "extra", Entries: []Entry{{ID: "a"}}}, + } + if _, err := StackBundles([]string{"nope"}, bundles); err == nil { + t.Fatal("expected unknown bundle") + } + if _, err := StackBundles([]string{"base", "extra"}, bundles); err == nil { + t.Fatal("expected duplicate id") + } + got, err := StackBundles([]string{"base"}, bundles) + if err != nil || len(got) != 1 || got[0].ID != "a" { + t.Fatalf("got = %v, %v", got, err) + } +} + +func TestLoadProfile(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "profile.yaml") + if err := os.WriteFile(path, []byte("name: lab\nbundles: [base]\n"), 0o644); err != nil { + t.Fatal(err) + } + p, err := LoadProfile(path, false) + if err != nil { + t.Fatal(err) + } + if p.Name != "lab" || len(p.Bundles) != 1 || p.Bundles[0] != "base" { + t.Fatalf("profile = %+v", p) + } + missing := filepath.Join(dir, "absent.yaml") + p, err = LoadProfile(missing, true) + if err != nil || p != nil { + t.Fatalf("missing allow = %+v, %v", p, err) + } + if _, err = LoadProfile(missing, false); err == nil { + t.Fatal("expected read error") + } +} + +func TestEntryFactoryName(t *testing.T) { + if (Entry{ID: "a"}).FactoryName() != "a" { + t.Fatal("fallback") + } + if (Entry{ID: "a", Plugin: "b"}).FactoryName() != "b" { + t.Fatal("plugin field") + } +} diff --git a/internal/plugin/context.go b/internal/plugin/context.go new file mode 100644 index 0000000000..9897cf304a --- /dev/null +++ b/internal/plugin/context.go @@ -0,0 +1,114 @@ +package plugin + +import ( + "fmt" + "sync" +) + +// Context is a repository of services plus an event bus and an effect stack. +// Child contexts created by Isolate inherit parent lookups and the shared +// event bus, but own their own effects so one plugin can unload alone. +type Context struct { + mu sync.RWMutex + parent *Context + services map[string]any + events *EventBus + effects *EffectStack +} + +// NewContext creates a root context. +func NewContext() *Context { + return &Context{ + services: make(map[string]any), + events: NewEventBus(), + effects: newEffectStack(), + } +} + +// Isolate returns a child context. Services are shared on the root so peer +// plugins can inject them; Effect stays local so Unload only unwinds this +// child. A later per-agent realm can add a true isolated service map. +func (c *Context) Isolate() *Context { + return &Context{ + parent: c, + services: make(map[string]any), + events: c.events, + effects: newEffectStack(), + } +} + +func (c *Context) root() *Context { + for c.parent != nil { + c = c.parent + } + return c +} + +// Provide installs a service under key on the root context. The previous +// value (if any) is restored when the current isolate unloads. +func (c *Context) Provide(key string, svc any) { + if key == "" { + return + } + root := c.root() + c.Effect(func() Disposable { + root.mu.Lock() + prev, had := root.services[key] + root.services[key] = svc + root.mu.Unlock() + return DisposeFunc(func() { + root.mu.Lock() + defer root.mu.Unlock() + if had { + root.services[key] = prev + } else { + delete(root.services, key) + } + }) + }) +} + +// Get looks up a service on this context, then parents. +func (c *Context) Get(key string) (any, bool) { + for cur := c; cur != nil; cur = cur.parent { + cur.mu.RLock() + v, ok := cur.services[key] + cur.mu.RUnlock() + if ok { + return v, true + } + } + return nil, false +} + +// Service returns a typed service or an error if missing / wrong type. +func Service[T any](ctx *Context, key string) (T, error) { + var zero T + v, ok := ctx.Get(key) + if !ok { + return zero, fmt.Errorf("plugin: missing service %q", key) + } + t, ok := v.(T) + if !ok { + return zero, fmt.Errorf("plugin: service %q has type %T, want %T", key, v, zero) + } + return t, nil +} + +// Effect records a reversible registration. fn runs immediately and its +// disposer is unwound on Unload. +func (c *Context) Effect(fn func() Disposable) error { + if fn == nil { + return nil + } + c.effects.Push(fn()) + return nil +} + +// Events returns the shared event bus. +func (c *Context) Events() *EventBus { return c.events } + +// Unload disposes every effect recorded on this context (not parents). +func (c *Context) Unload() { + c.effects.Close() +} diff --git a/internal/plugin/discover.go b/internal/plugin/discover.go new file mode 100644 index 0000000000..0cc194352d --- /dev/null +++ b/internal/plugin/discover.go @@ -0,0 +1,108 @@ +package plugin + +import ( + "fmt" + "os" + "path/filepath" + "sort" + "strings" +) + +const manifestName = "plugin.yaml" + +// Discover walks each directory for plugin.yaml (the dir itself or one +// level of children). Results are sorted by id. Invalid files return error. +func Discover(dirs []string) ([]Manifest, error) { + seen := map[string]string{} + var out []Manifest + for _, dir := range dirs { + dir = strings.TrimSpace(dir) + if dir == "" || dir == "-" || dir == "none" { + continue + } + info, err := os.Stat(dir) + if err != nil { + if os.IsNotExist(err) { + continue + } + return nil, fmt.Errorf("plugin: stat %s: %w", dir, err) + } + if !info.IsDir() { + return nil, fmt.Errorf("plugin: %s is not a directory", dir) + } + found, err := scanDir(dir) + if err != nil { + return nil, err + } + for _, m := range found { + if prev, ok := seen[m.ID]; ok { + return nil, fmt.Errorf("plugin: duplicate id %q (%s and %s)", m.ID, prev, m.Dir) + } + seen[m.ID] = m.Dir + out = append(out, m) + } + } + sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID }) + return out, nil +} + +func scanDir(dir string) ([]Manifest, error) { + var out []Manifest + root := filepath.Join(dir, manifestName) + if st, err := os.Stat(root); err == nil && !st.IsDir() { + m, err := LoadManifest(root) + if err != nil { + return nil, err + } + out = append(out, m) + } + entries, err := os.ReadDir(dir) + if err != nil { + return nil, fmt.Errorf("plugin: read %s: %w", dir, err) + } + for _, e := range entries { + if !e.IsDir() { + continue + } + path := filepath.Join(dir, e.Name(), manifestName) + if st, err := os.Stat(path); err != nil || st.IsDir() { + continue + } + m, err := LoadManifest(path) + if err != nil { + return nil, err + } + out = append(out, m) + } + return out, nil +} + +// BundleFromManifests builds the external bundle. Disabled / auto_enable +// false rows are still listed so a profile patch can turn them on. +func BundleFromManifests(manifests []Manifest) Bundle { + entries := make([]Entry, 0, len(manifests)) + for _, m := range manifests { + entries = append(entries, Entry{ + ID: m.ID, + Plugin: m.ID, + Config: m.Config, + Disabled: !m.Enabled(), + }) + } + return Bundle{Name: ExternalBundle, Entries: entries} +} + +// ParsePluginDirs splits WEKNORA_PLUGIN_DIR (os.PathListSeparator). +func ParsePluginDirs(raw string) []string { + if strings.TrimSpace(raw) == "" { + return []string{"plugins.d"} + } + var out []string + for _, p := range filepath.SplitList(raw) { + p = strings.TrimSpace(p) + if p != "" { + out = append(out, p) + } + } + return out +} diff --git a/internal/plugin/discover_test.go b/internal/plugin/discover_test.go new file mode 100644 index 0000000000..973ef75e66 --- /dev/null +++ b/internal/plugin/discover_test.go @@ -0,0 +1,135 @@ +package plugin + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +func TestDiscoverNestedAndRoot(t *testing.T) { + dir := t.TempDir() + nested := filepath.Join(dir, "bravo") + if err := os.Mkdir(nested, 0o755); err != nil { + t.Fatal(err) + } + writeManifest(t, filepath.Join(nested, "plugin.yaml"), ` +id: websearch.bravo +seam: web_search +runtime: js +entry: search.js +`) + writeManifest(t, filepath.Join(dir, "plugin.yaml"), ` +id: websearch.alpha +seam: web_search +runtime: http +endpoint: http://127.0.0.1:9/search +`) + got, err := Discover([]string{dir}) + if err != nil { + t.Fatal(err) + } + if len(got) != 2 || got[0].ID != "websearch.alpha" || got[1].ID != "websearch.bravo" { + t.Fatalf("got = %+v", got) + } + if !got[0].Enabled() || got[1].ProviderID() != "bravo" { + t.Fatalf("fields = %+v %+v", got[0], got[1]) + } +} + +func TestDiscoverMissingDir(t *testing.T) { + got, err := Discover([]string{filepath.Join(t.TempDir(), "nope"), "none", "-"}) + if err != nil || len(got) != 0 { + t.Fatalf("got = %v, %v", got, err) + } +} + +func TestDiscoverDuplicateID(t *testing.T) { + a := t.TempDir() + b := t.TempDir() + writeManifest(t, filepath.Join(a, "plugin.yaml"), ` +id: websearch.same +seam: web_search +runtime: http +endpoint: http://127.0.0.1:9/a +`) + writeManifest(t, filepath.Join(b, "plugin.yaml"), ` +id: websearch.same +seam: web_search +runtime: http +endpoint: http://127.0.0.1:9/b +`) + if _, err := Discover([]string{a, b}); err == nil { + t.Fatal("expected duplicate id") + } +} + +func TestManifestStdioExecAndValidate(t *testing.T) { + m := Manifest{ + ID: "websearch.py", Seam: ServiceWebSearch, Runtime: RuntimeStdio, + Command: "python3", Entry: "plugin.py", Args: []string{"-u"}, Dir: "/tmp/p", + } + if err := m.Validate(); err != nil { + t.Fatal(err) + } + name, args := m.Exec() + if name != "python3" || len(args) != 2 || args[0] != "-u" || args[1] != "/tmp/p/plugin.py" { + t.Fatalf("exec = %s %v", name, args) + } + bad := Manifest{ID: "x", Seam: ServiceWebSearch, Runtime: RuntimeStdio} + if err := bad.Validate(); err == nil { + t.Fatal("expected command-or-entry") + } +} + +func TestParsePluginDirs(t *testing.T) { + if got := ParsePluginDirs(""); len(got) != 1 || got[0] != "plugins.d" { + t.Fatalf("default = %v", got) + } + got := ParsePluginDirs("a" + string(os.PathListSeparator) + " b ") + if len(got) != 2 || got[0] != "a" || got[1] != "b" { + t.Fatalf("split = %v", got) + } +} + +func TestBundleFromManifestsHonorsAutoEnable(t *testing.T) { + off := false + b := BundleFromManifests([]Manifest{ + {ID: "on", Runtime: RuntimeHTTP, Endpoint: "http://x", Seam: ServiceWebSearch}, + {ID: "off", Runtime: RuntimeHTTP, Endpoint: "http://x", Seam: ServiceWebSearch, AutoEnable: &off}, + }) + if len(b.Entries) != 2 || b.Entries[0].Disabled || !b.Entries[1].Disabled { + t.Fatalf("entries = %+v", b.Entries) + } +} + +func TestConfigMerge(t *testing.T) { + got := Config{"a": "1", "b": "2"}.Merge(Config{"b": "3", "c": "4"}) + if got.String("a") != "1" || got.String("b") != "3" || got.String("c") != "4" { + t.Fatalf("merge = %+v", got) + } +} + +func TestHostDump(t *testing.T) { + Register("dump.alpha", func(Config) (Plugin, error) { + return Func{ID: "dump.alpha"}, nil + }) + h := NewHost() + if err := h.Compose(Profile{Bundles: []string{"b"}}, map[string]Bundle{ + "b": {Entries: []Entry{{ID: "dump.alpha"}, {ID: "skip", Disabled: true}}}, + }, nil); err != nil { + t.Fatal(err) + } + dump := h.Dump() + if dump == "" || !strings.Contains(dump, "dump.alpha") || !strings.Contains(dump, "skip") { + t.Fatalf("dump = %s", dump) + } + h.Unload() +} + +func writeManifest(t *testing.T, path, body string) { + t.Helper() + if err := os.WriteFile(path, []byte(body), 0o644); err != nil { + t.Fatal(err) + } +} diff --git a/internal/plugin/dump.go b/internal/plugin/dump.go new file mode 100644 index 0000000000..64eaac44e1 --- /dev/null +++ b/internal/plugin/dump.go @@ -0,0 +1,34 @@ +package plugin + +import ( + "bytes" + "fmt" + + "gopkg.in/yaml.v3" +) + +// DumpRow is one composed row, for dsh --dump-config style inspection. +type DumpRow struct { + ID string `yaml:"id" json:"id"` + Plugin string `yaml:"plugin" json:"plugin"` + Disabled bool `yaml:"disabled,omitempty" json:"disabled,omitempty"` + Inject []string `yaml:"inject,omitempty" json:"inject,omitempty"` +} + +// Dump returns YAML of the mounted tree. Every row can be targeted by a patch. +func (h *Host) Dump() string { + rows := make([]DumpRow, 0, len(h.mounted)) + for _, m := range h.mounted { + rows = append(rows, DumpRow{ + ID: m.ID, Plugin: m.Plugin, Disabled: m.Disabled, Inject: m.Inject, + }) + } + var buf bytes.Buffer + enc := yaml.NewEncoder(&buf) + enc.SetIndent(2) + if err := enc.Encode(rows); err != nil { + return fmt.Sprintf("# dump error: %v\n", err) + } + _ = enc.Close() + return buf.String() +} diff --git a/internal/plugin/effect.go b/internal/plugin/effect.go new file mode 100644 index 0000000000..db47875868 --- /dev/null +++ b/internal/plugin/effect.go @@ -0,0 +1,62 @@ +package plugin + +import "sync" + +// Disposable undoes a registration. Dispose must be idempotent. +type Disposable interface { + Dispose() +} + +// DisposeFunc adapts a function to Disposable. +type DisposeFunc func() + +// Dispose implements Disposable. +func (f DisposeFunc) Dispose() { + if f != nil { + f() + } +} + +// EffectStack records disposers and unwinds them in reverse order. +type EffectStack struct { + mu sync.Mutex + items []Disposable + closed bool +} + +func newEffectStack() *EffectStack { + return &EffectStack{} +} + +// Push records a disposer. After Close, Push is a no-op and immediately +// disposes the new item so a late registration cannot leak. +func (s *EffectStack) Push(d Disposable) { + if d == nil { + return + } + s.mu.Lock() + if s.closed { + s.mu.Unlock() + d.Dispose() + return + } + s.items = append(s.items, d) + s.mu.Unlock() +} + +// Close disposes every recorded effect in reverse order. Safe to call twice. +func (s *EffectStack) Close() { + s.mu.Lock() + if s.closed { + s.mu.Unlock() + return + } + s.closed = true + items := s.items + s.items = nil + s.mu.Unlock() + + for i := len(items) - 1; i >= 0; i-- { + items[i].Dispose() + } +} diff --git a/internal/plugin/event.go b/internal/plugin/event.go new file mode 100644 index 0000000000..f58e245fd2 --- /dev/null +++ b/internal/plugin/event.go @@ -0,0 +1,145 @@ +package plugin + +import ( + "context" + "sync" + "sync/atomic" +) + +// Handler is around-middleware for a named event. Call next to delegate; +// return without next to short-circuit (waterfall / serial). +type Handler func(ctx context.Context, payload any, next func(any) (any, error)) (any, error) + +type listener struct { + id uint64 + handler Handler + prepend bool +} + +// EventBus dispatches named events. Mode is chosen by the caller and is +// part of each event's public contract. +type EventBus struct { + mu sync.RWMutex + seq atomic.Uint64 + listeners map[string][]listener +} + +// NewEventBus creates an empty bus. +func NewEventBus() *EventBus { + return &EventBus{listeners: make(map[string][]listener)} +} + +// On registers a listener. The returned disposer removes it. +func (b *EventBus) On(name string, h Handler) Disposable { + return b.on(name, h, false) +} + +// Prepend registers a listener that runs before ordinary registrations. +func (b *EventBus) Prepend(name string, h Handler) Disposable { + return b.on(name, h, true) +} + +func (b *EventBus) on(name string, h Handler, prepend bool) Disposable { + if name == "" || h == nil { + return DisposeFunc(nil) + } + item := listener{id: b.seq.Add(1), handler: h, prepend: prepend} + b.mu.Lock() + b.listeners[name] = append(b.listeners[name], item) + b.mu.Unlock() + return DisposeFunc(func() { + b.mu.Lock() + defer b.mu.Unlock() + cur := b.listeners[name] + out := make([]listener, 0, len(cur)) + for _, l := range cur { + if l.id == item.id { + continue + } + out = append(out, l) + } + if len(out) == 0 { + delete(b.listeners, name) + } else { + b.listeners[name] = out + } + }) +} + +// snapshot copies listeners so dispatch does not hold the lock. +func (b *EventBus) snapshot(name string) []Handler { + b.mu.RLock() + defer b.mu.RUnlock() + raw := b.listeners[name] + prepend := make([]Handler, 0, len(raw)) + normal := make([]Handler, 0, len(raw)) + for _, l := range raw { + if l.prepend { + prepend = append(prepend, l.handler) + } else { + normal = append(normal, l.handler) + } + } + return append(prepend, normal...) +} + +// Emit notifies listeners in registration order and ignores return values. +func (b *EventBus) Emit(ctx context.Context, name string, payload any) { + for _, h := range b.snapshot(name) { + _, _ = h(ctx, payload, func(p any) (any, error) { return p, nil }) + } +} + +// Waterfall is around-middleware. Each listener receives next and must call +// it to delegate. The first listener that returns without next short-circuits. +func (b *EventBus) Waterfall(ctx context.Context, name string, payload any) (any, error) { + handlers := b.snapshot(name) + var run func(int, any) (any, error) + run = func(i int, p any) (any, error) { + if i >= len(handlers) { + return p, nil + } + return handlers[i](ctx, p, func(nextPayload any) (any, error) { + return run(i+1, nextPayload) + }) + } + return run(0, payload) +} + +// Parallel runs every listener concurrently. next is a no-op identity. +func (b *EventBus) Parallel(ctx context.Context, name string, payload any) error { + handlers := b.snapshot(name) + if len(handlers) == 0 { + return nil + } + errCh := make(chan error, len(handlers)) + for _, h := range handlers { + go func(handler Handler) { + _, err := handler(ctx, payload, func(p any) (any, error) { return p, nil }) + errCh <- err + }(h) + } + var first error + for range handlers { + if err := <-errCh; err != nil && first == nil { + first = err + } + } + return first +} + +// Serial runs listeners in order. A listener that returns without next +// replaces the payload seen by later listeners. +func (b *EventBus) Serial(ctx context.Context, name string, payload any) (any, error) { + cur := payload + for _, h := range b.snapshot(name) { + out, err := h(ctx, cur, func(p any) (any, error) { + return p, nil + }) + if err != nil { + return cur, err + } + cur = out + } + return cur, nil +} diff --git a/internal/plugin/event_test.go b/internal/plugin/event_test.go new file mode 100644 index 0000000000..7d94db73f3 --- /dev/null +++ b/internal/plugin/event_test.go @@ -0,0 +1,131 @@ +package plugin + +import ( + "context" + "errors" + "sync/atomic" + "testing" +) + +func TestEmitAndDispose(t *testing.T) { + bus := NewEventBus() + var n atomic.Int32 + d := bus.On("ping", func(ctx context.Context, payload any, next func(any) (any, error)) (any, error) { + n.Add(1) + return next(payload) + }) + bus.Emit(context.Background(), "ping", nil) + d.Dispose() + bus.Emit(context.Background(), "ping", nil) + if n.Load() != 1 { + t.Fatalf("n = %d, want 1", n.Load()) + } +} + +func TestWaterfallShortCircuit(t *testing.T) { + bus := NewEventBus() + var seen []string + bus.On("wf", func(ctx context.Context, payload any, next func(any) (any, error)) (any, error) { + seen = append(seen, "outer") + return next(payload) + }) + bus.On("wf", func(ctx context.Context, payload any, next func(any) (any, error)) (any, error) { + seen = append(seen, "block") + return "stopped", nil + }) + bus.On("wf", func(ctx context.Context, payload any, next func(any) (any, error)) (any, error) { + seen = append(seen, "inner") + return next(payload) + }) + out, err := bus.Waterfall(context.Background(), "wf", "start") + if err != nil { + t.Fatal(err) + } + if out != "stopped" { + t.Fatalf("out = %v", out) + } + if len(seen) != 2 || seen[0] != "outer" || seen[1] != "block" { + t.Fatalf("seen = %v", seen) + } +} + +func TestWaterfallRewrite(t *testing.T) { + bus := NewEventBus() + bus.On("wf", func(ctx context.Context, payload any, next func(any) (any, error)) (any, error) { + return next(payload.(string) + "-a") + }) + bus.On("wf", func(ctx context.Context, payload any, next func(any) (any, error)) (any, error) { + return next(payload.(string) + "-b") + }) + out, err := bus.Waterfall(context.Background(), "wf", "x") + if err != nil { + t.Fatal(err) + } + if out != "x-a-b" { + t.Fatalf("out = %v", out) + } +} + +func TestPrependRunsFirst(t *testing.T) { + bus := NewEventBus() + var seen []string + bus.On("e", func(ctx context.Context, payload any, next func(any) (any, error)) (any, error) { + seen = append(seen, "normal") + return next(payload) + }) + bus.Prepend("e", func(ctx context.Context, payload any, next func(any) (any, error)) (any, error) { + seen = append(seen, "pre") + return next(payload) + }) + _, _ = bus.Waterfall(context.Background(), "e", nil) + if len(seen) != 2 || seen[0] != "pre" || seen[1] != "normal" { + t.Fatalf("seen = %v", seen) + } +} + +func TestParallelCollectsError(t *testing.T) { + bus := NewEventBus() + want := errors.New("boom") + bus.On("p", func(ctx context.Context, payload any, next func(any) (any, error)) (any, error) { + return nil, want + }) + bus.On("p", func(ctx context.Context, payload any, next func(any) (any, error)) (any, error) { + return next(payload) + }) + if err := bus.Parallel(context.Background(), "p", nil); !errors.Is(err, want) { + t.Fatalf("err = %v", err) + } +} + +func TestSerialReplacesPayload(t *testing.T) { + bus := NewEventBus() + bus.On("s", func(ctx context.Context, payload any, next func(any) (any, error)) (any, error) { + return payload.(int) + 1, nil + }) + bus.On("s", func(ctx context.Context, payload any, next func(any) (any, error)) (any, error) { + return payload.(int) * 10, nil + }) + out, err := bus.Serial(context.Background(), "s", 2) + if err != nil { + t.Fatal(err) + } + if out != 30 { + t.Fatalf("out = %v, want 30", out) + } +} + +func TestEmptyDispatch(t *testing.T) { + bus := NewEventBus() + bus.Emit(context.Background(), "none", nil) + out, err := bus.Waterfall(context.Background(), "none", "keep") + if err != nil || out != "keep" { + t.Fatalf("waterfall empty = %v %v", out, err) + } + if err := bus.Parallel(context.Background(), "none", nil); err != nil { + t.Fatal(err) + } + out, err = bus.Serial(context.Background(), "none", "keep") + if err != nil || out != "keep" { + t.Fatalf("serial empty = %v %v", out, err) + } +} diff --git a/internal/plugin/host.go b/internal/plugin/host.go new file mode 100644 index 0000000000..0304eb3b6b --- /dev/null +++ b/internal/plugin/host.go @@ -0,0 +1,145 @@ +package plugin + +import ( + "context" + "fmt" +) + +// Mounted describes one mounted (or disabled) row for dumps and logs. +type Mounted struct { + ID string + Plugin string + Disabled bool + Inject []string +} + +type liveScope struct { + id string + scope *Context +} + +// Host mounts plugins onto a root Context. Each plugin gets its own isolate +// so Unload of the host (or a future per-plugin unload) only unwinds that +// plugin's effects. +type Host struct { + ctx *Context + mounted []Mounted + scopes []liveScope +} + +// NewHost creates an empty host with a root context. +func NewHost() *Host { + return &Host{ctx: NewContext()} +} + +// Context returns the root context. Call Provide here for host-owned services +// (existing WeKnora registries) before Compose. +func (h *Host) Context() *Context { return h.ctx } + +// Mounted returns a snapshot of composed rows. +func (h *Host) Mounted() []Mounted { + out := make([]Mounted, len(h.mounted)) + copy(out, h.mounted) + return out +} + +// Compose stacks bundles, applies patches, then mounts enabled entries. +func (h *Host) Compose(profile Profile, bundles map[string]Bundle, extra []Patch) error { + entries, err := StackBundles(profile.Bundles, bundles) + if err != nil { + return err + } + entries = ApplyPatches(entries, profile.Patch) + entries = ApplyPatches(entries, extra) + return h.mountEntries(entries) +} + +func (h *Host) mountEntries(entries []Entry) error { + pending := append([]Entry(nil), entries...) + for len(pending) > 0 { + progress := false + var next []Entry + for _, e := range pending { + if e.Disabled { + h.mounted = append(h.mounted, Mounted{ + ID: e.ID, Plugin: e.FactoryName(), Disabled: true, + }) + progress = true + continue + } + ok, err := h.tryMount(e) + if err != nil { + return err + } + if ok { + progress = true + continue + } + next = append(next, e) + } + if !progress { + return fmt.Errorf("plugin: unsatisfied inject for %s", describePending(next)) + } + pending = next + } + return nil +} + +func (h *Host) tryMount(e Entry) (bool, error) { + factory, err := MustLookupFactory(e.FactoryName()) + if err != nil { + return false, err + } + p, err := factory(e.Config) + if err != nil { + return false, fmt.Errorf("plugin: construct %s: %w", e.ID, err) + } + if !h.injectSatisfied(p.Inject()) { + return false, nil + } + scope := h.ctx.Isolate() + if err := p.Apply(scope); err != nil { + scope.Unload() + return false, fmt.Errorf("plugin: apply %s: %w", e.ID, err) + } + h.scopes = append(h.scopes, liveScope{id: e.ID, scope: scope}) + h.mounted = append(h.mounted, Mounted{ + ID: e.ID, Plugin: e.FactoryName(), Inject: append([]string(nil), p.Inject()...), + }) + h.ctx.Events().Emit(context.Background(), EventPluginMounted, e.ID) + return true, nil +} + +func (h *Host) injectSatisfied(keys []string) bool { + for _, key := range keys { + if _, ok := h.ctx.Get(key); !ok { + return false + } + } + return true +} + +// Unload closes every plugin scope and then the root context. +func (h *Host) Unload() { + for i := len(h.scopes) - 1; i >= 0; i-- { + id := h.scopes[i].id + h.scopes[i].scope.Unload() + h.ctx.Events().Emit(context.Background(), EventPluginUnloaded, id) + } + h.scopes = nil + h.ctx.Unload() +} + +func describePending(entries []Entry) string { + if len(entries) == 0 { + return "(none)" + } + out := entries[0].ID + for i := 1; i < len(entries) && i < 5; i++ { + out += ", " + entries[i].ID + } + if len(entries) > 5 { + out += ", ..." + } + return out +} diff --git a/internal/plugin/host_test.go b/internal/plugin/host_test.go new file mode 100644 index 0000000000..8f0324460f --- /dev/null +++ b/internal/plugin/host_test.go @@ -0,0 +1,118 @@ +package plugin + +import ( + "errors" + "testing" +) + +func TestHostComposeMountAndUnload(t *testing.T) { + var applied, disposed int + Register("host.alpha", func(Config) (Plugin, error) { + return Func{ + ID: "host.alpha", + InjectKeys: []string{"core"}, + ApplyFn: func(ctx *Context) error { + applied++ + ctx.Provide("alpha", true) + return ctx.Effect(func() Disposable { + return DisposeFunc(func() { disposed++ }) + }) + }, + }, nil + }) + Register("host.beta", func(Config) (Plugin, error) { + return Func{ + ID: "host.beta", + InjectKeys: []string{"alpha"}, + ApplyFn: func(ctx *Context) error { + applied++ + if _, err := Service[bool](ctx, "alpha"); err != nil { + return err + } + return nil + }, + }, nil + }) + + h := NewHost() + h.Context().Provide("core", struct{}{}) + err := h.Compose(Profile{Name: "t", Bundles: []string{"base"}}, map[string]Bundle{ + "base": {Name: "base", Entries: []Entry{ + {ID: "host.beta", Plugin: "host.beta"}, + {ID: "host.alpha", Plugin: "host.alpha"}, + }}, + }, nil) + if err != nil { + t.Fatal(err) + } + if applied != 2 { + t.Fatalf("applied = %d", applied) + } + mounted := h.Mounted() + if len(mounted) != 2 { + t.Fatalf("mounted = %+v", mounted) + } + h.Unload() + if disposed != 1 { + t.Fatalf("disposed = %d", disposed) + } +} + +func TestHostDisabledAndMissingInject(t *testing.T) { + Register("host.needs", func(Config) (Plugin, error) { + return Func{ID: "host.needs", InjectKeys: []string{"missing-svc"}, ApplyFn: func(*Context) error { + return nil + }}, nil + }) + h := NewHost() + err := h.Compose(Profile{Bundles: []string{"base"}}, map[string]Bundle{ + "base": {Entries: []Entry{ + {ID: "skip.me", Plugin: "host.needs", Disabled: true}, + {ID: "host.needs", Plugin: "host.needs"}, + }}, + }, nil) + if err == nil { + t.Fatal("expected unsatisfied inject") + } + h2 := NewHost() + if err := h2.Compose(Profile{Bundles: []string{"base"}}, map[string]Bundle{ + "base": {Entries: []Entry{{ID: "skip.me", Plugin: "host.needs", Disabled: true}}}, + }, nil); err != nil { + t.Fatal(err) + } + if got := h2.Mounted(); len(got) != 1 || !got[0].Disabled { + t.Fatalf("mounted = %+v", got) + } +} + +func TestHostApplyErrorUnwinds(t *testing.T) { + var disposed int + Register("host.fail", func(Config) (Plugin, error) { + return Func{ID: "host.fail", ApplyFn: func(ctx *Context) error { + _ = ctx.Effect(func() Disposable { + return DisposeFunc(func() { disposed++ }) + }) + return errors.New("apply failed") + }}, nil + }) + h := NewHost() + err := h.Compose(Profile{Bundles: []string{"base"}}, map[string]Bundle{ + "base": {Entries: []Entry{{ID: "host.fail"}}}, + }, nil) + if err == nil { + t.Fatal("expected apply error") + } + if disposed != 1 { + t.Fatalf("disposed = %d, want 1 (partial apply unwound)", disposed) + } +} + +func TestHostUnknownFactory(t *testing.T) { + h := NewHost() + err := h.Compose(Profile{Bundles: []string{"base"}}, map[string]Bundle{ + "base": {Entries: []Entry{{ID: "no.such.plugin"}}}, + }, nil) + if err == nil { + t.Fatal("expected missing factory") + } +} diff --git a/internal/plugin/manifest.go b/internal/plugin/manifest.go new file mode 100644 index 0000000000..e3af115199 --- /dev/null +++ b/internal/plugin/manifest.go @@ -0,0 +1,151 @@ +package plugin + +import ( + "fmt" + "os" + "path/filepath" + "strings" + + "gopkg.in/yaml.v3" +) + +// Runtime identifiers for disk-loaded plugins. native stays compile-time. +const ( + RuntimeStdio = "stdio" + RuntimeJS = "js" + RuntimeHTTP = "http" +) + +// ExternalBundle is the profile bundle assembled from WEKNORA_PLUGIN_DIR. +const ExternalBundle = "external" + +// Manifest is the on-disk contract (package.json analogue). Drop a folder +// with plugin.yaml and WeKnora will register it without a blank import. +type Manifest struct { + ID string `yaml:"id"` + Name string `yaml:"name,omitempty"` + Version string `yaml:"version,omitempty"` + Description string `yaml:"description,omitempty"` + Seam string `yaml:"seam"` + Runtime string `yaml:"runtime"` + Command string `yaml:"command,omitempty"` + Args []string `yaml:"args,omitempty"` + Entry string `yaml:"entry,omitempty"` + Endpoint string `yaml:"endpoint,omitempty"` + Provider string `yaml:"provider,omitempty"` + DocsURL string `yaml:"docs_url,omitempty"` + RequiresKey bool `yaml:"requires_api_key,omitempty"` + AutoEnable *bool `yaml:"auto_enable,omitempty"` + Disabled bool `yaml:"disabled,omitempty"` + TimeoutMS int `yaml:"timeout_ms,omitempty"` + Env map[string]string `yaml:"env,omitempty"` + Config Config `yaml:"config,omitempty"` + Dir string `yaml:"-"` +} + +// Enabled reports whether the manifest should be mounted by default. +func (m Manifest) Enabled() bool { + if m.Disabled { + return false + } + if m.AutoEnable == nil { + return true + } + return *m.AutoEnable +} + +// ProviderID is the seam-specific registration key. +func (m Manifest) ProviderID() string { + if m.Provider != "" { + return m.Provider + } + id := m.ID + if i := strings.LastIndex(id, "."); i >= 0 { + return id[i+1:] + } + return id +} + +// DisplayName is the UI label. +func (m Manifest) DisplayName() string { + if m.Name != "" { + return m.Name + } + return m.ID +} + +// Timeout returns the request timeout, default 10s. +func (m Manifest) Timeout() int { + if m.TimeoutMS > 0 { + return m.TimeoutMS + } + return 10000 +} + +// EntryPath resolves Entry against the plugin directory. +func (m Manifest) EntryPath() string { + if m.Entry == "" || m.Dir == "" { + return m.Entry + } + if filepath.IsAbs(m.Entry) { + return m.Entry + } + return filepath.Join(m.Dir, m.Entry) +} + +// Exec returns the argv host will launch for runtime: stdio. +// command + args, with entry appended when both are set. +func (m Manifest) Exec() (name string, args []string) { + args = append([]string{}, m.Args...) + if strings.TrimSpace(m.Command) != "" { + if m.Entry != "" { + args = append(args, m.EntryPath()) + } + return m.Command, args + } + return m.EntryPath(), args +} + +// Validate checks required fields for a disk plugin. +func (m Manifest) Validate() error { + if m.ID == "" { + return fmt.Errorf("plugin.yaml: id is required") + } + if m.Seam == "" { + return fmt.Errorf("plugin %s: seam is required", m.ID) + } + switch m.Runtime { + case RuntimeStdio: + if strings.TrimSpace(m.Command) == "" && strings.TrimSpace(m.Entry) == "" { + return fmt.Errorf("plugin %s: stdio runtime requires command or entry", m.ID) + } + case RuntimeJS: + if m.Entry == "" { + return fmt.Errorf("plugin %s: js runtime requires entry", m.ID) + } + case RuntimeHTTP: + if strings.TrimSpace(m.Endpoint) == "" { + return fmt.Errorf("plugin %s: http runtime requires endpoint", m.ID) + } + default: + return fmt.Errorf("plugin %s: unsupported runtime %q (stdio|js|http)", m.ID, m.Runtime) + } + return nil +} + +// LoadManifest reads one plugin.yaml. +func LoadManifest(path string) (Manifest, error) { + data, err := os.ReadFile(path) + if err != nil { + return Manifest{}, fmt.Errorf("plugin: read %s: %w", path, err) + } + var m Manifest + if err := yaml.Unmarshal(data, &m); err != nil { + return Manifest{}, fmt.Errorf("plugin: parse %s: %w", path, err) + } + m.Dir = filepath.Dir(path) + if err := m.Validate(); err != nil { + return Manifest{}, err + } + return m, nil +} diff --git a/internal/plugin/plugin.go b/internal/plugin/plugin.go new file mode 100644 index 0000000000..88e7b907fc --- /dev/null +++ b/internal/plugin/plugin.go @@ -0,0 +1,81 @@ +// Package plugin is a Cordis-inspired composition kernel for WeKnora. +// +// A plugin contributes services, typed events, and reversible registrations +// to a shared Context. There is no privileged core to patch: new behavior +// mounts beside existing plugins. See docs/dev/plugin-architecture.md. +package plugin + +// Plugin is a unit of composition. Name identifies the plugin in dumps and +// logs. Inject lists service keys that must exist before Apply runs. Apply +// registers services, event listeners, and other reversible effects. +type Plugin interface { + Name() string + Inject() []string + Apply(ctx *Context) error +} + +// Func is a function-shaped plugin. Use it for small in-tree adapters that +// do not need their own type. +type Func struct { + ID string + InjectKeys []string + ApplyFn func(ctx *Context) error +} + +// Name implements Plugin. +func (f Func) Name() string { return f.ID } + +// Inject implements Plugin. +func (f Func) Inject() []string { return f.InjectKeys } + +// Apply implements Plugin. +func (f Func) Apply(ctx *Context) error { + if f.ApplyFn == nil { + return nil + } + return f.ApplyFn(ctx) +} + +// Factory constructs a plugin from declarative config (YAML / env overlay). +type Factory func(cfg Config) (Plugin, error) + +// Config is a plugin's declarative configuration bag. +type Config map[string]any + +// String returns a string config value, or empty if missing / not a string. +func (c Config) String(key string) string { + if c == nil { + return "" + } + v, ok := c[key] + if !ok { + return "" + } + s, _ := v.(string) + return s +} + +// Bool returns a bool config value, or false if missing / not a bool. +func (c Config) Bool(key string) bool { + if c == nil { + return false + } + v, ok := c[key] + if !ok { + return false + } + b, _ := v.(bool) + return b +} + +// Merge returns a new Config with overlay keys winning. +func (c Config) Merge(overlay Config) Config { + out := Config{} + for k, v := range c { + out[k] = v + } + for k, v := range overlay { + out[k] = v + } + return out +} diff --git a/internal/plugin/plugin_test.go b/internal/plugin/plugin_test.go new file mode 100644 index 0000000000..f72eaea853 --- /dev/null +++ b/internal/plugin/plugin_test.go @@ -0,0 +1,115 @@ +package plugin + +import ( + "testing" +) + +func TestContextProvideOverrideAndRestore(t *testing.T) { + root := NewContext() + root.Provide("reg", "root-value") + + child := root.Isolate() + child.Provide("reg", "child-value") + child.Provide("local", 42) + + if v, err := Service[string](root, "reg"); err != nil || v != "child-value" { + t.Fatalf("shared override = %q, %v", v, err) + } + if v, err := Service[int](root, "local"); err != nil || v != 42 { + t.Fatalf("shared local = %d, %v", v, err) + } + + child.Unload() + if v, err := Service[string](root, "reg"); err != nil || v != "root-value" { + t.Fatalf("after unload, reg = %q, %v", v, err) + } + if _, err := Service[int](root, "local"); err == nil { + t.Fatal("child-provided service should be gone after unload") + } +} + +func TestContextEffectUnwinds(t *testing.T) { + ctx := NewContext() + var order []string + _ = ctx.Effect(func() Disposable { + order = append(order, "a") + return DisposeFunc(func() { order = append(order, "dispose-a") }) + }) + _ = ctx.Effect(func() Disposable { + order = append(order, "b") + return DisposeFunc(func() { order = append(order, "dispose-b") }) + }) + ctx.Unload() + want := []string{"a", "b", "dispose-b", "dispose-a"} + if len(order) != len(want) { + t.Fatalf("order = %v, want %v", order, want) + } + for i := range want { + if order[i] != want[i] { + t.Fatalf("order = %v, want %v", order, want) + } + } +} + +func TestServiceWrongType(t *testing.T) { + ctx := NewContext() + ctx.Provide("n", 1) + if _, err := Service[string](ctx, "n"); err == nil { + t.Fatal("expected type error") + } +} + +func TestRegisterFirstWins(t *testing.T) { + name := "test.first-wins" + Register(name, func(Config) (Plugin, error) { + return Func{ID: "first"}, nil + }) + Register(name, func(Config) (Plugin, error) { + return Func{ID: "second"}, nil + }) + f, err := MustLookupFactory(name) + if err != nil { + t.Fatal(err) + } + p, err := f(nil) + if err != nil { + t.Fatal(err) + } + if p.Name() != "first" { + t.Fatalf("name = %q, want first", p.Name()) + } +} + +func TestMustLookupFactoryMissing(t *testing.T) { + if _, err := MustLookupFactory("does-not-exist"); err == nil { + t.Fatal("expected error") + } +} + +func TestConfigHelpers(t *testing.T) { + var empty Config + if empty.String("x") != "" || empty.Bool("y") { + t.Fatal("empty config should return zero values") + } + c := Config{"name": "echo", "on": true, "n": 1} + if c.String("name") != "echo" || !c.Bool("on") || c.String("n") != "" { + t.Fatalf("helpers = %q %v %q", c.String("name"), c.Bool("on"), c.String("n")) + } +} + +func TestFuncPluginNilApply(t *testing.T) { + p := Func{ID: "noop"} + if err := p.Apply(NewContext()); err != nil { + t.Fatal(err) + } + if p.Name() != "noop" || p.Inject() != nil { + t.Fatalf("unexpected func plugin fields") + } +} + +func TestServiceMissing(t *testing.T) { + _, err := Service[string](NewContext(), "missing") + if err == nil { + t.Fatal("expected missing service error") + } +} diff --git a/internal/plugin/protocol/jsonrpc.go b/internal/plugin/protocol/jsonrpc.go new file mode 100644 index 0000000000..2624eadf67 --- /dev/null +++ b/internal/plugin/protocol/jsonrpc.go @@ -0,0 +1,224 @@ +// Package protocol is the WeKnora plugin ABI: JSON-RPC 2.0 over a +// newline-delimited byte stream (the same framing MCP stdio uses). +// +// Transports (stdio subprocess, in-process JS, HTTP fallback) only bind +// this ABI. Plugin authors implement methods, not servers. +package protocol + +import ( + "bufio" + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "strconv" + "sync" + "sync/atomic" +) + +const ( + jsonrpcVersion = "2.0" + maxLineBytes = 4 << 20 +) + +// Request is a JSON-RPC 2.0 request or notification (notification has no ID). +type Request struct { + JSONRPC string `json:"jsonrpc"` + ID any `json:"id,omitempty"` + Method string `json:"method,omitempty"` + Params json.RawMessage `json:"params,omitempty"` +} + +// Response is a JSON-RPC 2.0 response. +type Response struct { + JSONRPC string `json:"jsonrpc"` + ID any `json:"id,omitempty"` + Result json.RawMessage `json:"result,omitempty"` + Error *Error `json:"error,omitempty"` +} + +// Error is a JSON-RPC 2.0 error object. +type Error struct { + Code int `json:"code"` + Message string `json:"message"` +} + +func (e *Error) Error() string { + if e == nil { + return "jsonrpc: empty error" + } + return fmt.Sprintf("jsonrpc %d: %s", e.Code, e.Message) +} + +// Conn is a serialized JSON-RPC client over a bidirectional byte stream. +type Conn struct { + r *bufio.Reader + w *bufio.Writer + mu sync.Mutex + next atomic.Int64 +} + +// NewConn wraps a reader/writer pair. Writes are flushed after each message. +func NewConn(r io.Reader, w io.Writer) *Conn { + return &Conn{ + r: bufio.NewReaderSize(r, 64*1024), + w: bufio.NewWriterSize(w, 64*1024), + } +} + +// Call sends a request and waits for the matching response. Non-JSON lines +// (beginner console.log on stdout) are skipped so logs do not break the stream. +func (c *Conn) Call(ctx context.Context, method string, params, result any) error { + if c == nil { + return fmt.Errorf("jsonrpc: nil conn") + } + id := c.next.Add(1) + raw, err := marshalParams(params) + if err != nil { + return err + } + req := Request{JSONRPC: jsonrpcVersion, ID: id, Method: method, Params: raw} + + c.mu.Lock() + defer c.mu.Unlock() + if err := c.writeLocked(req); err != nil { + return err + } + + type outcome struct { + resp Response + err error + } + ch := make(chan outcome, 1) + go func() { + resp, err := c.readMatchLocked(id) + ch <- outcome{resp: resp, err: err} + }() + + select { + case <-ctx.Done(): + return ctx.Err() + case out := <-ch: + if out.err != nil { + return out.err + } + if out.resp.Error != nil { + return out.resp.Error + } + if result == nil || len(out.resp.Result) == 0 || string(out.resp.Result) == "null" { + return nil + } + if err := json.Unmarshal(out.resp.Result, result); err != nil { + return fmt.Errorf("jsonrpc: decode result: %w", err) + } + return nil + } +} + +// Notify writes a notification (no response expected). +func (c *Conn) Notify(method string, params any) error { + if c == nil { + return fmt.Errorf("jsonrpc: nil conn") + } + raw, err := marshalParams(params) + if err != nil { + return err + } + c.mu.Lock() + defer c.mu.Unlock() + return c.writeLocked(Request{JSONRPC: jsonrpcVersion, Method: method, Params: raw}) +} + +func (c *Conn) writeLocked(v any) error { + line, err := json.Marshal(v) + if err != nil { + return err + } + if _, err := c.w.Write(line); err != nil { + return err + } + if err := c.w.WriteByte('\n'); err != nil { + return err + } + return c.w.Flush() +} + +func (c *Conn) readMatchLocked(id int64) (Response, error) { + for { + line, err := readLine(c.r) + if err != nil { + return Response{}, err + } + line = bytes.TrimSpace(line) + if len(line) == 0 || line[0] != '{' { + continue + } + var resp Response + if err := json.Unmarshal(line, &resp); err != nil { + continue + } + if resp.ID == nil { + continue + } + got, ok := asInt64(resp.ID) + if !ok || got != id { + continue + } + return resp, nil + } +} + +func readLine(r *bufio.Reader) ([]byte, error) { + var buf []byte + for { + chunk, err := r.ReadSlice('\n') + if len(buf)+len(chunk) > maxLineBytes { + return nil, fmt.Errorf("jsonrpc: line exceeds %d bytes", maxLineBytes) + } + buf = append(buf, chunk...) + if err == nil { + return bytes.TrimSuffix(buf, []byte("\n")), nil + } + if err == bufio.ErrBufferFull { + continue + } + if err == io.EOF && len(buf) > 0 { + return buf, nil + } + return nil, err + } +} + +func marshalParams(params any) (json.RawMessage, error) { + if params == nil { + return nil, nil + } + if raw, ok := params.(json.RawMessage); ok { + return raw, nil + } + b, err := json.Marshal(params) + if err != nil { + return nil, err + } + return b, nil +} + +func asInt64(v any) (int64, bool) { + switch n := v.(type) { + case float64: + return int64(n), true + case int: + return int64(n), true + case int64: + return n, true + case json.Number: + i, err := n.Int64() + return i, err == nil + case string: + i, err := strconv.ParseInt(n, 10, 64) + return i, err == nil + default: + return 0, false + } +} diff --git a/internal/plugin/protocol/jsonrpc_test.go b/internal/plugin/protocol/jsonrpc_test.go new file mode 100644 index 0000000000..4bd0b5c958 --- /dev/null +++ b/internal/plugin/protocol/jsonrpc_test.go @@ -0,0 +1,145 @@ +package protocol + +import ( + "bufio" + "bytes" + "context" + "encoding/json" + "io" + "strings" + "testing" + "time" + + "github.com/Tencent/WeKnora/internal/types" +) + +func TestConnCallRoundTrip(t *testing.T) { + cr, cw := io.Pipe() + sr, sw := io.Pipe() + t.Cleanup(func() { + _ = cr.Close() + _ = cw.Close() + _ = sr.Close() + _ = sw.Close() + }) + go serveEcho(cr, sw) + + conn := NewConn(sr, cw) + var out SearchResponse + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + err := conn.Call(ctx, MethodWebSearchSearch, SearchRequest{Query: "q"}, &out) + if err != nil { + t.Fatal(err) + } + if len(out.Results) != 1 || out.Results[0].Title != "t" || out.Results[0].Snippet != "q" { + t.Fatalf("out = %+v", out) + } +} + +func TestConnSkipsNoiseOnStdout(t *testing.T) { + cr, cw := io.Pipe() + sr, sw := io.Pipe() + t.Cleanup(func() { + _ = cr.Close() + _ = cw.Close() + _ = sr.Close() + _ = sw.Close() + }) + go func() { + req, ok := readRequest(cr) + if !ok { + return + } + _, _ = sw.Write([]byte("debug: starting\n")) + _, _ = sw.Write([]byte("not-json\n")) + writeResult(sw, req.ID, SearchResponse{Results: []*types.WebSearchResult{}}) + }() + + conn := NewConn(sr, cw) + var out SearchResponse + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + if err := conn.Call(ctx, MethodWebSearchSearch, SearchRequest{}, &out); err != nil { + t.Fatal(err) + } + if len(out.Results) != 0 { + t.Fatalf("out = %+v", out) + } +} + +func TestConnCallTimeout(t *testing.T) { + cr, cw := io.Pipe() + sr, sw := io.Pipe() + t.Cleanup(func() { + _ = cr.Close() + _ = cw.Close() + _ = sr.Close() + _ = sw.Close() + }) + stop := make(chan struct{}) + go func() { + _, _ = readRequest(cr) + <-stop + }() + t.Cleanup(func() { close(stop) }) + + conn := NewConn(sr, cw) + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond) + defer cancel() + err := conn.Call(ctx, MethodWebSearchSearch, SearchRequest{}, &SearchResponse{}) + if err == nil { + t.Fatal("expected timeout") + } +} + +func TestNotifyWritesNoID(t *testing.T) { + var buf bytes.Buffer + conn := NewConn(strings.NewReader(""), &buf) + if err := conn.Notify(MethodShutdown, nil); err != nil { + t.Fatal(err) + } + var req Request + if err := json.Unmarshal(bytes.TrimSpace(buf.Bytes()), &req); err != nil { + t.Fatal(err) + } + if req.Method != MethodShutdown || req.ID != nil { + t.Fatalf("req = %+v", req) + } +} + +func serveEcho(r io.Reader, w io.Writer) { + req, ok := readRequest(r) + if !ok { + return + } + var in SearchRequest + _ = json.Unmarshal(req.Params, &in) + out := SearchResponse{} + if q := strings.TrimSpace(in.Query); q != "" { + out.Results = []*types.WebSearchResult{{ + Title: "t", URL: "https://x", Snippet: q, + }} + } + writeResult(w, req.ID, out) +} + +func readRequest(r io.Reader) (Request, bool) { + br := bufio.NewReader(r) + line, err := readLine(br) + if err != nil { + return Request{}, false + } + var req Request + if err := json.Unmarshal(line, &req); err != nil { + return Request{}, false + } + return req, true +} + +func writeResult(w io.Writer, id any, result any) { + raw, _ := json.Marshal(result) + resp := Response{JSONRPC: jsonrpcVersion, ID: id, Result: raw} + line, _ := json.Marshal(resp) + _, _ = w.Write(append(line, '\n')) +} diff --git a/internal/plugin/protocol/search.go b/internal/plugin/protocol/search.go new file mode 100644 index 0000000000..9f3a1bd432 --- /dev/null +++ b/internal/plugin/protocol/search.go @@ -0,0 +1,25 @@ +package protocol + +import "github.com/Tencent/WeKnora/internal/types" + +// Method names on the plugin ABI. New seams add methods; they do not add +// a new wire format. +const ( + MethodWebSearchSearch = "websearch.search" + MethodShutdown = "shutdown" +) + +// SearchRequest is the params object for websearch.search. +type SearchRequest struct { + Query string `json:"query"` + MaxResults int `json:"max_results"` + IncludeDate bool `json:"include_date"` + Parameters types.WebSearchProviderParameters `json:"parameters"` +} + +// SearchResponse is the result object for websearch.search. +// Error is for the HTTP fallback body; stdio plugins should use JSON-RPC errors. +type SearchResponse struct { + Results []*types.WebSearchResult `json:"results"` + Error string `json:"error,omitempty"` +} diff --git a/internal/plugin/runtime/bind.go b/internal/plugin/runtime/bind.go new file mode 100644 index 0000000000..1f6a892122 --- /dev/null +++ b/internal/plugin/runtime/bind.go @@ -0,0 +1,108 @@ +package runtime + +import ( + "fmt" + "os" + "strings" + + infra_web_search "github.com/Tencent/WeKnora/internal/infrastructure/web_search" + "github.com/Tencent/WeKnora/internal/plugin" + "github.com/Tencent/WeKnora/internal/types" + "github.com/Tencent/WeKnora/internal/types/interfaces" +) + +// RegisterManifests adds a factory for each disk plugin. First-wins so a +// dropped file cannot hijack a built-in id. +func RegisterManifests(manifests []plugin.Manifest) error { + for _, m := range manifests { + m := m + plugin.Register(m.ID, func(cfg plugin.Config) (plugin.Plugin, error) { + return newDiskPlugin(m, m.Config.Merge(cfg)) + }) + } + return nil +} + +func newDiskPlugin(m plugin.Manifest, cfg plugin.Config) (plugin.Plugin, error) { + if m.Seam != plugin.ServiceWebSearch { + return nil, fmt.Errorf("plugin %s: seam %q is not wired for disk load yet", m.ID, m.Seam) + } + return plugin.Func{ + ID: m.ID, + InjectKeys: []string{plugin.ServiceWebSearch}, + ApplyFn: func(ctx *plugin.Context) error { + reg, err := plugin.Service[*infra_web_search.Registry](ctx, plugin.ServiceWebSearch) + if err != nil { + return err + } + factory, closer, err := providerFactory(m, cfg) + if err != nil { + return err + } + id := m.ProviderID() + info := types.WebSearchProviderTypeInfo{ + ID: id, + Name: m.DisplayName(), + RequiresAPIKey: m.RequiresKey, + Description: strings.TrimSpace(m.Description), + DocsURL: m.DocsURL, + } + return ctx.Effect(func() plugin.Disposable { + reg.Register(id, factory) + types.RegisterWebSearchProviderType(info) + return plugin.DisposeFunc(func() { + if closer != nil { + _ = closer.Close() + } + reg.Unregister(id) + types.UnregisterWebSearchProviderType(id) + }) + }) + }, + }, nil +} + +func providerFactory(m plugin.Manifest, cfg plugin.Config) (infra_web_search.ProviderFactory, ioCloser, error) { + timeout := clampTimeout(m.Timeout()) + switch m.Runtime { + case plugin.RuntimeStdio: + session, err := newStdioSession(m) + if err != nil { + return nil, nil, err + } + return func(params types.WebSearchProviderParameters) (interfaces.WebSearchProvider, error) { + return session.withParams(params), nil + }, session, nil + case plugin.RuntimeHTTP: + endpoint := cfg.String("endpoint") + if endpoint == "" { + endpoint = m.Endpoint + } + base, err := newHTTPProvider(m.ProviderID(), endpoint, timeout) + if err != nil { + return nil, nil, err + } + return func(params types.WebSearchProviderParameters) (interfaces.WebSearchProvider, error) { + return base.withParams(params), nil + }, nil, nil + case plugin.RuntimeJS: + path := m.EntryPath() + src, err := os.ReadFile(path) + if err != nil { + return nil, nil, fmt.Errorf("plugin %s: read %s: %w", m.ID, path, err) + } + base, err := newJSProvider(m.ProviderID(), string(src), timeout) + if err != nil { + return nil, nil, err + } + return func(params types.WebSearchProviderParameters) (interfaces.WebSearchProvider, error) { + return base.withParams(params), nil + }, nil, nil + default: + return nil, nil, fmt.Errorf("plugin %s: runtime %q", m.ID, m.Runtime) + } +} + +type ioCloser interface { + Close() error +} diff --git a/internal/plugin/runtime/http.go b/internal/plugin/runtime/http.go new file mode 100644 index 0000000000..0229fd5912 --- /dev/null +++ b/internal/plugin/runtime/http.go @@ -0,0 +1,104 @@ +package runtime + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "net/url" + "strings" + "time" + + "github.com/Tencent/WeKnora/internal/plugin/protocol" + "github.com/Tencent/WeKnora/internal/types" + "github.com/Tencent/WeKnora/internal/types/interfaces" +) + +// httpProvider is a fallback for a search API that already exists as a +// remote service. New plugins should use stdio (JSON-RPC), not start HTTP. +// The endpoint comes from plugin.yaml (trusted); user query is JSON only. +type httpProvider struct { + name string + endpoint string + client *http.Client + params types.WebSearchProviderParameters +} + +func newHTTPProvider(name, endpoint string, timeout time.Duration) (*httpProvider, error) { + u, err := url.Parse(strings.TrimSpace(endpoint)) + if err != nil || u.Scheme == "" || u.Host == "" { + return nil, fmt.Errorf("plugin: invalid http endpoint %q", endpoint) + } + if u.Scheme != "http" && u.Scheme != "https" { + return nil, fmt.Errorf("plugin: http endpoint must be http(s)") + } + return &httpProvider{ + name: name, + endpoint: strings.TrimRight(endpoint, "/"), + client: &http.Client{Timeout: timeout}, + }, nil +} + +func (p *httpProvider) withParams(params types.WebSearchProviderParameters) *httpProvider { + cp := *p + cp.params = params + return &cp +} + +func (p *httpProvider) Name() string { return p.name } + +func (p *httpProvider) Search( + ctx context.Context, query string, maxResults int, includeDate bool, +) ([]*types.WebSearchResult, error) { + return p.search(ctx, query, maxResults, includeDate, p.params) +} + +func (p *httpProvider) search( + ctx context.Context, query string, maxResults int, includeDate bool, + params types.WebSearchProviderParameters, +) ([]*types.WebSearchResult, error) { + body, err := json.Marshal(protocol.SearchRequest{ + Query: query, MaxResults: maxResults, IncludeDate: includeDate, Parameters: params, + }) + if err != nil { + return nil, err + } + req, err := http.NewRequestWithContext(ctx, http.MethodPost, p.endpoint, bytes.NewReader(body)) + if err != nil { + return nil, err + } + req.Header.Set("Content-Type", "application/json") + req.Header.Set("Accept", "application/json") + resp, err := p.client.Do(req) + if err != nil { + return nil, fmt.Errorf("plugin %s: http: %w", p.name, err) + } + defer resp.Body.Close() + raw, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) + if err != nil { + return nil, err + } + if resp.StatusCode >= 300 { + return nil, fmt.Errorf("plugin %s: http status %d: %s", p.name, resp.StatusCode, truncate(raw, 200)) + } + var out SearchResponse + if err := json.Unmarshal(raw, &out); err != nil { + return nil, fmt.Errorf("plugin %s: decode: %w", p.name, err) + } + if out.Error != "" { + return nil, fmt.Errorf("plugin %s: %s", p.name, out.Error) + } + return out.Results, nil +} + +func truncate(b []byte, n int) string { + s := strings.TrimSpace(string(b)) + if len(s) > n { + return s[:n] + "..." + } + return s +} + +var _ interfaces.WebSearchProvider = (*httpProvider)(nil) diff --git a/internal/plugin/runtime/js.go b/internal/plugin/runtime/js.go new file mode 100644 index 0000000000..a3e0a66f51 --- /dev/null +++ b/internal/plugin/runtime/js.go @@ -0,0 +1,193 @@ +package runtime + +import ( + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "strings" + "sync" + "time" + + "github.com/dop251/goja" + + infra_web_search "github.com/Tencent/WeKnora/internal/infrastructure/web_search" + "github.com/Tencent/WeKnora/internal/types" + "github.com/Tencent/WeKnora/internal/types/interfaces" +) + +// jsProvider evaluates a dropped-in search.js. Network goes through +// WeKnora's SSRF-safe client via the host function httpRequest. +type jsProvider struct { + name string + source string + timeout time.Duration + client *http.Client + params types.WebSearchProviderParameters + mu sync.Mutex +} + +func newJSProvider(name, source string, timeout time.Duration) (*jsProvider, error) { + client, err := infra_web_search.NewSearchHTTPClient(timeout, "") + if err != nil { + return nil, err + } + p := &jsProvider{name: name, source: source, timeout: timeout, client: client} + if err := p.ensureSearchFn(); err != nil { + return nil, err + } + return p, nil +} + +func (p *jsProvider) withParams(params types.WebSearchProviderParameters) *jsProvider { + cp := *p + cp.params = params + return &cp +} + +func (p *jsProvider) Name() string { return p.name } + +func (p *jsProvider) Search( + ctx context.Context, query string, maxResults int, includeDate bool, +) ([]*types.WebSearchResult, error) { + p.mu.Lock() + defer p.mu.Unlock() + + vm, err := p.newVM() + if err != nil { + return nil, err + } + fn, ok := goja.AssertFunction(vm.Get("search")) + if !ok { + return nil, fmt.Errorf("plugin %s: search.js must define function search(...)", p.name) + } + + done := make(chan struct{}) + go func() { + select { + case <-ctx.Done(): + vm.Interrupt(ctx.Err().Error()) + case <-time.After(p.timeout): + vm.Interrupt("js timeout") + case <-done: + } + }() + defer close(done) + + val, err := fn( + goja.Undefined(), + vm.ToValue(query), + vm.ToValue(maxResults), + vm.ToValue(includeDate), + vm.ToValue(paramsToMap(p.params)), + ) + if err != nil { + return nil, fmt.Errorf("plugin %s: %w", p.name, err) + } + return decodeJSResults(val.Export()) +} + +func (p *jsProvider) ensureSearchFn() error { + vm, err := p.newVM() + if err != nil { + return err + } + if _, ok := goja.AssertFunction(vm.Get("search")); !ok { + return fmt.Errorf("plugin %s: search.js must define function search(...)", p.name) + } + return nil +} + +func (p *jsProvider) newVM() (*goja.Runtime, error) { + vm := goja.New() + if err := vm.Set("httpRequest", func(call goja.FunctionCall) goja.Value { + return p.doHTTP(vm, call) + }); err != nil { + return nil, err + } + if _, err := vm.RunString(p.source); err != nil { + return nil, fmt.Errorf("plugin %s: load js: %w", p.name, err) + } + return vm, nil +} + +func (p *jsProvider) doHTTP(vm *goja.Runtime, call goja.FunctionCall) goja.Value { + opts, _ := call.Argument(0).Export().(map[string]any) + if opts == nil { + panic(vm.NewTypeError("httpRequest expects an object")) + } + method := strings.ToUpper(stringFrom(opts["method"])) + if method == "" { + method = http.MethodGet + } + rawURL := stringFrom(opts["url"]) + if rawURL == "" { + panic(vm.NewTypeError("httpRequest.url is required")) + } + var body io.Reader + if b := stringFrom(opts["body"]); b != "" { + body = strings.NewReader(b) + } + req, err := http.NewRequest(method, rawURL, body) + if err != nil { + return httpResult(vm, 0, err.Error()) + } + if headers, ok := opts["headers"].(map[string]any); ok { + for k, v := range headers { + req.Header.Set(k, fmt.Sprint(v)) + } + } + resp, err := p.client.Do(req) + if err != nil { + return httpResult(vm, 0, err.Error()) + } + defer resp.Body.Close() + raw, err := io.ReadAll(io.LimitReader(resp.Body, 2<<20)) + if err != nil { + return httpResult(vm, 0, err.Error()) + } + return httpResult(vm, resp.StatusCode, string(raw)) +} + +func httpResult(vm *goja.Runtime, status int, body string) goja.Value { + o := vm.NewObject() + _ = o.Set("status", status) + _ = o.Set("body", body) + return o +} + +func stringFrom(v any) string { + if v == nil { + return "" + } + s, _ := v.(string) + return s +} + +func paramsToMap(p types.WebSearchProviderParameters) map[string]any { + return map[string]any{ + "api_key": p.APIKey, + "engine_id": p.EngineID, + "base_url": p.BaseURL, + "proxy_url": p.ProxyURL, + "extra_config": p.ExtraConfig, + } +} + +func decodeJSResults(v any) ([]*types.WebSearchResult, error) { + if v == nil { + return nil, nil + } + raw, err := json.Marshal(v) + if err != nil { + return nil, err + } + var out []*types.WebSearchResult + if err := json.Unmarshal(raw, &out); err != nil { + return nil, fmt.Errorf("search() must return an array of results: %w", err) + } + return out, nil +} + +var _ interfaces.WebSearchProvider = (*jsProvider)(nil) diff --git a/internal/plugin/runtime/protocol.go b/internal/plugin/runtime/protocol.go new file mode 100644 index 0000000000..277ceb1d2f --- /dev/null +++ b/internal/plugin/runtime/protocol.go @@ -0,0 +1,24 @@ +// Package runtime loads disk plugins onto WeKnora seams without a +// compile-time blank import. Language plugins speak protocol (JSON-RPC +// over stdio). JS runs in-process. HTTP is a fallback for an already +// running remote service — not the way to write a new plugin. +package runtime + +import ( + "time" + + "github.com/Tencent/WeKnora/internal/plugin/protocol" +) + +// SearchRequest is the JSON body posted to an HTTP search plugin. +type SearchRequest = protocol.SearchRequest + +// SearchResponse is the JSON body returned by an HTTP search plugin. +type SearchResponse = protocol.SearchResponse + +func clampTimeout(ms int) time.Duration { + if ms <= 0 { + ms = 10000 + } + return time.Duration(ms) * time.Millisecond +} diff --git a/internal/plugin/runtime/runtime_test.go b/internal/plugin/runtime/runtime_test.go new file mode 100644 index 0000000000..db073bb1e1 --- /dev/null +++ b/internal/plugin/runtime/runtime_test.go @@ -0,0 +1,166 @@ +package runtime + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "testing" + + infra_web_search "github.com/Tencent/WeKnora/internal/infrastructure/web_search" + "github.com/Tencent/WeKnora/internal/plugin" + "github.com/Tencent/WeKnora/internal/types" + "github.com/Tencent/WeKnora/internal/utils" +) + +func TestHTTPProviderSearch(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + t.Fatalf("method = %s", r.Method) + } + var req SearchRequest + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + t.Fatal(err) + } + if req.Query != "hello" || req.Parameters.APIKey != "k" { + t.Fatalf("req = %+v", req) + } + _ = json.NewEncoder(w).Encode(SearchResponse{Results: []*types.WebSearchResult{ + {Title: "T", URL: "https://example.com", Snippet: req.Query, Source: "http-echo"}, + }}) + })) + defer srv.Close() + + p, err := newHTTPProvider("http-echo", srv.URL, clampTimeout(2000)) + if err != nil { + t.Fatal(err) + } + p = p.withParams(types.WebSearchProviderParameters{APIKey: "k"}) + got, err := p.Search(context.Background(), "hello", 3, false) + if err != nil { + t.Fatal(err) + } + if len(got) != 1 || got[0].Snippet != "hello" { + t.Fatalf("got = %+v", got) + } +} + +func TestJSProviderSearch(t *testing.T) { + src := ` +function search(query, maxResults, includeDate, params) { + if (!query) return []; + return [{ + title: params.api_key || "js", + url: "https://weknora.local/js", + snippet: query, + source: "js-echo" + }]; +} +` + p, err := newJSProvider("js-echo", src, clampTimeout(2000)) + if err != nil { + t.Fatal(err) + } + p = p.withParams(types.WebSearchProviderParameters{APIKey: "from-params"}) + got, err := p.Search(context.Background(), " q ", 1, false) + if err != nil { + t.Fatal(err) + } + if len(got) != 1 || got[0].Title != "from-params" || got[0].Snippet != " q " { + t.Fatalf("got = %+v", got) + } +} + +func TestJSProviderHTTPRequest(t *testing.T) { + utils.ResetSSRFWhitelistForTest() + t.Setenv("SSRF_WHITELIST", "127.0.0.1,localhost") + defer utils.ResetSSRFWhitelistForTest() + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Write([]byte(`{"ok":true,"q":"` + r.URL.Query().Get("q") + `"}`)) + })) + defer srv.Close() + + src := fmt.Sprintf(` +function search(query, maxResults) { + var resp = httpRequest({ method: "GET", url: %q + "?q=" + encodeURIComponent(query) }); + if (resp.status !== 200) return []; + var data = JSON.parse(resp.body); + return [{ title: "remote", url: "https://example.com", snippet: data.q, source: "js-http" }]; +} +`, srv.URL) + p, err := newJSProvider("js-http", src, clampTimeout(3000)) + if err != nil { + t.Fatal(err) + } + got, err := p.Search(context.Background(), "hi", 1, false) + if err != nil { + t.Fatal(err) + } + if len(got) != 1 || got[0].Snippet != "hi" { + t.Fatalf("got = %+v", got) + } +} + +func TestRegisterManifestsMountsJS(t *testing.T) { + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, "search.js"), []byte(` +function search(query) { + return [{ title: "disk", url: "https://weknora.local/disk", snippet: query, source: "disk" }]; +} +`), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "plugin.yaml"), []byte(` +id: websearch.disk +name: Disk JS +seam: web_search +runtime: js +entry: search.js +description: runtime disk plugin +`), 0o644); err != nil { + t.Fatal(err) + } + ms, err := plugin.Discover([]string{dir}) + if err != nil { + t.Fatal(err) + } + if err := RegisterManifests(ms); err != nil { + t.Fatal(err) + } + reg := infra_web_search.NewRegistry() + h := plugin.NewHost() + h.Context().Provide(plugin.ServiceWebSearch, reg) + if err := h.Compose(plugin.Profile{Bundles: []string{plugin.ExternalBundle}}, map[string]plugin.Bundle{ + plugin.ExternalBundle: plugin.BundleFromManifests(ms), + }, nil); err != nil { + t.Fatal(err) + } + if !reg.Has("disk") { + t.Fatalf("list = %v", reg.List()) + } + if !types.IsKnownWebSearchProviderType("disk") { + t.Fatal("type catalog missing disk") + } + p, err := reg.CreateProvider("disk", types.WebSearchProviderParameters{}) + if err != nil { + t.Fatal(err) + } + got, err := p.Search(context.Background(), "ping", 1, false) + if err != nil || len(got) != 1 || got[0].Snippet != "ping" { + t.Fatalf("search = %+v, %v", got, err) + } + h.Unload() + if types.IsKnownWebSearchProviderType("disk") { + t.Fatal("type should unload with the plugin") + } +} + +func TestHTTPEndpointRejectsBadScheme(t *testing.T) { + if _, err := newHTTPProvider("x", "file:///etc/passwd", clampTimeout(1)); err == nil { + t.Fatal("expected scheme error") + } +} diff --git a/internal/plugin/runtime/stdio.go b/internal/plugin/runtime/stdio.go new file mode 100644 index 0000000000..d9c17db894 --- /dev/null +++ b/internal/plugin/runtime/stdio.go @@ -0,0 +1,202 @@ +package runtime + +import ( + "bufio" + "context" + "fmt" + "io" + "os" + "os/exec" + "sync" + "time" + + "github.com/Tencent/WeKnora/internal/logger" + "github.com/Tencent/WeKnora/internal/plugin" + "github.com/Tencent/WeKnora/internal/plugin/protocol" + "github.com/Tencent/WeKnora/internal/types" + "github.com/Tencent/WeKnora/internal/types/interfaces" +) + +// stdioSession is one host-launched subprocess that speaks the plugin ABI +// on stdin/stdout. The author does not open a port or serve HTTP. +type stdioSession struct { + name string + command string + args []string + dir string + env []string + timeout time.Duration + + mu sync.Mutex + closed bool + cmd *exec.Cmd + conn *protocol.Conn + runCancel context.CancelFunc + done <-chan struct{} +} + +func newStdioSession(m plugin.Manifest) (*stdioSession, error) { + name, args := m.Exec() + path, err := exec.LookPath(name) + if err != nil { + return nil, fmt.Errorf("plugin %s: command %q: %w", m.ID, name, err) + } + return &stdioSession{ + name: m.ProviderID(), + command: path, + args: args, + dir: m.Dir, + env: extraEnv(m.Env), + timeout: clampTimeout(m.Timeout()), + }, nil +} + +func (s *stdioSession) withParams(params types.WebSearchProviderParameters) *stdioProvider { + return &stdioProvider{session: s, params: params} +} + +func (s *stdioSession) Close() error { + s.mu.Lock() + defer s.mu.Unlock() + s.closed = true + s.stopLocked() + return nil +} + +func (s *stdioSession) search( + ctx context.Context, query string, maxResults int, includeDate bool, + params types.WebSearchProviderParameters, +) ([]*types.WebSearchResult, error) { + s.mu.Lock() + defer s.mu.Unlock() + if s.closed { + return nil, fmt.Errorf("plugin %s: stdio session closed", s.name) + } + if err := s.ensureLocked(); err != nil { + return nil, err + } + + callCtx, cancel := context.WithTimeout(ctx, s.timeout) + defer cancel() + + var out protocol.SearchResponse + err := s.conn.Call(callCtx, protocol.MethodWebSearchSearch, protocol.SearchRequest{ + Query: query, MaxResults: maxResults, IncludeDate: includeDate, Parameters: params, + }, &out) + if err != nil { + s.stopLocked() + return nil, fmt.Errorf("plugin %s: stdio: %w", s.name, err) + } + if out.Error != "" { + return nil, fmt.Errorf("plugin %s: %s", s.name, out.Error) + } + return out.Results, nil +} + +func (s *stdioSession) ensureLocked() error { + if s.cmd != nil { + select { + case <-s.done: + s.clearLocked() + default: + return nil + } + } + return s.startLocked() +} + +func (s *stdioSession) startLocked() error { + runCtx, cancel := context.WithCancel(context.Background()) + cmd := exec.CommandContext(runCtx, s.command, s.args...) + cmd.Dir = s.dir + if len(s.env) > 0 { + cmd.Env = append(os.Environ(), s.env...) + } + stdin, err := cmd.StdinPipe() + if err != nil { + cancel() + return err + } + stdout, err := cmd.StdoutPipe() + if err != nil { + cancel() + return err + } + stderr, err := cmd.StderrPipe() + if err != nil { + cancel() + return err + } + if err := cmd.Start(); err != nil { + cancel() + return fmt.Errorf("plugin %s: start %s: %w", s.name, s.command, err) + } + done := make(chan struct{}) + go func() { + _ = cmd.Wait() + close(done) + }() + go drainStderr(s.name, stderr) + + s.cmd = cmd + s.conn = protocol.NewConn(stdout, stdin) + s.runCancel = cancel + s.done = done + return nil +} + +func (s *stdioSession) stopLocked() { + if s.conn != nil { + _ = s.conn.Notify(protocol.MethodShutdown, nil) + } + if s.runCancel != nil { + s.runCancel() + } + if s.done != nil { + <-s.done + } + s.clearLocked() +} + +func (s *stdioSession) clearLocked() { + s.cmd = nil + s.conn = nil + s.runCancel = nil + s.done = nil +} + +func extraEnv(kv map[string]string) []string { + if len(kv) == 0 { + return nil + } + out := make([]string, 0, len(kv)) + for k, v := range kv { + if k == "" { + continue + } + out = append(out, k+"="+v) + } + return out +} + +func drainStderr(name string, r io.Reader) { + sc := bufio.NewScanner(r) + for sc.Scan() { + logger.Debugf(context.Background(), "[plugin %s] %s", name, sc.Text()) + } +} + +type stdioProvider struct { + session *stdioSession + params types.WebSearchProviderParameters +} + +func (p *stdioProvider) Name() string { return p.session.name } + +func (p *stdioProvider) Search( + ctx context.Context, query string, maxResults int, includeDate bool, +) ([]*types.WebSearchResult, error) { + return p.session.search(ctx, query, maxResults, includeDate, p.params) +} + +var _ interfaces.WebSearchProvider = (*stdioProvider)(nil) diff --git a/internal/plugin/runtime/stdio_test.go b/internal/plugin/runtime/stdio_test.go new file mode 100644 index 0000000000..d5f3e8e29f --- /dev/null +++ b/internal/plugin/runtime/stdio_test.go @@ -0,0 +1,230 @@ +package runtime + +import ( + "bufio" + "context" + "encoding/json" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + + infra_web_search "github.com/Tencent/WeKnora/internal/infrastructure/web_search" + "github.com/Tencent/WeKnora/internal/plugin" + "github.com/Tencent/WeKnora/internal/plugin/protocol" + "github.com/Tencent/WeKnora/internal/types" +) + +func TestMain(m *testing.M) { + if os.Getenv("WEKNORA_STDIO_HELPER") != "" { + runStdioHelper(os.Getenv("WEKNORA_STDIO_HELPER")) + os.Exit(0) + } + os.Exit(m.Run()) +} + +func runStdioHelper(kind string) { + sc := bufio.NewScanner(os.Stdin) + enc := json.NewEncoder(os.Stdout) + for sc.Scan() { + var req protocol.Request + if err := json.Unmarshal(sc.Bytes(), &req); err != nil { + continue + } + if req.Method == protocol.MethodShutdown { + return + } + if kind == "sleep" { + select {} + } + var in protocol.SearchRequest + _ = json.Unmarshal(req.Params, &in) + out := protocol.SearchResponse{} + if q := strings.TrimSpace(in.Query); q != "" { + title := "stdio-helper" + if in.Parameters.APIKey != "" { + title = in.Parameters.APIKey + } + out.Results = []*types.WebSearchResult{{ + Title: title, URL: "https://weknora.local/stdio", + Snippet: q, Source: "stdio-helper", + }} + } + raw, _ := json.Marshal(out) + _ = enc.Encode(protocol.Response{JSONRPC: "2.0", ID: req.ID, Result: raw}) + } +} + +func helperManifest(t *testing.T, kind string, timeoutMS int) plugin.Manifest { + t.Helper() + return plugin.Manifest{ + ID: "websearch.stdio-helper", + Seam: plugin.ServiceWebSearch, + Runtime: plugin.RuntimeStdio, + Command: os.Args[0], + Args: []string{"-test.run=^$"}, + Provider: "stdio-helper", + TimeoutMS: timeoutMS, + Env: map[string]string{"WEKNORA_STDIO_HELPER": kind}, + Dir: t.TempDir(), + } +} + +func TestStdioMissingCommand(t *testing.T) { + _, err := newStdioSession(plugin.Manifest{ + ID: "websearch.missing", Seam: plugin.ServiceWebSearch, Runtime: plugin.RuntimeStdio, + Command: "weknora-no-such-bin-xyz", Dir: t.TempDir(), + }) + if err == nil { + t.Fatal("expected lookpath error") + } +} + +func TestStdioProviderSearch(t *testing.T) { + session, err := newStdioSession(helperManifest(t, "echo", 3000)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = session.Close() }) + + p := session.withParams(types.WebSearchProviderParameters{APIKey: "keyed"}) + got, err := p.Search(context.Background(), "hello", 3, false) + if err != nil { + t.Fatal(err) + } + if len(got) != 1 || got[0].Title != "keyed" || got[0].Snippet != "hello" { + t.Fatalf("got = %+v", got) + } + + got, err = p.Search(context.Background(), "again", 1, false) + if err != nil || len(got) != 1 || got[0].Snippet != "again" { + t.Fatalf("reuse = %+v, %v", got, err) + } +} + +func TestStdioProviderTimeoutRestarts(t *testing.T) { + session, err := newStdioSession(helperManifest(t, "sleep", 50)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = session.Close() }) + + _, err = session.withParams(types.WebSearchProviderParameters{}). + Search(context.Background(), "x", 1, false) + if err == nil { + t.Fatal("expected timeout") + } + + session.timeout = clampTimeout(3000) + session.env = extraEnv(map[string]string{"WEKNORA_STDIO_HELPER": "echo"}) + got, err := session.withParams(types.WebSearchProviderParameters{}). + Search(context.Background(), "recovered", 1, false) + if err != nil || len(got) != 1 || got[0].Snippet != "recovered" { + t.Fatalf("after restart = %+v, %v", got, err) + } +} + +func TestStdioCloseRejectsSearch(t *testing.T) { + session, err := newStdioSession(helperManifest(t, "echo", 3000)) + if err != nil { + t.Fatal(err) + } + p := session.withParams(types.WebSearchProviderParameters{}) + if _, err := p.Search(context.Background(), "x", 1, false); err != nil { + t.Fatal(err) + } + if err := session.Close(); err != nil { + t.Fatal(err) + } + if _, err := p.Search(context.Background(), "x", 1, false); err == nil { + t.Fatal("expected closed session") + } +} + +func TestRegisterManifestsMountsStdio(t *testing.T) { + m := helperManifest(t, "echo", 3000) + if err := RegisterManifests([]plugin.Manifest{m}); err != nil { + t.Fatal(err) + } + reg := infra_web_search.NewRegistry() + h := plugin.NewHost() + h.Context().Provide(plugin.ServiceWebSearch, reg) + if err := h.Compose(plugin.Profile{Bundles: []string{plugin.ExternalBundle}}, map[string]plugin.Bundle{ + plugin.ExternalBundle: plugin.BundleFromManifests([]plugin.Manifest{m}), + }, nil); err != nil { + t.Fatal(err) + } + p, err := reg.CreateProvider("stdio-helper", types.WebSearchProviderParameters{}) + if err != nil { + t.Fatal(err) + } + got, err := p.Search(context.Background(), "mounted", 1, false) + if err != nil || len(got) != 1 || got[0].Snippet != "mounted" { + t.Fatalf("search = %+v, %v", got, err) + } + h.Unload() + if reg.Has("stdio-helper") { + t.Fatal("unload should drop stdio provider") + } +} + +func TestStdioPythonSample(t *testing.T) { + if _, err := exec.LookPath("python3"); err != nil { + t.Skip("python3 not installed") + } + dir := filepath.Join(repoRoot(t), "plugins.d", "websearch-stdio-echo") + ms, err := plugin.Discover([]string{dir}) + if err != nil || len(ms) != 1 { + t.Fatalf("discover = %v, %v", ms, err) + } + session, err := newStdioSession(ms[0]) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = session.Close() }) + got, err := session.withParams(types.WebSearchProviderParameters{}). + Search(context.Background(), "from-python", 1, false) + if err != nil || len(got) != 1 || got[0].Source != "stdio-echo" { + t.Fatalf("python sample = %+v, %v", got, err) + } +} + +func TestStdioNodeSample(t *testing.T) { + if _, err := exec.LookPath("node"); err != nil { + t.Skip("node not installed") + } + dir := filepath.Join(repoRoot(t), "plugins.d", "websearch-node-echo") + ms, err := plugin.Discover([]string{dir}) + if err != nil || len(ms) != 1 { + t.Fatalf("discover = %v, %v", ms, err) + } + session, err := newStdioSession(ms[0]) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = session.Close() }) + got, err := session.withParams(types.WebSearchProviderParameters{}). + Search(context.Background(), "from-node", 1, false) + if err != nil || len(got) != 1 || got[0].Source != "node-echo" { + t.Fatalf("node sample = %+v, %v", got, err) + } +} + +func repoRoot(t *testing.T) string { + t.Helper() + dir, err := os.Getwd() + if err != nil { + t.Fatal(err) + } + for { + if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil { + return dir + } + parent := filepath.Dir(dir) + if parent == dir { + t.Fatal("go.mod not found") + } + dir = parent + } +} diff --git a/internal/plugin/seams.go b/internal/plugin/seams.go new file mode 100644 index 0000000000..2a103a5f76 --- /dev/null +++ b/internal/plugin/seams.go @@ -0,0 +1,28 @@ +package plugin + +// Well-known Context service keys. A seam is a swappable capability: the +// key names the Service Definition, a plugin Provides or Registers a +// Service Provider, and existing WeKnora services consume it. +// +// Keys stay stable so out-of-tree plugins can depend on them without +// importing a concrete implementation. +const ( + ServiceWebSearch = "web_search" + ServiceRetriever = "retriever" + ServiceStorage = "storage" + ServiceConnector = "datasource" + ServiceIM = "im" + ServiceModel = "model" + ServiceChunker = "chunker" + ServiceParser = "parser" + ServiceAgentTool = "agent_tool" + ServiceChat = "chat_pipeline" +) + +// Well-known event names. Dispatch mode is part of the public contract. +const ( + // EventPluginMounted is emitted after a plugin Apply succeeds. Mode: emit. + EventPluginMounted = "plugin/mounted" + // EventPluginUnloaded is emitted after a plugin context is closed. Mode: emit. + EventPluginUnloaded = "plugin/unloaded" +) diff --git a/internal/plugin/websearch/plugins.go b/internal/plugin/websearch/plugins.go new file mode 100644 index 0000000000..da9d7c37bd --- /dev/null +++ b/internal/plugin/websearch/plugins.go @@ -0,0 +1,83 @@ +// Package websearch registers in-tree web search engines as plugins. +// Adding a new in-tree engine means a factory here plus an Entry in Bundle; +// container.go no longer lists providers. +package websearch + +import ( + "github.com/Tencent/WeKnora/internal/infrastructure/web_search" + "github.com/Tencent/WeKnora/internal/plugin" +) + +const prefix = "websearch." + +// BundleName is the profile bundle that mounts built-in search engines. +const BundleName = "base" + +type spec struct { + id string + factory web_search.ProviderFactory +} + +func builtins() []spec { + return []spec{ + {"duckduckgo", web_search.NewDuckDuckGoProvider}, + {"google", web_search.NewGoogleProvider}, + {"bing", web_search.NewBingProvider}, + {"tavily", web_search.NewTavilyProvider}, + {"ollama", web_search.NewOllamaProvider}, + {"baidu", web_search.NewBaiduProvider}, + {"searxng", web_search.NewSearxngProvider}, + {"keenable", web_search.NewKeenableProvider}, + {"zhipu", web_search.NewZhipuProvider}, + {"exa", web_search.NewExaProvider}, + {"metaso", web_search.NewMetasoProvider}, + } +} + +func init() { + for _, s := range builtins() { + id, factory := prefix+s.id, s.factory + providerID := s.id + plugin.Register(id, func(plugin.Config) (plugin.Plugin, error) { + return providerPlugin(id, providerID, factory), nil + }) + } +} + +func providerPlugin(pluginID, providerID string, factory web_search.ProviderFactory) plugin.Plugin { + return plugin.Func{ + ID: pluginID, + InjectKeys: []string{plugin.ServiceWebSearch}, + ApplyFn: func(ctx *plugin.Context) error { + reg, err := plugin.Service[*web_search.Registry](ctx, plugin.ServiceWebSearch) + if err != nil { + return err + } + return ctx.Effect(func() plugin.Disposable { + reg.Register(providerID, factory) + return plugin.DisposeFunc(func() { reg.Unregister(providerID) }) + }) + }, + } +} + +// Bundle returns the base bundle of in-tree web search plugins. +func Bundle() plugin.Bundle { + entries := make([]plugin.Entry, 0, len(builtins())) + for _, s := range builtins() { + id := prefix + s.id + entries = append(entries, plugin.Entry{ID: id, Plugin: id}) + } + return plugin.Bundle{Name: BundleName, Entries: entries} +} + +// Bundles is the default bundle map used by the WeKnora plugin host. +func Bundles() map[string]plugin.Bundle { + b := Bundle() + return map[string]plugin.Bundle{b.Name: b} +} + +// DefaultProfile stacks the base bundle. YAML / env overlays patch it. +func DefaultProfile() plugin.Profile { + return plugin.Profile{Name: "standard", Bundles: []string{BundleName}} +} diff --git a/internal/plugin/websearch/plugins_test.go b/internal/plugin/websearch/plugins_test.go new file mode 100644 index 0000000000..16b897747c --- /dev/null +++ b/internal/plugin/websearch/plugins_test.go @@ -0,0 +1,63 @@ +package websearch + +import ( + "testing" + + "github.com/Tencent/WeKnora/internal/infrastructure/web_search" + "github.com/Tencent/WeKnora/internal/plugin" + "github.com/Tencent/WeKnora/internal/types" +) + +func TestBundleMatchesBuiltins(t *testing.T) { + b := Bundle() + if len(b.Entries) != len(builtins()) { + t.Fatalf("bundle %d != builtins %d", len(b.Entries), len(builtins())) + } + for i, s := range builtins() { + want := prefix + s.id + if b.Entries[i].ID != want || b.Entries[i].Plugin != want { + t.Fatalf("entry[%d] = %+v, want %s", i, b.Entries[i], want) + } + if _, ok := plugin.LookupFactory(want); !ok { + t.Fatalf("factory %s not registered", want) + } + } +} + +func TestProviderPluginRegistersAndUnregisters(t *testing.T) { + reg := web_search.NewRegistry() + h := plugin.NewHost() + h.Context().Provide(plugin.ServiceWebSearch, reg) + if err := h.Compose(DefaultProfile(), Bundles(), nil); err != nil { + t.Fatal(err) + } + if !reg.Has("duckduckgo") || !reg.Has("exa") { + t.Fatalf("list = %v", reg.List()) + } + if _, err := reg.CreateProvider("duckduckgo", types.WebSearchProviderParameters{}); err != nil { + t.Fatal(err) + } + h.Unload() + if reg.Has("duckduckgo") { + t.Fatal("duckduckgo should unload with the plugin") + } +} + +func TestPatchDisablesProvider(t *testing.T) { + off := true + reg := web_search.NewRegistry() + h := plugin.NewHost() + h.Context().Provide(plugin.ServiceWebSearch, reg) + profile := DefaultProfile() + profile.Patch = []plugin.Patch{{ID: "websearch.exa", Disabled: &off}} + if err := h.Compose(profile, Bundles(), nil); err != nil { + t.Fatal(err) + } + if reg.Has("exa") { + t.Fatal("exa should be disabled by patch") + } + if !reg.Has("bing") { + t.Fatal("bing should still be mounted") + } + h.Unload() +} diff --git a/internal/types/web_search_provider.go b/internal/types/web_search_provider.go index 0a21cfbc6b..f22ae00263 100644 --- a/internal/types/web_search_provider.go +++ b/internal/types/web_search_provider.go @@ -4,6 +4,7 @@ import ( "database/sql/driver" "encoding/json" "log" + "sync" "time" "github.com/Tencent/WeKnora/internal/utils" @@ -169,8 +170,68 @@ type WebSearchProviderConfigFieldOption struct { Value string `json:"value"` } +var ( + extraProviderMu sync.RWMutex + extraProviderTypes = map[string]WebSearchProviderTypeInfo{} +) + +// RegisterWebSearchProviderType adds a runtime plugin type to the UI catalog. +// Built-in ids are left untouched. +func RegisterWebSearchProviderType(info WebSearchProviderTypeInfo) { + if info.ID == "" { + return + } + extraProviderMu.Lock() + extraProviderTypes[info.ID] = info + extraProviderMu.Unlock() +} + +// UnregisterWebSearchProviderType removes a runtime plugin type. +func UnregisterWebSearchProviderType(id string) { + extraProviderMu.Lock() + delete(extraProviderTypes, id) + extraProviderMu.Unlock() +} + +// LookupWebSearchProviderType finds built-in or runtime type metadata. +func LookupWebSearchProviderType(id string) (WebSearchProviderTypeInfo, bool) { + for _, info := range builtinWebSearchProviderTypes() { + if info.ID == id { + return info, true + } + } + extraProviderMu.RLock() + info, ok := extraProviderTypes[id] + extraProviderMu.RUnlock() + return info, ok +} + +// IsKnownWebSearchProviderType reports whether id is a built-in or disk plugin. +func IsKnownWebSearchProviderType(id string) bool { + _, ok := LookupWebSearchProviderType(id) + return ok +} + // GetWebSearchProviderTypes returns metadata for all supported provider types. func GetWebSearchProviderTypes() []WebSearchProviderTypeInfo { + out := builtinWebSearchProviderTypes() + seen := make(map[string]struct{}, len(out)) + for _, info := range out { + seen[info.ID] = struct{}{} + } + extraProviderMu.RLock() + extras := make([]WebSearchProviderTypeInfo, 0, len(extraProviderTypes)) + for id, info := range extraProviderTypes { + if _, ok := seen[id]; !ok { + extras = append(extras, info) + } + } + extraProviderMu.RUnlock() + out = append(out, extras...) + return out +} + +func builtinWebSearchProviderTypes() []WebSearchProviderTypeInfo { return []WebSearchProviderTypeInfo{ { ID: "duckduckgo", diff --git a/internal/types/web_search_provider_extra_test.go b/internal/types/web_search_provider_extra_test.go new file mode 100644 index 0000000000..623658e688 --- /dev/null +++ b/internal/types/web_search_provider_extra_test.go @@ -0,0 +1,30 @@ +package types + +import "testing" + +func TestRuntimeProviderTypeCatalog(t *testing.T) { + id := "runtime-catalog-test" + RegisterWebSearchProviderType(WebSearchProviderTypeInfo{ID: id, Name: "Runtime"}) + t.Cleanup(func() { UnregisterWebSearchProviderType(id) }) + + if !IsKnownWebSearchProviderType(id) { + t.Fatal("expected runtime type") + } + if !IsKnownWebSearchProviderType("bing") { + t.Fatal("builtin bing") + } + found := false + for _, info := range GetWebSearchProviderTypes() { + if info.ID == id { + found = true + break + } + } + if !found { + t.Fatal("GetWebSearchProviderTypes should include runtime type") + } + UnregisterWebSearchProviderType(id) + if IsKnownWebSearchProviderType(id) { + t.Fatal("unregistered type should be gone") + } +} diff --git a/plugins.d/websearch-js-echo/plugin.yaml b/plugins.d/websearch-js-echo/plugin.yaml new file mode 100644 index 0000000000..928b048968 --- /dev/null +++ b/plugins.d/websearch-js-echo/plugin.yaml @@ -0,0 +1,11 @@ +# Drop-in JS search plugin. No WeKnora rebuild, no blank import. +# Copied into WEKNORA_PLUGIN_DIR (default: ./plugins.d). +id: websearch.js-echo +name: JS Echo +version: 0.1.0 +description: Returns the query as a single result. Template for disk plugins. +seam: web_search +runtime: js +entry: search.js +provider: js-echo +auto_enable: true diff --git a/plugins.d/websearch-js-echo/search.js b/plugins.d/websearch-js-echo/search.js new file mode 100644 index 0000000000..9460781c2b --- /dev/null +++ b/plugins.d/websearch-js-echo/search.js @@ -0,0 +1,17 @@ +// search(query, maxResults, includeDate, params) → [{title,url,snippet,content,source}] +// httpRequest({method,url,headers,body}) is provided by the host and is SSRF-safe. +function search(query, maxResults, includeDate, params) { + var q = String(query || "").trim(); + if (!q || maxResults === 0) { + return []; + } + return [ + { + title: (params && params.api_key) ? "js-echo (keyed)" : "js-echo", + url: "https://weknora.local/plugin/js-echo", + snippet: q, + content: q, + source: "js-echo", + }, + ]; +} diff --git a/plugins.d/websearch-node-echo/index.js b/plugins.d/websearch-node-echo/index.js new file mode 100644 index 0000000000..97bd25a8d6 --- /dev/null +++ b/plugins.d/websearch-node-echo/index.js @@ -0,0 +1,46 @@ +#!/usr/bin/env node +// WeKnora stdio plugin: JSON-RPC 2.0, one message per line. Logs go to stderr. +const readline = require("node:readline"); + +const rl = readline.createInterface({ input: process.stdin }); +rl.on("line", (line) => { + let msg; + try { + msg = JSON.parse(line); + } catch { + return; + } + if (msg.method === "shutdown") { + process.exit(0); + } + if (msg.method !== "websearch.search") { + write({ + jsonrpc: "2.0", + id: msg.id, + error: { code: -32601, message: "method not found" }, + }); + return; + } + const q = String((msg.params && msg.params.query) || "").trim(); + write({ + jsonrpc: "2.0", + id: msg.id, + result: { + results: q + ? [ + { + title: "node-echo", + url: "https://weknora.local/plugin/node-echo", + snippet: q, + content: q, + source: "node-echo", + }, + ] + : [], + }, + }); +}); + +function write(obj) { + process.stdout.write(JSON.stringify(obj) + "\n"); +} diff --git a/plugins.d/websearch-node-echo/plugin.yaml b/plugins.d/websearch-node-echo/plugin.yaml new file mode 100644 index 0000000000..4a816e8a77 --- /dev/null +++ b/plugins.d/websearch-node-echo/plugin.yaml @@ -0,0 +1,11 @@ +# TypeScript / Node authors: implement handlers, do not listen on a port. +id: websearch.node-echo +name: Node Echo +version: 0.1.0 +description: Node sample for runtime stdio. Off by default. +seam: web_search +runtime: stdio +command: node +entry: index.js +provider: node-echo +auto_enable: false diff --git a/plugins.d/websearch-stdio-echo/plugin.py b/plugins.d/websearch-stdio-echo/plugin.py new file mode 100644 index 0000000000..f8512e0592 --- /dev/null +++ b/plugins.d/websearch-stdio-echo/plugin.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +"""WeKnora stdio plugin: JSON-RPC 2.0, one message per line, logs on stderr.""" + +from __future__ import annotations + +import json +import sys + + +def handle(msg: dict) -> dict | None: + method = msg.get("method") + if method == "shutdown": + return None + if method != "websearch.search": + return { + "jsonrpc": "2.0", + "id": msg.get("id"), + "error": {"code": -32601, "message": "method not found"}, + } + params = msg.get("params") or {} + query = str(params.get("query") or "").strip() + results = [] + if query: + results.append( + { + "title": "stdio-echo", + "url": "https://weknora.local/plugin/stdio-echo", + "snippet": query, + "content": query, + "source": "stdio-echo", + } + ) + return {"jsonrpc": "2.0", "id": msg.get("id"), "result": {"results": results}} + + +def main() -> None: + for raw in sys.stdin: + line = raw.strip() + if not line: + continue + try: + msg = json.loads(line) + except json.JSONDecodeError: + continue + resp = handle(msg) + if resp is None: + return + sys.stdout.write(json.dumps(resp, ensure_ascii=False) + "\n") + sys.stdout.flush() + + +if __name__ == "__main__": + main() diff --git a/plugins.d/websearch-stdio-echo/plugin.yaml b/plugins.d/websearch-stdio-echo/plugin.yaml new file mode 100644 index 0000000000..2529c2f366 --- /dev/null +++ b/plugins.d/websearch-stdio-echo/plugin.yaml @@ -0,0 +1,12 @@ +# Language-agnostic search plugin. Host launches the process and talks +# JSON-RPC 2.0 on stdin/stdout — no HTTP server, no port. +id: websearch.stdio-echo +name: Stdio Echo +version: 0.1.0 +description: Python sample for runtime stdio. Off by default. +seam: web_search +runtime: stdio +command: python3 +entry: plugin.py +provider: stdio-echo +auto_enable: false diff --git a/plugins/README.md b/plugins/README.md new file mode 100644 index 0000000000..aa7c3cb4c1 --- /dev/null +++ b/plugins/README.md @@ -0,0 +1,15 @@ +# WeKnora plugins + +Add a capability without editing `container.go`: + +| Path | Language | Rebuild WeKnora? | How it loads | +| --- | --- | --- | --- | +| `../plugins.d//plugin.yaml` | any (`runtime: stdio`) or JS (`runtime: js`) | No | Host scans `WEKNORA_PLUGIN_DIR` | +| `websearch-echo/` | Go | Yes (blank import) | `WEKNORA_PLUGINS=websearch.echo` | +| `sdk-ts/websearch/` | TypeScript | No | `serve()` on stdin/stdout | + +`runtime: http` is a fallback for an already-running remote service, not +the way to write a new plugin. + +In-tree engines stay in `internal/plugin/websearch` (bundle `base`). +Design: `docs/dev/plugin-architecture.md`. diff --git a/plugins/sdk-ts/websearch/README.md b/plugins/sdk-ts/websearch/README.md new file mode 100644 index 0000000000..3a38183063 --- /dev/null +++ b/plugins/sdk-ts/websearch/README.md @@ -0,0 +1,56 @@ +# TypeScript web search plugin + +WeKnora cannot `import()` an npm package into the Go process. The +language-agnostic path is the same one MCP, LSP, and Dify local plugins +use: **the host launches your process and speaks JSON-RPC 2.0 on +stdin/stdout**. You implement handlers. You do not start an HTTP server. + +## plugin.yaml + +```yaml +id: websearch.brave-ts +name: Brave (TS) +seam: web_search +runtime: stdio +command: node +entry: index.js +provider: brave-ts +requires_api_key: true +``` + +Drop that folder in `WEKNORA_PLUGIN_DIR` (default `plugins.d//`). + +## Author surface + +```ts +import { serve } from "./serve.ts" + +serve({ + async search(req) { + return { results: [/* title, url, snippet */] } + }, +}) +``` + +`npx --yes tsx plugins/sdk-ts/websearch/example.ts` is a one-file echo. +A copy that needs no tsx lives at `plugins.d/websearch-node-echo/`. + +## Wire format + +One JSON-RPC 2.0 object per line. Logs go to **stderr** (stdout is the +protocol). Methods: + +| Method | Kind | Payload | +| --- | --- | --- | +| `websearch.search` | request | `{ query, max_results, include_date, parameters }` | +| `shutdown` | notification | none | + +`parameters` is the tenant bag (`api_key`, `engine_id`, `base_url`, …). + +## What about HTTP? + +`runtime: http` is a **fallback** for a search API that already exists +as a remote service. Do not invent a sidecar just to write a plugin. +If you later need a shared multi-tenant endpoint, the JSON body is the +same `SearchRequest` / `SearchResponse` as `plugin.invoke` result. +`example-server.ts` stays only as that fallback illustration. diff --git a/plugins/sdk-ts/websearch/example-server.ts b/plugins/sdk-ts/websearch/example-server.ts new file mode 100644 index 0000000000..71ad558b4b --- /dev/null +++ b/plugins/sdk-ts/websearch/example-server.ts @@ -0,0 +1,44 @@ +// Fallback only: use this when the search API already exists as a remote +// HTTP service. New plugins should use serve.ts (stdio JSON-RPC). +import { createServer, type IncomingMessage, type ServerResponse } from "node:http" +import type { SearchRequest, SearchResponse } from "./protocol.ts" + +const port = Number(process.env.PORT || 9101) + +function readBody(req: IncomingMessage): Promise { + return new Promise((resolve, reject) => { + const chunks: Buffer[] = [] + req.on("data", (c) => chunks.push(c)) + req.on("end", () => resolve(Buffer.concat(chunks).toString("utf8"))) + req.on("error", reject) + }) +} + +const server = createServer(async (req: IncomingMessage, res: ServerResponse) => { + if (req.method !== "POST" || req.url !== "/search") { + res.writeHead(404) + res.end() + return + } + const body = JSON.parse(await readBody(req)) as SearchRequest + const q = (body.query || "").trim() + const out: SearchResponse = { + results: q + ? [ + { + title: "ts-echo", + url: "https://weknora.local/plugin/ts-echo", + snippet: q, + content: q, + source: "ts-echo", + }, + ] + : [], + } + res.writeHead(200, { "content-type": "application/json" }) + res.end(JSON.stringify(out)) +}) + +server.listen(port, "127.0.0.1", () => { + console.log(`weknora ts websearch sidecar on http://127.0.0.1:${port}/search`) +}) diff --git a/plugins/sdk-ts/websearch/example.ts b/plugins/sdk-ts/websearch/example.ts new file mode 100644 index 0000000000..a92167f4b8 --- /dev/null +++ b/plugins/sdk-ts/websearch/example.ts @@ -0,0 +1,20 @@ +import { serve } from "./serve.ts" + +serve({ + search(req) { + const q = (req.query || "").trim() + return { + results: q + ? [ + { + title: "ts-echo", + url: "https://weknora.local/plugin/ts-echo", + snippet: q, + content: q, + source: "ts-echo", + }, + ] + : [], + } + }, +}) diff --git a/plugins/sdk-ts/websearch/protocol.ts b/plugins/sdk-ts/websearch/protocol.ts new file mode 100644 index 0000000000..89b9e35273 --- /dev/null +++ b/plugins/sdk-ts/websearch/protocol.ts @@ -0,0 +1,24 @@ +export type SearchRequest = { + query: string + max_results: number + include_date: boolean + parameters: { + api_key?: string + engine_id?: string + base_url?: string + extra_config?: Record + } +} + +export type SearchHit = { + title: string + url: string + snippet?: string + content?: string + source?: string +} + +export type SearchResponse = { + results: SearchHit[] + error?: string +} diff --git a/plugins/sdk-ts/websearch/serve.ts b/plugins/sdk-ts/websearch/serve.ts new file mode 100644 index 0000000000..342bf7a68d --- /dev/null +++ b/plugins/sdk-ts/websearch/serve.ts @@ -0,0 +1,60 @@ +import { createInterface } from "node:readline" +import { writeSync } from "node:fs" +import type { SearchRequest, SearchResponse } from "./protocol.ts" + +export type Handlers = { + search: (req: SearchRequest) => SearchResponse | Promise +} + +type JsonRpcRequest = { + jsonrpc?: string + id?: number | string + method?: string + params?: SearchRequest +} + +/** + * Speak the WeKnora plugin ABI on stdin/stdout. + * Do not open a port. Logs must go to stderr. + */ +export function serve(handlers: Handlers): void { + const rl = createInterface({ input: process.stdin }) + rl.on("line", (line) => { + void handleLine(line, handlers) + }) +} + +async function handleLine(line: string, handlers: Handlers): Promise { + const trimmed = line.trim() + if (!trimmed) { + return + } + let msg: JsonRpcRequest + try { + msg = JSON.parse(trimmed) as JsonRpcRequest + } catch { + return + } + if (msg.method === "shutdown") { + process.exit(0) + } + if (msg.method !== "websearch.search") { + write({ + jsonrpc: "2.0", + id: msg.id, + error: { code: -32601, message: `method not found: ${msg.method}` }, + }) + return + } + try { + const result = await handlers.search(msg.params || ({} as SearchRequest)) + write({ jsonrpc: "2.0", id: msg.id, result }) + } catch (err) { + const message = err instanceof Error ? err.message : String(err) + write({ jsonrpc: "2.0", id: msg.id, error: { code: -32000, message } }) + } +} + +function write(obj: unknown): void { + writeSync(1, JSON.stringify(obj) + "\n") +} diff --git a/plugins/websearch-echo/plugin.go b/plugins/websearch-echo/plugin.go new file mode 100644 index 0000000000..30ca8ec5cb --- /dev/null +++ b/plugins/websearch-echo/plugin.go @@ -0,0 +1,69 @@ +// Package websearchecho is a template out-of-tree web search plugin. +// It registers as type "echo" and returns the query as a single result. +// Enable it with WEKNORA_PLUGINS=websearch.echo or a profile patch. +package websearchecho + +import ( + "context" + "strings" + + "github.com/Tencent/WeKnora/internal/infrastructure/web_search" + "github.com/Tencent/WeKnora/internal/plugin" + "github.com/Tencent/WeKnora/internal/types" + "github.com/Tencent/WeKnora/internal/types/interfaces" +) + +const ( + FactoryName = "websearch.echo" + ProviderID = "echo" +) + +func init() { + plugin.Register(FactoryName, func(cfg plugin.Config) (plugin.Plugin, error) { + return New(cfg), nil + }) +} + +// New constructs the echo plugin. Config key "title" customizes the result. +func New(cfg plugin.Config) plugin.Plugin { + title := cfg.String("title") + if title == "" { + title = "echo" + } + return plugin.Func{ + ID: FactoryName, + InjectKeys: []string{plugin.ServiceWebSearch}, + ApplyFn: func(ctx *plugin.Context) error { + reg, err := plugin.Service[*web_search.Registry](ctx, plugin.ServiceWebSearch) + if err != nil { + return err + } + return ctx.Effect(func() plugin.Disposable { + reg.Register(ProviderID, func(types.WebSearchProviderParameters) (interfaces.WebSearchProvider, error) { + return &echoProvider{title: title}, nil + }) + return plugin.DisposeFunc(func() { reg.Unregister(ProviderID) }) + }) + }, + } +} + +type echoProvider struct { + title string +} + +func (p *echoProvider) Name() string { return ProviderID } + +func (p *echoProvider) Search( + _ context.Context, query string, maxResults int, _ bool, +) ([]*types.WebSearchResult, error) { + q := strings.TrimSpace(query) + if q == "" || maxResults == 0 { + return nil, nil + } + return []*types.WebSearchResult{{ + Title: p.title, + URL: "https://weknora.local/plugin/echo", + Content: q, + }}, nil +} diff --git a/plugins/websearch-echo/plugin_test.go b/plugins/websearch-echo/plugin_test.go new file mode 100644 index 0000000000..baa2e44852 --- /dev/null +++ b/plugins/websearch-echo/plugin_test.go @@ -0,0 +1,50 @@ +package websearchecho + +import ( + "context" + "testing" + + "github.com/Tencent/WeKnora/internal/infrastructure/web_search" + "github.com/Tencent/WeKnora/internal/plugin" + "github.com/Tencent/WeKnora/internal/types" +) + +func TestEchoPluginSearchAndUnload(t *testing.T) { + reg := web_search.NewRegistry() + h := plugin.NewHost() + h.Context().Provide(plugin.ServiceWebSearch, reg) + + on := false + profile := plugin.Profile{Name: "t", Bundles: []string{"empty"}} + bundles := map[string]plugin.Bundle{"empty": {Name: "empty"}} + patch := []plugin.Patch{{ + ID: FactoryName, Plugin: FactoryName, Insert: true, Disabled: &on, + Config: plugin.Config{"title": "from-config"}, + }} + if err := h.Compose(profile, bundles, patch); err != nil { + t.Fatal(err) + } + p, err := reg.CreateProvider(ProviderID, types.WebSearchProviderParameters{}) + if err != nil { + t.Fatal(err) + } + got, err := p.Search(context.Background(), " hello ", 3, false) + if err != nil { + t.Fatal(err) + } + if len(got) != 1 || got[0].Title != "from-config" || got[0].Content != "hello" { + t.Fatalf("result = %+v", got) + } + h.Unload() + if reg.Has(ProviderID) { + t.Fatal("echo should unload with the plugin") + } +} + +func TestEchoEmptyQuery(t *testing.T) { + p := &echoProvider{title: "echo"} + got, err := p.Search(context.Background(), " ", 3, false) + if err != nil || got != nil { + t.Fatalf("empty query = %v, %v", got, err) + } +} diff --git a/website-docs/01-getting-started/04-configuration.md b/website-docs/01-getting-started/04-configuration.md index 2839a5f316..f46348075f 100644 --- a/website-docs/01-getting-started/04-configuration.md +++ b/website-docs/01-getting-started/04-configuration.md @@ -252,6 +252,10 @@ AWS S3 的 `S3_ACCESS_KEY` / `S3_SECRET_KEY` 可以**同时留空**,此时走 | --- | --- | --- | | Sandbox 配置 | 设置页按空间维护 | 后端、凭据、模板、超时和私网访问策略不再读取 `WEKNORA_SANDBOX_*` | | `WEKNORA_SKILLS_DIR` | 空(镜像内 /app/skills/preloaded) | 自定义 Skills 目录 | +| `WEKNORA_PLUGIN_PROFILE` | `config/plugin_profile.yaml` | 插件组合 profile(bundle + patch),见[插件化架构](../06-development/04-plugin-architecture.md) | +| `WEKNORA_PLUGIN_PATCH` | 空 | 额外 overlay YAML(只读其中的 `patch`) | +| `WEKNORA_PLUGINS` | 空 | 逗号分隔 factory id,启动时 insert-if-missing(如 `websearch.echo`) | +| `WEKNORA_PLUGIN_DIR` | `plugins.d` | 运行时扫描 `plugin.yaml`(JS / HTTP 插件,无需重新编译);`none` 关闭 | | `WEKNORA_AGENT_LLM_TIMEOUT` | 120s | Agent 单次 LLM 调用超时(Go duration 或纯数字秒) | | `WEKNORA_AGENT_TOOL_APPROVAL_TIMEOUT` / `_FAIL_OPEN` | 600s / fail-close | MCP 工具人工审批等待与失败策略 | | `WEKNORA_CHAT_ATTACHMENT_TTL_HOURS` / `_WAIT_TIMEOUT_SEC` / `_OCR_CONCURRENCY` / `_OCR_MAX_PAGES` | 24 / 60 / 8 / 8 | 聊天附件解析保留时长、等待超时与 OCR 并发/页数上限 | diff --git a/website-docs/02-architecture/02-backend-design.md b/website-docs/02-architecture/02-backend-design.md index 67c955abbb..1f4a6fc0b9 100644 --- a/website-docs/02-architecture/02-backend-design.md +++ b/website-docs/02-architecture/02-backend-design.md @@ -68,7 +68,7 @@ func BuildContainer(container *dig.Container) *dig.Container { | `dig.As` | 把具体类型绑定为接口:`container.Provide(NewResourceCleaner, dig.As(new(interfaces.ResourceCleaner)))`;`router.NewAsyncqClient` 绑定为 `interfaces.TaskEnqueuer` | | `dig.Name` 命名依赖 | 同一接口多实例:4 个抽取服务(`chunkExtractor`/`dataTableSummary`/`imageMultimodal`/`knowledgePostProcess`)、6 个 Asynq server(`coreAsynqServer`/`postProcessAsynqServer`/`enrichmentAsynqServer`/`maintenanceAsynqServer`/`sharedAsynqServer`/`wikiAsynqServer`)、`wikiIngest` | | `dig.In` 参数结构体 | `router.RouterParams` 内嵌 `dig.In`,一次性注入约 60 个 Handler/Service 依赖,避免超长构造函数签名 | -| `container.Invoke` 执行副作用 | 注册即启动的后台组件:`registerPoolCleanup`、`registerWebSearchProviders`、`startDataSourceScheduler`、`startHousekeepingService`、`startAuditLogRetention`、`startTemporaryDocumentCleanup`、15 个 `chatpipeline.NewPluginXxx`(Search/Rerank/WebFetch/Merge/DataAnalysis/QueryUnderstand/LoadHistory/ChatCompletionStream 等插件自注册到 EventManager)、`router.RunAsynqServer`、`recoverPendingWikiTasks` 等 | +| `container.Invoke` 执行副作用 | 注册即启动的后台组件:`registerPoolCleanup`、`pluginboot.Start`(联网搜索等插件树)、`startDataSourceScheduler`、`startHousekeepingService`、`startAuditLogRetention`、`startTemporaryDocumentCleanup`、15 个 `chatpipeline.NewPluginXxx`(Search/Rerank/WebFetch/Merge/DataAnalysis/QueryUnderstand/LoadHistory/ChatCompletionStream 等插件自注册到 EventManager)、`router.RunAsynqServer`、`recoverPendingWikiTasks` 等 | | 适配器 Provide | 用闭包做接口转换:`func(s *service.StorageBackendService) interfaces.StorageBackendService { return s }`;`RetrieveEngineRegistry` 同实例同时暴露为 `StoreRegistry` | ### 2.2 注册顺序与条件装配 diff --git a/website-docs/06-development/03-extension-points.md b/website-docs/06-development/03-extension-points.md index 597df99199..3dbb55ad0d 100644 --- a/website-docs/06-development/03-extension-points.md +++ b/website-docs/06-development/03-extension-points.md @@ -2,6 +2,8 @@ WeKnora 在文档解析、分块、检索、模型接入、联网搜索、数据源、IM 渠道、Agent 工具、对象存储九个层面都预留了清晰的扩展点。本章逐个给出:**核心接口定义(真实源码)→ 现有实现列表 → 新增实现步骤(含注册点文件)**。所有接口代码均摘自当前仓库源码。 +进程级组合(profile / bundle / 可撤销注册)见[插件化架构](./04-plugin-architecture.md)。联网搜索已经迁到 `plugin.Host`;其它缝仍按本章的旧注册点接入,迁移顺序写在 `docs/dev/plugin-architecture.md`。 + ## 0. 扩展点总览 ```mermaid @@ -31,12 +33,12 @@ graph LR P1 -.->|"文件读写"| P9 P2 -.-> P9 CT["container.go
(依赖注入 / 注册中枢)"] -.->|"注册"| P3 - CT -.->|"注册"| P5 + PH["plugin.Host
(profile / bundle)"] -.->|"注册"| P5 CT -.->|"注册"| P6 CT -.->|"注册"| P7 ``` -Go 侧绝大多数扩展点的**注册中枢**是 `internal/container/container.go`(依赖注入容器):检索引擎 `initRetrieveEngineRegistry()`、联网搜索 `registerWebSearchProviders()`、IM 适配器 `registerIMAdapterFactories()`、数据源连接器 `initConnectorRegistry()`。 +Go 侧多数扩展点的**注册中枢**仍是 `internal/container/container.go`(依赖注入容器):检索引擎 `initRetrieveEngineRegistry()`、IM 适配器 `registerIMService()`、数据源连接器 `initConnectorRegistry()`。联网搜索改为 `internal/plugin/boot` 的 `plugin.Host`(`registerWebSearchProviders` 已删除)。 --- @@ -391,17 +393,19 @@ func (r *Registry) CreateProvider(providerType string, params types.WebSearchPro 1. 在 `internal/types/web_search_provider.go` 增加 `WebSearchProviderType` 常量; 2. 在 `internal/infrastructure/web_search/` 新建 `mysearch.go`,实现 `WebSearchProvider` 并暴露工厂 `func NewMySearchProvider(params types.WebSearchProviderParameters) (interfaces.WebSearchProvider, error)`; -3. **注册点:`internal/container/container.go` 的 `registerWebSearchProviders()`**: +3. **注册点:`internal/plugin/websearch/plugins.go` 的 `builtins()`**(不要再改 `container.go`): ```go -func registerWebSearchProviders(registry *infra_web_search.Registry) { - registry.Register("duckduckgo", infra_web_search.NewDuckDuckGoProvider) - registry.Register("google", infra_web_search.NewGoogleProvider) - // ... 在此追加: - registry.Register("mysearch", infra_web_search.NewMySearchProvider) +func builtins() []spec { + return []spec{ + // ... + {"mysearch", web_search.NewMySearchProvider}, + } } ``` +树外插件优先丢 `plugins.d/mysearch/plugin.yaml`(`runtime: stdio` 或 `js`),无需重新编译;见[插件化架构](./04-plugin-architecture.md)。 + 4. 前端的 provider 下拉与参数表单如需展示新引擎,同步 `frontend/` 相应配置页组件;租户配置持久化在 `web_search_providers` 表。 --- @@ -697,7 +701,7 @@ default: | 分块策略 | tier 函数 `func(text, cfg, profile) []Chunk` | `internal/infrastructure/chunker/strategy.go` | 同文件 `runTier()` + 策略常量 | | 检索引擎 | `RetrieveEngineRepository` | `internal/types/interfaces/retriever.go` | `container.go` `initRetrieveEngineRegistry()`(`RETRIEVE_DRIVER` 门控) | | 模型 Provider | `Provider` / `providerAdapter` / `Embedder` / `Reranker` | `internal/models/provider/provider.go` 等 | `provider.Register()` + `internal/models/chat/provider.go` | -| 联网搜索 | `WebSearchProvider` | `internal/types/interfaces/web_search.go` | `container.go` `registerWebSearchProviders()` | +| 联网搜索 | `WebSearchProvider` | `internal/types/interfaces/web_search.go` | `internal/plugin/websearch` `builtins()` + `plugin.Host` | | 数据源连接器 | `Connector` / `StreamingConnector` | `internal/datasource/connector.go` | `container.go` `initConnectorRegistry()` + `ConnectorMetadataRegistry` | | IM 适配器 | `Adapter`(+`StreamSender`/`FileDownloader`) | `internal/im/adapter.go` | `container.go` `registerIMAdapterFactories()` | | Agent 工具 | `types.Tool` | `internal/types/agent.go` | `internal/agent/tools/definitions.go` + `ToolRegistry.RegisterTool` | diff --git a/website-docs/06-development/04-plugin-architecture.md b/website-docs/06-development/04-plugin-architecture.md new file mode 100644 index 0000000000..886da64729 --- /dev/null +++ b/website-docs/06-development/04-plugin-architecture.md @@ -0,0 +1,71 @@ +# 插件化架构 + +WeKnora 正在把「扩展点写进主仓库、注册写进 `container.go`」收成一套 **Everything is a Plugin** 组合核,思路来自 [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) / Cordis,实现按 Go 单体做了裁剪。完整对照与分阶段计划见仓库内 [`docs/dev/plugin-architecture.md`](https://github.com/Tencent/WeKnora/blob/main/docs/dev/plugin-architecture.md)。 + +九条业务缝的接口清单仍以[扩展点指南](./03-extension-points.md)为准。本章只讲 **进程级组合**:如何在不改 `internal/container/container.go`、也不重新编译的前提下启用、关闭、替换一条能力。 + +## 1. 和「扩展点」的差别 + +扩展点回答「要实现哪个接口」。插件核回答: + +- 谁在启动时被挂上(profile / bundle / 磁盘目录) +- 依赖哪个服务 key(`inject`,例如 `web_search`) +- 卸载时如何撤销注册(reversible effect) + +Go 不能像 TypeScript 那样 `pnpm add` 完在同一进程 `import()`。接近的手感、也是 MCP / LSP / Dify 本地插件的做法:往 `plugins.d/` 丢 `plugin.yaml`,Host 拉起进程,在 **stdin/stdout** 上讲 JSON-RPC。作者实现 handler,不要自己开 HTTP。轻逻辑可以用 `runtime: js`(goja)。问答流水线里的 `chat_pipeline.Plugin` 仍是请求级 waterfall,暂时不动。 + +```mermaid +flowchart TB + D["WEKNORA_PLUGIN_DIR / plugins.d"] --> H["plugin.Host"] + P["config/plugin_profile.yaml"] --> H + B["bundle base"] --> H + ST["runtime: stdio JSON-RPC"] --> D + JS["runtime: js search.js"] --> D + H --> R["web_search.Registry"] + H --> T["GetWebSearchProviderTypes()"] + R --> C["WebSearchService / Agent"] +``` + +## 2. 当前已迁到 Host 的缝 + +| 缝 | 内置插件 id | 服务 key | 旧注册点 | +| --- | --- | --- | --- | +| 联网搜索 | `websearch.duckduckgo` … `websearch.metaso` | `web_search` | `container.registerWebSearchProviders`(已删除) | + +磁盘插件还会把自己的 `provider` 写进 `/api/v1/web-search-providers/types`,前端下拉能看到。 + +## 3. 组合文件与环境变量 + +| 变量 | 默认 | 作用 | +| --- | --- | --- | +| `WEKNORA_PLUGIN_DIR` | `plugins.d` | 扫描 `plugin.yaml`;`none` 关闭 | +| `WEKNORA_PLUGIN_PROFILE` | `config/plugin_profile.yaml` | 主 profile | +| `WEKNORA_PLUGIN_PATCH` | 空 | 额外 overlay YAML | +| `WEKNORA_PLUGINS` | 空 | 逗号分隔 factory id,insert-if-missing | + +仓库自带 `plugins.d/websearch-js-echo/`(进程内 JS)、`plugins.d/websearch-stdio-echo/`(Python stdio,默认关闭)和 `plugins/sdk-ts/websearch/`(`serve()`,stdin/stdout)。 + +## 4. 新增一个联网搜索插件 + +**免编译(推荐):** 新建 `plugins.d/mysearch/plugin.yaml`: + +```yaml +id: websearch.mysearch +name: My Search +seam: web_search +runtime: stdio # 任意语言;轻脚本用 js +command: node # 或 python3 +entry: index.js +provider: mysearch +``` + +stdio 插件在 stdin 读一行 JSON-RPC,在 stdout 写一行应答;日志打 stderr。TS 用 `plugins/sdk-ts/websearch` 的 `serve({ search })`。`search.js`(`runtime: js`)导出 `function search(...)`,出网走宿主 `httpRequest`(SSRF 白名单)。`runtime: http` 只留给已经在跑的远程服务。 + +**内置引擎:** `internal/plugin/websearch` 的 `builtins()` 加一行。不要改 `container.go`。 + +## 5. 明确不抄的部分 + +- 不用 Go `plugin.Open`(`.so`)做社区分发。 +- 不替换 uber/dig:dig 继续管 Handler / Service / DB。 +- 安全策略(RBAC、SSRF、配额)不是可卸载插件。 +- 没有 Cordis 式 HMR:改磁盘插件后重启进程。 diff --git a/website-docs/README.md b/website-docs/README.md index 14a5d32908..9cee014759 100644 --- a/website-docs/README.md +++ b/website-docs/README.md @@ -168,6 +168,7 @@ npm run preview # 预览构建产物 | [开发指南](06-development/01-dev-guide.md) | 环境要求、Makefile 全目标、开发模式、四条测试线、CI 与代码规范、调试技巧 | | [数据库与迁移](06-development/02-database-schema.md) | 40+ 张表结构与 ER 图、golang-migrate 双路径(versioned / sqlite)、新增迁移步骤、故障排查 | | [扩展点指南](06-development/03-extension-points.md) | 9 大扩展点:解析器/分块策略/检索引擎/模型 Provider/搜索引擎/数据源连接器/IM 适配器/Agent 工具/存储后端 | +| [插件化架构](06-development/04-plugin-architecture.md) | Cordis 风格组合核:profile / bundle / 可撤销注册;联网搜索已迁到 `plugin.Host` | ## 系统组件速览