feat(windows): harden and productionize the maka-cu Computer Use executor

[liugddx]

Relationship

Part of #2142, Phase 5 (Windows Computer Use).

Context

The Windows support roadmap currently lists Computer Use as deferred work: define a Windows backend using UI Automation plus an appropriate capture API, design consent/secure-desktop/elevation/multi-monitor/scaling/session-lock behavior, reuse the platform-neutral Computer Use host event contract, and add Windows integration/E2E evidence.

This issue proposes the concrete executor work needed to make that phase implementable and reviewable. It is based on a code review of maka-agent/maka-cu, revision 4a9787d, especially its experimental apps/OpenComputerUseWindows runtime.

The current implementation is useful as a functional prototype: Go owns the CLI/MCP/tool schema and an in-process snapshot cache; an embedded PowerShell bridge uses Windows UI Automation for discovery/tree rendering/semantic actions and falls back to Win32 window messages for some input paths. It has been validated against basic Notepad flows. It is not yet a safe or reliable equivalent of the macOS maka.cu/2 executor.

Review Findings

1. Observation/action binding is too weak

The Windows runtime caches snapshots by a lower-cased app query/name/bundle-like process name/PID and action calls reuse a numeric element_index. The PowerShell bridge then re-enumerates the current process tree. If a UIA runtime id is unavailable, it falls back to the first matching AutomationId/name and control type:

• main.go#L438-L486
• runtime.ps1#L676-L700

This can target the wrong window/control after an app restart, process recycling, modal-window creation, duplicate controls, or a reflow. The macOS host protocol already has the stronger model: opaque per-snapshot element tokens, snapshot state (live, spent, superseded, expired, evicted), process identity, window identity, and an element digest:

• HostSnapshotRegistry.swift#L33-L80
• HostProtocolServer+Observe.swift#L447-L499

2. The screenshot path can disagree with the UIA tree

Capture-WindowPngBase64 uses Graphics.CopyFromScreen on the window rectangle:

• runtime.ps1#L565-L632

When the target is covered by another window, the image contains the occluding window while the tree describes the target app. Minimized, layered, hardware-accelerated, and some redirected windows can also produce incomplete or black captures.

The production path should prefer a target-window capture API such as Windows Graphics Capture (IGraphicsCaptureItemInterop::CreateForWindow(HWND, ...)) and report an explicit degraded capability when only a screen-rectangle fallback is available. PrintWindow can be evaluated as a compatibility fallback, but it is synchronous and application-dependent, so it must not be treated as universally correct.

3. Win32 input currently reports success without verification

The bridge ignores PostMessage return values and returns ok=true after a fixed 120 ms delay:

• runtime.ps1#L135-L203
• runtime.ps1#L912-L999

A message can fail because of UIPI/integrity level, an invalid or recycled HWND, a toolkit that ignores the message, or a target that has not processed it yet. Windows documents that PostMessage and SendInput are subject to UIPI. The executor must distinguish at least:

• refused before dispatch;
• dispatched and verified;
• dispatched but outcome unknown;
• dispatch failed;
• unsupported for this toolkit/window.

It must never silently turn an explicitly background-safe path into foreground SendInput or global pointer input.

4. The wheel fallback uses the wrong coordinate space

Send-Scroll converts the target point to client coordinates before putting it in WM_MOUSEWHEEL.lParam:

• runtime.ps1#L188-L203

WM_MOUSEWHEEL expects screen coordinates in lParam, unlike ordinary client-area mouse messages. This is incorrect for a window not positioned at the screen origin and is especially visible on multi-monitor layouts with negative coordinates.

5. DPI, threading, and per-call process startup need a production decision

The UIA bounding rectangle is in physical screen coordinates, while the runtime does not establish a clear Per-Monitor-V2 DPI contract for the executor and coordinate conversion. Mixed-DPI monitors can therefore make the screenshot, UIA frame, and Win32 input disagree.

Every tool call currently writes a temporary script and starts a new Windows PowerShell process. This is simple for a prototype but expensive and makes long-lived UIA element/cache/event ownership difficult. Microsoft recommends using a dedicated non-UI MTA thread for UI Automation clients and provides cache requests to reduce cross-process property calls.

Proposed Direction

A. Reuse the platform-neutral host contract

Windows should implement the same native host protocol used by macOS (maka.cu/2) rather than growing a second, Windows-only nine-tool executor contract. The Maka runtime/host remains responsible for model-facing Anthropic Computer Use semantics; the native executor remains responsible for observation, target binding, dispatch, capture, and verification.

The protocol should carry:

• executor version and capabilities/limits;
• session lifecycle and cancellation;
• observe results with opaque snapshot IDs and element tokens;
• target identity including PID, process start time, and HWND/window generation;
• element/window digests and explicit stale/unknown/expired/spent errors;
• dispatch result fields for outcome, tier, path, effect, and verification;
• an explicit capability/degraded result for missing capture, locked desktop, secure desktop, elevation/UIPI, or toolkit limitations.

B. Use stable target identity, not app-name identity

The minimum Windows target identity should be:

session + snapshotId + pid + processStartTime + hwnd + windowGeneration

HWND must be revalidated at dispatch time (IsWindow, owning PID, and current process start time). A recycled PID or HWND must fail closed. Element tokens must be opaque and resolved only inside the quoted snapshot; numeric indexes can remain a display convenience for the model/runtime, never the dispatch authority.

C. Make capture and input capability-driven

Recommended dispatch tiers:

1. UIA semantic pattern (Invoke, Toggle, SelectionItem, Value, Scroll, Text);
2. target HWND/window-message path, only when the target control and message contract are known;
3. Windows Graphics Capture / target-window coordinate path;
4. foreground SendInput, only as explicit opt-in with user-visible policy and verification.

Each result should identify the selected path and whether the operation was verified. Unsupported or unsafe paths should return typed errors, not fallback silently.

D. Prefer a long-lived native Windows bridge

Keep the Go/Node integration boundary if useful, but replace per-call PowerShell startup with a long-lived C#/.NET or native helper. It should own:

• UIA COM initialization on a dedicated MTA worker;
• cache requests for bulk tree properties;
• event-driven invalidation for window/control changes;
• Windows Graphics Capture sessions;
• Win32/DPI/monitor identity and coordinate conversion;
• structured HRESULT/Win32/UIPI errors.

A C#/.NET helper is likely the lowest-risk first production step; a direct Go COM/WinRT implementation can be evaluated later if packaging and maintenance justify it.

E. Define the Windows security/session contract

The implementation must explicitly handle and test:

• normal interactive desktop versus service/SSH/session-0 execution;
• locked workstation and unavailable input desktop;
• UAC secure desktop and elevated target processes;
• UIPI/integrity-level mismatches;
• multi-monitor and mixed-DPI layouts;
• minimized, occluded, redirected, and hardware-accelerated windows;
• sensitive apps and credential/password fields;
• screenshot and input consent/approval;
• no automatic app launch, focus stealing, or foreground fallback by default.

The product must surface unsupported/deferred Computer Use capability in the Windows preview instead of claiming parity.

Proposed Deliverables

• Add a Windows-side maka.cu/2 protocol adapter and capability report.
• Define and implement Windows snapshot lifecycle and target identity (PID + process start + HWND/window generation).
• Replace numeric-index dispatch with opaque snapshot-bound element tokens and digest validation.
• Implement target-window capture with Windows Graphics Capture; retain a typed degraded fallback if necessary.
• Fix coordinate-space handling for wheel, mouse, DPI, virtual-screen, and negative-monitor coordinates.
• Return structured dispatch outcomes and verify semantic mutations where possible.
• Move UIA work to a long-lived dedicated MTA bridge and add property/pattern caching.
• Add consent/locked-desktop/UIPI/elevation/sensitive-app policy and diagnostics.
• Add a deterministic Windows fixture and interactive desktop E2E suite.
• Add CI evidence for Windows 11 x64 at minimum, including multi-monitor/mixed-DPI where the runner permits it.
• Update roadmap(windows): make Windows a supported platform #2142 and Windows support documentation only after the above capability boundaries are explicit.

Acceptance Criteria

Protocol and safety

• An action planned against snapshot A cannot execute against snapshot B, a recycled PID, or a recycled HWND.
• A spent, expired, superseded, evicted, unknown, or mismatched snapshot returns a distinct typed result.
• No unsupported path silently falls back to foreground/global input.
• Every action result states the attempted path, outcome, effect, and verification status.
• Locked/secure desktop/UIPI/elevated-target conditions are reported explicitly.

Observation and capture

• UIA tree and screenshot refer to the same PID/HWND/window generation.
• Target-window capture is correct when another window covers the target, or the result explicitly declares the degraded capture mode.
• Coordinates remain correct under Per-Monitor-V2 scaling, mixed-DPI monitors, and negative virtual-screen coordinates.
• Tree rendering has bounded node/depth/time budgets and does not hang on a provider that stops responding.

Actions

• Semantic actions work without foreground activation for supported Win32, WPF/WinUI, Electron/WebView2, and browser fixtures where the toolkit exposes the required pattern.
• Message/input fallbacks have return-value and post-action verification; failures are not reported as success.
• type_text, key combinations, click, drag, horizontal/vertical scroll, and set-value have explicit capability coverage per toolkit.

Release evidence

• A Windows 11 x64 CI lane runs protocol, fixture, and packaged smoke tests.
• Interactive desktop tests cover Notepad, a Chromium/Electron/WebView2 fixture, modal windows, app restart, occlusion, locked session, elevation/UIPI, two monitors, and mixed DPI.
• The Windows preview documentation states exactly what is supported, what is foreground-only, and what remains deferred.

Non-goals

• This issue does not make every Windows GUI toolkit background-controllable.
• This issue does not require global physical mouse movement as the default path.
• This issue does not make the current PowerShell prototype production-ready by adding more heuristics alone.
• This issue does not claim that Windows Computer Use is supported before the acceptance evidence exists.

References

• Parent roadmap: #2142
• Executor: maka-agent/maka-cu
• Windows prototype: apps/OpenComputerUseWindows
• macOS host protocol: OpenComputerUseKit/HostProtocol
• Microsoft UI Automation threading guidance: https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-threading
• Microsoft UI Automation caching guidance: https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-cachingforclients
• Microsoft UI Automation scaling guidance: https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-screenscaling
• Microsoft Windows Graphics Capture window interop: https://learn.microsoft.com/en-us/windows/win32/api/windows.graphics.capture.interop/nf-windows-graphics-capture-interop-igraphicscaptureiteminterop-createforwindow
• Microsoft WM_MOUSEWHEEL coordinate contract: https://learn.microsoft.com/en-us/windows/win32/inputdev/wm-mousewheel
• Microsoft PostMessage/UIPI behavior: https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-postmessagew

[M4n5ter]

The five prototype gaps described here are real, and the Windows Computer Use problem is worth tracking. However, I would keep this issue as an epic rather than use the current checklist as one implementation contract. It currently combines a protocol redesign, durable helper lifecycle, UI Automation, capture, security/session policy, fallback input, toolkit coverage, DPI/multi-monitor behavior, and packaged E2E into one acceptance boundary. That boundary is too broad to close safely.

Three design changes are needed before implementation proceeds:

1. Do not make Windows pretend to implement the existing literal maka.cu/2 wire. That wire is a closed, macOS-specific contract: its paths, capture/session fields, permissions, and key semantics include AX/CG/Skylight, ScreenCaptureKit, TCC, and macOS modifiers. Windows should reuse the platform-neutral host boundary and the important invariants—bounded snapshots, opaque tokens, cancellation, fail-closed dispatch, and effect verification—while its child protocol remains private until there is a demonstrated need for a new public cross-platform revision. The current host only needs to stop assuming TCC-shaped preflight, a screenRecording lease, and Darwin-only selector names.

2. Separate observation from actuation. Windows Graphics Capture is a capture authority, not a dispatch tier. The result must independently report the capture path and the action path. In v1, capture should use CreateForWindow(HWND) and return capture_unavailable if that cannot be established; it must not silently fall back to a screen rectangle that may show a different window.

3. Define one narrow, testable vertical slice. I recommend the first child issue support only:

   • an unlocked Windows 11 interactive desktop;
   • one explicitly selected, already-running top-level HWND;
   • a bounded UIA tree and WGC frame bound to the same validated process incarnation and HWND;
   • set_value through ValuePattern;
   • click_element only when the element explicitly advertises Invoke, Toggle, or SelectionItem;
   • typed verified, refused, or unknown outcomes after re-observing the same HWND.

Every mutating action should spend its snapshot. Before and after the action, the helper should revalidate the owning PID's creation time, HWND, and retained UIA root. A modal or replacement HWND requires a new explicit selection. The tree and frame need to identify the same target generation; they do not need to claim impossible temporal atomicity for dynamic content.

The implementation should be a host-supervised long-lived child with one MTA UIA lane, bounded registry/tree/time budgets, cancellation, parent-death handling, and restart after a hung provider. It should reuse the existing product approval flow rather than introduce a second Windows consent state machine.

For v1, I would explicitly defer keyboard input, scrolling, window manipulation, PostMessage, SendInput, all coordinate paths, mixed-DPI/multi-monitor guarantees, and broad toolkit parity. Unsupported paths must refuse rather than fall through to global input. Evidence should be one deterministic adversarial fixture plus one packaged Windows 11 E2E that exercises the full observe → opaque token → action → readback chain.

With that boundary, this issue remains useful as the problem/epic record. The implementation should be split into smaller child issues, beginning with the bounded semantic preview above; the broader checklist should not be treated as the acceptance criteria for a single change.

简体中文

这里列出的五个原型缺口都真实存在，Windows Computer Use 也值得继续推进。不过，我建议把本 issue 保留为 epic，而不是把当前清单直接当成一个实现单的验收合同。当前边界同时包含协议重构、长驻 helper 生命周期、UI Automation、截图、安全/会话策略、输入 fallback、toolkit 覆盖、DPI/多屏和打包 E2E，范围过大，无法作为一个可安全闭合的交付。

实现前需要先调整三点设计：

1. Windows 不应冒充实现现有字面意义上的 maka.cu/2 wire。 该协议是闭集且明显面向 macOS，其中的路径、截图/会话字段、权限和按键语义包含 AX/CG/Skylight、ScreenCaptureKit、TCC 和 macOS 修饰键。Windows 应复用平台中立的 Host 边界，以及 bounded snapshot、opaque token、取消、fail-closed dispatch、效果验证等关键不变量；Windows child protocol 暂时保持私有，等确实需要新的公共跨平台 revision 时再引入。当前 Host 只需去掉对 TCC-shaped preflight、screenRecording lease 和 Darwin-only selector 的硬编码假设。

2. 观察与执行必须分开。 Windows Graphics Capture 是 capture authority，不是 dispatch tier。结果应分别报告 capture path 和 action path。v1 截图只使用 CreateForWindow(HWND)；无法建立时返回 capture_unavailable，不能静默退化成可能截到其他窗口的屏幕矩形。

3. 先定义一条窄而可验收的垂直切片。 首个子单建议只支持：

   • 已解锁的 Windows 11 交互桌面；
   • 一个由用户显式选择、已经运行的顶层 HWND；
   • 绑定同一进程 incarnation 与 HWND 的 bounded UIA tree 和 WGC frame；
   • 通过 ValuePattern 执行 set_value；
   • 仅当元素明确声明 Invoke、Toggle 或 SelectionItem 时执行 click_element；
   • 动作后重新观察同一 HWND，并只返回 typed verified / refused / unknown。

每个可能改变状态的动作都应消费 snapshot。动作前后都要重新校验进程创建时间、HWND 和保留的 UIA root；出现 modal 或新的 HWND 时必须重新显式选择。tree 与 frame 只承诺属于同一个 target generation，不宣称动态内容在时间上原子一致。

实现应为 Host 监管的长驻 child：单 MTA UIA lane、bounded registry/tree/time、取消、parent-death，以及 provider 卡死后的 kill/restart。继续复用现有产品 approval，不新增第二套 Windows consent 状态机。

v1 应明确后移 keyboard、scroll、window 操作、PostMessage、SendInput、所有坐标路径、mixed-DPI/多屏保证和广泛 toolkit parity。unsupported path 必须拒绝，不能 fallback 到全局输入。验收证据至少需要一个确定性 adversarial fixture，以及一条真实打包 Windows 11 E2E，覆盖 observe → opaque token → action → readback 全链路。

按这个边界，本 issue 可以继续作为问题/epic 记录；实现应拆成更小的子单，先交付上述 bounded semantic preview，而不是把当前大清单作为单个变更的验收标准。