Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,8 @@ macOS platform block:
- Workspace targets supply bundle/application identity. `FSQ_MACOS_APPIUM_SERVER_URL` is the optional process-environment endpoint override; presets own backend, snapshot, and timeout policy.
- Current action surface exposes desktop aliases through the existing PlatformTool registry: `launchApp`, `killApp`, `clickOn`, `doubleClickOn`, `rightClickOn`, `typeText`, `pressKey`, `hoverOn`, `dragTo`, `takeScreenshot`, `uiSnapshot`, `assertVisible`, `assertElementsOrder`, and `assertWithAI`.
- Explicit observation capability: `ui_snapshot` with alias `uiSnapshot`; macOS must not expose Android `ui_tree`/`uiTree` naming. Automatic runner evidence captures `screenshot` plus normalized `ui_snapshot` using a bounded compact semantic Appium Mac2 control tree that preserves useful locator, text, state, and geometry signals.
- macOS `ui_snapshot` also supports bounded structured element queries over current, unabridged backend page-source attributes before display compaction. Query results distinguish display previews from complete locator values, expose ambiguity and incomplete coverage, and do not treat a missing snapshot match as proof that a control is absent from the application. Existing unfiltered snapshot fields and raw artifact-search semantics remain compatible.
- macOS `ui_snapshot` returns only the bounded compact tree and has no platform-specific element-query mode. Dynamic agents use the shared bounded artifact search/slice helpers when snapshot output is externalized by the runtime output budget.
- macOS `type_text` verifies text entry against a readable resolved or active element value. A failed primary element input retries once through system-level `macos: keys`; a still-unsatisfied postcondition is an action failure rather than a successful tool transport. Untargeted input without a readable active element retains system-level key delivery without inventing a value assertion.
- macOS element resolution preserves all supplied locator constraints, safely handles literal text, and rejects ambiguous matches before acting. Locator syntax, missing targets, ambiguous targets, unavailable sessions, and backend failures remain distinguishable in safe diagnostics.
- Harness skill: `macos-harness.md`.
- The Appium MCP reference project may guide Mac2 session mechanics and action semantics, but fsq-agent must not wrap or depend on that MCP server as a runtime capability source.
Expand Down
4 changes: 2 additions & 2 deletions fsq_agent/adapters/coding_agent/_harness_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
from typing import Any

from fsq_agent._capability_bootstrap import build_capability_registry
from fsq_agent.agent_engine import ToolBinding, ToolCall, ToolInputFailure
from fsq_agent.agent_engine import EngineError, ToolBinding, ToolCall, ToolInputFailure
from fsq_agent.core import HarnessInterface, RuntimeSecretStore, StepRunner
from fsq_agent.core.interfaces import EvidenceJournalSink
from fsq_agent.models import CapabilityDefinition, ConfigurationError, ExecutableStep, HarnessFunctionSchema, HarnessPlatform, PostActionDelaySettings, RunnerStepResult
Expand Down Expand Up @@ -118,7 +118,7 @@ async def invoke(call: ToolCall) -> str:
# Tool transport must convert arbitrary capability failures into structured results.
except Exception as exc:
if getattr(exc, "fsq_evidence_fatal", False):
raise
raise EngineError("evidence", "Durable execution evidence failed.") from None
if type(exc).__name__ in {"TaskCancelledError", "ExecutionCancelled", "RunCancelled"}:
raise asyncio.CancelledError() from exc
return self._format_failure(schema, exc, int((time.perf_counter() - started) * 1000))
Expand Down
6 changes: 6 additions & 0 deletions fsq_agent/adapters/coding_agent/_runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,12 @@


def _runtime_failure_metadata(exc: BaseException) -> dict[str, str]:
if isinstance(exc, EngineError) and exc.category == "evidence":
return {
"failure_category": "artifact_error",
"failure_reason": "evidence_persistence",
"failure_summary": "Agent runtime stopped because durable execution evidence failed.",
}
if isinstance(exc, EngineError) and exc.category == "incomplete" and exc.reason == "content_filter":
return {
"failure_category": "provider_content_filter",
Expand Down
6 changes: 4 additions & 2 deletions fsq_agent/core/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,11 +141,13 @@ Windows owns public `WindowsDriverInterface`, private `WindowsHarness`, private

### macOS Platform Block

macOS LLM-exposed Appium Mac2 capabilities include inherited CommonTool `wait_ms` plus driver-backed PlatformTools `launch_app`, `kill_app`, `click_on`, `double_click_on`, `right_click_on`, `type_text`, `press_key`, `hover_on`, `drag_to`, `take_screenshot`, `ui_snapshot`, `assert_visible`, `assert_elements_order`, and `assert_with_ai`. Authored FSQ aliases include `waitMs`, `launchApp`, `killApp`, `clickOn`, `doubleClickOn`, `rightClickOn`, `typeText`, `pressKey`, `hoverOn`, `dragTo`, `takeScreenshot`, `uiSnapshot`, `assertVisible`, `assertElementsOrder`, and `assertWithAI`. macOS `type_text` accepts `textType="runtimeSecret"` to reference allowlisted workspace secret values resolved by `StepRunner`. macOS observation uses `ui_snapshot`/`uiSnapshot` and must not reuse Android `ui_tree`/`uiTree` naming. macOS target resolution uses a locator built from accessibility id, name, label, value, role/control type, class name, XPath, predicate string, semantic target text, and explicit coordinates.
macOS LLM-exposed Appium Mac2 capabilities include inherited CommonTool `wait_ms` plus driver-backed PlatformTools `launch_app`, `kill_app`, `click_on`, `double_click_on`, `right_click_on`, `type_text`, `press_key`, `hover_on`, `drag_to`, `take_screenshot`, `ui_snapshot`, `assert_visible`, `assert_elements_order`, and `assert_with_ai`. Authored FSQ aliases include `waitMs`, `launchApp`, `killApp`, `clickOn`, `doubleClickOn`, `rightClickOn`, `typeText`, `pressKey`, `hoverOn`, `dragTo`, `takeScreenshot`, `uiSnapshot`, `assertVisible`, `assertElementsOrder`, and `assertWithAI`. macOS `type_text` accepts `textType="runtimeSecret"` to reference allowlisted workspace secret values resolved by `StepRunner`. macOS observation uses `ui_snapshot`/`uiSnapshot` and must not reuse Android `ui_tree`/`uiTree` naming. macOS target resolution uses a locator built from accessibility id, name, label, value, title, role/control type, class name, XPath, predicate string, semantic target text, and explicit coordinates. Semantic target text matches title in addition to the other supported semantic identity fields.

macOS owns public `MacOSDriverInterface`, private `MacOSHarness`, private `AppiumMac2Driver`, macOS catalog-backed platform declarations, and macOS default capability definitions selected through public factories. `MacOSDriverInterface` extends the shared driver observation contract; automatic runner capture writes `screenshot` and normalized `ui_snapshot` artifacts. `AppiumMac2Driver.assert_with_ai` is a decorated backend tool that calls shared AI assertion support. The explicit `ui_snapshot`/`uiSnapshot` observation capability remains available for authored dynamic and strict commands. Appium imports, Mac2 option construction, Appium server connection, and application launch are lazy runtime/backend concerns, not registry-bootstrap concerns. FSQ platform/backend names are `macos` and `appium_mac2`; the driver maps those to Appium native `platformName: Mac` and `automationName: Mac2` internally.

`AppiumMac2Driver.ui_snapshot` owns conversion of Appium page-source XML into the existing bounded structured macOS control-tree payload. Parsed payload metadata continues to report original source length and node count, while the structured `root` is compact: empty attributes and low-value default states are removed; element text plus `name`, `label`, and `value` are clipped to 50 characters; stable identifiers, semantic type/role, non-default `enabled`/`visible`/`selected` state, and geometry on otherwise meaningful nodes are retained. Signal-free wrapper nodes are removed and their meaningful descendants are lifted in source order. Invisible or zero-size nodes remain only when they carry meaningful text, a stable identifier, or non-default state; geometry alone is not meaningful. `include_attributes=false` uses the compact semantic allowlist, while `include_attributes=true` also retains additional non-empty, non-default attributes subject to text clipping. `max_depth` bounds the emitted compact tree, signal-free wrappers do not consume emitted depth, and omitted immediate meaningful children are reported through `children_truncated`. Empty input preserves the empty parsed payload. Rejected declarations, XML parse failures, and unexpected local compaction failures return the existing bounded `unparsed_xml` preview payload instead of failing observation capture.
`AppiumMac2Driver.ui_snapshot` owns conversion of Appium page-source XML into the existing bounded structured macOS control-tree payload and exposes no platform-specific query mode. Parsed payload metadata continues to report original source length and node count, while the structured `root` is compact: empty attributes and low-value default states are removed; element text plus `name`, `label`, `value`, and `title` are clipped to 50 characters; stable identifiers, semantic type/role, non-default `enabled`/`visible`/`selected` state, and geometry on otherwise meaningful nodes are retained. Signal-free wrapper nodes are removed and their meaningful descendants are lifted in source order. Invisible or zero-size nodes remain only when they carry meaningful text, a stable identifier, or non-default state; geometry alone is not meaningful. `include_attributes=false` uses the compact semantic allowlist including `title`, while `include_attributes=true` also retains additional non-empty, non-default attributes subject to text clipping. `max_depth` bounds the emitted compact tree, signal-free wrappers do not consume emitted depth, and omitted immediate meaningful children are reported through `children_truncated`. Empty input preserves the empty parsed payload. Rejected declarations, XML parse failures, and unexpected local compaction failures return the existing bounded `unparsed_xml` preview payload instead of failing observation capture.

`AppiumMac2Driver.type_text` preserves target-resolution constraints and requested clear behavior. When a readable resolved or active element is available, it uses element text input first, reads back the element value, retries once through system-level `macos: keys` when the value does not reflect the request, and returns an action failure if the second readable value still does not satisfy the input postcondition. Untargeted input without a readable active element retains system-level key delivery without inventing a value assertion.

`AppiumMac2Driver` maps configured `new_command_timeout_seconds` to the Appium `newCommandTimeout` session capability. This command-idle lifetime defaults to 300 seconds and is independent from `action_timeout_seconds`; the action timeout must not determine Appium session idle expiry.

Expand Down
39 changes: 26 additions & 13 deletions fsq_agent/core/evidence/_recorder.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,35 +61,49 @@ def record_planned_step(self, step: ExecutableStep) -> None:
key = (step.source_step_id or step.step_id, step.invocation_path)
if key in self._planned_keys:
return
self._append(planned_step=step)
self._planned.append(step)
record = self._canonical_record(planned_step=step)
canonical = record.planned_step
if canonical is None:
raise ValueError("Canonical planned step is unavailable.")
self._append_record(record)
self._planned.append(canonical)
self._planned_keys.add(key)

def record_event(self, event: RunnerEvent) -> None:
if event.run_id != self.run_id:
raise ValueError("Evidence journal Run identity mismatch.")
with self._lock:
_validate_execution_facts([*self._events, event], self._steps)
self._append(event=event)
self._events.append(event)
record = self._canonical_record(event=event)
canonical = record.event
if canonical is None:
raise ValueError("Canonical evidence event is unavailable.")
_validate_execution_facts([*self._events, canonical], self._steps)
self._append_record(record)
self._events.append(canonical)

def record_step_result(self, result: RunnerStepResult) -> None:
with self._lock:
_validate_execution_facts(self._events, [*self._steps, result])
self._append(step_result=result)
self._steps.append(result)
record = self._canonical_record(step_result=result)
canonical = record.step_result
if canonical is None:
raise ValueError("Canonical evidence step result is unavailable.")
_validate_execution_facts(self._events, [*self._steps, canonical])
self._append_record(record)
self._steps.append(canonical)
self.write_manifest()

def _append(self, **fact) -> None:
def _canonical_record(self, **fact) -> EvidenceJournalRecord:
record = EvidenceJournalRecord(sequence=self._sequence + 1, event_id=uuid.uuid4().hex, run_id=self.run_id, **fact)
return EvidenceJournalRecord.model_validate(self._sanitize(record.model_dump(mode="json")))

def _append_record(self, record: EvidenceJournalRecord) -> None:
if self._write_failed:
error = OSError("Evidence journal is unavailable after a failed append.")
error.fsq_evidence_fatal = True
raise error
try:
self.output_dir.mkdir(parents=True, exist_ok=True)
path = self._contained("evidence-events.jsonl")
record = EvidenceJournalRecord(sequence=self._sequence + 1, event_id=uuid.uuid4().hex, run_id=self.run_id, **fact)
record = EvidenceJournalRecord.model_validate(self._sanitize(record.model_dump(mode="json")))
with path.open("ab") as stream:
stream.write(record.model_dump_json().encode("utf-8") + b"\n")
stream.flush()
Expand All @@ -105,8 +119,7 @@ def _sanitize(self, value):

def build_bundle(self) -> EvidenceBundle:
with self._lock:
bundle = _bundle(self.bundle_id, self.run_id, self._events, self._steps, self._planned, self._sequence, self.metadata, include_unresolved=True)
return EvidenceBundle.model_validate(self._sanitize(bundle.model_dump(mode="json")))
return _bundle(self.bundle_id, self.run_id, self._events, self._steps, self._planned, self._sequence, self.metadata, include_unresolved=True)

def _contained(self, filename: str) -> Path:
path = self.output_dir / filename
Expand Down
Loading
Loading