feat: Document automation run configuration options

[malhotra5]

• A human has tested these changes.

---

Why

The automation service recently added clearer support for configurable run options, including execution timeouts and sandbox cleanup behavior. The OpenHands automation skill should surface those options so agents can create and update automations with the current API.

Summary

• Added a run configuration section covering model, timeout, keep_alive, and repos support.
• Updated prompt, plugin, custom automation, and A/B testing references with current field tables and examples.
• Documented timeout defaults/max values and keep_alive sandbox cleanup behavior.

Issue Number

N/A

How to Test

• uv run --with pyyaml python scripts/sync_extensions.py --check
  • Passed with a pre-existing non-blocking coverage warning for plugins/issue-duplicate-checker.
• git diff --check
  • Passed.

Video/Screenshots

N/A - documentation-only change.

Notes

This PR was created by an AI agent (OpenHands) on behalf of the user.

@malhotra5 can click here to continue refining the PR

[enyst]

I realize this is just documenting for the skill, I'm a bit confused though: why A/B testing, and maybe why is it part of this skill.

First, running the reviewer agent on GitHub with a different LLM profile seem to me like stretching the term "A/B testing".

(1) Idk, what the user (PR author, PR reviewer) wants, is for the reviewer agent to work. It's not really testing for user preferences or whatnot.

(2) It would be nice to take into account writing the LLM profile into the comment: Slack proposal

(3) For different LLMs, it happens that we can intuit what kind of problems they're likely to see and what they are less likely to see. This intuition, and the problems posed by bypassing it, is maybe not as strong in this case, as it is in regular conversations... but, still 🤔 . I'm not sure, I feel it would be more useful if we can control what LLM gets used. But I don't know how, I'll make an issue on that...

[enyst]

@OpenHands Read the entire skill, and clone yourself automation/ repo from same org to see how exactly it works:

• how is A/B testing designed and how do you configure it?

Respond as a comment here. Be CONCISE.

[openhands-ai]

I'm on it! enyst can track my progress at all-hands.dev

[enyst]

Read the full skill + OpenHands/automation implementation.

A/B testing is currently plugin preset only: POST /api/automation/v1/preset/plugin. Configure it by replacing top-level plugins with variants and adding experiment_id.

Each variant has: name, positive relative weight, plugins[], and optional model. Shared fields remain name, prompt, trigger, automation-level model, timeout, keep_alive, and repos.

Service design: validates exactly one of plugins/variants, requires 2-10 unique variants, writes experiment_config.json into the generated tarball, then runtime sdk_main.py uses random.choices(..., weights=...) per run to select a variant and load that variant plugins/model. No stickiness or built-in stats.

One doc mismatch I found: runtime conversation tags are experimentid, variant, and optional modelprofile; the PR doc currently says experiment_id.

This comment was created by an AI agent (OpenHands) on behalf of the user.

[openhands-ai]

Posted the concise PR comment: #378 (comment)

[enyst]

@malhotra5 is this maybe something to fix?

One doc mismatch I found: runtime conversation tags are experimentid, variant, and optional modelprofile; the PR doc currently says experiment_id.