Computer Control

AI Butler can act on your computer. Automation is organized as a ladder of tiers, and the agent prefers the most reliable, most auditable surface available before falling back to coarser ones:

Tier	Surface	Tools
0–1	Native APIs & MCP integrations	provider/MCP tools
2	Native OS scripting	shell.applescript, shell.shortcuts, shell.dbus, shell.powershell, shell.script
—	Real browser automation	JavaScript-capable click / type / submit
3	Accessibility tree (read-only)	accessibility.read_ui
4	Vision + synthetic input	screen.capture, input.click / input.type / input.key

Lower-numbered tiers are cheaper to audit and harder to break; Tier 4 exists for the long tail nothing else can reach.

Honest status labels, per tier: native OS scripting, shell.script, clipboard, wait.until, cost.forecast, vault.request, and permissions.check are ready. The accessibility reader and the vision + input tier are shipped and unit-tested everywhere, with the Linux backends validated end-to-end in CI and the Windows backends still awaiting validation on a real interactive desktop — treat them as beta on Windows. Browser automation is shipped; it requires a local Chrome/Chromium at runtime.

Native OS scripting

Four script tools, one per platform automation surface:

Tool	Platform	Drives
shell.applescript	macOS	Mail, Calendar, Music, Notification Center — anything scriptable
shell.shortcuts	macOS / iOS-synced	Apple Shortcuts by name
shell.dbus	Linux	MPRIS players, desktop notifications, session services
shell.powershell	Windows	The existing PowerShell surface

For cross-platform work there’s a dispatcher, shell.script: the agent supplies per-OS payloads and the dispatcher routes to the right one by the host OS, so one skill can carry macOS, Linux, and Windows variants.

Safety rails

• AppleScript target-app allowlist — grants are per target application (tell:Mail, tell:Music*, or tell:*), so “can script Music” doesn’t mean “can script everything.”
• Action recording — every native-script call lands in a fine-grained actions audit log with target, payload (credentials redacted), duration, and result.
• Capability-gated like every other tool: no grant, no execution.

On macOS, ask the agent to run the permissions.check tool before first use — it probes Automation and Screen Recording permissions and deep-links you to the right System Settings panes.

Browser automation

AI Butler can drive a real, JavaScript-capable browser — clicks, typing, and form submission actually execute instead of being described. It requires a local Chrome/Chromium installation; for plain page fetches the HTTP client remains the cheaper path.

Accessibility-tree reader (Tier 3)

accessibility.read_ui {app, depth} returns a read-only snapshot of an application’s live UI hierarchy — roles, names, and values of buttons, fields, and labels — in one consistent shape on macOS (System Events), Windows (UIAutomation), and Linux (AT-SPI2 over D-Bus). Zero CGO on all three.

Because reading another app’s UI is information disclosure, it’s gated even though it’s read-only: an allowlist of inspectable application names (an empty allowlist denies everything), an app-name injection guard, bounded execution time, capped output, and the tool.accessibility.read capability.

Vision + synthetic input (Tier 4)

The last resort, for when no finer-grained path can do the job:

• screen.capture — screenshots via each OS’s native tooling (macOS screencapture, common Linux capture tools on X11/Wayland, PowerShell on Windows).
• input.click / input.type / input.key — synthetic mouse and keyboard on all three platforms.

Synthetic input sits behind a double gate: it requires both the tool.input.control capability grant and the explicit AIBUTLER_ENABLE_SYNTHETIC_INPUT=1 environment flag. Both default to off — a fresh install cannot move your mouse.

Vision input

Model adapters accept images (base64 or URL) alongside text — supported on Ollama vision models, GPT-4o, and Claude. Paste a screenshot into the web chat and ask “what’s wrong with this dialog?”, or let a scheduled task attach an image for review.

Clipboard

clipboard.read and clipboard.write, cross-platform. Privacy default: reads are audited by byte count only — clipboard contents are never written to the audit log.

Waiting on the real world

wait.until blocks a step until an actual readiness condition holds, instead of racing UIs with sleeps:

• file_exists — a path appears
• process_running — a process is up
• port_open / http_ready — a service is accepting connections / answering
• duration — a plain timed wait when that’s genuinely what you want

Cost forecasting

cost.forecast returns a token + USD estimate for a planned model call before it happens — useful inside missions and scheduled jobs where you want spend visibility up front.

Just-in-time credentials

vault.request is a default-deny credential broker: tools don’t hold secrets; they request them just-in-time, the broker decides per policy, and every issuance is audited. A denied request is an answer, not an error.

Choosing a tier

Prefer the lowest-numbered tier that can do the job: native scripting is the most reliable and the cheapest to audit; the accessibility reader answers “what’s on screen” without screenshots; vision + input is for the long tail nothing else reaches. Every tier is capability-gated and every action lands in the audit log.