All guides

Guide · 9 min read

How an AI assistant can safely control Mac apps.

An agent clicking around your Mac is only safe if it knows what it's clicking. Axiom asks macOS for the button by name first. Coordinates are the last resort, and a bad coordinate gets refused, not nudged.

Begin with the requested outcome

Internal steps aren't permission to open windows or click controls. The visible effect has to belong to the thing you asked for. Reading an attachment quietly and popping Preview in your face are different products, even when both can see the pixels.

Under the hoodHow it works
  • Attachment preparation uses a non-foreground data path when the requested outcome is an answer.
  • Foreground app launches remain available for explicit visible actions instead of being globally blocked.
Foreground rule

Background work stays quiet. Visible actions happen when you ask for them or when the visible task cannot finish without them.

Act on known elements before coordinates

macOS can identify any button, menu item, or field as a named object. Acting on that reference beats estimating a point, because the reference carries identity and state. If a predicted point lands inside a known element, use the element. Coordinates only apply to things that have no name, like a drawing canvas.

Under the hoodHow it works
  • An accessibility reference carries identity, role, state, and current geometry.
  • If a predicted point falls inside a known element, route the action through that element reference.

A coordinate must be valid or refused

A wrong coordinate that lands somewhere plausible looks exactly like success. That's what makes them dangerous. Validate the screen, window, scale, and bounds before acting. And never clamp an out-of-range point to the nearest edge. That turns a wrong guess into a confident-looking click somewhere you didn't ask for.

  • Confirm the target screen and window
  • Reject points outside the valid range
  • Refresh geometry after a move or resize
  • Record which action path actually ran
Under the hoodHow it works
  • Reject coordinates outside the measured range; never clamp them to a nearby edge.
  • Invalidate geometry after window moves, display changes, scaling changes, and relevant animation.

Scale confirmation to consequence

Read-only and destructive actions don't deserve the same friction. Sending, publishing, spending, and deleting need a confirmation that names exactly what will happen and what can't be undone. Not a generic Allow button.

Under the hoodHow it works
  • Confirmation depth should follow consequence, reversibility, audience, and data exposure.
  • The confirmation must name the exact target and effect rather than use a generic permission label.

Design the failure path before the demo path

Permissions get denied and elements vanish mid-action. Stop at the uncertain step, say what's missing, offer the precise next move. Retrying blind isn't intelligence, it's risk with extra steps.

Under the hoodHow it works
  • Stop at the first unverified boundary and retain the task state needed to resume.
  • Do not treat repeated retries as evidence that an unavailable element or permission has become safe.

Sources and scope

We wrote this from Axiom's current code, regression tests, and recorded product checks. If a behavior has only passed a source test, we do not describe it as proven on every Mac. There are no invented benchmark numbers here.