Start with one task you already repeat
Run the same task three times over a few days. The first run tells you how well it finds things. The later runs tell you whether it remembered anything, or whether you're doing the same setup dance every single time. Count four things: time until you have something useful, how many times you had to intervene, whether it found the right file, and whether you can undo what it did.
- Time to first useful result
- Number of manual handoffs
- Whether it found the right file
- Whether the final state was easy to undo
Under the hoodHow it works
- Run the task once from a cold start and again after the files and apps have been used.
- Record the number of manual selections, corrections, and app switches, not only the response time.
Separate answers from actions
A chat app can explain how to rename files without being able to rename one. A launcher can open an app without understanding the project. Figure out which one your task needs before comparing products. And more agency isn't automatically better: if the assistant isn't sure what it's about to click, it should stop, not guess.
Under the hoodHow it works
- Separate read-only answers, deterministic commands, and multi-step actions in the test plan.
- Verify that visible effects map to the request and that uncertain targets stop before acting.
Test the context boundary
Close and reopen the app and check that the right project comes back. Ask a follow-up that depends on an earlier decision. Context is only valuable when it stays scoped to the right conversation. One frame of the wrong client's chat is a leak.
Under the hoodHow it works
- Presentation state should be keyed by record identity rather than shared by the transcript component.
- Slow and out-of-order loads must be rejected when their record identity is no longer selected.
Draw the data path
Trace all four paths: local, model provider, device sync, public publishing. If the product can't explain them, a 'private' badge means nothing. Check diagnostics and analytics too. Privacy doesn't stop at the model request.
Under the hoodHow it works
- Trace model inference, account sync, diagnostics, and public sharing as separate data paths.
- Check the active provider and the exact attachments selected for the request.
What exact data leaves this Mac when I perform this exact action?
Deliberately create a failure
The recovery path tells you more about a product than the demo does. A good one explains what's missing, keeps your work intact, and offers the next safe step. A bad one fabricates an answer, retries blind, or shows stale content. Break things on purpose before you trust an assistant with real work.
Under the hoodHow it works
- Test permission denial, timeout, cancellation, missing index data, and stale responses independently.
- A recovery path should state the reason, the effect on the task, and the next safe action.
Choose the tool that removes the most repeated friction
The right answer might be a chat app, a launcher, or a plain automation tool. Keep what already works. Replace a workflow only when the new way is clearly faster or clearer. Our bet is that most Mac tasks start in the same place and get better with shared context. The trial proves whether that's true for you.
Under the hoodHow it works
- Compare complete task time and correction rate rather than isolated model latency.
- Keep a deterministic tool when predictable repetition matters more than flexible reasoning.
Sources and scope
We wrote this from Axiom's current code, regression tests, and recorded product checks. If a behavior has only passed a source test, we do not describe it as proven on every Mac. There are no invented benchmark numbers here.