agent-desktop · architecture

Computer use, done natively.

Most computer-use stacks feed the agent a screenshot and hope it can find a button by pixel. agent-desktop reads the OS accessibility tree directly — every element has a deterministic name, every action goes through the same native API a screen reader uses. Faster, cheaper, repeatable. Works with any harness, any model.

computer use native accessibility open source · apache-2.0
command
1
4
refs · json
a–d
ax query
2
3
tree
Image Upload
AI agent LLM · script · host
agent-desktop cli · cdylib · ffi
Target app macOS · windows · linux
1
Agent calls the binary
An LLM or script invokes agent-desktop snapshot --app MySwiftApp.
2
Binary queries the OS
Reads the live accessibility tree of the target app via native AXUIElement APIs.
3
Tree returns
Every interactive element gets a stable handle: @e1 … @eN, scoped to a snapshot id.
4
Agent acts on a ref
Follow-up call like click @e7 drives the app through the same native AX path.