diff --git a/README.md b/README.md index 04d52d8..a1dc9df 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,11 @@ ## Architecture

- agent-desktop architecture diagram + agent-desktop architecture diagram +

+ +

+ agent-desktop real-world example — Slack accessibility tree with 97% token savings

diff --git a/assets/Tutorial.gif b/assets/Tutorial.gif index 520157e..f0bd1c3 100644 Binary files a/assets/Tutorial.gif and b/assets/Tutorial.gif differ diff --git a/docs/architecture.html b/docs/architecture.html new file mode 100644 index 0000000..5db1276 --- /dev/null +++ b/docs/architecture.html @@ -0,0 +1,949 @@ + + + + + + agent-desktop · architecture + + + + + + +
+
+ agent-desktop · architecture +

Computer use, done natively.

+

Most computer-use stacks feed the agent a screenshot and hope it can find a button by pixel. agent-desktop reads the OS accessibility tree directly — every element has a deterministic name, every action goes through the same native API a screen reader uses. Faster, cheaper, repeatable. Works with any harness, any model.

+
+ computer use + native accessibility + open source · apache-2.0 +
+
+ +
+ + +
+ +
+
+
+
+
+
+
+
+
+
+ +
+
+
+
+
+ +
+
+ command +
+ + 1 + + +
+
+
+
+ + + 4 + +
+ refs · json +
+
+ +
+
+
+ a–d +
+
+
+ +
+
+ ax query +
+ + 2 + + +
+
+
+
+ + + 3 + +
+ tree +
+
+ +
+
+
+
+
+
Image Upload
+
+
+
+
+
+
+
+ + +
+ AI agent + LLM · script · host +
+
+ agent-desktop + cli · cdylib · ffi +
+
+ Target app + macOS · windows · linux +
+
+ + +
+
+ 1 +
+
Agent calls the binary
+
An LLM or script invokes agent-desktop snapshot --app MySwiftApp.
+
+
+
+ 2 +
+
Binary queries the OS
+
Reads the live accessibility tree of the target app via native AXUIElement APIs.
+
+
+
+ 3 +
+
Tree returns
+
Every interactive element gets a stable handle: @e1 … @eN, scoped to a snapshot id.
+
+
+
+ 4 +
+
Agent acts on a ref
+
Follow-up call like click @e7 drives the app through the same native AX path.
+
+
+
+ + + + diff --git a/docs/architecture.png b/docs/architecture.png index 9da00b8..b568feb 100644 Binary files a/docs/architecture.png and b/docs/architecture.png differ diff --git a/docs/example.html b/docs/example.html new file mode 100644 index 0000000..3a07b8a --- /dev/null +++ b/docs/example.html @@ -0,0 +1,292 @@ + + + + + + agent-desktop · real example + + + + + + +
+
+ real example · dense desktop app +

Slack — a window with 1,432 nodes.

+

A full snapshot of Slack's main window is a wall of text. Skeleton mode (depth 3) returns the same actionable refs at ~3% of the token cost. Same approach works for VS Code, Figma, Notion, Excel — anything with an accessibility tree.

+
+ +
+
+ full tree + 1,432nodes + ~28,000 tokens +
+
+ skeleton · depth 3 + 41nodes + ~900 tokens +
+
+ savings + 97% + same refs · same actions +
+
+ +
+
+ $ agent-desktop snapshot --skeleton --app Slack + snapshot k3p9w2x4 +
+ 
▾ Window "Slack | #engineering"
+  ▾ Toolbar
+    · Button "Workspace switcher"                        @e1
+    · SearchField "Search Acme"                          @e3
+  ▾ Group "sidebar"                                          children_count: 247
+      [+247 more · drill @e5]
+  ▾ Group "active channel"                                  @e9
+    ▾ Group "messages"                                    children_count: 1,043 · virtualized
+    ▾ Group "compose"
+      · TextField "Message #engineering"                @e12
+      · Button "Send"                                       @e15
+  ▾ Group "thread pane"                                     children_count: 89
+  · StatusBar "connected"
+
+ Then act: agent-desktop type @e12 "ship it" → agent-desktop click @e15. + native AX · no cursor · no screenshot +
+
+ + +
+ + diff --git a/docs/example.png b/docs/example.png new file mode 100644 index 0000000..5b7fbaa Binary files /dev/null and b/docs/example.png differ