How to automate macOS with pi-computer-use

If you’ve been looking for a way to give your AI agents actual hands on your desktop, you’ve likely hit the same wall I did: most automation frameworks are either too clunky or require you to surrender control of your mouse entirely. The pi-computer-use package changes that by integrating Codex-style tools directly into the Pi coding agent environment. It’s a clean, native approach to letting an agent see your screen and interact with your apps without turning your workflow into a chaotic mess.

The real magic here isn't just the ability to click buttons; it’s the "invisible" nature of the implementation. By using a native macOS helper, the agent handles input dispatching and screen capture with a level of precision that standard browser-based automation scripts simply can't match. You aren't just running a script; you're teaching an agent how to navigate your specific environment.

Getting your agent up and running

Before you start, ensure you’re running macOS 15 or higher. This is a hard requirement because the underlying accessibility and screen recording APIs have evolved significantly in recent versions. If you try to force this on an older OS, you’ll spend more time debugging permissions than actually automating your tasks.

To install the package, skip the standard registry search—it currently resolves to an unrelated project. Instead, pull directly from the source:

1. Open your terminal and run pi install git:github.com/injaneity/pi-computer-use.
2. If you’re working on a specific project, use the -l flag to keep the dependency local.
3. Once installed, the agent will automatically load the necessary extensions and skills.

Handling the permission hurdle

Here is where most people get tripped up: the first time you ask your agent to perform an action, it will fail. This isn't a bug; it’s macOS security doing its job. You must manually grant Accessibility and Screen Recording permissions to the helper binary located at ~/.pi/agent/helpers/pi-computer-use/bridge.

Don't try to automate the permission grant itself—it’s a security boundary you have to cross manually. Once you’ve toggled those switches in System Settings, the agent gains the ability to inspect windows and execute commands like click, type_text, and scroll with high reliability.

Why this approach beats the alternatives

Most automation tools rely on global cursor hijacking, which makes it impossible for you to use your computer while the agent is working. pi-computer-use prefers a non-intrusive strategy. It attempts to set values via the Accessibility API before falling back to raw key presses or mouse events. This means the agent can often interact with your UI elements in the background, leaving your actual mouse cursor free for you to use.

If you’re wondering how to fix common interaction failures, always start by calling screenshot first. By forcing the agent to "see" the frontmost window before it attempts a click or drag, you provide the context it needs to map coordinates accurately. If the agent is struggling to find a button, it’s usually because it hasn't refreshed its visual state.

This is the most efficient way to build autonomous workflows on macOS today. Try this today and share what you find in the comments, or read our breakdown of advanced agentic workflows next to see how you can chain these tools for complex tasks.