The gist

ESP32 iPhone Agent is an open-source tool that allows vision AI models to control a physical iPhone like a human. By combining an inexpensive microcontroller board with an iOS companion app, it sees the screen via AirPlay and sends taps and keystrokes over USB. It is designed for small teams and creators who want to automate repetitive tasks on a real iPhone without a permanent computer connection or jailbreak.

What it does

Streams the iPhone display to a microcontroller board using built-in AirPlay screen mirroring
Controls the phone by simulating a standard USB mouse and keyboard via iOS AssistiveTouch
Sends screen captures to any OpenAI-compatible vision AI model to determine the next action
Executes actions including tapping buttons, swiping, typing text, and pressing keys
Replays and reviews past runs step by step with touch paths drawn directly on the phone

How it works

The microcontroller board plugs into the iPhone and creates a local Wi-Fi hotspot for AirPlay screen mirroring. An app on the phone captures this video feed and sends screen images to a vision AI model over cellular data. The AI decides what action to take next, and the app instructs the board to move the on-screen pointer and tap or type through a USB connection.

Best for

Automating routine multi-step workflows directly inside native iPhone apps
Testing mobile applications on real iOS devices without cloud device farms
Creators and small agencies needing physical phone automation without jailbreaking

Watch out for

Requires a Mac with Xcode and developer tools for the initial one-time flashing and app installation
Needs a specific $7 ESP32-S3 board with PSRAM and native USB, plus an adapter for Lightning iPhones
Consumes API credits from your chosen vision model provider for each visual action step
The iPhone must use cellular data during runs because its Wi-Fi connection is dedicated to the board

What you need: Install it on your own computer. Cost: free, open source.