droid-cua

What is droid-cua? •
Quick Start •
Features •
Usage •
Assertions •
Command Line Options •
How It Works •
License

---

AI-powered Android testing using OpenAI's computer-use model

Create and run automated Android tests using natural language. The AI explores your app and generates executable test scripts.

https://github.com/user-attachments/assets/36b2ea7e-820a-432d-9294-8aa61dceb4b0

---

💡 What is droid-cua?

droid-cua gives you three core components for Android testing:

* Interactive Shell – Design and run tests with real-time feedback and visual status indicators
* Test Scripts – Simple text files with natural language instructions and assertions
* AI Agent – Autonomous exploration powered by OpenAI's computer-use model

Together, these let you create and execute Android tests without writing traditional test code.

---

🚀 Quick Start

1. Install

Globally (recommended):
``sh
npm install -g @loadmill/droid-cua
`

Or from source:
`sh
git clone https://github.com/loadmill/droid-cua
cd droid-cua
npm install
npm run build
`

2. Set your OpenAI API key

Using environment variable:
`sh
export OPENAI_API_KEY=your-api-key
`

Or create a .env file:
`sh
echo "OPENAI_API_KEY=your-api-key" > .env
`

3. Ensure ADB is available

`sh
adb version
`

4. Run

`sh
droid-cua
`

The emulator will auto-launch if not already running.

---

✨ Features

- Design Mode - Describe what to test, AI explores and creates test scripts
- Execution Mode - Run tests with real-time feedback and assertion handling
- Headless Mode - Run tests in CI/CD pipelines
- Test Management - Create, edit, view, and run test scripts
- Smart Actions - Automatic wait detection and coordinate mapping

---

📚 Usage

$3

| Command | Description |
|---------|-------------|
| /create | Create a new test |
| /run | Execute a test |
| /list | List all tests |
| /view | View test contents |
| /edit | Edit a test |
| /help | Show help |
| /exit | Exit shell |

$3

`sh
droid-cua
> /create login-test
> Test the login flow with valid credentials
`

The AI will explore your app and generate a test script. Review and save it.

$3

Interactive:
`sh
droid-cua
> /run login-test
`

Headless (CI/CD):
`sh
droid-cua --instructions tests/login-test.dcua
`

$3

One instruction per line:

`
Open the Calculator app
assert: Calculator app is visible
Type "2"
Click the plus button
Type "3"
Click the equals button
assert: result shows 5
exit
`

Assertions

Assertions validate the app state during test execution. Add them anywhere in your test script.

Syntax (all valid):
`
assert: the login button is visible
Assert: error message appears
ASSERT the result shows 5
`

Interactive Mode - When an assertion fails:
- retry - Retry the same assertion
- skip - Continue to next instruction
- stop - Stop test execution

Headless Mode - Assertions fail immediately and exit with code 1.

Examples:
`
assert: Calculator app is open
assert: the result shows 8
assert: error message is displayed in red
assert: login button is enabled
`

---

💻 Command Line Options

| Option | Description |
|--------|-------------|
| --avd=NAME | Specify emulator |
| --instructions=FILE | Run test headless |
| --record | Save screenshots |
| --debug | Enable debug logs |

---

Requirements

- Node.js 18.17.0+
- Android Debug Bridge (ADB)
- Android Emulator (AVD)
- OpenAI API Key (Tier 3 for computer-use-preview model)

---

🔧 How It Works

1. Connects to a running Android emulator
2. Captures full-screen device screenshots
3. Scales down the screenshots for OpenAI model compatibility
4. Sends screenshots and user instructions to OpenAI's computer-use-preview model
5. Receives structured actions (click, scroll, type, keypress, wait, drag)
6. Rescales model outputs back to real device coordinates
7. Executes the actions on the device via ADB
8. Validates assertions and handles failures
9. Repeats until task completion

---

🎞️ Convert Screenshots to Video

If you run with --record, screenshots are saved to:
`
droid-cua-recording-/
`

Convert to video with ffmpeg:
`sh
ffmpeg -framerate 1 -pattern_type glob -i 'droid-cua-recording-/frame_.png' \
-vf "pad=ceil(iw/2)2:ceil(ih/2)2" \
-c:v libx264 -pix_fmt yuv420p session.mp4
``

---

📄 License

© 2025 Loadmill. All rights reserved.