Model Context Protocol (MCP) server for Runhuman - Human-powered QA testing for AI agents
npm install @runhuman/mcp-serverA Model Context Protocol (MCP) server that allows AI agents to interact with the Runhuman QA testing service.
This MCP server provides tools for creating and managing human QA jobs through the Runhuman API. AI agents can use this server to:
- Create new QA jobs with custom schemas
- Check the status of running jobs
- Retrieve completed job results
1. Get your API key at: https://runhuman.com/dashboard/api-keys
2. Add to your Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json on Mac):
``json`
{
"mcpServers": {
"runhuman": {
"command": "npx",
"args": ["-y", "@runhuman/mcp-server", "--api-key=qa_live_xxxxxxxxxxxxx"]
}
}
}
3. Restart Claude Desktop
That's it! The server will be automatically downloaded and run by Claude.
From the monorepo root:
`bash
npm install
npm run build --workspace=@runhuman/mcp-server
$3
####
Create a new QA job with human testers.Parameters:
-
(string): The URL to test
- description (string): Instructions for the human tester describing what to test
- schema (object): Expected result schema that the tester response will be extracted into
- targetDurationMinutes (number, optional): Time limit for tester (default: 5, range: 1-60)####
wait_for_result
Check status, wait, and retrieve results for a QA job in a single convenient call.Parameters:
-
(string): The ID of the job to check
- waitSeconds (number, optional): How long to wait before checking again (default: 30, range: 1-300)Behavior:
- Checks status BEFORE waiting (returns immediately if already complete)
- Waits for the specified duration
- Checks status AFTER waiting
- Returns results if complete, otherwise suggests calling again with longer wait time
Usage Pattern:
`javascript
// After creating a job, call repeatedly with increasing wait times:
let result = await wait_for_result(jobId, { waitSeconds: 30 });
if (result.status !== 'completed') {
result = await wait_for_result(jobId, { waitSeconds: 45 });
}
if (result.status !== 'completed') {
result = await wait_for_result(jobId, { waitSeconds: 60 });
}
`Returns:
-
: Structured test results extracted from tester's response
- status: Job status (completed, failed, timeout, pending, claimed, in_progress)
- costUsd: Exact cost in USD with full precision (e.g., 0.396)
- testDurationSeconds: Time spent by tester in seconds (rounded up)
- testerResponse: Raw natural language feedback from the human tester
- testerAlias: Anonymized tester name (e.g., "Tester Alpha")
- testerAvatarUrl: Avatar image URL for UI display
- testerColor: Hex color code for theming (e.g., "#4A90E2")
- Additional metadata (timestamps, etc.)Cost Calculation:
- Costs are calculated as:
duration × $0.0018/second (general-use tier)
- Duration is always rounded UP using Math.ceil (any partial second counts)
- Costs are never $0 unless the tester never actually worked on it
- Full precision maintained (not rounded to cents)Configuration
The MCP server needs to be configured with your Runhuman API credentials.
$3
Option A: Via Dashboard
1. Start the API server:
npm run dev --workspace=@runhuman/api
2. Open http://localhost:3400/api-keys
3. Click "Create API Key"
4. Copy the key (starts with )Option B: Use Default Test Key
- For local development, you can use:
qa_live_test_key_123
- This key exists in $3
Create a
file in the MCP server directory:`bash
For local development
RUNHUMAN_API_URL=http://localhost:3400
RUNHUMAN_API_KEY=qa_live_test_key_123For production
RUNHUMAN_API_URL=https://api.runhuman.com
RUNHUMAN_API_KEY=qa_live_xxxxxxxxxxxxxxxxxxxxx
`Important: Never commit
files to git! They're already in .gitignore.$3
Test your API key works:
`bash
curl http://localhost:3400/api/jobs \
-H "Authorization: Bearer qa_live_test_key_123" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com","description":"test","outputSchema":{}}'
`Should return a job ID if authentication works.
For more details, see docs/API-AUTHENTICATION.md
Testing
The MCP server includes automated tests to verify it's working correctly:
bash
Build first
npm run build --workspace=@runhuman/mcp-serverRun simple automated test
npm run test --workspace=@runhuman/mcp-serverOr use the MCP Inspector (interactive testing)
npm run test:inspector --workspace=@runhuman/mcp-server
`The test script will:
1. ✅ Initialize a connection to the MCP server
2. ✅ List all available tools (create_job, wait_for_result)
3. ✅ Test calling the create_job tool
$3
✅ Server initialized successfully
✅ Tools listed: create_job, wait_for_result
✅ create_job tool called successfully
Development
bash
Watch mode (auto-rebuild on changes)
npm run dev --workspace=@runhuman/mcp-serverBuild
npm run build --workspace=@runhuman/mcp-serverTest after building
npm run test --workspace=@runhuman/mcp-server
`Integration with Claude Desktop
To use this MCP server with Claude Desktop, add it to your configuration:
json
{
"mcpServers": {
"runhuman": {
"command": "node",
"args": ["/path/to/qa-experiment/packages/mcp-server/dist/index.js"]
}
}
}
`Example Usage
Once connected to an AI agent (like Claude), the agent can use these tools naturally:
User: "Can someone test my checkout page at https://myapp.com/checkout?"
Agent uses create_job:
✅ Job created successfully!
Job ID: job_abc123
Status: pending
...
Agent calls wait_for_result repeatedly until complete:
⏳ Job Status: in_progress
Waited 30s, job not complete yet.
💡 Suggestion: Call wait_for_result again with waitSeconds: 45
Finally:
✅ Test completed!
Results Summary:
- Checkout Flow: ✅ Working
- Payment Processing: ✅ Successful
...
For developers working on this MCP server:
- docs/HOW-AGENTS-USE-MCP.md - How AI agents discover and use MCP servers
- docs/TOOL-RESPONSE-BEST-PRACTICES.md - Best practices for tool responses
- Model Context Protocol Documentation
- Runhuman API Documentation