📦 test-mcp

What's test-mcp? •
Installation •
Getting Started •
Configuration & Test Format •
How to Run •
CLI Flags •
Test Discovery •
Interactive Mode •
Programmatic API •
Roadmap •
Contributing •
License

---

test-mcp is a headless MCP client for automated testing of MCP servers and agents.

If you’re building an MCP server or agent, test-mcp lets you run natural-language test scripts and assertions end-to-end, so you can validate behavior in a fast and repeatable way.

https://github.com/user-attachments/assets/c4e5295b-7217-47f2-96d9-76befcea8b21

---

💡 What's test-mcp?

test-mcp gives you three core components:

* Configuration – define your MCP servers and LLM provider in a single JSON file.
* Test Files – write flows of natural-language prompts and assertions in YAML.
* Runner – run tests from the CLI, get clear pass/fail results.

Together, these let you automate and validate MCP server behavior with simple, repeatable tests.

Supported Transports & Providers:
- MCP Servers: STDIO (local) and HTTP (remote)
- LLM Providers: Anthropic Claude and OpenAI GPT models
- MCP Features: Tools (done), Resources/Prompts/Sampling (planned)

---

🏗️ Installation

``bash


using npm

npm install -g @loadmill/test-mcp

or with pnpm

pnpm add -g @loadmill/test-mcp
``

> [!NOTE]
> While test-mcp itself can run on Node.js 18 or higher, many popular MCP servers require Node.js 20.
> For the smoothest experience, we recommend using Node.js 20.

When running from source:

`bash
git clone https://github.com/loadmill/test-mcp
cd test-mcp
npm install


For OpenAI (default example)

echo "OPENAI_API_KEY=your_api_key_here" > .env

Or for Anthropic

echo "ANTHROPIC_API_KEY=your_api_key_here" > .env
npm run build
`

---

🚀 Getting Started

To try test-mcp quickly with the included examples:

`bash


from source


node build/index.js
`

This will run a demonstration that shows both local STDIO and remote HTTP MCP servers working together. The test rolls a local dice server and queries a remote MCP server registry.

---

📑 Configuration & Test Format

1) Example config (mcp.config.json)

`json
{
"mcpClient": {
"provider": "openai",
"model": "gpt-4o-mini",
"api_key": "${env:OPENAI_API_KEY}"
},
"mcpServers": {
"loadmill": {
"type": "stdio",
"command": "npx",
"args": ["@loadmill/mcp"],
"env": {
"LOADMILL_API_TOKEN": "${env:LOADMILL_API_TOKEN}"
}
}
}
}
`

OpenAI is also supported - see configuration variations in the examples/ folder.

2) Example test (tests/bank-transaction.test.yaml)

`yaml
description: "Maker Checker Bank - Transaction Creation and Rejection Flow"

steps:
- prompt: "Login with username alice and password alice123 and transfer $100 to Bob"
- prompt: "Login with username bob and password bob456, reject transaction from Alice"
- assert: "Validate the transaction was created and rejected successfully"
`

---

▶️ How to run

By default, test-mcp looks for mcp.config.json in the project root and runs tests in the tests/ folder.

Globally installed:

`bash
test-mcp
`

From source:

`bash
node build/index.js
`

Point to a specific config or tests directory:

`bash
test-mcp --config mcp.config.json --tests-dir ./tests
`

---

💻 CLI Flags

`
Options:
-c, --config Path to config file (default: mcp.config.json)
-t, --tests-dir

Directory containing test files (default: tests)
-i, --interactive Run in interactive chat mode
--trace Enable detailed tracing output
-h, --help Show help
`

---

🔎 Test Discovery

All files ending in .test.yaml under the tests/ directory are executed.
Recursive discovery and full glob patterns are planned for later.

---

💬 Interactive Mode

Run the client without tests and chat with your MCP servers:

`bash
test-mcp -i
`

---

🔧 Programmatic API

You can use test-mcp programmatically in your Node.js code:

`javascript
import { TestMCPClient } from '@loadmill/test-mcp';

const client = new TestMCPClient({
llm: {
provider: 'openai',
model: 'gpt-4o-mini',
apiKey: process.env.OPENAI_API_KEY
},
servers: {
myServer: {
type: 'stdio',
command: 'node',
args: ['./server.js']
}
}
});

await client.connect();
const response = await client.prompt('Your question');
const assertion = await client.assert('Expected behavior');
await client.disconnect();
`

See examples/api-example.js for a complete example.

---

🛣️ Roadmap

* [x] Headless MCP client with Anthropic support
* [x] Support for stdio transport
* [x] Support for MCP tools
* [x] Evaluator for natural-language assertions
* [x] OpenAI support
* [x] Support for http transport
* [ ] Test parameterization with ${}`
* [ ] CI-friendly reports
* [ ] Support for MCP resources
* [ ] Support for MCP prompts
* [ ] Support for MCP sampling

---

🤝 Contributing

Contributions, ideas, and bug reports are welcome! See CONTRIBUTING.md.

---

📄 License

Apache License 2.0 © Loadmill