Playwright AI Reporter


Transform test failures into actionable insights with AI-powered analysis and auto-healing

Playwright AI Reporter is an enterprise-grade, production-ready test reporter that combines artificial intelligence with comprehensive test automation workflows. Built on a flexible provider-based architecture, it automatically analyzes test failures, creates detailed bug reports, generates fix suggestions, and can even submit auto-healing pull requests—all while integrating seamlessly with your existing development tools.

$3

- 🧠 AI-Powered Analysis - Multiple AI providers (Azure OpenAI, Anthropic Claude, Google Gemini, Mistral, OpenAI) analyze failures and suggest intelligent fixes
- 🔌 Plug & Play Architecture - Swap bug trackers, databases, AI providers, and notification systems without code changes
- 🚀 Auto-Healing Tests - Automatically generate and submit PRs with AI-suggested fixes for flaky or failing tests
- 📊 Enterprise Integration - Native support for GitHub, Azure DevOps, Jira, MySQL, SQLite, SMTP, and more
- 🎨 Rich Reporting - Colorized console output, comprehensive metrics, historical analysis, and build integration
- ⚡ Production-Ready - TypeScript, fully tested, extensive documentation, CI/CD workflows included

> 💡 Perfect for - CI/CD pipelines • Enterprise test automation • Multi-team projects • Flaky test management • Test debugging at scale

---

📖 Table of Contents

- Features
- Architecture
- Quick Start
- Installation
- Configuration
- Provider Support
- Usage Examples
- Output Examples
- FAQs
- Documentation
- Contributing
- License

---

✨ Features

$3

- ✅ Colorized console output (Passed, Failed, Retries, Skipped)
- 📊 Comprehensive test metrics and statistics
- 🎯 Slowest test identification and ranking
- ⏱️ Average test duration analysis
- 📈 Test history tracking and comparison
- 🏗️ CI/CD integration with build information
- 🌐 Interactive HTML Report: Self-contained HTML dashboard with charts, test details, and AI fix suggestions

$3

- 🧠 Multi-AI Provider Support: Azure OpenAI, OpenAI, Anthropic (Claude), Google AI (Gemini), Mistral AI
- 🔧 Automatic Fix Suggestions: AI analyzes failures and suggests fixes
- 📝 Context-Aware Analysis: Includes test code, error details, and stack traces
- 🎓 Smart Categorization: Automatic error categorization (Timeout, Selector, Network, etc.)
- 💡 Best Practices: Suggestions follow Playwright best practices

$3

- 🎫 Multi-Platform Bug Creation: GitHub Issues, Azure DevOps Work Items, Jira Tickets
- 📋 Rich Bug Details: Test info, error details, AI suggestions, environment data
- 🏷️ Smart Labeling: Automatic labels and priority assignment
- 🔗 Integrated Tracking: Links bugs to test runs and failures

$3

- 🚀 Automatic PR Creation: Generate PRs with AI-suggested fixes (set generatePR: true)
- 🌿 Branch Management: Auto-create topic branches (autofix/test-name-timestamp)
- 💾 Smart Commits: Commit fixes to topic branch with detailed messages
- 📤 Push & PR: Push changes and create pull request from topic to base branch
- 📝 Rich PR Descriptions: Include error analysis, test details, and fix rationale
- 🎯 Draft PRs: Created as drafts for mandatory code review
- 🏷️ Auto Labels: auto-fix, test-failure, ai-generated labels
- ⚙️ Platform Support: GitHub and Azure DevOps

$3

- 📊 Test Run Tracking: Store complete test run metadata (environment, branch, commit, totals, duration)
- 📈 Result History: Track individual test results over time with full details
- 🔍 Failure Analysis: Query and analyze failure patterns with indexed searches
- 🏗️ Build Integration: Link results to CI/CD builds with metadata
- 💾 Multi-Database: SQLite (file-based), MySQL, PostgreSQL support
- 🗃️ Schema: 2 tables (test_runs, test_results) with 4 performance indexes

---

🏗️ Architecture

$3

``mermaid
graph TB
subgraph "Playwright Test Runner"
Tests[Test Execution]
end

subgraph "AI Test Reporter"
Reporter[Reporter Core]
Registry[Provider Registry]
Workflow[Test Workflow Engine]
end

subgraph "AI Providers"
Azure[Azure OpenAI]
OpenAI[OpenAI]
Anthropic[Anthropic Claude]
Google[Google Gemini]
Mistral[Mistral AI]
end

subgraph "Bug Trackers"
GitHub[GitHub Issues]
ADO[Azure DevOps]
Jira[Jira]
end

subgraph "Databases"
SQLite[SQLite]
MySQL[MySQL]
Postgres[PostgreSQL]
end

subgraph "Notifications"
Email[Email/SMTP]
Slack[Slack]
Teams[MS Teams]
end

subgraph "PR Providers"
GHPR[GitHub PRs]
ADOPR[Azure Repos PRs]
end

Tests --> Reporter
Reporter --> Registry
Registry --> Workflow

Workflow --> Azure
Workflow --> OpenAI
Workflow --> Anthropic
Workflow --> Google
Workflow --> Mistral

Workflow --> GitHub
Workflow --> ADO
Workflow --> Jira

Workflow --> SQLite
Workflow --> MySQL
Workflow --> Postgres

Workflow --> Email
Workflow --> Slack
Workflow --> Teams

Workflow --> GHPR
Workflow --> ADOPR

style Reporter fill:#4CAF50
style Registry fill:#2196F3
style Workflow fill:#FF9800
`

$3

The reporter uses a provider-based architecture for maximum flexibility:

`mermaid
graph LR
subgraph "Application Layer"
Reporter[Test Reporter]
Utils[Utilities]
end

subgraph "Provider Registry"
Registry[Provider Registry
Singleton Manager]
end

subgraph "Factory Layer"
AIFactory[AI Factory]
BugFactory[Bug Tracker Factory]
DBFactory[Database Factory]
NotifyFactory[Notification Factory]
PRFactory[PR Factory]
end

subgraph "Provider Interfaces"
IAI[IAIProvider]
IBug[IBugTrackerProvider]
IDB[IDatabaseProvider]
INotify[INotificationProvider]
IPR[IPRProvider]
end

subgraph "Concrete Implementations"
AzureAI[Azure OpenAI]
OpenAI[OpenAI]
GitHubBug[GitHub Issues]
SQLite[SQLite]
EmailNotify[Email]
GitHubPR[GitHub PRs]
end

Reporter --> Registry
Utils --> Registry

Registry --> AIFactory
Registry --> BugFactory
Registry --> DBFactory
Registry --> NotifyFactory
Registry --> PRFactory

AIFactory --> IAI
BugFactory --> IBug
DBFactory --> IDB
NotifyFactory --> INotify
PRFactory --> IPR

IAI --> AzureAI
IAI --> OpenAI
IBug --> GitHubBug
IDB --> SQLite
INotify --> EmailNotify
IPR --> GitHubPR

style Registry fill:#FF6B6B
style IAI fill:#4ECDC4
style IBug fill:#4ECDC4
style IDB fill:#4ECDC4
style INotify fill:#4ECDC4
style IPR fill:#4ECDC4
`

$3

`mermaid
sequenceDiagram
participant PT as Playwright Test
participant R as Reporter
participant AI as AI Provider
participant BT as Bug Tracker
participant PR as PR Provider
participant DB as Database
participant N as Notification

PT->>R: Test Failed
R->>R: Categorize Error
R->>R: Extract Test Code

R->>AI: Generate Fix Suggestion
AI-->>R: AI Analysis & Fix

par Parallel Operations
R->>BT: Create Bug/Issue
BT-->>R: Bug Created
and
R->>DB: Save Test Result
DB-->>R: Result Saved
end

alt Auto-Healing Enabled
R->>PR: Create Fix PR
PR-->>R: PR Created
end

R->>N: Send Notification
N-->>R: Notification Sent

R->>PT: Report Complete
`

$3

| Component | Description |
| --------------------- | ------------------------------------------------------------- |
| Reporter | Main entry point implementing Playwright's Reporter interface |
| Provider Registry | Centralized provider management with lazy initialization |
| AI Providers | Multiple AI service implementations for fix suggestions |
| Bug Trackers | Issue/ticket creation across platforms |
| Databases | Test result storage and historical analysis |
| PR Providers | Automated pull request creation |
| Notifications | Alert delivery across channels |
| Factories | Provider instantiation with configuration |
| Workflow Engine | Orchestrates the test failure handling process |

$3

- 🔌 Provider Independence - Not locked into any single service
- 🏭 Factory Pattern - Clean, standardized provider creation
- ⚡ Lazy Initialization - Resources loaded only when needed
- 🛡️ Type Safety - Full TypeScript support
- 🧪 Testable - Easy mocking for unit tests
- 📦 Modular - Import only what you need

---

⚡ Quick Start

$3

- Node.js 18 or higher
- Playwright 1.51 or higher
- An AI provider API key (Azure OpenAI, OpenAI, Anthropic, Google AI, or Mistral)

$3

`bash


Install the reporter

`bash


GitHub + Mistral AI + SQLite

Edit your .env file with your API keys and settings:

`env


AI Provider (choose one)

Update your playwright.config.ts:

`typescript
import {defineConfig} from '@playwright/test';

export default defineConfig({
reporter: [
['list'],
[
'playwright-ai-reporter',
{
// Test thresholds
slowTestThreshold: 3,
maxSlowTestsToShow: 5,

// Output
outputDir: './test-results',
showStackTrace: true,

// AI & Automation features
generateFix: true, // Generate AI fix suggestions
createBug: false, // Auto-create bugs for failures
generatePR: false, // Auto-create PRs with fixes
publishToDB: false, // Save to database
sendEmail: false, // Send email notifications
},
],
],
});
`

$3

`bash


From the examples folder

`bash


From the examples folder

`bash
npm install playwright-ai-reporter --save-dev
`

$3

`bash
yarn add -D playwright-ai-reporter
`

$3

`bash
pnpm add -D playwright-ai-reporter
`

---

⚙️ Configuration

$3

| Option | Type | Default | Description |
| ------------------------- | --------- | ---------------- | -------------------------------------------------------- |
| slowTestThreshold | number | 5 | Tests slower than this (seconds) are flagged as slow |
| maxSlowTestsToShow | number | 3 | Maximum number of slow tests to display in report |
| timeoutWarningThreshold | number | 30 | Warn if tests approach this timeout value (seconds) |
| showStackTrace | boolean | true | Include full stack traces in error reports |
| outputDir | string | ./test-results | Directory for JSON output files and AI-generated fixes |
| generateFix | boolean | false | Generate AI-powered fix suggestions (saves to files) |
| createBug | boolean | false | Auto-create bugs for failures (requires bug tracker) |
| generatePR | boolean | false | Auto-create PRs with fixes (requires generateFix=true) |
| publishToDB | boolean | false | Publish test results to database (requires DB provider) |
| sendEmail | boolean | false | Send email notifications (requires email configuration) |

$3

#### Generate Fix Only (Default)

`typescript
{ generateFix: true, createBug: false, generatePR: false, publishToDB: false }
// → Creates AI fix suggestions in test-results/fixes/
`

#### Create Bugs for Failures

`typescript
{ generateFix: false, createBug: true, generatePR: false, publishToDB: false }
// → Creates bugs in GitHub/Jira/Azure DevOps for each failure
`

#### Generate Fix + Auto PR

`typescript
{ generateFix: true, createBug: false, generatePR: true, publishToDB: false }
// → Creates fix files + topic branch + draft PR with fixes
`

#### Full Stack (All Features)

`typescript
{ generateFix: true, createBug: true, generatePR: true, publishToDB: true, sendEmail: true }
// → AI fixes + bug tracking + PRs + database logging + email notifications
`

$3

1. Test Fails → AI analyzes failure and generates fix suggestion
2. Generate Fix (generateFix: true) → Creates fix files in test-results/fixes/
3. Create Branch (generatePR: true) → Creates topic branch autofix/test-name-{timestamp}
4. Commit Changes → Commits AI fix to topic branch with detailed message
5. Create PR → Opens draft PR from topic branch → base branch with:
- Error details and AI analysis
- Labels: auto-fix, test-failure, ai-generated
- Links to commit and test details
6. Review & Merge → Team reviews draft PR before merging

---

🔌 Provider Support

$3

| Provider | Status | Configuration |
| ---------------------- | ------------------- | ------------------------------------------------------- |
| Azure OpenAI | ✅ Production Ready | AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_DEPLOYMENT_NAME |
| OpenAI | ✅ Production Ready | OPENAI_API_KEY, OPENAI_MODEL |
| Anthropic (Claude) | ✅ Production Ready | ANTHROPIC_API_KEY, ANTHROPIC_MODEL |
| Google AI (Gemini) | ✅ Production Ready | GOOGLE_AI_API_KEY, GOOGLE_AI_MODEL |
| Mistral AI | ✅ Production Ready | MISTRAL_API_KEY, MISTRAL_MODEL |

$3

| Provider | Status | Configuration |
| ----------------- | ------------------- | ------------------------------------------------------------------ |
| GitHub Issues | ✅ Production Ready | GITHUB_TOKEN, GITHUB_OWNER, GITHUB_REPO |
| Azure DevOps | ✅ Production Ready | AZURE_DEVOPS_ORG_URL, AZURE_DEVOPS_PROJECT, AZURE_DEVOPS_PAT |
| Jira | ✅ Production Ready | JIRA_HOST, JIRA_EMAIL, JIRA_API_TOKEN, JIRA_PROJECT_KEY |

$3

| Provider | Status | Configuration |
| -------------- | ------------------- | -------------------------------------------------------------- |
| SQLite | ✅ Production Ready | SQLITE_DB_PATH (optional) |
| MySQL | ✅ Production Ready | MYSQL_HOST, MYSQL_USER, MYSQL_PASSWORD, MYSQL_DATABASE |
| PostgreSQL | 🚧 Coming Soon | - |

$3

| Provider | Status | Configuration |
| ------------------- | ------------------- | ---------------------------------------------------------------------- |
| Email (SMTP) | ✅ Production Ready | EMAIL_HOST, EMAIL_USER, EMAIL_PASSWORD, EMAIL_FROM, EMAIL_TO |
| Slack | 🚧 Coming Soon | SLACK_WEBHOOK_URL |
| Microsoft Teams | 🚧 Coming Soon | TEAMS_WEBHOOK_URL |

$3

| Provider | Status | Configuration |
| ---------------- | ------------------- | ----------------------------------- |
| GitHub | ✅ Production Ready | Uses GITHUB_* configuration |
| Azure DevOps | ✅ Production Ready | Uses AZURE_DEVOPS_* configuration |

---

💻 Usage Examples

$3

`typescript
// playwright.config.ts
import {defineConfig} from '@playwright/test';

export default defineConfig({
reporter: [
[
'playwright-ai-reporter',
{
generateFix: true,
categorizeFailures: true,
slowTestThreshold: 3,
maxSlowTestsToShow: 5,
},
],
],
});
`

$3

`typescript
import {ProviderRegistry} from 'playwright-ai-reporter';

// Initialize providers
await ProviderRegistry.initialize({
ai: {type: 'openai'},
bugTracker: {type: 'github'},
database: {type: 'sqlite'},
});

// Get providers
const ai = await ProviderRegistry.getAIProvider();
const bugTracker = await ProviderRegistry.getBugTrackerProvider();
const db = await ProviderRegistry.getDatabaseProvider();
`

$3

`typescript
import {AIProviderFactory} from 'playwright-ai-reporter';

// Create specific provider
const provider = await AIProviderFactory.createProvider('anthropic');

// Generate completion
const response = await provider.generateCompletion([
{role: 'system', content: 'You are a test engineer.'},
{role: 'user', content: 'Analyze this test failure...'},
]);

console.log(response.content);
`

$3

`typescript
import {ReporterWorkflow} from 'playwright-ai-reporter';

// Initialize
await ReporterWorkflow.initialize();

// Process test failure
await ReporterWorkflow.processTestFailure(failure, sourceCode);

// Save test run
const runId = await ReporterWorkflow.saveTestRun(summary);

// Send notifications
await ReporterWorkflow.sendNotification(summary, failures);

// Cleanup
await ReporterWorkflow.cleanup();
`

For more examples, check the examples folder.

---

📋 Output Examples

$3

`plaintext
🚀 Starting test run: 3 tests using 2 workers
✅ Login test passed in 1.23s
✅ API integration test passed in 2.45s
⚠️ Payment test was skipped

✅ All 3 tests passed | 1 skipped | ⏱ Total: 3.68s

🖥️ Running locally

Additional Metrics:
- Average passed test time: 1.84s
- Slowest test took: 2.45s
- Top 3 slowest tests:
1. API integration test: 2.45s
2. Login test: 1.23s

⚠️ Warning: 1 test was skipped.
Please ensure to test the skipped scenarios manually before deployment.
`

$3

`plaintext
🚀 Starting test run: 3 tests using 2 workers
✅ Login test passed in 1.23s
❌ API test failed in 2.45s
🔄 Retry attempt for "API test" (failed) in 2.50s
⚠️ Payment test was skipped

❌ 1 of 3 tests failed | 1 passed | 1 skipped | ⏱ Total: 6.18s

🤖 Generating AI-powered fix suggestions...
Generating fix suggestion for: API test
✅ Fix suggestion generated:
- Prompt: ./test-results/prompts/api-test.md
- Fix: ./test-results/fixes/fix-api-test.md

🔄 Generating pull request with fix...
Creating topic branch: autofix/api-test-2025-12-30T10-30-45
✅ Branch created successfully
Committing changes to autofix/api-test-2025-12-30T10-30-45
✅ Changes committed: a1b2c3d
Creating pull request: autofix/api-test-2025-12-30T10-30-45 → main
✅ Pull request created successfully:
PR #42: https://github.com/yourorg/yourrepo/pull/42
Branch: autofix/api-test-2025-12-30T10-30-45 → main
Status: open (draft)

AI fix suggestion generation complete

Additional Metrics:
- Average passed test time: 1.23s
- Slowest test took: 1.23s

Test Failures:
--- Failure #1 ---
Test: API test
Category: NetworkError
Error: Connection timeout
Stack Trace:
at Connection.connect (/src/api/connection.ts:45:7)
`

$3

After each test run, a self-contained HTML report is automatically generated:

`plaintext
📊 Generating self-contained test health report...
✓ Loaded testSummary.json (25 tests)
✓ Loaded testFailures.json (2 failures)
✓ Loaded HTML template
✓ Loaded CSS
✓ Loaded JavaScript
✓ Generated standalone report: E:\project\test-results\test-health-report.html

╔══════════════════════════════════════════════════════════════╗
║ 📊 Test Health Report ║
╠══════════════════════════════════════════════════════════════╣
║ Open in browser: file:///E:/project/test-results/test-health-report.html
║ ║
║ Or run: npx playwright show-report (if using Playwright) ║
╚══════════════════════════════════════════════════════════════╝
`

The HTML report includes:
- 📈 Interactive Charts: Test results overview, failure categories, duration analysis
- 📋 Test Details Grid: Searchable/filterable list of all tests with status badges
- 🔍 Failed Tests Section: Detailed error messages with AI fix suggestions
- ⚠️ Flaky Tests Analysis: Pattern detection and stability recommendations
- 🐌 Slowest Tests: Performance analysis and optimization targets
- 📎 Artifact Links: Screenshots, videos, and trace files for failed tests

---

❓ Frequently Asked Questions (FAQs)

$3

$3

$3

$3

$3

$3

$3

$3

---

📚 Documentation

$3

#### Quick Links

- Quick Start Guide - Get started in 5 minutes
- Environment Configuration - Complete setup guide with sample configurations
- Provider Documentation - Detailed provider documentation and usage
- Architecture & Design - System architecture and design decisions
- Implementation Details - Technical implementation overview
- API Reference - Complete API documentation
- Troubleshooting Guide - Common issues and solutions

#### Examples

- Environment Config Examples - Pre-configured .env files for different stacks
- Test Examples - Sample test files demonstrating reporter usage
- Workflow Examples - Code examples for common workflows

> 💡 New here? Start with the Quick Start Guide and ENV_CONFIG_GUIDE

---

🤝 Contributing

We welcome contributions! Here's how you can help:

1. Fork the repository
2. Create your feature branch: git checkout -b feature/amazing-feature
3. Make your changes and commit: git commit -m 'Add amazing feature'
4. Push to your fork: git push origin feature/amazing-feature`
5. Open a Pull Request

Please ensure your PR:

- Follows the existing code style
- Includes appropriate tests
- Updates documentation as needed
- Describes the changes made

$3

See docs/PROVIDERS.md for instructions on adding new provider implementations.

---

🌟 Why Choose This Reporter?

✅ Provider Independence - Not locked into any single service
✅ Enterprise Ready - Azure integration, managed identity, MySQL support
✅ Cost Optimized - Choose the most cost-effective AI provider
✅ Flexible - Use only the features you need
✅ Extensible - Easy to add new providers
✅ Type Safe - Full TypeScript support
✅ Production Tested - Battle-tested in real-world scenarios
✅ Well Documented - Comprehensive docs and examples
✅ Active Development - Regular updates and improvements
✅ Open Source - MIT licensed, community-driven

---

🎯 Use Cases

$3

- Automatic bug creation in Azure DevOps for test failures
- Store historical test data in MySQL for trend analysis
- Email notifications to QA team
- Azure OpenAI for fix suggestions with enterprise security

$3

- GitHub Issues for bug tracking
- SQLite for lightweight data storage
- GitHub PRs for automated fixes
- OpenAI or Anthropic for AI suggestions

$3

- Slack notifications for instant alerts
- Jira integration for sprint planning
- Google AI (Gemini) for cost-effective analysis
- Quick iteration with auto-healing

---

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🙏 Acknowledgments

- Built with ❤️ for the Playwright community
- Inspired by the need for better test reporting and automatic debugging in CI/CD pipelines
- Multi-provider AI support: Azure OpenAI, OpenAI, Anthropic, Google AI, Mistral AI
- Thanks to all contributors who help make this reporter better

---

📞 Support

- 📖 Documentation
- 🐛 Issue Tracker
- 💬 Discussions
- 📧 Email Support
- 🐦 Twitter

---

Made with ❤️ by Deepak Kamboj for the Playwright community

⭐ Star us on GitHub if you find this useful!