JMeter test orchestrator - config-driven JMX modification and execution
npm install sonar-forgeJMeter test orchestrator - YAML-driven JMX modification and execution.
Inspired by Blazemeter Taurus, Sonar-Forge has been built as a simplified and streamlined alternative to work within Sonar's performance testing framework and ecosystem. It enables clean, version-controlled test and tool configuration specific to each test without needting to modify JMX files and Jmeter properties, or embedding complex properties and variables within each script.
Sonar-Forge separates test configuration from test scripts by:
- Reading existing JMX files to generate an initial YAML configuration specifying test and thread properties/configuration
- Users are able to extend the configuration, or make several versions (e.g. smoke test, peak test, soak test, etc.) by updating the initial configuration without needing to modify the JMX itself.
- Executing tests with automatic result collection and error logging
- Refreshing the YAML configuration file when the JMX gets updated (e.g. new thread groups get added, etc.)
``bash`
npm install -g sonar-forge
`bash`
sonar-forge init my-test.jmx
This creates a project structure:
``
my-test/
├── config/
│ └── my-test.yaml # Test configuration
├── scripts/
│ └── my-test.jmx # Original JMX file
├── data/ # Test data files
├── logs/ # JMeter logs
└── results/ # Test results
Edit config/my-test.yaml:
`yaml
test_script: ../scripts/my-test.jmx
output:
results_dir: ../results
logs_dir: ../logs
jmeter:
properties:
jmeter.save.saveservice.output_format: csv
jmeter.save.saveservice.timestamp_format: ms
# ... additional JMeter properties
error_log:
enabled: true
filename: errors.xml
thread_groups:
- name: Load Test
enabled: true
threads: 50
rampup: 10
iterations: -1
duration: 300
lifecycle:
EnableLifeCycleController: true
GenerateParentSample: true
UseFixedPacing: true
PacingInterval: 5000
`
`bash`
cd my-test
sonar-forge run config/my-test.yaml
Results are saved with timestamps:
- results/results_2025-01-15_14-30-00.jtl - CSV format with all metricsresults/errors_2025-01-15_14-30-00.xml
- - XML format with full error details
`bash`
sonar-forge init my-test.jmx
sonar-forge init my-test.jmx -o custom-dir
`bash`
sonar-forge validate config/my-test.yaml
`bash`
sonar-forge build config/my-test.yaml
sonar-forge build config/my-test.yaml --dry-run
sonar-forge build config/my-test.yaml --run-id "test-001" # Set unique run ID
`bash`
sonar-forge run config/my-test.yaml
sonar-forge run config/my-test.yaml -v # Verbose output
sonar-forge run config/my-test.yaml --build-only # Build only, don't run
sonar-forge run config/my-test.yaml --keep-jmx # Keep effective JMX after run
sonar-forge run config/my-test.yaml --run-id "test-001" # Set unique run ID
`bash
cd my-test
Merge mode: Preserves your configuration values, adds new thread groups from JMX.
Replace mode: Completely replaces configuration with current JMX state.
--all flag: Processes all YAML config files in the
config/ directory, showing a summary of successes and failures.Configuration
$3
Control JMeter's console and log file verbosity:
`yaml
JMeter logging level (DEBUG, INFO, WARN, ERROR)
log_level: WARN
`Available levels (from most to least verbose):
-
DEBUG - Detailed debugging information
- INFO - General informational messages (default if not specified)
- WARN - Warning messages only
- ERROR - Error messages only$3
`yaml
thread_groups:
- name: Thread Group Name # Must match JMX testname
enabled: true # Enable/disable thread group
threads: 50 # Number of threads
rampup: 10 # Ramp-up period in seconds
iterations: -1 # Number of iterations (-1 = infinite)
duration: 300 # Thread lifetime in seconds (null to disable)
`Execution modes:
- Duration only: Set duration, leave iterations at -1 (runs until time limit)
- Iterations only: Set iterations, set duration to null (runs until iteration count)
- Both: Stops when either limit is reached
- Infinite: iterations: -1, duration: null (runs forever)
$3
For tests using Sonar's LifecycleController:
`yaml
thread_groups:
- name: My Test
# ... thread configuration
lifecycle:
EnableLifeCycleController: true
GenerateParentSample: true
TroubleshootingMode: false
UseFixedPacing: true
PacingInterval: 5000
PacingRandomizationPct: 5
`$3
The
run_id provides a unique identifier for test runs that is automatically applied to all backend listeners. This enables tracking and correlating metrics across multiple backend systems.Three-tier precedence (highest to lowest):
1. CLI flag:
--run-id "my-run-id"
2. YAML config: run_id: "my-run-id"
3. Auto-generated: UUID automatically generated if not specifiedExamples:
`bash
CLI flag (highest precedence)
sonar-forge run config/my-test.yaml --run-id "production-2025-01-15"YAML config
cat config/my-test.yaml
run_id: "test-run-001"Auto-generated (no configuration needed)
Automatically generates UUID like "550e8400-e29b-41d4-a716-446655440000"
`The run_id is automatically injected into the
runId property of all enabled backend listeners, overriding any values configured in the YAML backend listener properties.$3
Sonar-Forge automatically detects and configures backend listeners for real-time test result reporting.
#### Supported Listeners
TimescaleDB Listener (recommended for Sonar framework):
`yaml
backend_listeners:
- name: Sonar - TimescaleDB Listener
type: timescaledb
enabled: true properties:
dbHost: timescaledb.example.com
dbPort: 6432
dbName: tsdb
dbUser: username
dbPassword: password
runId: test-run
reportLabel: ""
batchSize: 5000
flushInterval: 4000
poolSize: 20
errorThreshold: 5
sampleTableName: performance_metrics_full
threadTableName: performance_threads_full
recordSubSamples: true
saveResponseBodyOfFailures: true
responseBodyLength: 2000
`InfluxDB v2 Listener:
`yaml
backend_listeners:
- name: InfluxDBv2 Backend Listener
type: influxdb2
enabled: false properties:
testName: Test
nodeName: Test-Node
runId: test-run
influxDBURL: "http://localhost:8086/"
influxDBToken: "your-token-here"
influxDBOrganization: performance_testing
influxDBBucket: jmeter
influxDBFlushInterval: 4000
influxDBMaxBatchSize: 2000
influxDBThresholdError: 5
samplersList: .*
useRegexForSamplerList: true
recordSubSamples: true
saveResponseBodyOfFailures: false
responseBodyLength: 2000
`Other Listeners:
For other backend listener types, only name, classname, and enabled status are captured:
`yaml
backend_listeners:
- name: Custom Backend Listener
type: other
classname: com.example.CustomListener
enabled: false
`Warning: If no TimescaleDB or InfluxDB2 listeners are detected, a warning comment appears in the YAML:
`yaml
WARNING: No real-time reporting listeners (TimescaleDB/InfluxDB2) detected in this test
backend_listeners: []
`$3
Configure JTL output format and other JMeter settings:
`yaml
jmeter:
properties:
jmeter.save.saveservice.output_format: csv
jmeter.save.saveservice.time: true
jmeter.save.saveservice.latency: true
jmeter.save.saveservice.bytes: true
# ... additional properties
`$3
Capture full request/response details for failed samples:
`yaml
error_log:
enabled: true
filename: errors.xml
`Generates timestamped XML files with complete error diagnostics.
$3
`yaml
output:
results_dir: ../results # JTL results directory
logs_dir: ../logs # JMeter log directory
`Workflow
$3
After modifying your JMX file in
scripts/, sync your configuration using the refresh command.#### Single Configuration File
If you have one config file, the refresh command auto-detects it:
`bash
cd my-test
sonar-forge refresh # Merge: preserves your config values
sonar-forge refresh --replace # Replace: overwrites with JMX values
sonar-forge refresh -v # Verbose output shows details
`#### Multiple Configuration Files
If you maintain multiple configurations (e.g., smoke, load, soak tests), you have several options:
Refresh a specific config:
`bash
cd my-test
sonar-forge refresh smoke-test.yaml
sonar-forge refresh load-test.yaml --replace
`Refresh all configs at once:
`bash
cd my-test
sonar-forge refresh --all # Merge mode for all
sonar-forge refresh --all --replace # Replace mode for all
sonar-forge refresh --all -v # Verbose for all
`The
--all flag processes all YAML files in config/ and shows a summary:
`
Found 3 config file(s) to refresh
...
Summary
============================================================
Mode: merge
Total files: 3
✓ Successful: 2
✗ Failed: 1
`#### What Refresh Detects
Merge mode (default):
- ✅ Adds new thread groups from JMX
- ✅ Removes thread groups deleted from JMX
- ✅ Preserves your custom configuration values (threads, duration, etc.)
- ✅ Keeps your lifecycle controller settings
- ✅ Maintains all YAML comments and formatting
Replace mode (
--replace):
- Completely replaces thread group configuration with current JMX state
- Useful when you want to reset config to match JMX exactly
- ⚠️ Overwrites your custom settings#### Common Scenarios
Scenario 1: Added new thread group to JMX
`bash
Edit scripts/my-test.jmx, add new thread group "API Tests"
sonar-forge refresh # Auto-adds "API Tests" to config
Edit config/my-test.yaml to customize the new thread group
`Scenario 2: Multiple test profiles
`bash
You have: smoke.yaml, load.yaml, soak.yaml
All reference scripts/my-test.jmx
JMX was updated with new thread group
sonar-forge refresh --all # Updates all 3 configs
Each config preserves its custom settings (threads, duration, etc.)
`Scenario 3: Reset config to match JMX
`bash
Your config has experimental changes, want to reset
sonar-forge refresh --replace # Overwrites config with JMX defaults
`$3
Commit these files:
-
config/*.yaml - Test configuration (source of truth)
- scripts/*.jmx - Test scriptsAdd to .gitignore:
-
results/ - Test results
- logs/ - JMeter logs
- data/ - Test data files (unless needed)
- *_effective.jmx` - Generated effective JMX files