Sonar-Forge

JMeter test orchestrator - YAML-driven JMX modification and execution.

Inspired by Blazemeter Taurus, Sonar-Forge has been built as a simplified and streamlined alternative to work within Sonar's performance testing framework and ecosystem. It enables clean, version-controlled test and tool configuration specific to each test without needting to modify JMX files and Jmeter properties, or embedding complex properties and variables within each script.

Overview

Sonar-Forge separates test configuration from test scripts by:
- Reading existing JMX files to generate an initial YAML configuration specifying test and thread properties/configuration
- Users are able to extend the configuration, or make several versions (e.g. smoke test, peak test, soak test, etc.) by updating the initial configuration without needing to modify the JMX itself.
- Executing tests with automatic result collection and error logging
- Refreshing the YAML configuration file when the JMX gets updated (e.g. new thread groups get added, etc.)

Installation

``bash
npm install -g sonar-forge
`

Quick Start

$3

`bash
sonar-forge init my-test.jmx
`

This creates a project structure:
`
my-test/
├── config/
│ └── my-test.yaml # Test configuration
├── scripts/
│ └── my-test.jmx # Original JMX file
├── data/ # Test data files
├── logs/ # JMeter logs
└── results/ # Test results
`

$3

Edit config/my-test.yaml:

`yaml
test_script: ../scripts/my-test.jmx

output:
results_dir: ../results
logs_dir: ../logs

jmeter:
properties:
jmeter.save.saveservice.output_format: csv
jmeter.save.saveservice.timestamp_format: ms
# ... additional JMeter properties

error_log:
enabled: true
filename: errors.xml

thread_groups:
- name: Load Test
enabled: true
threads: 50
rampup: 10
iterations: -1
duration: 300

lifecycle:
EnableLifeCycleController: true
GenerateParentSample: true
UseFixedPacing: true
PacingInterval: 5000
`

$3

`bash
cd my-test
sonar-forge run config/my-test.yaml
`

Results are saved with timestamps:
- results/results_2025-01-15_14-30-00.jtl - CSV format with all metrics
- results/errors_2025-01-15_14-30-00.xml - XML format with full error details

Commands

$3

Generate project from existing JMX file.

`bash
sonar-forge init my-test.jmx
sonar-forge init my-test.jmx -o custom-dir
`

$3

Validate configuration file.

`bash
sonar-forge validate config/my-test.yaml
`

$3

Build effective JMX without running test.

`bash
sonar-forge build config/my-test.yaml
sonar-forge build config/my-test.yaml --dry-run
sonar-forge build config/my-test.yaml --run-id "test-001" # Set unique run ID
`

$3

Build and execute JMeter test.

`bash
sonar-forge run config/my-test.yaml
sonar-forge run config/my-test.yaml -v # Verbose output
sonar-forge run config/my-test.yaml --build-only # Build only, don't run
sonar-forge run config/my-test.yaml --keep-jmx # Keep effective JMX after run
sonar-forge run config/my-test.yaml --run-id "test-001" # Set unique run ID
`

$3

Update configuration from modified JMX file.

`bash
cd my-test

Single config file (auto-detect)

sonar-forge refresh # Merge mode (default) - preserves config
sonar-forge refresh --replace # Replace mode - overwrites with JMX data

Multiple config files (specify which one)

sonar-forge refresh test1.yaml
sonar-forge refresh test2.yaml --replace

Process all config files

sonar-forge refresh --all # Merge mode for all configs
sonar-forge refresh --all --replace # Replace mode for all configs

Verbose output

sonar-forge refresh -v
`

Merge mode: Preserves your configuration values, adds new thread groups from JMX.

Replace mode: Completely replaces configuration with current JMX state.

--all flag: Processes all YAML config files in the config/ directory, showing a summary of successes and failures.

Configuration

$3

Control JMeter's console and log file verbosity:

`yaml


JMeter logging level (DEBUG, INFO, WARN, ERROR)


log_level: WARN
`

Available levels (from most to least verbose):
- DEBUG - Detailed debugging information
- INFO - General informational messages (default if not specified)
- WARN - Warning messages only
- ERROR - Error messages only

$3

`yaml
thread_groups:
- name: Thread Group Name # Must match JMX testname
enabled: true # Enable/disable thread group
threads: 50 # Number of threads
rampup: 10 # Ramp-up period in seconds
iterations: -1 # Number of iterations (-1 = infinite)
duration: 300 # Thread lifetime in seconds (null to disable)
`

Execution modes:
- Duration only: Set duration, leave iterations at -1 (runs until time limit)
- Iterations only: Set iterations, set duration to null (runs until iteration count)
- Both: Stops when either limit is reached
- Infinite: iterations: -1, duration: null (runs forever)

$3

For tests using Sonar's LifecycleController:

`yaml
thread_groups:
- name: My Test
# ... thread configuration
lifecycle:
EnableLifeCycleController: true
GenerateParentSample: true
TroubleshootingMode: false
UseFixedPacing: true
PacingInterval: 5000
PacingRandomizationPct: 5
`

$3

The run_id provides a unique identifier for test runs that is automatically applied to all backend listeners. This enables tracking and correlating metrics across multiple backend systems.

Three-tier precedence (highest to lowest):
1. CLI flag: --run-id "my-run-id"
2. YAML config: run_id: "my-run-id"
3. Auto-generated: UUID automatically generated if not specified

Examples:

`bash


CLI flag (highest precedence)

sonar-forge run config/my-test.yaml --run-id "production-2025-01-15"

YAML config

cat config/my-test.yaml
run_id: "test-run-001"

Auto-generated (no configuration needed)

Automatically generates UUID like "550e8400-e29b-41d4-a716-446655440000"

`

The run_id is automatically injected into the runId property of all enabled backend listeners, overriding any values configured in the YAML backend listener properties.

$3

Sonar-Forge automatically detects and configures backend listeners for real-time test result reporting.

#### Supported Listeners

TimescaleDB Listener (recommended for Sonar framework):
`yaml
backend_listeners:
- name: Sonar - TimescaleDB Listener
type: timescaledb
enabled: true

properties:
dbHost: timescaledb.example.com
dbPort: 6432
dbName: tsdb
dbUser: username
dbPassword: password
runId: test-run
reportLabel: ""
batchSize: 5000
flushInterval: 4000
poolSize: 20
errorThreshold: 5
sampleTableName: performance_metrics_full
threadTableName: performance_threads_full
recordSubSamples: true
saveResponseBodyOfFailures: true
responseBodyLength: 2000
`

InfluxDB v2 Listener:
`yaml
backend_listeners:
- name: InfluxDBv2 Backend Listener
type: influxdb2
enabled: false

properties:
testName: Test
nodeName: Test-Node
runId: test-run
influxDBURL: "http://localhost:8086/"
influxDBToken: "your-token-here"
influxDBOrganization: performance_testing
influxDBBucket: jmeter
influxDBFlushInterval: 4000
influxDBMaxBatchSize: 2000
influxDBThresholdError: 5
samplersList: .*
useRegexForSamplerList: true
recordSubSamples: true
saveResponseBodyOfFailures: false
responseBodyLength: 2000
`

Other Listeners:
For other backend listener types, only name, classname, and enabled status are captured:
`yaml
backend_listeners:
- name: Custom Backend Listener
type: other
classname: com.example.CustomListener
enabled: false
`

Warning: If no TimescaleDB or InfluxDB2 listeners are detected, a warning comment appears in the YAML:
`yaml


WARNING: No real-time reporting listeners (TimescaleDB/InfluxDB2) detected in this test


backend_listeners: []
`

$3

Configure JTL output format and other JMeter settings:

`yaml
jmeter:
properties:
jmeter.save.saveservice.output_format: csv
jmeter.save.saveservice.time: true
jmeter.save.saveservice.latency: true
jmeter.save.saveservice.bytes: true
# ... additional properties
`

$3

Capture full request/response details for failed samples:

`yaml
error_log:
enabled: true
filename: errors.xml
`

Generates timestamped XML files with complete error diagnostics.

$3

`yaml
output:
results_dir: ../results # JTL results directory
logs_dir: ../logs # JMeter log directory
`

Workflow

$3

After modifying your JMX file in scripts/, sync your configuration using the refresh command.

#### Single Configuration File

If you have one config file, the refresh command auto-detects it:

`bash
cd my-test
sonar-forge refresh # Merge: preserves your config values
sonar-forge refresh --replace # Replace: overwrites with JMX values
sonar-forge refresh -v # Verbose output shows details
`

#### Multiple Configuration Files

If you maintain multiple configurations (e.g., smoke, load, soak tests), you have several options:

Refresh a specific config:
`bash
cd my-test
sonar-forge refresh smoke-test.yaml
sonar-forge refresh load-test.yaml --replace
`

Refresh all configs at once:
`bash
cd my-test
sonar-forge refresh --all # Merge mode for all
sonar-forge refresh --all --replace # Replace mode for all
sonar-forge refresh --all -v # Verbose for all
`

The --all flag processes all YAML files in config/ and shows a summary:
`
Found 3 config file(s) to refresh
...
Summary
============================================================
Mode: merge
Total files: 3
✓ Successful: 2
✗ Failed: 1
`

#### What Refresh Detects

Merge mode (default):
- ✅ Adds new thread groups from JMX
- ✅ Removes thread groups deleted from JMX
- ✅ Preserves your custom configuration values (threads, duration, etc.)
- ✅ Keeps your lifecycle controller settings
- ✅ Maintains all YAML comments and formatting

Replace mode (--replace):
- Completely replaces thread group configuration with current JMX state
- Useful when you want to reset config to match JMX exactly
- ⚠️ Overwrites your custom settings

#### Common Scenarios

Scenario 1: Added new thread group to JMX
`bash


Edit scripts/my-test.jmx, add new thread group "API Tests"

sonar-forge refresh # Auto-adds "API Tests" to config

Edit config/my-test.yaml to customize the new thread group

`

Scenario 2: Multiple test profiles
`bash


You have: smoke.yaml, load.yaml, soak.yaml

All reference scripts/my-test.jmx

JMX was updated with new thread group

sonar-forge refresh --all # Updates all 3 configs

Each config preserves its custom settings (threads, duration, etc.)

`

Scenario 3: Reset config to match JMX
`bash


Your config has experimental changes, want to reset


sonar-forge refresh --replace # Overwrites config with JMX defaults
`

$3

Commit these files:
- config/*.yaml - Test configuration (source of truth)
- scripts/*.jmx - Test scripts

Add to .gitignore:
- results/ - Test results
- logs/ - JMeter logs
- data/ - Test data files (unless needed)
- *_effective.jmx` - Generated effective JMX files

Upcoming Features

- Disable section that allow specific elements at the Test Plan level to be explictly enabled/disabled
- Add jmeter properties to support extending backend listener queue depth
- Improved test execution UI (similar to Taurus, but customised for Sonar framework)