A mutation testing tool for Ethereum smart contracts
npm install @morenabarboni/sumoSuMo was designed to run mutation testing on Solidity projects in a NodeJS environment. It can run test using Hardhat, Brownie and Forge, hybrid test suites, and custom test scripts.
To install sumo run ``npm install @morenabarboni/sumo`
sumo-config.js` is automatically generated when SuMo is installed.Here's a simple example of
`sumo-config.js`:`
module.exports = {
buildDir: "auto", //build directory of the SUT (auto detect)
contractsDir: "auto", //contract directory of the SUT (auto detect)
testDir: "auto", //test directory of the SUT (auto detect)
skipContracts: ["interfaces", "mock", "test"], // Relative paths from contractsDir
skipTests: [], // Relative paths from testsDir
testingFramework: "auto", //testing framework (auto detect)
minimalOperators: false, // use minimal mutation rules
randomSampling: false, //use random mutant sampling
randomMutants: 100, //if random sampling is enabled, generate 100 mutants max
testingTimeOutInSec: 500 //testing time-out for a mutant
}
$3
SuMo will try to automatically find your project directories based on standard naming conventions (e.g., /contracts and /test). These can be overriden in the sumo-config.js` file. Contracts and test files/folders to be ignored by SuMo can be specified as well.| Field | Description | Default Value |
| ------ | ------ | :----: |
|
`contractsDir`| relative path to the directory of the contracts to be mutated | `auto` |
| `testDir`| relative path to the directory of the tests to be evaluated | `auto` |
| `buildDir`| relative path to the directory of the compilation artifacts | `auto` |
| `skipContracts`| blacklist of relative paths to contract files (or folders) | `["interfaces", "mock", "test"]` |
| `skipTests`| blacklist of relative paths to test files (or folders) | `[]` |$3
These fields allow to customize the testing frameworks used by SuMo.
By default,
`testingFramework` is set to `auto`: SuMo will automatically select the testing framework(s) to be used based on the configuration files (e.g., `foundry.toml`) present in your workspace. If multiple configuration files are found, SuMo will try to run a hybrid testing process.| Field | Description | Available Options | Default Value |
| ------ | ------ | ------ | :----: |
|
`testingFramework`| the testing framework to be used for compiling and testing the smart contracts | `auto`, `brownie`, `forge`, `hardhat`, `custom` | `auto` |#### Brownie
When choosing
`brownie`:
* SuMo will rely on a local/global `brownie` installation;
* The smart contracts will be compiled with a minimal compile command (e.g., `brownie compile` );
* The smart contracts will be tested with a minimal test command and (optionally) by a list of test files to be executed (e.g., `brownie test ...testFiles --exitfirst`) .#### Forge
When choosing
`forge` :
* SuMo will rely on the global installation of `foundry`;
* The smart contracts will be compiled with a minimal compile command (e.g., `forge build`);
* The smart contracts will be tested with a minimal test command and (optionally) by a list of test files to be executed (e.g., `forge test ...testFiles --fail-fast`).
* Make sure that your `forge` installation is up-to-date to enable `--fail-fast`.#### Custom
If you set
`testingFramework` to `custom`, SuMo will invoke the `compile` and `test` script defined in your `package.json`. This allows you to customize both scripts and have more control over the testing process. For example, you can define the scripts as follows:
`
//package.json
scripts: {
compile: "hardhat compile",
test "hardhat test --bail && forge test --fail-fast"
}//sumo-config.js
project: {
buildDir: "artifacts",
}
Additionally, you must also explicitly define a buildDir` (matching your compile command) in your sumo-config.js.`
ā ļø Limitations of Custom Test Scripts:
* buildDir: must be explicitly specified it in the sumo-config.js
* skipTests: will be ignored. You have to specify them in your custom script.
$3
These fields allow you to further customize the mutation testing process:
| Field | Description | Default Value |
| ------ | ------ | :----: |
|
minimalOperators`| use minimal mutation rules | `false` |
| `randomSampling`| use Random Mutant Sampling | `false` |
| `randomMutants`| the maximum number of mutants to be tested (only if `randomSampling` is enabled) | `100` |
| `testingTimeOutInSec`| seconds after which a mutant is marked as timed-out during testing | `500` | CLI Usage
Selecting the Mutation Operators
Before starting the mutation process you can choose which mutation operators to use:| Command | Description | Usage | Example |
|---------------|------------------------------------|--------------------------|-------------------------------------|
|
list | Shows the enabled mutation operators. | npx/yarn sumo list | $ npx sumo list |
| enable | Enables one or more mutation operators. If no operator IDs are specified, all of them are enabled. | npx/yarn sumo enable [...ID] | $ npx sumo enable
$ npx sumo enable AOR BOR |
| disable | Disables one or more mutation operators. If no operator IDs are specified, all of them are disabled. | npx/yarn sumo disable [...ID] | $ npx sumo disable
$ npx sumo disable FVR |Viewing the available mutations
| Command | Description | Usage | Example |
|---------------|------------------------------------|--------------------------|-------------------------------------|
|
lookup | Generates the mutations and creates reports without starting mutation testing. | npx/yarn sumo lookup | $ npx sumo lookup |
| mutate | Generates the mutations and saves a copy of each .sol mutant to to ./sumo/mutants. | npx/yarn sumo mutate | $ npx sumo mutate |Running Mutation Testing
| Command | Description | Usage | Example |
|---------------|------------------------------------|--------------------------|-------------------------------------|
|
pretest | Runs the test suite on the original smart contracts to check if all tests pass and can be successfully evaluated. Pretest is automatically run when sumo test is executed. | npx/yarn sumo pretest | $ npx sumo pretest |
| test | Starts the mutation testing process. You can also choose a single mutant / an interval of mutants to be tested by sepcifying ` and (optionally) `.| npx/yarn sumo test | $ npx sumo test
$ npx sumo test mbc5e8f56 mbg5t86o6|
| restore | Restores the SUT files to a clean version. This should be executed if you suddenly interrupt the mutation process. Note that the restore command overwrites your codebase with the files stored in the `sumo/baseline` folder. If you need to restore the project files, make sure to do so before performing other operations as the baseline is automatically refreshed on subsequent preflight or test runs.| $ npx/yarn sumo restore | $ npx sumo restore|Viewing the results
SuMo automatically creates a `sumo\results` folder in the root directory of the project with the following reports:
* `mutations.json`: List of mutations in json format, synchronoysly updated during testing.
* `index.html`: A simple web display of the results (you can view this using VSCode extensions like `Live Server`). From here, you can also download a csv with the results.
* `\mutants`: Folder with mutated `.sol` source files (only if generated with `sumo mutate`)Mutation Operators š¾
SuMo includes the following Traditional and Solidity-specific operators. Note that not all mutation operators are enabled by default.
Traditional Mutation Operators
| Operator | Name | Mutation Example | Enabled by Default | Minimal Available |
| ------ | ------ | ------ | ------ | :----: |
| ACM| Argument Change of overloaded Method call |
`overloadedFunc(a,b);` → `overloadedFunc(a,b,c);` | Y | N |
| AOR | Assignment Operator Replacement | `+= ` → `=` | Y | N |
| BCRD | Break and Continue Replacement
and Deletion | `break` →
`continue` → `break` | Y | N |
| BLR | Boolean Literal Replacement | `true` → `false` | Y | N |
| BOR | Binary Operator Replacement | `+` → `-`
`<` → `>=` | Y | Y |
| CBD | Catch Block Deletion | `catch{}` → ` ` | Y | N |
| CSC | Conditional Statement Change | `if(condition)` → `if(false)`
`else{}` → ` ` | Y | N |
| ER | Enum Replacemet | `enum.member1` → `enum.member2` | Y | Y |
| ECS | Explicit Conversion to Smaller type | `uint256` → `uint8` | Y | N |
| FCD | Function Call Deletion | `foo()` → ` `| Y | N |
| HLR | Hexadecimal Literal Replacement | `hex\"01\"` → `hex\"random\"`| Y | N |
| ILR | Integer Literal Replacement | `1` → `0` | Y | N |
| LCS | Loop Statement Change | `while(condition)` → `while(false)` | Y | N |
| OLFD | Overloaded Function Deletion | `function overloadedF(){}` → ` ` | Y | N |
| ORFD | Overridden Function Deletion | `function f() override {}` → ` ` | Y | N |
| SKR | Super Keyword Replacement | `x = getData()` → `x = super.getData()` | Y | N |
| SLR | String Literal Replacement | `"string"` → `""` | Y | N |
| UORD | Unary Operator Replacement and Deletion | `++` → `--`
`!` → ` ` | Y | Y |
Solidity Mutation Operators
| Operator | Name | Mutation Example |Enabled by Default | Minimal version available |
| ------ | ------ | ------ | ------ | :----: |
| AVR | Address Value Replacement | `0x67ED2e5dD3d0...` → ` address.this()`| Y | Y |
| CCD | Contract Constructor Deletion | `constructor(){}` → ` ` | Y | N |
| DLR | Data Location Keyword Replacement | `memory` → `storage` | N | N |
| DOD | Delete Operator Deletion | `delete` → | Y | N |
| ETR | Ether Transfer function Replacement | `delegatecall()` → `call()` | Y | Y |
| EED | Event Emission Deletion | `emit Deposit(...)` → `/emit Deposit(...)/` | Y | N |
| EHD | Exception Handling Deletion | `require(...)` → `/require(...)/` | Y | N |
| FVR | Function Visibility Replacement | `function f() public` → `function f() private` | N | Y |
| GVR | Global Variable Replacement | `msg.value()` → `tx.gasprice()` | Y | Y |
| MCR | Mathematical and Cryptographic
function Replacement | `addmod` → `mulmod`
`keccak256` → `sha256` | Y | Y |
| MOD | Modifier Deletion | `function f() onlyOwner` → `function f()` | Y | Y |
| MOI | Modifier Insertion | `function f()` → `function f() onlyOwner` | N | Y |
| OMD | Overridden Modifier Deletion | `modifier m() override {}` → ` ` | Y | N |
| PKD | Payable Keyword Deletion | `function f() payable` → `function f()` | Y | N |
| RSD | Return Statement Deletion | `return amount;` → `//return amount;` | Y | N |
| RVS | Return Values Swap | `return (1, "msg", 100);` → `return (100, "msg", 1);` | Y | Y |
| SCD | Selfdestruct Call Deletion | `selfdestruct();` → `//selfdestruct();` | Y | N |
| SFR | SafeMath Function Replacement | `SafeMath.add` → `SafeMath.sub` | Y | Y |
| SCEC | Switch Call Expression Casting | `Contract c = Contract(0x86C9...);` → `Contract c = Contract(0x67ED...); ` | Y | N |
| TOR | Transaction Origin Replacement | `msg.sender` → `tx.origin` | Y | N |
| VUR | Variable Unit Replacement | `wei` → `ether`
`minutes` → `hours` | Y | Y |
| VVR | Variable Visibility Replacement | `uint private data;` → `uint public data;` | N | Y |
Minimal Mutation Rules
Some mutation operators foresee a minimal version:
* The extended operators generate a more comprehensive set of mutants. These guarantee a more in-depth test adequacy assessment, but they can generate more than one replacement per target (e.g.,
`+` is mutated in both `-` and `*`), which can lead to longer execution times.
* The minimal operators define simplified rules that only inject one replacement per target (e.g., `+` is mutated in `-`), limiting the generation of subsumed mutants and speeding up the testing process.By default, SuMo employs the extended operators. However, you can enable the minimal rules in the
`sumo-config.js` file.Publications
To cite SuMo, please use the following:
`
@article{BARBONI2022111445,
title = {SuMo: A mutation testing approach and tool for the Ethereum blockchain},
journal = {Journal of Systems and Software},
volume = {193},
pages = {111445},
year = {2022},
issn = {0164-1212},
doi = {https://doi.org/10.1016/j.jss.2022.111445},
author = {Morena Barboni and Andrea Morichetta and Andrea Polini}
}