Benchmark

> This is an independently maintained fork of @dapplion/benchmark.
This repo now maintains it's own versioning as @chainsafe/benchmark and release schedule. It was forked from the base of @dapplion/benchmark@1.0.0 and new functionality was added.

Ensures that new code does not introduce performance regressions with CI. Tracks:

- Do PR against the base branch include a performance regression?
- Do new commits in the main branch include a performance regression?

This tooling provides both a easy to use runner for benchmarking and easy integrations to persist past benchmark data.

Quick start

Create a benchmark test file but use itBench instead of it

``ts
import {describe, itBench, setBenchOpts} from "@chainsafe/benchmark";

describe("Sum array benchmark", () => {
itBench("sum array with reduce", () => {
arr.reduce((total, curr) => total + curr, 0);
});
});
`

Then run the CLI.

`
benchmark 'test/perf/*/.perf.ts' --local
`

Inspect benchmark results in the terminal

`
Sum array benchmark
✔ sum array with reduce 826.0701 ops/s 1.210551 ms/op x0.993 578 runs 1.21 s
`

How does it work?

This tool is a CLI tool, example usage:

`
benchmark 'test/perf/*/.perf.ts' --s3
`

The above command will:

- Read benchmark history from the specified provider (AWS S3)
- Figure out the prev benchmark based on your option (defaults to latest commit in main branch)
- Run benchmark comparing with previous
- Collect benchmark data in-memory while streaming results
- Add result to benchmark history and persist them to the specified provider (AWS S3)
- If in CI, post a PR or commit comment with an expandable summary of the benchmark results comparision
- If a performance regression was detected, exit 1

$3

Below is a suggested Github action to run this tool with s3 history provider:

`yaml
name: Benchmark


Ensure a single benchmark is run at a time

# Run benchmark with custom tooling and stores the output to a file
- name: Run performance tests
run: yarn benchmark --s3
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# S3 credentials
S3_ACCESS_KEY: ${{ secrets.S3_ACCESS_KEY }}
S3_SECRET_KEY: ${{ secrets.S3_SECRET_KEY }}
S3_REGION: ${{ secrets.S3_REGION }}
S3_BUCKET: ${{ secrets.S3_BUCKET }}
S3_ENDPOINT: ${{ secrets.S3_ENDPOINT }}
# Key prefix to separate benchmark data from multiple repositories
S3_KEY_PREFIX: ${{ github.repository }}/${{ runner.os }}
`

$3

When working on optimizing a function you may want to know if your code is actually faster than the previous implementation.

To do that you can keep a benchmark history file locally and run the benchmark first against previous code

`
git checkout master
benchmark test/perf/func.perf.ts --local
`

- Runs benchmark without comparing with previous
- Writes single benchmark data to ./benchmark_data

Then measure performance with the new code

`
git checkout fix1
benchmark test/perf/func.perf.ts --local
`

- Reads single benchmark data from ./benchmark_data
- Run benchmark comparing with prev
- Does not write benchmark data

Config

$3

Provide the default branch of this repository to prevent fetching from Github

- type: string
- default:

$3

Choose what branches to persist benchmark data

- type: array
- default: default-branch

$3

Limit number of benchmarks persisted per branch

- type: number
- default: Infinity

$3

Ratio of new average time per run vs previos time per run to consider a failure. Set to 'Infinity' to disable it.

- type: number
- default: 2

$3

Compare new benchmark data against the latest available benchmark in this branch

- type: string
- default: default-branch

$3

Compare new benchmark data against the benchmark data associated with a specific commit

- type: string
- default:

$3

When persisting history, delete benchmark data associated with commits that are no longer in the current git history

- type: boolean
- default:

$3

Force persisting benchmark data in history

- type: boolean
- default:

$3

Exit cleanly even if a preformance regression was found

- type: boolean
- default:

$3

Persist benchmark history locally. May specify just a boolean to use a default path, or provide a path

- type: string
- default: ./benchmark_data

$3

Persist benchmark history in Github Actions cache. Requires Github authentication. May specify just a boolean to use a default cache key or provide a custom key

- type: string
- default: benchmark_data

$3

Persist benchmark history in an Amazon S3 bucket. Requires Github authentication

- type: string
- default:

Roadmap

$3

`
bench compare --from --to
``

- Retrieves benchmark history
- Don't run benchmark
- Print comparision of from benchmark with to benchmark

$3

- Github actions cache: Doesn't work due to Github's limitations
- Commited in the repository (gh-pages, or data branch)