The Guardian's AB Testing Library

Getting Started

1. Install the package with your manager of choice. e.g.
pnpm add @guardian/ab-core
2. Initialise the AB Class in your project
3. Consume the API

> Note There is also a (P)React framework-specific implementation,
> @guardian/ab-react

What does this client-side A/B testing library do?

This library:

- Is initialised with an multivariate identifier (MVT ID)
- This can be set by the server for fast user bucketing, e.g. as a cookie
- Is built upon the A/B testing code
from frontend
- Has the ability to force variants for testing
- Has simple integration with Ophan, with impression and success methods built
in to the library
- Can be integrated into a Typescript or Javascript project

There’s some background to the
early requirements to the library and some documentation in Frontend,
there’s some
notes about the migration of A/B tests.

How it works

1. Define the AB test: Each AB test and their variants are defined in code
with configuration such as audience size & offset and impression & success
listeners etc
2. Initialise the library: The AB Test library is initialised with
configuration values such as a user's MVT ID, an array of the above defined
A/B tests etc
3. Use the AB Test API: The intialisation returns an API that can be used to
check if the current user is in a variant of a test along with a variety of
other API methods

$3

``ts
import { AB } from '@guardian/ab-core';
import type { AbTestConfig } from '@guardian/ab-core';

const config: AbTestConfig = {
mvtId: 999_999,
pageIsSensitive: false,
abTestSwitches: {},
arrayOfTestObjects: [],
};

const abTests = new AB(config);

// Must be performed in the platform after initialisation to ensure tracking defined in ABTests is setup
// [tests] being an array of runnable ab tests
abTest.registerCompleteEvents([tests]);
abTest.registerImpressionEvents([tests]);
abTest.trackABTests([tests]);

// The API then provides access to the utility methods for use within modules
abTests.runnableTest(test);
abTests.firstRunnableTest([tests]);
abTests.isUserInVariant(testId, variantId);
`

$3

| Config | Type | Example |
| ------------------- | ------------------------------------------------- | --------------------------------------------------------------------------------- |
| mvtMaxValue | number | 10_000 |
| mvtCookieId | number | getCookie('mvt_id') |
| pageIsSensitive | boolean | guardian.config.page.isSensitive |
| abTestSwitches | Record | {'TestOne': true} |
| forcedTestVariants | Optional: { [key: string]: { variant: string } }; | { TestOne: { variant: 'myCoolVariant' }, TestTwo: { variant: 'myCoolVariant' } }; |
| forcedTestException | Optional: ABTest['id'] | |
| arrayOfTestObjects | ABTest[] | |
| ServerSideTets | ServerSideTests | |
| errorReporter | ErrorReporterFunc | |
| ophanRecord | OphanRecordFunction | |

$3

`ts
type CoreAPI = {
allRunnableTests: (
tests: ReadonlyArray,
) => ReadonlyArray> | [];
runnableTest: (
test: ABTest,
) => Runnable | null;
firstRunnableTest: (tests: ReadonlyArray) => Runnable | null;
isUserInVariant: (testId: ABTest['id'], variantId?: Variant['id']) => boolean;
};
`

| API Method | Params | Returns |
| ----------------- | ------------------------------ | ------------------------------------------------------------------------------------------ |
| allRunnableTests | Array of ab tests | Array of Runnable tests or empty array |
| runnableTest | A single AB test | A runnable ab test object with variantToRun property containing the variant to run or null |
| firstRunnableTest | Array of AB Tests | A runnable ab test object or null |
| isUserInVariant | A AB test ID, and a Variant ID | A boolean |

$3

Within your platforms, you should define the test this way. If the test needs to
run across platforms, then the test definition needs to be the same (as well as
the initialisation config).

`ts
import { ABTest } from '@guardian/ab-core';

export const abTestTest: ABTest = {
id: 'AbTestTest', // This ID must match the Server Side AB Test
start: '2020-05-20',
expiry: '2020-12-01', // Remember that the server side test expiry can be different
author: 'anemailaddress@theguardian.com',
description: 'This Test'
audience: 0.0001, // 0.01% (1 is 100%)
audienceOffset: 0.5, // 50% (1 is 100%). Prevent overlapping with other tests.
successMeasure: 'It works',
audienceCriteria: 'Everyone',
idealOutcome: 'It works',
showForSensitive: true, // Should this A/B test run on sensitive articles?
canRun: () => true, // Check for things like user or page sections
variants: [
{
id: 'control',
test: (): void => {}, // You can define what you want your variant to do in here or use the isUserInVariant API
impression: (impression) => {
// This will be immediate.
// You could also use eventListeners as below
// Make sure abTest.registerCompleteEvents([tests]); and abTest.registerImpressionEvents([tests]); have been called
impression();
},
success: (success) => {
// https://developer.mozilla.org/en-US/docs/Web/Guide/Events/Creating_and_triggering_events
window.addEventListener('guAbTestEvent', (e) => {
const detail = e?.detail;
if (
detail?.abTest === 'abTestTest' &&
detail?.variant === 'control' &&
detail?.event === 'success') {
success();
}
});

},
},
{
id: 'variant',
test: (): void => {},
impression: (impression) => {
impression();
},
success: (success) => {
//...
},
},
],
};

// If you're using event listeners for the impression and success events you can call them with CustomEvents
// Say a user clicked something
window.dispatchEvent(new CustomEvent('guAbTestEvent', {
detail: {
abTest: 'abTestTest',
variant: 'control'
event: 'success'
}
}))
`

$3

The initialisation values are populated on these platforms like so:

| Config | Note |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| mvtMaxValue | MVT % is calculated from 0 to mvtMaxValue |
| mvtCookieId | The user's MVT ID to calculate what tests and variants they fall into |
| pageIsSensitive | Sensitive pages must have explicit settings in AB tests |
| abTestSwitches | An object containing all of the boolean values of abTestSwitches, in Frontend from page.config.switches.abTests |
| forcedTestVariants | In Frontend this might be set by the URL override, but otherwise can be used to force a user into a test and variant at init time |
| forcedTestException | Can be used to force a user out of a test (in Frontend, again with url override) |
| arrayOfTestObjects | Pass all tests definitions into the config |
| ServerSideTets | ServerSideTests are accessed via client-side config in Frontend and DCR |
| errorReporter | Pass an error reporter, probably Sentry |
| ophanRecord | Probably Ophan's 'record' function |

Frontend: Difference and Integration with DCR

$3

- There is currently a requirement to copy and paste the AB test definitions
between the two platforms. Each platform has an experiments folder
(Frontend,
DCR)
and the test definition and structure of those folders should match. The
difference will be where to import - in Frontend in
ab-test.ts
and in DCR in
ab-tests.ts.
- For Frontend and DCR, you will need to have a switch as you would usually do
in Frontend. This will be passed through to DCR in the backend and be
accessible to the client-side code.

$3

- There is no local storage functionality in this library, unlike previously
where you could store the A/B test in Frontend. To persist an opted-in test,
the MVT cookie will need to be set to the correct value.
- _Forced Tests_ - used by the url-opt-in mechanism ignores canRun on both
the test _and_ the variant, so it will always run when forced. Previously on
Frontend it still listened to the variant's canRun.
- Handles only concurrent tests, this library does not concern itself with epics
or banner tests. There is no mechanism for A/B tests to interact or wait for
one another outside of audience size and offsets.
- The public API is reduced to only what was used in Frontend
- Some public methods have been renamed like isUserInVariant (this does make
it difficult to copy and paste an implementation between Frontend and DCR
right now until this library is integrated with Frontend)

MVTId calculator

Use this simple calculator to see what MVT ID
your test variant will fall into.

$3

Uses Jest, see .test.ts` files.

What's Next

- [x] Integrate into Frontend
- [ ] Review usability across other platforms and required APIs
- [ ] Investigate tree-shakeable-ness of methods
- [ ] Investigate exposing API methods outside of the configuration, to allow
usage inside of modules without passing a prop