Client side AB Testing

In this section we will go through the steps and considerations that you must have when you want to set up a new test.

Psst! you can also set up server-side AB tests by following this guide.

API

The AB test framework has the following methods:

init()

This function initialize the AB test framework by executing the following steps:

1. Read the MVT (multi variant testing) id from the users's cookie. If the user has no mvt in his or her browser, it generates a new MVTid and saves it in the cookie.

2. From the MVT, it generates the Participations object. The steps to build that object are:

• Read the participations from localStorage
• Check if the test is active.
• If the user is not already assigned to a variant in localStorage, assign them to a variant based on the value of the mvtId.
• Override the variant if the user passes the variant in the url using the form: #ab-[name_of_test]=[variant_name].

3. Save the Participations object in localStorage.

4. Submit an event to Ophan to track the initial impression for all active A/B tests.

getVariantsAsString(participations: Participations)

Takes the participation object and returns all variants as a string in the following format:

test1=variantA; test2=variantB

This method is used mainly when the developer wants to track the variants in GA. In order to achieve this, she or he has to set up the custom dimension called experiment with the value of the variants as a string. An example of this can be found here.

trackABOphan(participation: Participations, complete: boolean)

Track event using tracker-js from Ophan.

abTestReducer(state: Participations = {}, action: Action)

Reducer function to be included in the reducer of pages which import the AB test framework.

Set up the AB test framework in a new page of the site

Step 1: Initialize the AB test framework on the page you are working on

In order to use the AB test framework you have to initialize it in your page. To do this call the init function of the module, this will return a Participations object which should be passed to the Redux store.

// ----- AB Tests ----- //

const participation = abTest.init();
  
 
// ----- Redux Store ----- //
  
const store = createStore(reducer);
  
store.dispatch({ type: 'SET_AB_TEST_PARTICIPATION', payload: participation });
 

You can find a real example of this here.

Step 2: Add the AbTest reducer to your store

In order to be able to understand Redux AB actions, you have to add the abTestReducer to your page's reducer as follows:

export default combineReducers({
  contribution,
  intCmp,
  abTests,
});

You can find a real example of this here.

Implementation of a test

Step 0: Define your experiment

First design the experiment that you want to run. The experiment consist of a hypothesis and a number of variants. For example:

Hypothesis: By changing [element_to_test], it will [increase|decrease] the [put_some_metric] by number%

For example:

By changing the background colour to black, it will increase the conversion rate by 5%

The amount of change you expect your test to achieve will affect the length of the test. To detect a smaller change, you need more samples and therefore a longer running test. This is related to the statistical significance concept.

This tool can help to compute the sample size of your experiment, from the sample size, you can then estimate the duration of your test.

Step 1: Add your test to the Tests object

After your hypothesis is defined, you can implement the test in the codebase. First, define the test in the abtestDefinitions.js by adding a new field to the tests object.

// ----- Tests ----- //

export type Tests = { [testId: string]: Test }

export const tests: Tests = {
  yourTestId: {
    variants: ['control', 'variant'],
    audiences: {
      GB: {
        offset: 0,
        size: 1,
        breakpoint: {
            minWidth: 'tablet',
            maxWidth: 'desktop',
        },
      },
    },
    customSegmentCondition: () => window.location.pathname.indexOf('contribute') > 0, 
    isActive: true,
    independent: false,
    seed: 0,
  },
};

The tests object represents all of the tests running on the site, where each key is a test ID and each value is an object with the following fields:

• variants: This field is an array of strings, each string will be the name of a variant. One of these variant names has to be control.
• audiences: The audiences object contains a field for every country where the test will run. Then each audience object has three fields:
  • offset, a number from 0 to 1 which indicates the part of the audience that will be affected by the test.
  • size: a number from 0 to 1 which specifies the percentage of the audience to be included in the test. For example a test with offset 0.2 and size 0.5 will affect 50% of the audience starting from the 20%.
  • breakpoint: an object with two optional fields, minWidth and maxWidth. The possible values for these fields are: mobile, mobileMedium, mobileLandscape, phablet, tablet, desktop, leftCol and wide. With these two values, the AB test framework will generate a media query an validate it against the reader's device.
• isActive: Indicates whether the test is active or not. If the test is not active it is not possible to force any variant.
• independent: Expresses whether the algorithm uses the mvtId (independent: false) or a random number (independent: true) to assign a user to a variant.
• seed: It is used only when independent is equal to true. It is useful to sync different tests. In other words, the user will be in the same variant across different tests. For example, two tests with two variants (control and variantA) and the same seed, will cause that the user will be in control for both tests or variantA for both tests. Use this functionality carefully, if the tests are dependent on each other (e.g. part of the same flow), it is possible to invalidate both tests.
• customSegmentCondition: An optional function of type () => boolean which allows custom segmentation of the audience. For example to only opt in users on desktop we could use the following function:

  () => window.matchMedia('(min-width: 980px)').matches // matches the 'desktop' breakpoint defined in breakpoints.scss

  Users who visit the site in a browser with a width of less than 980 pixels will now receive a test allocation of notintest while those using a browser with a width of greater than or equal to 980 pixels will be allocated amongst the available variants.

Step 2: Read the variant from the state

The test and its variant(s), are now present in the abParticipations object. You can find that object inside the common object which is in every state of every page.

Once the container knows which variant is on, it will render the presentational component or element corresponding to that variant. Usually this can be achieved by creating a local (or global if the test is run across different pages) module that knows which component or element to instantiate depending on the variant.

An example of the above can be found on this line of the 'bundle landing' page which reads the variant and then pass the variant to the sub-components which will load the correct version on the variant. The module that knows which version to render can be found here.

Step 3: Track events with GA and Ophan

Now that you are rendering the correct component depending on the variant, the final step is to track user conversion. Conversion can mean different things depending on what you are testing, it could be a click on a video, a scroll action, a button click, whether the user writes something in a text field, etc. Basically, it can be any event that the user can produce. Test displays are automatically tracked to Ophan, but in order to use abacus as your test tool, you have to track the test's conversion action.

This tracking can be done using the trackABOphan function from the ABtest framework. This function receives an A/B participation and a flag indicating whether is a complete event or not (which should be true in the case of conversions).

If you are firing a conversion event for a specific test, be sure that the participations parameter you provide to trackABOphan only contains your test.