Clientside AB testing with Varnish (and Django)

Recently, we need to test some new features on DownDetector. We were looking into changing some copy and styling of certain elements, and wanted to make sure we choose the right option as we had several different options available.

We had a couple of requirements:

• No backend connection required. All of our pages are either cached, or are served directly from Varnish cache, we don’t want an active session to our backend servers as the frontend pages should be stateless. This saves a lot of traffic (and costs) to our backend servers.

• We don’t need multi page tests We’re not testing a multi page funnel or anything. We just want to change some tweaks to copy, layout or buttons, and test which one works best.

• Multiple experiments at once We want to be able to run multiple experiments, with multiple variants per experiment. The user should be only bothered by one variant for the test.

• We don’t want to setup another statistics system Either it should integrate with statsd (our internal statistics engine), or some other public (free) service (Google Analytics?)

• Should be as realtime as possible We want to see results as we go along, we don’t want to wait a day for results, but act fast.

Some of the commercial options we looked at included Google Analytics, Visual Website Optimizer and Optimizely.

All options have their pro’s and con’s. We really loved Optimizely, but considering that we needed to run this on pages which can serve up to millions of pageviews per month, it could get very costly, very quickly, and we had to look for other solutions.

We decided to look around for other solutions, and came across the vmod_abtest for Varnish. When used correctly, it can alleviate all the pain of AB testing, and you don’t have to worry about a stampede to your backends.

Magic, Ponies and Sparkledust

The magic of this solution, is that the full setup of the user and assigned group is done entirely in the http headers and in the client itself. The ABtest never hits our backend servers at any time in the process.

How does this work? The vmod is configure with a set of rules (loaded in memory) in Varnish. These rules determine whether a user should be in a certain test group. We assign the user a specific cookie which is only valid for a certain url. That cookie is checked in a generated javascript file, which is included on every page, and is cached on the client.

The generated javascript fires events to Google Analytics (event tracking) for realtime statistics, or for other analysis. Varnish can also be setup to measure into statsd, but we haven’t needed that yet.

This implementation is done in Varnish, but a similiar implementation should be doable in Nginx or Apache as well.

Flow

In this example, we’ll use our test rules file. You can find all the necessary files in the gist as well.

1.

First, the user requests a page, say /foo/:

$ curl -v http://127.0.0.1:80/
> GET /foo/ HTTP/1.1
> User-Agent: curl/7.30.0
> Host: downdetector.com
> Accept: */*

2.

Varnish checks whether a config is available for the given path, and if the user doesn’t have a cookie yet (in this case, it doesn’t).

3.

Varnish returns the page (from cache), and adds the cookie. Notice how a cookie ab=3 was added which is only valid for this path, and will expire when the experiment is done (in this case, 5 minutes from now)

< Date: Tue, 19 Nov 2013 13:35:37 GMT	
< Content-Type: text/html; charset=utf-8	
< Transfer-Encoding: chunked	
< Connection: keep-alive
< Vary: Accept-Encoding
< X-Varnish: 940285222 940285189
< Age: 7
< Via: 1.1 varnish
< Set-Cookie: ab=3; Path=/foo/; Expires=Tue, 19 Nov 2013 13:40:37 GMT
< X-Cache: HIT

4.

The client parses the returned html pages, and downloads the included experiments.js file. This javascript files checks if an experiment was defined for the request url (/foo/), and if the user has a cookie for that url. The client checks if the cookie matches any group in that experiment, and executes that javascript. Something like this (pseudocode):

5.

…

6. Profit!

Experiments

The experiments.js file is generated with the django-abtest package, which gives you an admin interface to define the different experiments and variations. You can use the provided middleware if you want to test locally, or use it in production, if you don’t want to use varnish.

Ofcourse you aren’t limited to Django, and you could use your own preferred framework to generate the experiments file.

Installation

Let’s assume you already have Varnish up and running. Make sure you have a local copy of the varnish source, and clone the vmod repository:

git clone git@github.com:Destination/libvmod-abtest.git

Change into the directory and build the vmod (I needed to supply the VMODDIR in Ubuntu):

./configure VARNISHSRC=SOURCE [VMODDIR=DIR]
make
make install

Now you can use the vmod. You can download our vcl, and config file and load those:

cd /etc/varnish/
wget https://gist.github.com/svdgraaf/7223211/raw/29e2d5a8dbacdaf3a6796245c8650af77ccaca4e/abtest.vcl
wget https://gist.github.com/svdgraaf/7223211/raw/7f5ba5c5b0c94b564df95df660e1602f1c30acf8/rules.cfg

And add the abtest.vcl to your default.vcl, by adding:

include "/etc/varnish/abtest.vcl";

Reload Varnish:

sudo /etc/init.d/varnish restart

And you should be good to go. If you check your /foo/ directory (see the abtest.cfg file), you should see the cookie getting set:

$ curl -v http://127.0.0.1/foo/
< Date: Tue, 19 Nov 2013 13:35:37 GMT	
< Content-Type: text/html; charset=utf-8	
< Transfer-Encoding: chunked	
< Connection: keep-alive
< Vary: Accept-Encoding
< X-Varnish: 940285222 940285189
< Age: 7
< Via: 1.1 varnish
< Set-Cookie: ab=3; Path=/foo/; Expires=Tue, 19 Nov 2013 13:40:37 GMT
< X-Cache: HIT