Clientside AB testing with Varnish (and Django)
Recently, we need to test some new features on DownDetector. We were looking into changing some copy and styling of certain elements, and wanted to make sure we choose the right option as we had several different options available.
We had a couple of requirements:
No backend connection required. All of our pages are either cached, or are served directly from Varnish cache, we don’t want an active session to our backend servers as the frontend pages should be stateless. This saves a lot of traffic (and costs) to our backend servers.
We don’t need multi page tests We’re not testing a multi page funnel or anything. We just want to change some tweaks to copy, layout or buttons, and test which one works best.
Multiple experiments at once We want to be able to run multiple experiments, with multiple variants per experiment. The user should be only bothered by one variant for the test.
We don’t want to setup another statistics system Either it should integrate with statsd (our internal statistics engine), or some other public (free) service (Google Analytics?)
Should be as realtime as possible We want to see results as we go along, we don’t want to wait a day for results, but act fast.
All options have their pro’s and con’s. We really loved Optimizely, but considering that we needed to run this on pages which can serve up to millions of pageviews per month, it could get very costly, very quickly, and we had to look for other solutions.
We decided to look around for other solutions, and came across the vmod_abtest for Varnish. When used correctly, it can alleviate all the pain of AB testing, and you don’t have to worry about a stampede to your backends.
Magic, Ponies and Sparkledust
The magic of this solution, is that the full setup of the user and assigned group is done entirely in the http headers and in the client itself. The ABtest never hits our backend servers at any time in the process.
This implementation is done in Varnish, but a similiar implementation should be doable in Nginx or Apache as well.
First, the user requests a page, say
$ curl -v http://127.0.0.1:80/ > GET /foo/ HTTP/1.1 > User-Agent: curl/7.30.0 > Host: downdetector.com > Accept: */*
Varnish checks whether a config is available for the given path, and if the user doesn’t have a cookie yet (in this case, it doesn’t).
Varnish returns the page (from cache), and adds the cookie. Notice how a cookie
ab=3 was added which is only valid for this path, and will expire when the experiment is done (in this case, 5 minutes from now)
< Date: Tue, 19 Nov 2013 13:35:37 GMT < Content-Type: text/html; charset=utf-8 < Transfer-Encoding: chunked < Connection: keep-alive < Vary: Accept-Encoding < X-Varnish: 940285222 940285189 < Age: 7 < Via: 1.1 varnish < Set-Cookie: ab=3; Path=/foo/; Expires=Tue, 19 Nov 2013 13:40:37 GMT < X-Cache: HIT
The experiments.js file is generated with the django-abtest package, which gives you an admin interface to define the different experiments and variations. You can use the provided middleware if you want to test locally, or use it in production, if you don’t want to use varnish.
Ofcourse you aren’t limited to Django, and you could use your own preferred framework to generate the experiments file.
Let’s assume you already have Varnish up and running. Make sure you have a local copy of the varnish source, and clone the vmod repository:
git clone firstname.lastname@example.org:Destination/libvmod-abtest.git
Change into the directory and build the vmod (I needed to supply the VMODDIR in Ubuntu):
./configure VARNISHSRC=SOURCE [VMODDIR=DIR] make make install
Now you can use the vmod. You can download our vcl, and config file and load those:
cd /etc/varnish/ wget https://gist.github.com/svdgraaf/7223211/raw/29e2d5a8dbacdaf3a6796245c8650af77ccaca4e/abtest.vcl wget https://gist.github.com/svdgraaf/7223211/raw/7f5ba5c5b0c94b564df95df660e1602f1c30acf8/rules.cfg
And add the abtest.vcl to your default.vcl, by adding:
sudo /etc/init.d/varnish restart
And you should be good to go. If you check your
/foo/ directory (see the abtest.cfg file), you should see the cookie getting set:
$ curl -v http://127.0.0.1/foo/ < Date: Tue, 19 Nov 2013 13:35:37 GMT < Content-Type: text/html; charset=utf-8 < Transfer-Encoding: chunked < Connection: keep-alive < Vary: Accept-Encoding < X-Varnish: 940285222 940285189 < Age: 7 < Via: 1.1 varnish < Set-Cookie: ab=3; Path=/foo/; Expires=Tue, 19 Nov 2013 13:40:37 GMT < X-Cache: HIT