The "Tech. Arch."

Architecting Forward ::>

A/B Test Your Ruby Camping Web App Using ABingo (Part 2)


In “A/B Test Your Ruby Camping Web App Using ABingo (Part 1)”, I introduced the ABingo A/B testing framework and the new Camping ABingo plugin. In this article I will cover the following topics:

  1. Look under the ABingo Hood to understand its API, object model, and underlying database tables
  2. Create a draft of the ABingo Test Application
  3. Integrate ABingo into our test app
  4. Run and analyze experiments

Under The ABingo Hood

Before we can dive in the integration of the Camping ABingo plugin with our prototype application, here is a quick overview of the basics of the ABingo framework.

ABingo API

The two most important APIs are ab_test and bingo!:

API Parameters Context Purpose/Usage

  • test name
  • optional: array of alternatives
  • optional: conversion name
View or Controller Select an alternative for the current user based on the alternatives specified for this test. If no alternatives are specified a boolean value is returned. Tracks the execution using the conversion name if provided otherwise use the test name.
  • conversion name (or test name)
Controller Tracks a conversion for the current user and the conversion specified. The test name is the fallback option.

Note: these 2 main APIs are implemented in Camping in a helper module being included in both controllers and views. The ab_test is syntactic sugar over the test method provided by the core Abingo class.


The basic object model for the ABingo framework looks like this:

Model Responsibilities
User The user account on the prototype app
Abingo ABingo’s main API class
Experiment Represents a specific test based on multiple alternatives
Alternative Tracks participants and conversions for a given alternative

ABingo Database Tables

ABingo will need to track executions of your Experiments and their corresponding Alternatives in the database associated with your application. So the ABingo Camping plugin will include the needed migration to add 2 tables to your schema. We’ll see this in more details later.

Create An ABingo Test Application

Install Camping-ABingo

First let’s install the gem. (If you don’t have Camping or any of its dependent gem they will be installed)

gem install camping-abingo

Note: You will find the fully functional test application under the examples/camping-abingo-test folder in the camping-abingo gem folder. If you want to start using it right away, skip the next sections and go straight to Setting Up Our First A/B Test. Otherwise follow along.

Running the Skeleton pre-ABingo Test Application

In the next sections I will show you how to integrate ABingo with the skeleton test app. But before let’s get the skeleton application up and running.

  1. Locate the examples/camping-abingo-test folder under your camping-abingo gem folder.

    There are two versions of the test app:

    • the skeleton (without ABingo support)
    • the fully implemented version
  2. Open camping-abingo-test-skeleton.rb in an editor.
  3. Then let’s start the Camping server with our skeleton:
    camping camping-abingo-test-skeleton.rb
  4. Open a browser and access the skeleton from the browser at the following url:

    Camping will run the first migration to create the table for our User model:

    And the basic test app should now come up:

  5. You can now test the basic flow of the application. When viewing the landing page, the sign-up button should always be labeled “Sign-Up Now!”. Once ABingo is integrated we will be experimenting with different alternatives.

Adding ABingo Support To The Skeleton Test App

The ABingo plugin is composed of several modules mirroring the standard modules a typical Camping application would contain.
Integrating the plugin will consist of linking the various modules using a combination of include and/or extend statements where appropriate. Here is a high-level diagram of the integration approach:

Here are the steps we will be following:

# To Do
A Reference and require the needed gems
B Customize the main module
C Plugging in the ABingo common helpers module
D Plugging in the ABingo Experience and Alternative models
E Plugging in the ABingo Dashboard controllers
F Plugging in the ABingo Dashboard views
G Add code to manage the ABingo user identity in our controllers

A. Reference and require the needed gems

We will need to reference 2 gems: filtering_camping and camping-abingo. Also we will require the corresponding files we need: filtering_camping and camping-abingo.So now the top of the source should look like this:

gem 'camping' , '>= 2.0'	
gem 'filtering_camping' , '>= 1.0'	
gem 'camping-abingo'

%w(rubygems active_record erb  fileutils json markaby md5 redcloth  
camping camping/session filtering_camping camping-abingo
).each { |lib| require lib }

Camping.goes :CampingABingoTest

B.Customizing the main module

Ok, so now we’re ready to enhance the main app module. First we’ll make sure to include the Camping::Session and CampingFilters modules, and to extend the app module with ABingoCampingPlugin and its Filters submodule, like so:

module CampingABingoTest
	include Camping::Session
	include CampingFilters

	extend  ABingoCampingPlugin
	include ABingoCampingPlugin::Filters
	# ...

We can also associate the logger with our camping-abingo plugin.

	Camping::Models::Base.logger = app_logger
	ABingoCampingPlugin.logger   = app_logger

Now let’s customize the create method by adding a call to ABingoCampingPlugin.create, so we can get the plugin to run any needed initialization.

	def CampingABingoTest.create

We need to more things: let’s link our logger to the ABingo cache logger (to make it easier to debug issues), and let’s define which User id represent the administrator who will have access to the ABingo Dashboard. Now the final version of the create method looks like this:

	def CampingABingoTest.create
		dbconfig = YAML.load('config/database.yml'))								
		environment = ENV['DATABASE_URL'] ? 'production' : 'development'
		Camping::Models::Base.establish_connection  dbconfig[environment]

		Abingo.cache.logger = Camping::Models::Base.logger
		Abingo.options[:abingo_administrator_user_id] = 1

		CampingABingoTest::Models.create_schema :assume => (CampingABingoTest::Models::User.table_exists? ? 1.1 : 0.0)

Ok, at this point we have a minimally configured application module.

C.Plugging in the ABingo common helpers module

The Helpers module is used in Camping to provide common utilities to both the Controllers and Views modules. Enhancing our Helpers module is very easy, we need to add both an extend and an include of the ABingoCampingPlugin::Helpers module so we can enhance both instance and class sides:

module CampingABingoTest::Helpers
	extend ABingoCampingPlugin::Helpers
	include ABingoCampingPlugin::Helpers

D.Plugging in the ABingo Experience and Alternative models

First, we’ll include the include ABingoCampingPlugin::Models module so we can get all the ABingo-specific models. Our model will look like this:

module CampingABingoTest::Models
	include ABingoCampingPlugin::Models

	# ...

Now we’ll enhance the CreateUserSchema migration class to plug in our ABingo model migration.

  • We’ll bump up the migration version number of the CreateUserSchema class to 1.1
  • In the up method we will add a call to ABingoCampingPlugin::Models.up so that the Experience, Alternative tables can be created.
  • In the down method we will add a call to ABingoCampingPlugin::Models.down to drop the ABingo tables.
	class CreateUserSchema < V 1.1
		def self.up
			create_table :CampingABingoTest_users, :force => true do |t|
				# ...
			# ...
		def self.down		
			drop_table :CampingABingoTest_users

Now if we restart the application, our version 1.1 migration should be executed:

E.Plugging in the ABingo Dashboard controllers

We will need to extend our app Controllers module with the ABingoCampingPlugin::Controllers module using the extend statement. Then just before the end of the Controllers module, we’ll add a call to the include_abingo_controllers method. This is how camping-abingo will inject and plugin the ABingo Dashboard controllers and helpers. It is important that this call always remaining the last statement of the module, even when you add new controller classes. So the module should look like so:

module CampingABingoTest::Controllers
	extend ABingoCampingPlugin::Controllers

	# ...

end #Controllers

F.Plugging in the ABingo Dashboard views

We will need to extend our app Views module with the ABingoCampingPlugin::Views module using the extend statement. Then just before the end of the Views module, we’ll add a call to the include_abingo_views method. This is how camping-abingo will inject and plugin the ABingo Dashboard views. It is important that this call always remaining the last statement of the module, even when you add new view methods. So the module should look like so:

module CampingABingoTest::Views
	extend ABingoCampingPlugin::Views

	# ...

G.Add code to manage the ABingo user identity in our controllers

ABingo tracks the selected experiment alternatives and potential conversions for a given user using a property named identity. Currently the ABingoCampingPlugin::Filters we included in our main module provides a filter that executes before all controllers. That filter is responsible for ensuring the Abingo.identity is set using either a large random integer if the user is anonymous, or the actual id of an existing user. Here is what the filter looks like:

	before :all do

To handle non-anonymous users and for our integration with our controllers to be complete we will need to:

  1. Set the identity to the new user’s id once signup is complete.

    So in the post action of our SignUp controller, let’s set the @state.abingo_identity to our user’s id before rendering the Welcome view – see line 14:

    	class SignUp < R '/signup'
    		# get ...
    		def post
    			@user = User.find_by_username(input.username)
    			if @user
    				@info = 'A user account already exist for this username.'
    				@user = :username => input.username,
    					:password => input.password
    				if @user
    					@state.user_id =
    					@state.abingo_identity =
    					redirect R(Welcome)
    					@info = @user.errors.full_messages unless @user.errors.empty?
    			render :signup
  2. Replace the abingo_identity with the user’s id ( when someone signs-in. See line 9:
    	class SignIn < R '/signin'
    		# ...
    		def post
    			@user = User.find_by_username_and_password(input.username, input.password)
    			if @user
    				@state.user_id =
    				@state.abingo_identity =
    				if @state.return_to.nil?
    					redirect R(Welcome)
    					return_to = @state.return_to
    					@state.return_to = nil
    				@info = 'Wrong username or password.'
    			render :signin		
  3. Generate a new random identity when the current user has signed-out.

    So let's set the @state.abingo_identity to another large random integer in the get action of our SignOut controller. See line 4:

    	class SignOut < R '/signout'		
    		def get
    			@state.user_id = nil
    			@state.abingo_identity = Abingo.identity = rand(10 ** 10).to_i
    			render :index
    1. Now our integration steps are complete. We're now ready to do some A/B testing!

      Setting Up Our First A/B Test

      Selecting A Random Alternative For a Given Experience

      For our first experiment named call_to_action we'll vary the text for our signup_btn button by selecting one of 3 alternatives based on the user.

      So let's modify the landing view of our test application. We'll change the assignment of the signup_text variable to the result of the ab_test call. So our code will look like the following (with the ab_test call on line 9):

      	def landing do
      			h1 'XYZ SAAS Application'
      			! do
      				# benefits ...
      				div.actnow! do
      					signup_text = ab_test("call_to_action", 
      											[ 	"Sign-Up Now!",
      												"Try It For Free Now!",
      												"Start Your Free Trial Now!",
      					a :href=>"/signup" do
      						div.signup_btn!  signup_text
      Recording A User Conversion

      If we ran the application now ABingo would choose an alternative and keep track of it in terms of user participation.
      So now we just need to record an actual conversion for the call_to_action test when the user clicks on the signup_btn button.

      To do that, we'll enhance the get action of our SignUp controller by calling the bingo! API with call_to_action as the name of the test. See line 3 below:

      At this stage, we have a basic Camping ABingo-enabled application, now let's test it!

      Running Our First A/B Test

      So let's refresh the browser - if you stopped the Camping server then restart it (as a note Camping automatically reloads your changes). You have two in three chances to get a different alternatives than our original hard-coded text:

      Let's look at the content of our camping-abingo-test.db SQLite database (preferably with Lita, an AIR-based SQLite administration tool):

      Note that the number of participants for the displayed alternative is 1.

      If you are using FireBug with FireCookie, delete the campingabingotest.state and refresh the page. You should notice that the abingo_identity should change as well as the button text. Repeat the process several times. And occasionally click on the button to cause a conversion to occur.

      Let's refresh our table contents in Lita:

      Note that the number of participants and conversions have increased across alternatives.

      Viewing A/B Test Results With The ABingo Dashboard

      Checking the database for our results is quite tedious (even with Lita!), luckily ABingo has a built-in Dashboard. Our Camping ABingo plugin has already defined a couple controllers and routes. We just need to expose the main /abingo/dashboard route.

      So let's go to the layout method of our ABingo Test app. Right below the link for Sign-Out let's add a code fragment to only show a link to if the id of the current signed-in user is the id of the allowed ABingo Administrator (using the abingo_administrator_user_id ABingo helper method). See lines 7-10:

      def layout
      	html do
      		# ...
      						a "Sign-Out",	:href=>'/signout'
      						if @state.user_id == abingo_administrator_user_id
      							span " | "
      							a "ABingo Dashboard", :href=>"/abingo/dashboard"
      		# ...

      Now let's sign-in as the default administrator for our app. Use admin for the id and camping as the password. You should now see a "ABingo Dashboard" link in the top right of the navigation bar. Click on it. The dashboard should now show you the results:

      Now it is easy to see the results and even terminate a given alternative if we want to.

      Note: each experiment you have defined will be listed with its corresponding details.

      Choosing Your Caching Mechanism

      By default the ABingo Plugin initialize its cache based on a :memory_store. This is fine and easy to troubleshoot our simple test app. But for your application you should consider a more advanced caching mechanisms - see the ActiveSupport Cache documentation. One suggestion is to use Memcached if possible (e.g. on Heroku).

      So What?

      1. Patrick McKenzie's ABingo is a powerful A/B testing framework for Ruby apps.
      2. ABingo makes it easy to define and execute experiments with multiple alternatives.
      3. The ABingo Dashboard makes interpreting test results a breeze!

      So hopefully this second post on ABingo (see here for the first post of the serie) will have given you a feel for how easy it is to A/B test-enable a Ruby Camping-based web application using the Camping ABingo plugin.

      Happy ABingo A/B experiments!!!

      Note: You can try the demo app yourself at: (including the ABingo Dashboard if you sign in as admin / camping).

      In a subsequent post I will cover some advanced topics (caching, troubleshooting, multi-variates, etc.) for Camping ABingo. Stay tuned!

      References and Resources

      A/B Testing
      My Other Related Posts:

      Source Code
      If you enjoyed this post, I would love it if you could check out mySkillsMap, my skills management app.

      December 14th, 2010 Posted by | Uncategorized | no comments

No Comments

No comments yet.

Sorry, the comment form is closed at this time.