A/B Test Your Ruby Camping Web App Using ABingo (Part 2)
Intro
In “A/B Test Your Ruby Camping Web App Using ABingo (Part 1)”, I introduced the ABingo A/B testing framework and the new Camping ABingo plugin. In this article I will cover the following topics:
- Look under the ABingo Hood to understand its API, object model, and underlying database tables
- Create a draft of the ABingo Test Application
- Integrate ABingo into our test app
- Run and analyze experiments
Under The ABingo Hood
Before we can dive in the integration of the Camping ABingo plugin with our prototype application, here is a quick overview of the basics of the ABingo framework.
ABingo API
The two most important APIs are ab_test and bingo!:
Note: these 2 main APIs are implemented in Camping in a helper module being included in both controllers and views. The ab_test is syntactic sugar over the test method provided by the core Abingo class.
Model
The basic object model for the ABingo framework looks like this:
ABingo Database Tables
ABingo will need to track executions of your Experiments and their corresponding Alternatives in the database associated with your application. So the ABingo Camping plugin will include the needed migration to add 2 tables to your schema. We’ll see this in more details later.
Create An ABingo Test Application
Install Camping-ABingo
First let’s install the gem. (If you don’t have Camping or any of its dependent gem they will be installed)
gem install camping-abingo
Note: You will find the fully functional test application under the examples/camping-abingo-test folder in the camping-abingo gem folder. If you want to start using it right away, skip the next sections and go straight to Setting Up Our First A/B Test. Otherwise follow along.
Running the Skeleton pre-ABingo Test Application
In the next sections I will show you how to integrate ABingo with the skeleton test app. But before let’s get the skeleton application up and running.
- Locate the examples/camping-abingo-test folder under your camping-abingo gem folder.
There are two versions of the test app:- the skeleton (without ABingo support)
- the fully implemented version
- Open camping-abingo-test-skeleton.rb in an editor.
- Then let’s start the Camping server with our skeleton:
camping camping-abingo-test-skeleton.rb
- Open a browser and access the skeleton from the browser at the following url:
http://localhost:3001/
Camping will run the first migration to create the table for our User model:
And the basic test app should now come up:
- You can now test the basic flow of the application. When viewing the landing page, the sign-up button should always be labeled “Sign-Up Now!”. Once ABingo is integrated we will be experimenting with different alternatives.
Adding ABingo Support To The Skeleton Test App
The ABingo plugin is composed of several modules mirroring the standard modules a typical Camping application would contain.
Integrating the plugin will consist of linking the various modules using a combination of include and/or extend statements where appropriate. Here is a high-level diagram of the integration approach:
Here are the steps we will be following:
| # | To Do |
|---|---|
| A | Reference and require the needed gems |
| B | Customize the main module |
| C | Plugging in the ABingo common helpers module |
| D | Plugging in the ABingo Experience and Alternative models |
| E | Plugging in the ABingo Dashboard controllers |
| F | Plugging in the ABingo Dashboard views |
| G | Add code to manage the ABingo user identity in our controllers |
A. Reference and require the needed gems
We will need to reference 2 gems: filtering_camping and camping-abingo. Also we will require the corresponding files we need: filtering_camping and camping-abingo.So now the top of the source should look like this:
gem 'camping' , '>= 2.0'
gem 'filtering_camping' , '>= 1.0'
gem 'camping-abingo'
%w(rubygems active_record erb fileutils json markaby md5 redcloth
camping camping/session filtering_camping camping-abingo
).each { |lib| require lib }
Camping.goes :CampingABingoTest
B.Customizing the main module
Ok, so now we’re ready to enhance the main app module. First we’ll make sure to include the Camping::Session and CampingFilters modules, and to extend the app module with ABingoCampingPlugin and its Filters submodule, like so:
module CampingABingoTest include Camping::Session include CampingFilters extend ABingoCampingPlugin include ABingoCampingPlugin::Filters # ... end
We can also associate the logger with our camping-abingo plugin.
Camping::Models::Base.logger = app_logger ABingoCampingPlugin.logger = app_logger
Now let’s customize the create method by adding a call to ABingoCampingPlugin.create, so we can get the plugin to run any needed initialization.
def CampingABingoTest.create ABingoCampingPlugin.create end
We need to more things: let’s link our logger to the ABingo cache logger (to make it easier to debug issues), and let’s define which User id represent the administrator who will have access to the ABingo Dashboard. Now the final version of the create method looks like this:
def CampingABingoTest.create
dbconfig = YAML.load(File.read('config/database.yml'))
environment = ENV['DATABASE_URL'] ? 'production' : 'development'
Camping::Models::Base.establish_connection dbconfig[environment]
ABingoCampingPlugin.create
Abingo.cache.logger = Camping::Models::Base.logger
Abingo.options[:abingo_administrator_user_id] = 1
CampingABingoTest::Models.create_schema :assume => (CampingABingoTest::Models::User.table_exists? ? 1.1 : 0.0)
end
Ok, at this point we have a minimally configured application module.
C.Plugging in the ABingo common helpers module
The Helpers module is used in Camping to provide common utilities to both the Controllers and Views modules. Enhancing our Helpers module is very easy, we need to add both an extend and an include of the ABingoCampingPlugin::Helpers module so we can enhance both instance and class sides:
module CampingABingoTest::Helpers extend ABingoCampingPlugin::Helpers include ABingoCampingPlugin::Helpers end
D.Plugging in the ABingo Experience and Alternative models
First, we’ll include the include ABingoCampingPlugin::Models module so we can get all the ABingo-specific models. Our model will look like this:
module CampingABingoTest::Models include ABingoCampingPlugin::Models # ... end
Now we’ll enhance the CreateUserSchema migration class to plug in our ABingo model migration.
- We’ll bump up the migration version number of the CreateUserSchema class to 1.1
- In the up method we will add a call to ABingoCampingPlugin::Models.up so that the Experience, Alternative tables can be created.
- In the down method we will add a call to ABingoCampingPlugin::Models.down to drop the ABingo tables.
class CreateUserSchema < V 1.1 def self.up create_table :CampingABingoTest_users, :force => true do |t| # ... end # ... ABingoCampingPlugin::Models.up end def self.down ABingoCampingPlugin::Models.down drop_table :CampingABingoTest_users end end
Now if we restart the application, our version 1.1 migration should be executed:
E.Plugging in the ABingo Dashboard controllers
We will need to extend our app Controllers module with the ABingoCampingPlugin::Controllers module using the extend statement. Then just before the end of the Controllers module, we’ll add a call to the include_abingo_controllers method. This is how camping-abingo will inject and plugin the ABingo Dashboard controllers and helpers. It is important that this call always remaining the last statement of the module, even when you add new controller classes. So the module should look like so:
module CampingABingoTest::Controllers extend ABingoCampingPlugin::Controllers # ... include_abingo_controllers end #Controllers
F.Plugging in the ABingo Dashboard views
We will need to extend our app Views module with the ABingoCampingPlugin::Views module using the extend statement. Then just before the end of the Views module, we’ll add a call to the include_abingo_views method. This is how camping-abingo will inject and plugin the ABingo Dashboard views. It is important that this call always remaining the last statement of the module, even when you add new view methods. So the module should look like so:
module CampingABingoTest::Views extend ABingoCampingPlugin::Views # ... include_abingo_views end
G.Add code to manage the ABingo user identity in our controllers
ABingo tracks the selected experiment alternatives and potential conversions for a given user using a property named identity. Currently the ABingoCampingPlugin::Filters we included in our main module provides a filter that executes before all controllers. That filter is responsible for ensuring the Abingo.identity is set using either a large random integer if the user is anonymous, or the actual id of an existing user. Here is what the filter looks like:
before :all do set_abingo_identity end
To handle non-anonymous users and for our integration with our controllers to be complete we will need to:
- Set the identity to the new user’s id once signup is complete.
So in the post action of our SignUp controller, let’s set the @state.abingo_identity to our user’s id before rendering the Welcome view – see line 14:class SignUp < R '/signup' # get ... def post @user = User.find_by_username(input.username) if @user @info = 'A user account already exist for this username.' else @user = User.new :username => input.username, :password => input.password @user.save if @user @state.user_id = @user.id @state.abingo_identity = @user.id redirect R(Welcome) else @info = @user.errors.full_messages unless @user.errors.empty? end end render :signup end end
- Replace the abingo_identity with the user’s id (@user.id) when someone signs-in. See line 9:
class SignIn < R '/signin' # ... def post @user = User.find_by_username_and_password(input.username, input.password) if @user @state.user_id = @user.id @state.abingo_identity = @user.id if @state.return_to.nil? redirect R(Welcome) else return_to = @state.return_to @state.return_to = nil redirect(return_to) end else @info = 'Wrong username or password.' end render :signin end end
- Generate a new random identity when the current user has signed-out.
So let's set the @state.abingo_identity to another large random integer in the get action of our SignOut controller. See line 4:class SignOut < R '/signout' def get @state.user_id = nil @state.abingo_identity = Abingo.identity = rand(10 ** 10).to_i render :index end end
- Patrick McKenzie's ABingo is a powerful A/B testing framework for Ruby apps.
- ABingo makes it easy to define and execute experiments with multiple alternatives.
- The ABingo Dashboard makes interpreting test results a breeze!
- A/B Testing Definition on Wikipedia
- Effective A/B Testing Basics (by Ben Tilly)
- Google Website Optmizer
- ABingo A/B Testing Framework (by Patrick McKenzie)
- Patrick McKenzie's Blog
- Patrick McKenzie Interview On TechZing Podcast
- A/B Test Your Ruby Camping Web App Using ABingo (Part 1)
- Visualize Application Metrics on NewRelic for your Ruby Camping Application
Now our integration steps are complete. We're now ready to do some A/B testing!
Setting Up Our First A/B Test
Selecting A Random Alternative For a Given Experience
For our first experiment named call_to_action we'll vary the text for our signup_btn button by selecting one of 3 alternatives based on the user.
So let's modify the landing view of our test application. We'll change the assignment of the signup_text variable to the result of the ab_test call. So our code will look like the following (with the ab_test call on line 9):
def landing
div.xyz do
h1 'XYZ SAAS Application'
div.marketing! do
# benefits ...
div.actnow! do
signup_text = ab_test("call_to_action",
[ "Sign-Up Now!",
"Try It For Free Now!",
"Start Your Free Trial Now!",
])
a :href=>"/signup" do
div.signup_btn! signup_text
end
end
end
end
end
Recording A User Conversion
If we ran the application now ABingo would choose an alternative and keep track of it in terms of user participation.
So now we just need to record an actual conversion for the call_to_action test when the user clicks on the signup_btn button.
To do that, we'll enhance the get action of our SignUp controller by calling the bingo! API with call_to_action as the name of the test. See line 3 below:
At this stage, we have a basic Camping ABingo-enabled application, now let's test it!
Running Our First A/B Test
So let's refresh the browser - if you stopped the Camping server then restart it (as a note Camping automatically reloads your changes). You have two in three chances to get a different alternatives than our original hard-coded text:
Let's look at the content of our camping-abingo-test.db SQLite database (preferably with Lita, an AIR-based SQLite administration tool):
Note that the number of participants for the displayed alternative is 1.
If you are using FireBug with FireCookie, delete the campingabingotest.state and refresh the page. You should notice that the abingo_identity should change as well as the button text. Repeat the process several times. And occasionally click on the button to cause a conversion to occur.
Let's refresh our table contents in Lita:
Note that the number of participants and conversions have increased across alternatives.
Viewing A/B Test Results With The ABingo Dashboard
Checking the database for our results is quite tedious (even with Lita!), luckily ABingo has a built-in Dashboard. Our Camping ABingo plugin has already defined a couple controllers and routes. We just need to expose the main /abingo/dashboard route.
So let's go to the layout method of our ABingo Test app. Right below the link for Sign-Out let's add a code fragment to only show a link to if the id of the current signed-in user is the id of the allowed ABingo Administrator (using the abingo_administrator_user_id ABingo helper method). See lines 7-10:
def layout html do # ... a "Sign-Out", :href=>'/signout' if @state.user_id == abingo_administrator_user_id span " | " a "ABingo Dashboard", :href=>"/abingo/dashboard" end # ... end end
Now let's sign-in as the default administrator for our app. Use admin for the id and camping as the password. You should now see a "ABingo Dashboard" link in the top right of the navigation bar. Click on it. The dashboard should now show you the results:
Now it is easy to see the results and even terminate a given alternative if we want to.
Note: each experiment you have defined will be listed with its corresponding details.
Choosing Your Caching Mechanism
By default the ABingo Plugin initialize its cache based on a :memory_store. This is fine and easy to troubleshoot our simple test app. But for your application you should consider a more advanced caching mechanisms - see the ActiveSupport Cache documentation. One suggestion is to use Memcached if possible (e.g. on Heroku).
So What?
So hopefully this second post on ABingo (see here for the first post of the serie) will have given you a feel for how easy it is to A/B test-enable a Ruby Camping-based web application using the Camping ABingo plugin.
Happy ABingo A/B experiments!!!
Note: You can try the demo app yourself at: camping-abingo.heroku.com (including the ABingo Dashboard if you sign in as admin / camping).
In a subsequent post I will cover some advanced topics (caching, troubleshooting, multi-variates, etc.) for Camping ABingo. Stay tuned!
References and Resources
A/B Testing
Camping
Miscellaneous
My Other Related Posts:
Source Code
