Fun with Rails A/B testing

Posted under: Behind the Scenes

When we launched VendorRisk.com a few weeks back, we added a very simple A/B experiment to test which sign up call to action worked better.  The first two options we chose were “View plans and pricing” and “Try free for 30 days!”.  We assumed that the latter would perform better because it sounded more enticing and contained the magical word “free.”

And we were wrong.  Of the clicks to the sign up page, 71% of them came from the “View plans and pricing” link, which according to the testing software is “99.9% likely to be significant”.

99.9% is good enough for us, so we changed the experiment to use “View plans and pricing” and “Pricing and signup.”  As of this writing, they’re running neck-and-neck, so we’ll continue the experiment for a while and see if one distinguishes itself from the other.

Tech talk:

We use the A/Bingo Rails plugin, which you can find at www.bingocardcreator.com/abingo.  It’s dead simple to setup and use.

In the views that contain the sign up link:

<a href="/signup"><%= get_signup_text %></a>

In the helper file:

def get_signup_text
  choices = ["View plans and pricing", "Pricing and signup"]
  ab_test("call_to_signup2", choices) { |text| text }
end

In the controller:

def signup
  bingo!("call_to_signup2")
end

Don’t forget the controller code, as that’s what logs the test result to the database.