SALESmanago Live Product Demo

Wednesday, 10:00 am CEST

SALESmanago Live Product Demo

Wednesday, 10:00 am CEST


A Complete Guide to A/B Testing: Lessons from Netflix, Barack Obama, and Optimizely

A/B testing: which version will your users choose?

Tests and experiments are the lifeblood of good marketing. This technique liberates you from clichés, common knowledge and following safe paths others go. Instead, you can explore what works for your audience and shape your actions according to data. In other words, you gain power and control. Are you ready for it?


Definition of A/B testing

An A/B test is a randomized controlled experiment: a type of hypothesis testing. You present two variants with just one difference to 2 groups of users to see which version performs better.

For example, you can send the same newsletter with two different subject lines to see which one will be opened more often.

Hey! Win the Zero Moment of Truth! Download free ebook

You can also test more than 2 variants.


5 steps to successful A/B tests: where to start

Perhaps you wonder what to test in the first place. I advocate a slightly different approach, based on the excellent book “A/B Testing. The Most Powerful Way to Turn Clicks into Customers” by Dan Siroker and Pete Koomen, founders of Optimizely. Begin with that 5-step formula:


Define success (and quantifiable success metrics)

What is the core aim of your website, the reason for its existence? Is it to drive leads or to sell products, make it easier to book a visit in the offline salon or to encourage donations? Find that major aim and put it clearly. Then think what could be a suitable metric for the success: the number of leads? Sales?


Identify bottlenecks

OK, now let’s move to the site analysis and find bottlenecks: major obstacles, or ”places when you lose the most momentum in moving users through your desired series of actions.“  For example, you might observe that customers in your e-store have problems with completing the transaction — they add products to the cart but rarely buy. It indicates that check-out is the thing to work on.


Construct a hypothesis

Two previous steps should guide you towards a working hypothesis on what should be improved. It’s science, remember? Take a critical look at your check-out process and decide which element you’d like to test first.

Bear in mind that you should test one thing and one thing only at the time: for example, test only the color of the button, not the copy.


The hypothesis should sound like this:

If I change … from … to …, I will achieve…

E.g.: If I change the color of the button from red to yellow, I’ll achieve higher conversion.


As you see, the hypothesis includes:

*    Tested element

*    How to alter it

*    What results will it bring



You have limited time, so assess what can have a bigger impact on the success. For the beginning, test the essential features or elements.  



OK, now you’re ready. Run the test!


What can I test?

Remember that you can experiment with every single element of your emails, website (contact forms, landing pages, recommendation frames) and UI. It includes pricing, feature list, streaming quality, time of page load… Anything that comes to your mind. Test layout, button, copy, colors, time of sending…

 Testing dynamic content with the Marketing Automation platform.   


In 2007 Dan Siroker quit his job at Google to join Barack Obama team. He focused on optimizing landing page with email sign-up form. He started from testing the copy on the button; then he checked various videos and photos (contrary to expectations, it turned out that audience responded better to photos, not videos). The winning combination (the image of the candidate with his wife and daughter + ”Change we believe in“ copy + ”Learn more“ button) improved the subscription rate by 40.6 percent.


Netflix on the other hand experiments with everything, including things they know from customers’ feedback, showing a discrepancy between users’ declarations and actions. Many Netflix visitors expressed the opinion that making the titles of platform’s content available to browse was the one thing they wanted to know before signing up. One could say it seemed legit: people want to make sure they’ll find valuable movies before they trade their contact data. But Netflix being Netflix refused just to stop at survey results; they ran tests. The team applied the following hypothesis:

If users can browse content titles before signing up, it will increase the number of users who register.


Then they created 5 variations with varying degrees of content searchability, each allowing visitors to know more before subscribing. The results? Against users’ declarations, the original option delivered the best results.


To sum up

Test everything.

Start from essential elements of your website, related to its raison d’etre, then move to details that add up indirectly to your success. At the same time make sure that you never optimize minor pieces at the expense of the crucial ones (like improving blog readership while lowering sales).

SALESmanago is a Customer Engagement Platform for impact-hungry eCommerce marketing teams who want to be lean yet powerful, trusted revenue growth partners for CEOs. Our AI-driven solutions have already been adopted by 2000+ mid-size businesses in 50 countries, as well as many well-known global brands such as Starbucks, Vodafone, Lacoste, KFC, New Balance and Victoria’s Secret.

SALESmanago delivers on its promise of maximizing revenue growth and improving eCommerce KPIs by leveraging three principles: (1) Customer Intimacy to create authentic customer relationships based on Zero and First Party Data; (2) Precision Execution to provide superior Omnichannel customer experience thanks to Hyperpersonalization; and (3) Growth Intelligence merging human and AI-based guidance enabling pragmatic and faster decision making for maximum impact.

More information:

How Focus Garden Cultivated Unprecedented Growth with SALESmanago
How Focus Garden Cultivated Unprecedented Growth with SALESmanago

    Focus Garden, the premier destination for high-quality garden furniture and supplies, cultivates success as meticulously as their customers cultivate their gardens. We’re proud to support the company in their digital journey, focused strategically overcoming challenges and setting new standards in customer engagement.   In less than a year, our collaborative efforts with Focus […]

Prepare your business for a cookieless world 🍪
Prepare your business for a cookieless world 🍪

    If you haven’t heard of zero-party data’s immense benefits for online businesses, you must’ve spent the last two years under a particularly cozy rock 🪨    But don’t fret – let us give you a quick rundown of the benefits before we dive into how you can harness this goldmine, maximize the utility […]

SALESmanago Announces Key Leadership Appointments: Welcoming New CEO and CRO
SALESmanago Announces Key Leadership Appointments: Welcoming New CEO and CRO

    SALESmanago, a leading no-code CDP & Marketing Automation platform, is excited to announce significant changes to its leadership team, aimed at enhancing our delivery of end-to-end solutions to the mid-market. With a focus on actionable data insights and a flexible workflow management engine, we enable our clients to run automated, omnichannel marketing campaigns […]

[New Feature] Shine the light on your hidden visitors with Spotlight
[New Feature] Shine the light on your hidden visitors with Spotlight

    The data noise can keep you in the dark – that’s why Spotlight shines the light on the hidden visitors and helps convert them into loyal Customers. Understand your anonymous shoppers and tailor personalized website campaigns based on their behavior.   With approximately 85% of website visitors remaining anonymous, it’s crucial to understand […]

The Essential Role of Product Catalogs in Marketing Evolution
The Essential Role of Product Catalogs in Marketing Evolution

    In the dynamic realm of digital marketing, Product Catalogs act as the vital link between your eCommerce platform and SALESmanago. They aren’t just useful but essential for maximizing your marketing potential.    You might be wondering, ‘I’ve already integrated XML Product Feeds, why consider another integration to Product Catalogs?’ While XML Product Feeds […]