But if you\u2019re making one of the common split testing mistakes listed below, your split tests might be doing more harm than good.<\/p>\n
Qubit says<\/a> that poorly done split tests can cause businesses to invest in unnecessary changes and can even hurt their profits.<\/p>\n The truth is, there\u2019s a lot more to A\/B testing than just setting up a test. To succeed with A\/B test results, you\u2019ll need to run them the right way and avoid the errors that can undermine your outcomes.<\/p>\n In this post, I\u2019ll share the most common A\/B testing mistakes many businesses make so you can avoid them. You\u2019ll also learn how to use split testing the right way so that you can discover the hidden strategies that can skyrocket your conversions.<\/p>\n Let\u2019s get started!<\/p>\n Here\u2019s the list of topics I\u2019ll cover in this post. You can use this table of contents to click on a topic that you\u2019re most interested in to jump directly to that section:<\/p>\n One of the biggest problems with A\/B testing is running your tests on the wrong pages. It\u2019s important to avoid wasting time and valuable resources with pointless split testing.<\/p>\n But how do you know which pages to test? If you\u2019re marketing a business, the answer is easy: the best pages to test are the ones that make a difference in conversions and sales.<\/p>\n Hubspot suggests<\/a> that the best pages to test are the ones that get the most traffic, such as:<\/p>\n Product pages are especially important for eCommerce sites to test because those tests can give you data on how you can improve your sales funnel<\/a>.<\/p>\n Simply put, if a page isn\u2019t part of your sales or digital marketing strategy, there\u2019s no point testing it. You can prioritize a page for your split testing experiments once you start noticing a spike in organic traffic or an improved engagement rate.<\/p>\n If making a change on a page won\u2019t affect the bottom line, move on. Instead, test a page that\u2019ll give you immediate results.<\/p>\n A common A\/B testing pitfall to avoid is not having a valid hypothesis.<\/p>\n An A\/B testing hypothesis is a theory about why you are getting particular results on a web page and how you can improve those results.<\/p>\n Let\u2019s break this down a little more. To form a hypothesis, you\u2019ll need to consider the following steps:<\/p>\n Here\u2019s how you can work it out in a real-life scenario:<\/p>\n Note that you need all the elements for a valid hypothesis: observing data, speculating about reasons, coming up with a theory for how to fix it, and measuring results after implementing a fix.<\/p>\n If the test doesn\u2019t yield satisfactory results, that means one of the variables was wrong. To succeed, you\u2019ll need to change it and iterate the experiment until you get a conclusive result.<\/p>\n Here\u2019s another common mistake in A\/B testing that many marketers make: trying to test too many items at once.<\/p>\n It might seem like testing multiple page elements at once saves time, but it doesn\u2019t. What happens is that you\u2019ll never know which change is responsible for the improved (or poor) results.<\/p>\n Good split testing involves changing one item on a page and testing it against another version of the same item.<\/p>\n The minute you change more than one item at a time, you need a\u00a0multivariate test<\/a>. A multivariate test lets you compare more than 2 variations of a\u00a0campaign<\/a>\u00a0at a time.<\/p>\n You can read more about it in detail in our comparison of\u00a0split testing versus multivariate testing<\/a>.<\/p>\n Here\u2019s a simple visual explanation of how a multivariate test works:<\/p>\n Multivariate testing is a great way to test a website redesign where you have to change lots of page elements. But you can end up with a lot of combinations to test, and that takes time you might not want to invest.<\/p>\n Multivariate testing also only works well for high-traffic sites and pages. In most cases, however, a simple split test will get you the most meaningful results.<\/p>\n When it comes to running A\/B tests, it\u2019s important to keep things simple.<\/p>\n While it\u2019s perfectly fine to run multiple split tests, it\u2019s not advisable to run more than 4 split tests at a time.<\/p>\n For example, you can get meaningful results by testing 3 different versions of your CTA button. Running these tests isn\u2019t the same as multivariate testing, because you\u2019re still only changing a single item for each test.<\/p>\n But the more variations you run, the bigger the A\/B testing sample size you need. That\u2019s because you have to send more traffic to each version to get reliable results.<\/p>\n This is known as\u00a0A\/B testing statistical significance<\/a>. In simple terms, it means making sure the numbers are large enough to actually have meaning.<\/p>\n You can check if you have a good enough sample size for your tests with the tools mentioned in our\u00a0split testing guide<\/a>.<\/p>\n Timing is everything with A\/B testing. But there are a couple of classic mistakes people make related to running a split test at the wrong time.<\/p>\n Let\u2019s understand this with an example. If a page gets most of its traffic on Wednesdays, it doesn\u2019t make sense to compare the test results for that day with the results on a low-traffic day.<\/p>\n This is an important point to remember if you\u2019re an eCommerce retailer. That\u2019s because it can be misleading to compare your test results from Thanksgiving or Christmas with the results you get during January’s sales slump.<\/p>\n It\u2019s also important to pay attention to external factors that might affect your split testing results. If you\u2019re marketing locally and natural disasters cause disruptions in people\u2019s lives, you won\u2019t get the results you expect.<\/p>\n Similarly, winter-related offers just won\u2019t have the same impact during summer months, according to\u00a0Small Business Sense<\/a>.<\/p>\n In any case, you shouldn\u2019t compare apples to oranges if you want reliable results. Instead, run your tests for comparable periods so you can accurately assess whether any change yields different outcomes.<\/p>\n You need to run an A\/B test for a certain amount of time to achieve statistical significance.<\/p>\n As an example, you should aim to get a 95% confidence rating in your results as an industry standard. That means you can be sure about accurate results and base your decision-making on the findings.<\/p>\n The test duration can vary depending on the number of variants and the expected conversions. For instance, if you\u2019re running 2 variants and expect 50 conversions, your testing period will be shorter than if you have 4 variants and are looking for 200 conversions.<\/p>\n Want to check if you have achieved statistical significance in your A\/B tests? Head over to\u00a0Visual Website Optimizer (VWO)<\/a>\u2019s Statistical Significance Calculator page.<\/p>\n The page allows you to enter the number of visitors and conversions for your test so that it can calculate the statistical significance.<\/p>\n When you\u2019re running split tests with OptinMonster, your campaign\u2019s timing can affect the success of your tests. One of the frequent mistakes I see people make in their tests is that they compare 2 different campaigns with different time delays.<\/p>\n For example, if a popup campaign appears after visitors have been on a landing page for 5 seconds, and another shows up after 20 seconds, that\u2019s not a fair comparison.<\/p>\n That\u2019s because you\u2019re not comparing similar audiences. Typically, a lot more people will spend 5 seconds on a page than 20 seconds.<\/p>\n As a result, you\u2019ll see different impressions for each campaign. And the results won\u2019t make sense or be useful to you.<\/p>\n For a true split test, you have to change 1 item on the page, not the timing. But if you want to experiment with timing your optins, this article on popups, welcome gates, and slide-in campaigns<\/a>\u00a0has some great suggestions.<\/p>\n To get meaningful results out of your A\/B tests, it\u2019s important that your site drives a healthy amount of traffic to its pages.<\/p>\n If you have a high-traffic site, you can complete split tests relatively fast because of the constant flow of visitors to your site.<\/p>\n But if you have a low-traffic site or sporadic visits, you\u2019ll need to run the test for longer.<\/p>\n It\u2019s also important to split your traffic the right way so you really are testing like against like. Some\u00a0A\/B testing tools<\/a>\u00a0allow you to manually allocate the traffic you are using for the test.<\/p>\n But it\u2019s much easier to split traffic automatically to avoid the possibility of getting unreliable results from the wrong kind of split.<\/p>\n If you\u2019re using\u00a0OptinMonster\u2019s A\/B testing functionality<\/a>, it\u2019s easy to get this right. That\u2019s because OptinMonster automatically segments your traffic according to the number of tests you\u2019re running:<\/p>\n Testing too soon is a recipe for failure in A\/B testing and can cloud your marketing judgment.<\/p>\n For example, if you start a new OptinMonster campaign, you should wait a bit before starting a split test.<\/p>\n That\u2019s because there\u2019s no point in creating a split test when you don\u2019t have enough data to compare. When you rush to test brand-new pages, you\u2019ll essentially be testing against nothing, which is a waste of time.<\/p>\n As a rule of thumb, run your new campaigns for at least a week and see how they perform before you start tweaking and testing.<\/p>\n Like I mentioned earlier in tip #5, the more conversions you want, the longer your test duration should be.<\/p>\n If you want to mess up your A\/B testing outcome, change your testing parameters or variables in the middle of the experiment.<\/p>\n This mistake usually happens if you:<\/p>\n Sudden changes invalidate your test and skew your results.<\/p>\n If you absolutely need to change something, then start your test again, or create a new one. It\u2019s the best way to get results you can rely on.<\/p>\n Measuring results correctly is as important as testing accurately. Yet, it\u2019s another area for marketers to make costly A\/B testing mistakes.<\/p>\n If you don\u2019t measure results properly, you can\u2019t rely on your data. This means you can\u2019t make data-driven decisions and finetune your marketing campaigns for better results.<\/p>\n One of the best ways to solve this is to ensure that your A\/B testing solution works with\u00a0Google Analytics<\/a>. That way, you\u2019ll have better control over understanding your test results and actionable insights on what to do next.<\/p>\n OptinMonster integrates with Google Analytics, so you can see accurate data about traffic and conversions in your dashboard:<\/p>\n Here\u2019s how you\u00a0integrate Google Analytics with OptinMonster<\/a>\u00a0so you can get real-time, actionable data points. You can also set up your own\u00a0Google Analytics dashboard<\/a>\u00a0to collect campaign data with the rest of your web metrics.<\/p>\n Display rules, in the context of marketing campaigns, are a set of conditions that let you control where and how your campaigns appear to visitors on your site.<\/p>\n OptinMonster has powerful display rules that control which campaigns show when, where, or based on specific user behaviors. These rules can include parameters such as:<\/p>\n Just like you shouldn\u2019t randomly change your testing parameters midway through a test, making arbitrary changes to the display rules can ruin your test outcomes.<\/p>\n Like I said earlier, split tests are about changing one element on the page. If you change the display rules so that one optin shows to people in the UK and another to people in the US, that\u2019s not an apples-to-apples comparison.<\/p>\n If one campaign is a fullscreen welcome mat<\/a>, and another is an Exit-Intent\u00ae campaign, you\u2019re setting up your test for failure. If one campaign shows up at 9 am and the other at 9 pm, you might get false positives that can jeopardize your marketing efforts in the future.<\/p>\n For more information, check out our guide to\u00a0using Display Rules with OptinMonster<\/a>. This will help you set rules for who to target your campaigns at, where, and at what time.<\/p>\n This is one of the silliest A\/B testing mistakes marketers fall for. Many of them test their marketing campaigns on a development site, which is not so bad by itself.<\/p>\n What\u2019s worse is that sometimes they forget to move their chosen campaigns over to the live site and they see that their split tests aren\u2019t working.<\/p>\n This is bad because the only people who visit a development site are web developers, not customers. Luckily, making the switch is an easy fix. If you\u2019re not seeing positive results, it\u2019s worth checking for this issue.<\/p>\n If you\u2019re using OptinMonster, here\u2019s how you fix this issue:<\/p>\n Step 1:\u00a0<\/strong>Log in to the OptinMonster dashboard, and click on your profile icon at the top-right of the screen. From the dropdown menu, click on\u00a0My Account<\/strong>:<\/p>\n Go to\u00a0Sites<\/strong>\u00a0from the top menu:<\/p>\n Choose the site you want to change and click\u00a0Edit<\/strong>:<\/p>\n Change the website URL from the dev site to the live site:<\/p>\n Once you\u2019re done, scroll down to the page\u2019s bottom to save your changes:<\/p>\n And that\u2019s it!<\/p>\n To learn more, read our help doc on\u00a0how to add, delete, or edit a website in OptinMonster<\/a>.<\/p>\n Some brands keep testing their sites, which means they are always running conversion rate optimization (CRO) tests to improve their site\u2019s user experience (UX). Others, not so much.<\/p>\n One of the A\/B testing mistakes you must avoid is ending your test prematurely. For example, don\u2019t stop your test before the 1-week time window that I mentioned earlier. You should also not stop a test halfway through assuming it has failed or if it\u2019s not giving you the desired results.<\/p>\n Quitting on a test too soon is a terrible error because it won\u2019t get optimization benefits.<\/p>\n It\u2019s important to know that there\u2019s no such thing as a failed split test because the goal of a test is to gather data.<\/p>\n You can learn something from an experiment even if it doesn\u2019t yield the outcomes you expected. Plus, the data you\u2019ll discover might help you fine-tune your next experiment.<\/p>\n You should also go with what the data says rather than relying on your gut feeling, also known as confirmation bias. You can always run a new test at the end of the testing period to see if different changes will achieve the results that you hoped for.<\/p>\n Also, don\u2019t stop the test abruptly if you haven\u2019t had enough time to get a decent sample size or achieve statistical significance and a 95% confidence rating. Otherwise, you\u2019ll waste your time.<\/p>\n It\u2019s great to\u00a0read case studies<\/a> and learn about the split testing techniques that have worked for different companies.<\/p>\n But one A\/B testing mistake you must avoid is copying what worked for others.<\/p>\n If that seems counterintuitive, hear me out.<\/p>\n It\u2019s fine to use case studies to get ideas for how and what to test. But be aware that what worked for another business might not work for yours, because every business is unique.<\/p>\n Instead, use A\/B testing case studies as a point of reference for creating your own A\/B testing strategy. That\u2019ll let you see what works best for your own target audience, not someone else\u2019s.<\/p>\n Now you know the most common A\/B testing mistakes that can waste your time and marketing efforts, you can get your split testing off to a good start.<\/p>\n If you liked this post, you might be interested in more resources related to A\/B testing:<\/p>\n OptinMonster makes it really easy to split test your marketing campaigns so you can get better results.<\/p>\n13 Most Common A\/B Testing Mistakes You Should Avoid<\/h2>\n
\n
1. Split Testing the Wrong Page<\/h3>\n
\n
2. Having an Invalid Hypothesis<\/h3>\n
\n
\n
3. Split Testing Too Many Items<\/h3>\n
![\"a\/b](\"https:\/\/optinmonster.com\/wp-content\/uploads\/2017\/09\/mistakes_multi-min.png\")
<\/p>\n4. Running Too Many Split Tests at Once<\/h3>\n
5. Getting the Timing Wrong<\/h3>\n
Comparing Different Time Periods<\/h4>\n
Not Running a Test for Long Enough<\/h4>\n
Testing Different Time Delays<\/h4>\n
6. Working With the Wrong Traffic<\/h3>\n
![\"split](\"https:\/\/optinmonster.com\/wp-content\/uploads\/2017\/09\/mistakes_edit-min.png\")
<\/p>\n7. Testing Too Early<\/h3>\n
8. Changing Parameters Mid-Test<\/h3>\n
\n
9. Measuring Results Inaccurately<\/h3>\n
![\"common](\"https:\/\/optinmonster.com\/wp-content\/uploads\/2017\/09\/mistakes_analytics-min.png\")
<\/p>\n10. Using Different Display Rules<\/h3>\n
\n
11. Running Tests on the Wrong Site<\/h3>\n
![\"common](\"https:\/\/optinmonster.com\/wp-content\/uploads\/2017\/09\/mistakes_change-1-min.png\")
<\/p>\n![\"top](\"https:\/\/optinmonster.com\/wp-content\/uploads\/2017\/09\/mistakes_change-2-min.png\")
<\/p>\n![\"top](\"https:\/\/optinmonster.com\/wp-content\/uploads\/2017\/09\/mistakes_change-3-min.png\")
<\/p>\n![\"mistakes](\"https:\/\/optinmonster.com\/wp-content\/uploads\/2017\/09\/mistakes_change-4-min.png\")
<\/p>\n![\"avoid](\"https:\/\/optinmonster.com\/wp-content\/uploads\/2017\/09\/mistakes_change-5-min.png\")
<\/p>\n12. Giving Up on Split Testing<\/h3>\n
13. Blindly Following Split Testing Case Studies<\/h3>\n
Put Your A\/B Tests to Good Use<\/h2>\n
\n