Why I don’t trust Optimizely (…blindly)
Let me start by saying that I really do like Optimizely. I have tried out quite a few testing solutions in my day and Optimizely is definitely one of the more pleasant ones out there. Quick to set up, easy to use. Just lovely. So this isn’t so much a knock against Optimizely, but rather a call to all conversion rate optimizers to use their instincts as they review the test results. Growth Hacking builds on past lessons. If your foundation isn’t solid, you’re asking for trouble.
Testing solutions don’t know your customers
Most packages will encourage you to push a variation live once it feels the test has reached significance — but remember, it doesn’t know the context of your business. The testing platform doesn’t have your instinct. It doesn’t know if your test has run for at least a buying cycle or not (tip: you should run your tests for at least one buying cycle — and long enough to rule out any wonky seasonality effects like day-of-the-week stuff).
Testing solutions won’t tell when you’re being stubborn
We’ve all been there. We have all created variations that we thought would be slam dunks and they don’t work out as well as we predicted. So what do we do? We let the test run longer. And I think that’s OK — to a point. Remember, we are growth hacking. Results matter. Impact matters above all else. If it takes longer than 2 or so weeks to see significance then you will have to accept one of 2 scenarios:
- Your variations don’t have any meaningful differences. I.e., your challenger made no difference.
- Your variation isn’t impactful enough. It may be that you will reach significance eventually in 3 months — but who has time for that? We need results now.
Either way, go back to the drawing board right away.
Finally, check your math
Beyond this, check your testing solution’s math. It’s just good practice. For simple A/B split-tests, it’s pretty straight-forward. For a great primer, check out this awesome post by Jason Cohen.
If you’re impatient like me, and you don’t feel like reading all that, I’ve created a simple online calculator (it’s nothing fancy but it gets the job done).
Just input the number of conversions per variation into the green boxes and boom, it tells you if you have a significant result. Note that you need to have at least 5 conversions for each variation.
Happy Hacking!
Originally published at www.linkedin.com
