Statistics is the most misunderstood tool in business today.
This misunderstanding is causing businesses to make poor decisions and to miss opportunities. You can reduce this risk in your own decision-making by improving your sense of when a statistic is unreliable.
A quick check of the sample size is often enough to head off trouble. We use statistics to make decisions. When we rely on false statistics, we make poor decisions.
A false statistic is one that claims to be representative of the situation, but is not.
Where do false statistics come from?
Very often they come from samples that are too small to be reliable.
I once had my sales director explain the basis for his opinion that specific sales reps were underperforming. He explained that he had randomly distributed 100 qualified leads to each rep and then measured their conversion rates after a suitable period of time. He now intended to fire the lowest performers based on this test.
The problem was that 100 leads were far too few to take the measurements he was interested in. Given that the average conversion rate was around 7 percent, much of the variation in the conversion rate that he saw among the reps based on only 100 leads was just random — it was not telling us enough about which reps were better.
When we used a sample-size calculator to find the minimum number of leads he needed to give each rep, we found that we needed to give each rep 700 leads. So we did that.
The conversion rates based on the 700 leads per rep told a very different story than the sales director saw in the first 100 leads. In fact, if he had made decisions based on his initial sample size of 100, he mistakenly would have fired many above-average sales reps — including two of our top 10. This single mistake would have cost us about $1 million in lost revenue that year.
How big is big enough?
How do you know the sample size required to derive a reliable statistic? You use a sample-size calculator. They are free online, like the one I sometimes use from Raosoft. You answer a few questions about how accurate you need the statistic to be and the calculator tells you what your sample size needs to be.
If you are going to make an important decision based in part on a statistic, ask about the sample size. If your experience is anything like mine, prepare to be horrified. Most people will reveal that they used that number because it was an easy number of data points to get, not because it was enough.
After the near miss with my sales director, I taught a short class on how to use a sample-size calculator and then required my team to use it before they greenlighted a test.
Determining the required sample size upfront allowed them to better estimate the cost and duration of a proposed test. But more importantly, it let them avoid coming to conclusions that were wrong. ●
Jerry McLaughlin is founder and CEO at Blow Birthday Cards.
