Why “Statistical Significance” Is Pointless | by Samuele Mazzanti | Dec, 2024
Right here’s a greater framework for data-driven decision-making
Knowledge scientists are within the enterprise of decision-making. Our work is targeted on make knowledgeable selections underneath uncertainty.
And but, relating to quantifying that uncertainty, we regularly lean on the concept of “statistical significance” — a software that, at greatest, gives a shallow understanding.
On this article, we’ll discover why “statistical significance” is flawed: arbitrary thresholds, a false sense of certainty, and a failure to handle real-world trade-offs.
Most vital, we’ll discover ways to transfer past the binary mindset of great vs. non-significant, and undertake a decision-making framework grounded in financial impression and danger administration.
Think about we simply ran an A/B check to guage a brand new function designed to spice up the time customers spend on our web site — and, consequently, their spending.
The management group consisted of 5,000 customers, and the therapy group included one other 5,000 customers. This provides us two arrays, named therapy and management, every of them containing 5,000 values representing the spending of particular person customers of their respective teams.
