When I was a grad student, we all knew that publishable results had to be statistically significant. One day a fellow student asked me a question about experiments he was running, and I was dumbfounded to realize he didn’t understand how statistical significance works. He didn’t grasp that if his program worked then running it on enough cases would eventually establish that it worked and give a significant result. To me it was obvious that if you sample the random variable more times you get more information about it, but all he knew about was this magic formula where you plugged your numbers in and prayed for favorable signs. The ignorance of knowledgeable people!
If he didn’t get that, imagine the difficulty of explaining that, no, it’s not valid statistical practice to stop at 34 cases because that’s the first time your formula says you’ve got statistical significance. The mystic ritual is pretty complicated!
Originally written 2003.
Updated and added here March 2011.