HELP! We really need this kind of criticism
From pb's comments on Kuro5hin (http://www.kuro5hin.org/story/2003/7/29/17659/1721) :
- I'd like to mention that just doing 10 runs isn't really going to generate reliable data. From a little ad-hoc experimentation, I can tell you that doing 100 runs could get you to about 2% of doing 1000 runs, which should be a lot closer to the actual values. However, only doing 10 runs could give you margins of error of 40% or so, easily.
- You could account for this by noting how much the data varies between runs, and doing more runs until the average stabilizes. Or you could do the math, and try to compute a confidence interval of some sort.
Accusations of Bias Damaging the Project
Other Methodological Problems
Back to WeNeedYourCriticism