Knowing whether a result is real

A sample tells you about the population only if every member had a known chance of being chosen. Start from a list of the whole population — the census, a register, a map of every field — and choose by random numbers, not by judgement or convenience.

Volunteers, people who answer the phone, or the plots nearest the road differ from the rest in ways you cannot measure, and no sample size cures that: a large biased sample is just confidently wrong. Record how many of those chosen could not be reached or refused, because they may differ too.

An estimate from a sample comes with a margin of error. For a proportion p from a random sample of n, the 95 % confidence interval is about ±1.96 × √(p × (1 − p) ÷ n):

| Sample size | Margin of error at p = 50 % | |---|---| | 100 | ±9.8 points | | 400 | ±4.9 points | | 1,000 | ±3.1 points | | 4,000 | ±1.5 points |

Quadrupling the sample only halves the margin. The interval covers chance alone; it says nothing about a biased list, refusals or badly worded questions, which must be reported separately.

Decide in advance the smallest effect worth knowing about, and calculate how many units are needed to detect it with high probability. A study too small for its question will usually miss a real effect, and its "no difference" means only that it could not tell.

If the calculation shows the study cannot be big enough, change the design, pool with other sites, or do not run it — a study that cannot answer its question wastes the participants' time.

Write down and publish, before collecting data: the question, the main outcome, the sample size, and exactly how the data will be analysed. Otherwise there are always more ways to slice the data — another outcome, a subgroup, a different test — and one of them will look significant by chance.

Medicine has made this a rule: journals require clinical trials to be registered in a public registry at or before the time of first patient enrolment as a condition of considering them for publication (ICMJE). Report the registered main outcome first, and label everything else as exploratory.

A p-value is the probability of data at least as extreme as those observed, if a stated model — usually "no effect" — were true. The American Statistical Association's six principles (ASA, 2016):

1. P-values can indicate how incompatible the data are with a specified statistical model.
2. They do not measure the probability that the hypothesis studied is true, or that the data were produced by chance alone.
3. Conclusions and decisions should not be based only on whether a p-value passes a threshold.
4. Proper inference requires full reporting and transparency.
5. A p-value does not measure the size of an effect or the importance of a result.
6. By itself, a p-value is not a good measure of evidence about a model or hypothesis.

So report the size of the effect with its confidence interval, and decide on what that size would mean in practice.

At the usual 5 % threshold, one test in twenty comes out "significant" by chance when there is no effect at all. Run twenty comparisons and expect one false finding.

Report how many tests were run, not only those that came out. Where many are run, tighten the threshold — the simplest correction divides it by the number of tests, so twenty tests at 5 % each need p below 0.0025 — or treat the results as leads to test again in a new study.

Two things that rise together may both be caused by a third. Assign treatments at random wherever possible, as controlled comparison requires: randomisation balances the unknown causes between the groups.

Where you cannot randomise — the effects of smoking, pollution or income — measure the likely common causes, adjust for them, and say plainly what could not be controlled. Beware regression to the mean: places or people picked because they scored worst will tend to score better next time with no intervention at all, so compare with a similar group left untreated.

A finding is not established until others have repeated it. In 2015 a collaboration repeated 100 studies from three leading psychology journals: 97 % of the originals had reported significant results, and 36 % of the replications did (Open Science Collaboration).

Publish the data and the analysis code with the paper, publish negative results through the journal, and base policy on findings replicated by independent teams.