Controlled comparison and the negative result
Establish how you decide whether something is actually true.
Everything for this step
- Field guide
- The sciences → 15 sections
- Builds
- None yet.
- Reference sheets
- None.
- Sciences it rests on
- Statistics
- Track
- Science and measurement, level 1 of 3 · next: Priority by publication
Why this matters
Craft knowledge accumulates by survival: what worked got repeated. That is powerful and it is also how a settlement acquires confident, expensive, wrong beliefs — because a practice that appears to work is indistinguishable from one that works, unless you compare it against not doing it. The method is the machinery for telling those apart: change one thing, hold the rest, compare against a control, and write down the result when it fails. The negative result is the part that gets dropped first and matters most, because the cost of not recording a failure is that everyone repeats it.
The shape of it
- Change one variable at a time. Two changes and a good outcome tells you nothing about which change caused it.
- Always run the control. The comparison against doing nothing, or doing it the old way, is the entire content of the method. Almost every discredited traditional remedy passed the test of "we did it and the patient recovered".
- Write down what you expected before you look. Recording the prediction first is what stops a result being retro-fitted into a success.
- Record failures with the same care as successes, in the same book, and never remove them. A record of only what worked is a sales document.
- Record the conditions, not just the outcome — quantities, times, temperatures, weather, who did it. A result you cannot reproduce is an anecdote, and the missing detail is usually a condition nobody thought worth writing.
- Repeat before believing. One result is a hypothesis. The same result from someone else, working from your written conditions, is a finding — which is the point at which it is ready for publication.
Doing it, step by step
The summary above says what has to happen. This says how, with the numbers that decide whether it works.
1 · Write the question and the prediction before you start
In the bound notebook, dated, before any work: what question you are testing, what you expect to happen, and what result would show you were wrong.
Recording the prediction first is what stops a result being retro-fitted into a success. A result that could not have come out any other way under your prediction tests nothing.
2 · Change one variable, and list what you hold fixed
Two changes and a good outcome tells you nothing about which change caused it.
Write down the one thing being changed, and list everything held the same: materials and their source, quantities, times, temperatures, the person doing it, the tools, the place. The list is also your checklist: if anything on it varied, record that it did.
3 · Run a control, side by side, at the same time
The comparison against doing nothing, or doing it the old way, is the entire content of the method. Almost every discredited traditional remedy passed the test of we did it and the patient recovered.
Run the control at the same time, in the same place, under the same conditions as the treatment — not last year's field against this year's. Weather, soil, season and people change, and a comparison across time mixes all of them into the result.
4 · Replicate, assign by lot, and judge blind where you can
- Replicate: several plots, batches or patients per group, at least four or five where you can. One plot of each tells you about that plot.
- Assign by lot: decide which gets the treatment by drawing lots or tossing a coin, not by choice. People unconsciously give the treatment to the promising cases.
- Judge blind: wherever a person judges the outcome — a taste, a symptom, a quality grade — they should not know which is which.
5 · Record conditions, outcomes and failures in a bound book
A bound book with numbered pages, in ink, never torn out. Record quantities, times, temperatures, weather, materials and who did it, then the outcome as a measurement — weights, counts, lengths — rather than an impression.
Record failures with the same care as successes, in the same book, and never remove them. A record of only what worked is a sales document, and the cost of not recording a failure is that everyone repeats it.
6 · Compare the difference with the scatter
A difference between groups means something only if it is large compared with the differences within each group.
Write down every replicate's result. If the treated plots range from 40 to 60 and the controls from 38 to 58, you have found nothing — the groups overlap almost entirely. If every treated plot beats every control, that is worth pursuing. When in doubt, repeat with more replicates rather than argue.
7 · Repeat before believing, then have someone else repeat it
One result is a hypothesis. Repeat it yourself. Then give someone else only your written record and have them repeat it. Believing a single result is the most common and most expensive error on this ladder.
When a second person, working from the page alone, gets the same result, it is a finding — and ready for publication. If they cannot, the missing detail is usually a condition nobody thought worth writing.