Test to disprove

To find out whether you are right, include tests that would prove you wrong, not only ones that would confirm you.

People test ideas with cases they expect to fit. In Wason’s 2-4-6 task (1960), people were told that 2, 4, 6 fits a rule and asked to find it by trying their own triples. Most guessed “numbers going up by 2” and tried 8, 10, 12 and 20, 22, 24. Every test got a yes, so they grew sure, and announced the wrong rule. The real rule was “any three numbers going up”. Only about one in five found it at the first try.

Their tests were not wrong; they were weak. A yes to a case your idea predicts fits many other ideas too. The informative test is one your idea says should fail, such as 1, 2, 3 or 6, 4, 2. If it fails, your idea survives a real test; if it passes, you have learnt you were wrong, cheaply.

The habit of looking for and weighing evidence that agrees with what you already think is called confirmation bias (Nickerson 1998). The answer to it is a question: what would change my mind? Then go and look for that.

Why it matters

Evidence that fits your idea usually fits many other ideas too, so it cannot tell them apart. Only a test that could fail can show you are wrong before it costs you.

Away from the game

Before backing a plan, ask what result would show it is wrong, and go and look for that result.