Survivorship Bias

November 29, 2024 at 12:00 AM

The heuristic

Survivorship bias happens when we focus only on the "survivors" or successful outcomes of a process, forgetting the countless failures that don’t get reported or remembered.

The real world

During World War II, engineers reinforced planes where bullet holes were most common—until they realized they needed to reinforce the areas where no bullet holes were found. Why? Because planes hit in those areas never made it back. Luckily, we're not dealing with bullet holes...

In that case, we have to ask ourselves: what are our planes?

Simply put: our projects (that make it).

Does this mean everything we've shipped to prod is standing on stilts and not worth anything? Not at all. What it means is that we have to be receptive to the fact that project x wasn't successful because of the stack that was used or the methodologies in the design, but perhaps despite them.

The success of a project often depends on variables we don’t immediately see—luck, timing, the team’s adaptability, or even external factors. Focusing only on the visible traits of successful projects can mislead us into thinking those are the magic ingredients. If we don’t study what didn't work—the projects that didn't make it—we risk reinforcing the wrong areas.

The lesson

The lesson here? Don’t just ask, What made this succeed? Also ask, What might have failed in another context—and how can we learn from that? By looking at both the planes that came back and the ones that didn't, we can make stronger, better-informed decisions.