So this graph is pretty famous, especially in quality circles. It’s meant to convey that fixing things earlier is better – by orders of magnitude. It’s a pretty stark visual of something we feel is intrinsically true.
Only problem is, it’s bullshit.
This specific visual is from The Journal Of Information Systems Technology And Planning by Dawson et al, and they reference it from another source, saying “A study was performed by the IBM System Science Institute”. There’s no citation for this study in the paper. Googling around, there’s very few references to the IBM System Science Institute at all. What’s going on?
This is not the first paper to use this 1/6.5/15/100x data, and Morendil on github has done the deep dive, but it turns out it was probably someone’s gut feel for a course taught internally at IBM. Not a study, not a published paper, just numbers pulled out of someone’s ass. Through a game of telephone and poor scholarship it’s become quoted as absolute truth.
So is there any real evidence for this?
Menzies et al did a study in 2016 that seems to be the only real attempt to figure out if delaying fixes is more costly (they call it the Delayed Issue Effect, DIE) with real data, and they find no conclusive evidence.
However they were working with projects with median duration of 60 days and programs only 4,000 lines long. It’s not surprising that any DIE wouldn’t manifest itself at this scale, so I don’t think it’s conclusive.
We might never have good evidence here! Large-scale computer science research seems like a really hard problem to crack. The data just isn’t shared from commercial development. But that doesn’t mean we should perpetuate bad graphs just because they correlate with our gut feel.
If anyone has any better data on the Delayed Issue Effect, let me know!
Andrew Fray 