Pair Programming - Empirical Studies

Empirical Studies

The Economist noted, "Laurie Williams of the University of Utah in Salt Lake City has shown that paired programmers are only 15% slower than two independent individual programmers, but produce 15% fewer bugs. Since testing and debugging are often many times more costly than initial programming, this is an impressive result." (Note: The original study showed that "error-free" code went from 70% to 85%; it may be more intuitive to call this a 50% (rather than 15%) decrease of errors, from 30% to 15%.)

The Williams et al. 2000 study showed an improvement in correctness of around 15% and a 20%–40% decrease in time, but between a 15% and 60% increase in effort—that is, total programmer-hours. Williams et al. 2000 also cites an earlier study (Nosek 1998) which also had a 40% decrease in time for a 60% increase in effort.

A study (Lui 2006) presents a rigorous scientific experiment in which novice–novice pairs against novice solos experience significantly greater productivity gains than expert–expert pairs against expert solos.

A larger recent study (Arisholm et al. 2007) had 48% increase in correctness for complex systems, but no significant difference in time, whilst simple systems had 20% decrease in time, but no significant difference in correctness. Overall there was no general reduction in time or increase in correctness, but an overall 84% increase in effort.

Lui, Chan, and Nosek (2008) shows that pair programming outperforms for design tasks.

A full-scale meta-analysis of pair programming experimental studies, from before or during 2007, (Hannay et al. 2009) confirms "that you cannot expect faster and better and cheaper". Higher quality for complex tasks costs higher effort, reduced duration for simpler tasks comes with noticeably lower quality – the meta-analysis "suggests that pair programming is not uniformly beneficial or effective".

However, a 2007 meta-analysis concluded that "pair programming is not uniformly beneficial or effective" because many other factors besides the choice of whether to use pair programming have large effects on the outcome of a programming task. The meta-study found that pair programming tends to reduce development time somewhat and produces marginal positive effects on code quality, but that pair programming requires significantly more developer effort; that is, it is significantly more expensive than solo programming. The authors suggest that studies of pair programming suffer from publication bias whereby studies that would not show that pair programming is beneficial were either not undertaken, not submitted for publication, or not accepted for publication. They conclude that "you cannot expect faster and better and cheaper."

This study suggests that even though coding is often completed faster than when one programmer works alone, the total amount of man-hours (number of programmers × time spent) increases. A manager needs to balance faster completion of the work and reduced testing and debugging time against the higher cost of coding. The relative weight of these factors can vary from project to project and task to task. The benefit of pairing is greatest on tasks that the programmers do not fully understand before they begin: that is, challenging tasks that call for creativity and sophistication. On simple tasks, which the pair already fully understands, pairing results in a net drop in productivity. Productivity can also drop when novice-novice pairing is used without sufficient availability of a mentor to coach them.