Numeric Precision
Despite the use of 15-figure precision, Excel can display many more figures (up to thirty) upon user request. But the displayed figures are not those actually used in its computations, and so, for example, the difference of two numbers may differ from the difference of their displayed values. Although such departures are usually beyond the 15th decimal, exceptions do occur, especially for very large or very small numbers. Serious errors can occur if decisions are made based upon automated comparisons of numbers (for example, using the Excel If function), as equality of two numbers can be unpredictable.
In the figure the fraction 1/9000 is displayed in Excel. Although this number has a decimal representation that is an infinite string of ones, Excel displays only the leading 15 figures. In the second line, the number one is added to the fraction, and again Excel displays only 15 figures. In the third line, one is subtracted from the sum using Excel. Because the sum in the second line has only eleven 1's after the decimal, the difference when 1 is subtracted from this displayed value is three 0's followed by a string of eleven 1's. However, the difference reported by Excel in the third line is three 0's followed by a string of thirteen 1's and two extra erroneous digits. This is because Excel calculates with about half a digit more than it displays.
Excel works with a modified 1985 version of the IEEE 754 specification. Excel's implementation involves conversions between binary and decimal representations, leading to accuracy that is on average better than one would expect from simple fifteen digit precision, but that can be worse. See the main article for details.
Besides accuracy in user computations, the question of accuracy in Excel-provided functions may be raised. Particularly in the arena of statistical functions, Excel has been criticized for sacrificing accuracy for speed of calculation.
As many calculations in Excel are executed using VBA, an additional issue is the accuracy of VBA, which varies with variable type and user-requested precision.
Read more about this topic: Microsoft Excel
Famous quotes containing the word precision:
“We are often struck by the force and precision of style to which hard-working men, unpracticed in writing, easily attain when required to make the effort. As if plainness and vigor and sincerity, the ornaments of style, were better learned on the farm and in the workshop than in the schools. The sentences written by such rude hands are nervous and tough, like hardened thongs, the sinews of the deer, or the roots of the pine.”
—Henry David Thoreau (1817–1862)