Spreadsheets and precision?

Jerry Feldman gaf at blu.org
Sun Sep 9 15:08:56 EDT 2007


On Sun, 9 Sep 2007 14:01:56 -0400
"Bill Ricker" <bill.n1vux at gmail.com> wrote:

> > Oh.  I do see that now that I look.  This strikes me as completely
> > counter-intuitive.  So, how to people do actual division in
> > spreadsheets?  Does Excel suffer from this as well?
> 
> Yes.
> 
> Every spreadsheet since VisiCalc has done floating point division and
> other basic numeric formuli in formulaic form as
> 
>   =3329.10/10
> 
> and used NamedFunctions() only for things without conventional (to
> business folks) infix notation.
> 
> VisiCalc and Lotus 1-2-3 allowed
>    +3329.10/10
> and might figure out that
>      3329.10/10
> was NOT a date and work, but OO.o.C and "Excel" insist on = prefix for
> any non-constant formula

Just to elaborate, floating point (doubles) uses 1 sign bit, 11
exponent bits and 53 mantissa bits. Note that the high order mantissa
bit is hidden except in denormals which is why 1 + 11 + 53 = 65. The
exponent is based on power of 2 which is why you might see a lot of 9
digits. You can use the format cell menu item to expand the visible
precision, but you are limited about 17 digits. Some chips use 80 bits
in their coprocessor. 
-- 
Jerry Feldman <gaf at blu.org>
Boston Linux and Unix user group
http://www.blu.org PGP key id:C5061EA9
PGP Key fingerprint:053C 73EC 3AC1 5C44 3E14 9245 FB00 3ED5 C506 1EA9
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mail.gnhlug.org/mailman/private/gnhlug-discuss/attachments/20070909/f43c99a1/attachment.bin 


More information about the gnhlug-discuss mailing list