A Microsoft Office (Excel, Word) forum. OfficeFrustration

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

Go Back   Home » OfficeFrustration forum » Microsoft Excel » Worksheet Functions
Site Map Home Register Authors List Search Today's Posts Mark Forums Read  

Fill/Extend Series adding decimal points to some numbers



 
 
Thread Tools Display Modes
  #1  
Old April 26th, 2010, 04:44 PM posted to microsoft.public.excel.worksheet.functions
awalker
external usenet poster
 
Posts: 15
Default Fill/Extend Series adding decimal points to some numbers

Hi,
I set up a worksheet for a Lookup Function that contains numbers increasing
by 0.1 from 100.0 to 50.0. I used the drag and fill to set up the numbers.
I noticed some incorrect entries in the results from the look up and found
that on certain numbers it added many decimal spots to the number for example
66.5 look like 66.5000000000001 when I click into the cell. I can not locate
a pattern for why this is occurring (i.e. does not occur in the 4-8 or 10-15
but does in the 9 and 16 series). Formatting is set to numerical with 1
decimal point.

Any help?

  #2  
Old April 26th, 2010, 05:06 PM posted to microsoft.public.excel.worksheet.functions
Pete_UK
external usenet poster
 
Posts: 8,780
Default Fill/Extend Series adding decimal points to some numbers

This is a limitation of any computer that uses binary representation
of decimal numbers - some fractions cannot be represented with
complete accuracy. A binary fraction is made up of digits to represent
1/2, 1/4, 1/8, 1/16 etc, but some simple decimal fractions (like 1/10
or 0.1) cannot be represented completely in binary as they are
recurring numbers (rather like 1/3 or 2/3 in decimal).

Another thing to consider is that Excel can only represent up to 15
digits of precision, so a recurring fraction will be truncated to this
precision and when it is added repeatedly then errors will accumulate.

I'm sure Joe will come along soon to give a more in-depth explanation.

Hope this helps.

Pete

On Apr 26, 4:44*pm, awalker wrote:
Hi,
I set up a worksheet for a Lookup Function that contains numbers increasing
by 0.1 from 100.0 to 50.0. I used the drag and fill to set up the numbers.. * *
I noticed some incorrect entries in the results from the look up and found
that on certain numbers it added many decimal spots to the number for example
66.5 look like 66.5000000000001 when I click into the cell. *I can not locate
a pattern for why this is occurring (i.e. does not occur in the 4-8 or 10-15
but does in the 9 and 16 series). *Formatting is set to numerical with 1
decimal point. *

Any help?


 




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump


All times are GMT +1. The time now is 10:47 AM.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 OfficeFrustration.
The comments are property of their posters.