If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. |
|
|
Thread Tools | Display Modes |
#1
|
|||
|
|||
subtraction returning incorrect calue
We have a spreadsheet that subtracts one number from another. We did not
understand why Excel is rounding the way it did so we decided to add decimal places until we see the value change. For instance, one of our number was going out to x,xxx.499999999999999999999999999999900000 while the other number was x,xxx.5000000000000000000000000000000000. When we rounded the numbers, one rounded up while the other rounded down. This is fine. Now that we fixed the formula to TRUNC instead of ROUND, I thought the issue would go away but now when I subtracted two numbers that were like x,xxx.xx000000000000000000000000 and another number that is also x,xxx.xx000000000000000000000000, why would I receive an answer that comes back as x,xxx.xx4999999999900000000000000000? This doesn't make sense to me. Has anyone else run into this issue? How did you fix it? -- Ashish |
Thread Tools | |
Display Modes | |
|
|