JiscMail Logo
Email discussion lists for the UK Education and Research communities

Help for EXCEL-VB-DISCUSS Archives


EXCEL-VB-DISCUSS Archives

EXCEL-VB-DISCUSS Archives


EXCEL-VB-DISCUSS@JISCMAIL.AC.UK


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

EXCEL-VB-DISCUSS Home

EXCEL-VB-DISCUSS Home

EXCEL-VB-DISCUSS  July 2011

EXCEL-VB-DISCUSS July 2011

Options

Subscribe or Unsubscribe

Subscribe or Unsubscribe

Log In

Log In

Get Password

Get Password

Subject:

Re: Accuracy of Variables

From:

Doug McNutt <[log in to unmask]>

Reply-To:

Excel Visual Basic for Applications.

Date:

Wed, 27 Jul 2011 10:54:46 -0600

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (34 lines)

At 17:15 +0100 7/27/11, James Button wrote:
>If you are calculating using Float, then fractional parts are held as negative powers of 2
>If you want to put the value as a decimal value into a cell, then you need to convert that value before you put it into the cell
>as in decimal_or Currency_variable = round_float
> 
>I also noted (Well Patrick O'Beirne pointed me to my incorrect assumption) that the problem persists with Trunc, and can then confound all by having the value as displayed rounded if the display format decimal place setting is less than the value.
> 
>BASIC approach -
> 
>Keep (Long)Integer, Decimal/currency and float number holding within Excel (and VBA) separate
> 
>Also - if the value may be float (as in the result of a division) then do not test for = value, but test for
>ABS(variable-testvalue)<0.0000001  (With enough zeros for whatever accuracy you want/need/can get away with!)
> 
>JimB
>
>----- From Original Message -----
>From: <mailto:[log in to unmask]>Fen Scott
>To: <mailto:[log in to unmask]>[log in to unmask]
>Sent: Wednesday, July 27, 2011 5:00 PM
>Subject: Accuracy of Variables
>
>
>Now  Im getting some issues with accuracy. As an example, when TotVA = 2 and CountVA = 10, AveVA is calculated as 0.2 (break mode lets me see this), but when it writes the info to the cell it writes 0.200000002980232


It looks as though you're dealing with 9 significant digits though it might be more if Excel VBA converts numerator and denominator to double or extended floats before division.

There is a really fundamental limit to double floats at approximately 14 decimal digits which comes up when the values are truncated to 52 bits in accordance with the IEEE floating point format. Some fractions - negative powers of 2 and sums of them - can be exact. Otherwise they are repeating fractions like 1/3 = .33333..... in decimal. 1/10 (10) = .00011001100,,,forever (base 2) is a real PITA because of its wide use in currency.

Intermediate results probably have an extra 16 bits as type extended but they cannot be brought back into a worksheet for formatting by Excel.
-- 

--> From the U S of A, the only socialist country that refuses to admit it. <--

Top of Message | Previous Page | Permalink

JISCMail Tools


RSS Feeds and Sharing


Advanced Options


Archives

February 2012
November 2011
October 2011
September 2011
July 2011
June 2011
April 2011
March 2011
February 2011
August 2010
June 2010
May 2010
January 2010
November 2009
September 2009
August 2009
July 2009
January 2009
September 2008
August 2008
July 2008
June 2008
May 2008
March 2008
February 2008
January 2008
November 2007
October 2007
September 2007
August 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
January 2007
December 2006
November 2006
October 2006
September 2006
August 2006
July 2006
June 2006
May 2006
April 2006
March 2006
February 2006
January 2006
December 2005
November 2005
October 2005
September 2005
August 2005
July 2005
May 2005
April 2005
March 2005
February 2005
January 2005
December 2004
November 2004
October 2004
September 2004
August 2004
July 2004
June 2004
May 2004
April 2004
March 2004
February 2004
January 2004
December 2003
November 2003
October 2003
September 2003
August 2003
July 2003
June 2003
May 2003
April 2003
March 2003
February 2003
January 2003
December 2002
November 2002
October 2002
September 2002
August 2002
July 2002
June 2002
May 2002
April 2002
March 2002
February 2002
January 2002
December 2001
November 2001
October 2001
September 2001
August 2001
July 2001
June 2001
May 2001
April 2001
March 2001
February 2001
January 2001
December 2000
November 2000
October 2000
September 2000
August 2000
July 2000
June 2000
May 2000
April 2000
March 2000
February 2000
January 2000
December 1999
November 1999
October 1999
September 1999
August 1999
July 1999
June 1999
May 1999
April 1999
March 1999
February 1999
January 1999
December 1998
November 1998
October 1998
September 1998
August 1998
July 1998
June 1998
May 1998
April 1998
March 1998


WWW.JISCMAIL.AC.UK

Secured by F-Secure Anti-Virus CataList Email List Search Powered by the LISTSERV Email List Manager