Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Deleted

Discussion in 'Scripting' started by JWLewis777, Sep 14, 2018.

  1. JWLewis777

    JWLewis777

    Joined:
    May 17, 2015
    Posts:
    46
    Post Deleted
     
    Last edited: Feb 23, 2022
  2. lordofduct

    lordofduct

    Joined:
    Oct 3, 2011
    Posts:
    8,514
    It's called floating point precision error. There are 8 bajillion articles on it on the internet:
    https://www.google.com/search?q=flo.....69i57j0l5.2309j0j7&sourceid=chrome&ie=UTF-8

    It's an inherent issue of trying to represent decimal values in binary with a finite amount of precision.

    Values that look otherwise normal in decimal, are actually repeating fractions in binary.

    Think like how 1/3 is 0.3333333333333... in decimal. Well in base3, it's just 0.1

    Well 0.1 in decimal is 0.0001100110011001100110011.... in binary.

    And 0.01 in decimal is 0.000000101000111101011100001... in binary.

    Since a float only has 24 bits of precision, it has to lop off the end making the value = to just shy of the actual value. Usually rounding tricks resolve this, but as you compound them repeatedly (by looping and adding over and over), the error becomes more pronounced.
     
    Ryiah likes this.
  3. lordofduct

    lordofduct

    Joined:
    Oct 3, 2011
    Posts:
    8,514
    If you specifically need it a certain number of digits when you 'ToString' it for display purposes.

    You can just pass in a format string which will do the precision rounding you want.

    Something like:
    Code (csharp):
    1.  
    2. someTextDisplay.text = a.ToString("0.00");
    3.  
    This breaks down the custom format string syntax:
    https://docs.microsoft.com/en-us/dotnet/standard/base-types/custom-numeric-format-strings

    And these are the standard built in ones:
    https://docs.microsoft.com/en-us/dotnet/standard/base-types/standard-numeric-format-strings
     
    Ryiah likes this.