Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Join us on March 30, 2023, between 5 am & 1 pm EST, in the Performance Profiling Dev Blitz Day 2023 - Q&A forum and Discord where you can connect with our teams behind the Memory and CPU Profilers.
    Dismiss Notice

Showcase Floating point Inaccuracy in a surprise location

Discussion in 'Scripting' started by LethalGenes, Mar 18, 2023.

  1. LethalGenes


    Jan 31, 2023

    It is found Inside the mesh of the unity editors plane object, all values but 1 return whole ;) That is index 6,

    A nice bit of a find here if I say so myself. What we do here is get the component mesh filters plane. Get vertices, and then iterate through the verts and ask to list the different X values of their positions and sort that list. The result is always 0.9999999 so it has something to do with the model itself.

    I suppose it could be considered a bug.
  2. orionsyndrome


    May 4, 2014
    Nice catch, but here's a fun fact: if this was in meters, this discrepancy would be 10 times smaller than the width of the most common bacteria (1-2 microns). I'm sure Unity team has better things to do :)
    Bunny83 likes this.
  3. Luxxuor


    Jul 18, 2019
  4. orionsyndrome


    May 4, 2014
    Actually, integer 1 is perfectly representable with IEEE-754 as it is equivalent to 0x3f800000 in memory.
    This issue has nothing to do with the fp arithmetics, and is indeed wrongly set in the mesh itself, but as I said, this is not to be seen as error, though it does offend OCD which is common among devs.

    More info:
    The eponymous number of that URL is represented with 0x3e99999a * and you can see that the repeating 9 gets abruptly truncated in the end, and this is where that rounding error comes from. 0.3 is simply irrational in this format, and can't be sanitized in a way that is compatible with the decimal system without truncation or rounding off. This is not the case with 1.

    * edit: btw I'm not even sure how is that site getting that value, because it's very easy to verify that .1f + .2f is approximately 0.300000011921 (or 0x3e99999a) in C#.
    Code (csharp):
    1. Console.WriteLine($"{.2f + .1f:F16}"); // 0.300000011921
    And even if you get 0.300000041723 (0x3e99999b) that's still far less zeros in between.
    (This is in agreement with multiple 32-bit IEEE-754 sites I've tested.)
    Last edited: Mar 19, 2023
    Bunny83 likes this.
  5. Bunny83


    Oct 18, 2010
    That page assumes double precision floats which is often the standard in many programming languages like Javascript and even in C#. The value
    is a double precision float. See this site. Decreasing the mantissa by 1 (changing the least significant bit) would bring you below 0.3 and the next larger value is

    Have a look at the following values. Those are the next larger and the next lower values of 0.1, 0.2 and 0.3. The issue with this classic FP arithmetic issue is that the closest number to 0.1 is slightly over 0.1. The same is true for 0.2. The error is smallest when we pick the slightly larger values. Even though 0.2999999.... would be closer to 3 that the result, adding 0.1 and 0.2 them would give you the number
    that's slightly closer to 0.300000000000000044409. Actually just by a single bit closer ^^. That's why the result is not the optimal approximation of 0.3

    Code (CSharp):
    1. // 0.1
    2. 0.100000000000000005551  // closest value
    3. 0.0999999999999999916733
    5. // 0.2
    6. 0.200000000000000011102 // closest value
    7. 0.199999999999999983347
    9. // 0.3
    10. 0.300000000000000044409
    11. 0.299999999999999988898 // closest value
    orionsyndrome likes this.
  6. orionsyndrome


    May 4, 2014
    @Bunny83 I considered it was a double, however the examples (on that site) clearly show single precision. Originally, I wanted to make my point across that it was a double, but then I also couldn't recreate the value at all. Probably I did something wrong, but hey, at least the value is explained (it definitely looks like a double precision error).