Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

Burst silently failing to create static readonly field

Discussion in 'Data Oriented Technology Stack' started by cjddmut, Oct 10, 2019 at 1:27 AM.

  1. cjddmut

    cjddmut

    Joined:
    Nov 19, 2012
    Posts:
    96
    The short of it is Burst appears to fail to properly initialize static readonly structs that reference other structs that are created with implicit casts. Not only fails but fails silently and leaves the values uninitialized. Below is a simple code sample that shows the failure.


    Code (CSharp):
    1. using Unity.Burst;
    2. using Unity.Collections;
    3. using Unity.Jobs;
    4. using UnityEngine;
    5.  
    6. public class TestStaticReadonly : MonoBehaviour
    7. {
    8.     // This value eq
    9.     private static readonly CustomStruct READONLY_FIELD = CustomStruct.VALUE;
    10.  
    11.     private struct CustomStruct
    12.     {
    13.         // Specifically appears to have to do with the implicit cast!
    14.         public static readonly CustomStruct VALUE = 5;
    15.    
    16.         // Below works just fine!
    17.         // public static readonly CustomStruct VALUE = new CustomStruct(5);
    18.    
    19.         public int internalValue;
    20.  
    21.         public CustomStruct(int internalValue)
    22.         {
    23.             this.internalValue = internalValue;
    24.         }
    25.  
    26.         public static implicit operator CustomStruct(int value)
    27.         {
    28.             return new CustomStruct(value);
    29.         }
    30.     }
    31.  
    32.     private struct Job : IJob
    33.     {
    34.         public NativeArray<int> result;
    35.    
    36.         public void Execute()
    37.         {
    38.             result[0] = READONLY_FIELD.internalValue;
    39.         }
    40.     }
    41.  
    42.     [BurstCompile]
    43.     private struct BurstJob : IJob
    44.     {
    45.         public NativeArray<int> result;
    46.    
    47.         public void Execute()
    48.         {
    49.             result[0] = READONLY_FIELD.internalValue;
    50.         }
    51.     }
    52.  
    53.     private void Update()
    54.     {
    55.         NativeArray<int> result = new NativeArray<int>(1, Allocator.TempJob);
    56.    
    57.         Job job = new Job();
    58.         job.result = result;
    59.    
    60.         job.Schedule().Complete();
    61.    
    62.         // Prints 5
    63.         Debug.Log("Regular Job: " + result[0]);
    64.  
    65.         result[0] = 0;
    66.    
    67.         BurstJob burstJob = new BurstJob();
    68.         burstJob.result = result;
    69.    
    70.         burstJob.Schedule().Complete();
    71.    
    72.         // Prints 0 but we're expecting 5
    73.         Debug.Log("Burst Job: " + result[0]);
    74.    
    75.         result.Dispose();
    76.     }
    77. }
    upload_2019-10-9_17-25-39.png

    Is this a known issue? The work around appears to not do implicit casts which I can probably find all instances and resolved but it'll be a VERY easy miss. I do have safety checks on and I put on synchronous compiling cause otherwise the burst job DOES return 5 until the compiling is done.

    I'm currently on Unity 2019.3.0b5

    *EDIT* The issue appears to be more nuanced. Removing the implicit cast in my project code still left a static readonly uninitialized.

    Thanks!
     
    Last edited: Oct 10, 2019 at 1:43 AM
  2. sheredom

    sheredom

    Unity Technologies

    Joined:
    Jul 15, 2019
    Posts:
    26
    Can you confirm what version of Burst you are using?
     
  3. sheredom

    sheredom

    Unity Technologies

    Joined:
    Jul 15, 2019
    Posts:
    26
    I've tested this locally on the latest Burst preview and it works as expected (I even tried to really confuse the compiler by making the constructee a struct, and it worked correctly!):

    Code (CSharp):
    1. private struct Constructee
    2. {
    3.     public int a;
    4.     public double b;
    5. }
    6.  
    7. private struct ImplicitConstructor
    8. {
    9.     public static readonly ImplicitConstructor ImplicitlyConstructed = new Constructee { a = 42, b = 13 };
    10.  
    11.     public int a;
    12.     public double b;
    13.  
    14.     public ImplicitConstructor(int a, double b)
    15.     {
    16.         this.a = a;
    17.         this.b = b;
    18.     }
    19.  
    20.     public static implicit operator ImplicitConstructor(Constructee x)
    21.     {
    22.         return new ImplicitConstructor(x.a, x.b);
    23.     }
    24. }
    25.  
    26. public static double TestImplicitConstructor()
    27. {
    28.     return ImplicitConstructor.ImplicitlyConstructed.a + ImplicitConstructor.ImplicitlyConstructed.b;
    29. }
    I fixed a bunch of readonly static problems in 1.1.3-preview.1, so if you update to at least that then it should work for you!
     
  4. cjddmut

    cjddmut

    Joined:
    Nov 19, 2012
    Posts:
    96
    I was on the latest verified (1.1.2)

    Updating to 1.2.0-preview.5 resolved that sample script I supplied (1.1.3-preview.1 did not). However for my actual project 1.2.0-preview.5 introduced a lot of instability with Burst.

    If I use Burst with synchronous compilation then the editor crashes when I hit play (going to the editor log doesn't give any real information).

    If I use Burst with asynchronous compilation then once the jobs finish compiling they throw a bunch of IndexOutOfRange exceptions (I assume with NativeArrays that are used by the job but currently unconfirmed).

    If I don't use Burst then everything works fine.

    Is that version just considered unstable? In that case turning off burst for now until the static readonly version becomes stable is an option.
     
  5. sheredom

    sheredom

    Unity Technologies

    Joined:
    Jul 15, 2019
    Posts:
    26
    Sorry to hear about the instabilities - we've been pushing the compiler as far as we can in the run up to our next release, always a higher risk of instabilities appearing.

    Is there any chance we could get a repro from you for the failures? We really would love to fix any issues you have before we cut the next release :)
     
    Krajca likes this.
  6. xoofx

    xoofx

    Unity Technologies

    Joined:
    Nov 5, 2016
    Posts:
    204
    Is it the only thing that changed in your project (burst upgrade) or did you have other changes? (new packages of other stuffs you are using...etc.)
     
  7. cjddmut

    cjddmut

    Joined:
    Nov 19, 2012
    Posts:
    96
    The only thing I changed in this test was upgrading burst from 1.1.2 to 1.2.0-preview.5.

    I'll see if I can isolate a reproduction and share the script.
     
  8. cjddmut

    cjddmut

    Joined:
    Nov 19, 2012
    Posts:
    96
    I've identified the cause of the issues in my primary project but it fails to cause a problem by itself in a fresh project. Since I've fixed it in the primary project I'm gonna have to move on from trying to get it to fail in the fresh project. However here's what I was doing. I was using unsafe pointers in order to keep a list of sorts. Without burst and before 1.2.0-preview.5 this functioned correctly but it's what is causing the errors in the burst preview. Below is the code that I tried to get to fail in the fresh project.


    Code (CSharp):
    1. using Unity.Burst;
    2. using Unity.Collections;
    3. using Unity.Jobs;
    4. using Unity.Mathematics;
    5. using UnityEngine;
    6. using UnityEngine.Assertions;
    7. using Debug = UnityEngine.Debug;
    8.  
    9. public class TestBurstJobException : MonoBehaviour
    10. {
    11.     internal struct UnsafeEntry
    12.     {
    13.         public byte entryCount;
    14.  
    15.         public int2 entry1;
    16.         public int2 entry2;
    17.         public int2 entry3;
    18.         public int2 entry4;
    19.  
    20.         public unsafe void AddEntry(int2 entry)
    21.         {
    22.             Assert.IsTrue(entryCount < 4);
    23.  
    24.             fixed (UnsafeEntry* pMe = &this)
    25.             {
    26.                 int2* start = (int2*) (((byte*) pMe) + sizeof(byte));
    27.                 *(start + entryCount) = entry;
    28.             }
    29.  
    30.             entryCount++;
    31.         }
    32.  
    33.         public unsafe int2 GetEntry(int index)
    34.         {
    35.             fixed (UnsafeEntry* pMe = &this)
    36.             {
    37.                 int2* start = (int2*) (((byte*) pMe) + sizeof(byte));
    38.                 return *(start + index);
    39.             }
    40.         }
    41.     }
    42.    
    43.     [BurstCompile]
    44.     private struct MyJob : IJob
    45.     {
    46.         public UnsafeEntry entry;
    47.  
    48.         [ReadOnly]
    49.         public NativeArray<int> entries;
    50.  
    51.         public NativeArray<int> result;
    52.  
    53.         public int width;
    54.        
    55.         public void Execute()
    56.         {
    57.             int r = 0;
    58.  
    59.             for (int i = 0; i < entry.entryCount; i++)
    60.             {
    61.                 int2 e = entry.GetEntry(i);
    62.                 r += entries[e.x + e.y * width];
    63.             }
    64.  
    65.             result[0] = r;
    66.         }
    67.     }
    68.  
    69.     private void Update()
    70.     {
    71.         int size = 100;
    72.        
    73.         NativeArray<int> entries = new NativeArray<int>(size * size, Allocator.TempJob);
    74.        
    75.         int2 e1 = new int2(0, 0);
    76.         int2 e2 = new int2(98, 23);
    77.         int2 e3 = new int2(50, 50);
    78.  
    79.         entries[e1.x + e1.y * size] = 1;
    80.         entries[e2.x + e2.y * size] = 2;
    81.         entries[e3.x + e3.y * size] = 3;
    82.        
    83.         UnsafeEntry myS = new UnsafeEntry();
    84.         myS.AddEntry(e1);
    85.         myS.AddEntry(e2);
    86.         myS.AddEntry(e3);
    87.  
    88.         MyJob job = default;
    89.         job.entries = entries;
    90.         job.entry = myS;
    91.         job.width = size;
    92.         job.result = new NativeArray<int>(1, Allocator.TempJob);
    93.        
    94.         job.Schedule().Complete();
    95.        
    96.         Debug.Log(job.result[0]);
    97.        
    98.         job.result.Dispose();
    99.         entries.Dispose();
    100.     }
    101. }
    Oddly enough (to me) the entry struct has bogus data when setting (according to the debugger) but correct data when pulling data out. I don't know why? And don't have the memory tools to dig deeper.
     
  9. tim_jones

    tim_jones

    Unity Technologies

    Joined:
    May 2, 2019
    Posts:
    25
    I'm not sure if this is the actual issue, but I did notice one problem with the test code: because of implicit padding, the entry1 field is not located at offset 1 (i.e. sizeof(byte)). It's actually located at offset 4 (i.e. sizeof(int)).

    As long as every bit of code that writes / reads this does the same thing with pointer calculations, then it should be fine, but it would look wrong in the debugger, and if any code directly accesses the fields then it would break.

    To fix it, replace this:

    Code (CSharp):
    1. int2* start = (int2*) (((byte*) pMe) + sizeof(byte));
    with this:

    Code (CSharp):
    1. int2* start = (int2*) (((byte*) pMe) + sizeof(int));
    or this:

    Code (CSharp):
    1. int2* start = (int2*)(((byte*)pMe) + Marshal.OffsetOf<UnsafeEntry>("entry1").ToInt64());
    If that doesn't fix the issue, please let me know, and we'll keep digging.
     
  10. cjddmut

    cjddmut

    Joined:
    Nov 19, 2012
    Posts:
    96
    Yep see that now (Marsh.OffsetOf is a pretty decent debug function!). I fixed the issue in my primary project by leveraging the UnsafeUtility functions instead of directly doing the pointer math myself and that resolved the Burst exceptions (which I still couldn't get to reproduce in a fresh project).

    Good to note that how the fields are laid out in memory may not necessarily match how are typed out in the struct (and I can imagine some potential frustrating issues using NativeArray.Reinterpret though looks like it does a size safety check). What's the general suggested guidelines there if we do find ourselves needing to play a little closer to the metal? If you're doing some memory specific stuff to use LayoutKind.Explicit and enforce offsets?

    Thanks for the assistance!
     
    tim_jones likes this.
  11. tim_jones

    tim_jones

    Unity Technologies

    Joined:
    May 2, 2019
    Posts:
    25
    LayoutKind.Sequential should be fine, because it does have deterministic rules, but they are not particularly obvious, especially when you take into account nested structs. Even if you use LayoutKind.Explicit, you still have to be aware of *some* of this stuff, because (for example) implicit padding can be inserted at the end of the struct to match the struct's alignment.

    I haven't tried using it with Unity, but ObjectLayoutInspector is pretty cool for peeking at how structs are actually laid out, taking into account implicit padding, packing, etc:
    https://github.com/SergeyTeplyakov/ObjectLayoutInspector#inspecting-a-value-type-layout-at-runtime

    In theory, https://sharplab.io/ can show struct layout, and this is what I've used previously for this exact thing, except for an unfortunate bug where it doesn't actually show intra-field padding in the right place at the moment. I've just logged a bug on the SharpLab repo: https://github.com/ashmind/SharpLab/issues/442. Once that's fixed that will be the tool I recommend for determining struct layout. Update: Already fixed. Here's an example showing the padding, but obviously it will work for more complex structs and nested structs: https://sharplab.io/#v2:C4LgTgrgdgN...DAA6y2AOj4BDFgGsAFFFkB3Tl5+Ni8ASlD9MWNcQyA===
     
    Last edited: Oct 15, 2019 at 9:31 AM
  12. cjddmut

    cjddmut

    Joined:
    Nov 19, 2012
    Posts:
    96
    I'll take a look at those!

    Thanks for the tips :)