Search Unity

Burst JIT Compile Times

Discussion in 'Data Oriented Technology Stack' started by threedots1, Aug 30, 2019.

  1. threedots1

    threedots1

    Joined:
    Oct 9, 2014
    Posts:
    26
    As our project is getting more complicated with more Burst jobs involved our time spent JIT compiling on each project run in the editor is increasing.

    We are using async compilation where possible, but for some of the more complicated jobs we have to force synchronous compilation or suffer a very very long first iteration in a ParallelFor loop (5s non burst vs around 0.006s once compiled!).

    As far as I can tell from the debugger however a lot of time is spent JIT compiling the jobs with no other work happening, this is sitting at about 30-40% of our time while entering play mode.

    Are there any ways to offload this work from the main thread?

    Is it possible to cache the compilation between runs, only recompiling if that job has changed?

    Forgive my probably stupid questions.
     
    Razmot and Xilinnilix like this.
  2. Lee_Hammerton

    Lee_Hammerton

    Unity Technologies

    Joined:
    Jul 26, 2018
    Posts:
    34
    Could you check it against the burst preview that just dropped today (1.1.3-preview.1), this contains a fix to the caching of assemblies, which should improve your experience. Burst has a set of caches in place at this time, The first level will check if the assembly has been modified and if not, the previously compiled version will be used. The second level will compare the IL, and if the IL hasn't changed, then it will use the previously compiled version.

    Note that at present these caches are reset if you close unity.
     
    SugoiDev and GilCat like this.
  3. Carpet_Head

    Carpet_Head

    Joined:
    Nov 27, 2014
    Posts:
    103
    That's great to hear!
     
  4. threedots1

    threedots1

    Joined:
    Oct 9, 2014
    Posts:
    26
    Thanks Lee and team. This has halved the time it takes to start our project. Much much much better.
     
    sheredom and fabrizio_unity like this.
  5. fabrizio_unity

    fabrizio_unity

    Unity Technologies

    Joined:
    May 3, 2018
    Posts:
    15
    For future readers, the package version publicly available is 1.1.3-preview.3 (unfortunately preview.1 is not available).
     
  6. threedots1

    threedots1

    Joined:
    Oct 9, 2014
    Posts:
    26
    While this has been working fantastic up until now, unfortunately I've just wasted a couple of hours trying to figure out why things were working fine when I had a job Burst compiled, and went to crap as soon as I enabled it.

    For some reason after updating the code it's not being detected as having changed, and therefore not recompiling and I'm stuck with older code while using Burst.

    The only way so far I've figured out how to fix it is to restart Unity.

    The job in question is a fairly large pathfinding job which may have something to do with it.
    Can't seem to find where the logs are for this to post though.
     
  7. tim_jones

    tim_jones

    Unity Technologies

    Joined:
    May 2, 2019
    Posts:
    19
    @threedots1 sorry to hear it's not working correctly. It sounds something like something is going wrong around cache invalidation. Please could you submit a bug report via the Unity Editor, using Help > Report a Bug?
     
  8. Lee_Hammerton

    Lee_Hammerton

    Unity Technologies

    Joined:
    Jul 26, 2018
    Posts:
    34
    I`m sorry to hear that, a bug did indeed sneak into the assembly caching which could result in the behavior you are describing, there is a newer preview package (1.2.0-preview.1) which contains a fix for exactly the issue described, which I believe should be out later today. If this doesn't correct the behavior, then please do file a report as Tim suggest.
     
  9. fabrizio_unity

    fabrizio_unity

    Unity Technologies

    Joined:
    May 3, 2018
    Posts:
    15
    Hello,
    we just published a new version of burst: the version is `1.2.0-preview.1`. Can you please try this one out and let us know if it fixes the issue?
    Thank you
     
  10. MostHated

    MostHated

    Joined:
    Nov 29, 2015
    Posts:
    809
    These are grayed out for me in the package manager. Am I supposed to change it manually in the manifest?
     
  11. threedots1

    threedots1

    Joined:
    Oct 9, 2014
    Posts:
    26
    Thanks for the quick response.

    Updated to the latest Burst package, but unfortunately I still have the same problem. I can reproduce it by simply commenting out some code with in a job with [BurstCompile].

    The code will continue to run on next launch as if unchanged, and will not compile correctly regardless of how many times the game is started. If I comment out [BurstCompile] the code executes correctly. If I renable [BurstCompile] and run again the code is now compiled correctly.

    I was running on 2019.3.0b1 initially, just updated to 2019.3.0.b2 and have the same issue.
     
  12. MostHated

    MostHated

    Joined:
    Nov 29, 2015
    Posts:
    809
    I had that happen to me the other day (2 days ago I believe). I could not figure out what the deal was. I ended up removing the file from my project, compiling, adding it back in and then it seemed ok. I thought I was losing my mind for a bit.
     
  13. tim_jones

    tim_jones

    Unity Technologies

    Joined:
    May 2, 2019
    Posts:
    19
    @threedots1 so just to confirm, you're using Burst 1.2.0-preview.1 and you're still getting the issue? That version did contain a fix specifically for cache invalidation, so I was hoping it would fix your issue. But if not, please could you submit a bug report via the Unity Editor, using Help > Report a Bug? And then please reply here with the bug number so I can follow it up.
     
  14. threedots1

    threedots1

    Joined:
    Oct 9, 2014
    Posts:
    26
    @tim_jones yeah that's correct.

    I've just submitted a bug report now. It's taking a while to pack and upload so I'll get back to you with the bug number in the morning.
     
  15. MostHated

    MostHated

    Joined:
    Nov 29, 2015
    Posts:
    809
    I just installed and tried this with 2019.3.0b3 using 1.2-preview.1 (was fine on 0b2) and I get this each time I try to play.

    Code (CSharp):
    1.  
    2. Caught fatal signal - signo:11 code:2 errno:0 addr:0x7fe24c4e5000
    3. Obtained 13 stack frames.
    4. #0  0x007fe20c09d0bd in (Unknown)
    5. #1  0x00000041026954 in Burst.Compiler.IL.Jit.JitCompiler:CompileMethod (Mono.Cecil.MethodReference,Burst.Compiler.IL.Jit.JitOptions)
    6. #2  0x00000040f56740 in Burst.Compiler.IL.Jit.JitCompilerService:Compile (Burst.Compiler.IL.Jit.JitCompilerService/CompileJob)
    7. #3  0x00000040ea4760 in Burst.Compiler.IL.Jit.JitCompilerService:CompilerThreadLoop ()
    8. #4  0x00000040ea3e36 in System.Threading.ThreadHelper:ThreadStart_Context (object)
    9. #5  0x00000040ea2c34 in System.Threading.ExecutionContext:Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
    10. #6  0x007fe26c0da8d9 in mono_print_method_from_ip
    11. #7  0x007fe26c246def in mono_perfcounter_foreach
    12. #8  0x007fe26c260a81 in mono_threads_set_shutting_down
    13. #9  0x007fe26c2c20d7 in GC_inner_start_routine
    14. #10 0x007fe26c2b6e42 in GC_call_with_stack_base
    15. #11 0x007fe452697182 in start_thread
    16. #12 0x007fe4525c0b1f in clone
    17. Launching bug reporter
    18. QObject: Cannot create children for a parent that is in a different thread.
    19. (Parent is QNetworkAccessManager(0x1a7c270), parent's thread is QThread(0x196a060), current thread is Thread(0x1a20a70)
    20. [0910/172216:ERROR:browser_main_loop.cc(207)] GLib-GObject: invalid unclassed pointer in cast to 'GtkWidget'
    21. [0910/172216:ERROR:browser_main_loop.cc(207)] Gtk: gtk_widget_set_sensitive: assertion 'GTK_IS_WIDGET (widget)' failed
    22. [0910/172231:ERROR:browser_main_loop.cc(207)] GLib-GObject: invalid unclassed pointer in cast to 'GtkWidget'
    23. [0910/172231:ERROR:browser_main_loop.cc(207)] Gtk: gtk_widget_set_sensitive: assertion 'GTK_IS_WIDGET (widget)' failed
    24. [0910/172233:ERROR:browser_main_loop.cc(207)] GLib-GObject: invalid unclassed pointer in cast to 'GtkWidget'
    25. [0910/172233:ERROR:browser_main_loop.cc(207)] Gtk: gtk_widget_set_sensitive: assertion 'GTK_IS_WIDGET (widget)' failed
    26. [Unity Package Manager (Upm)]
    27. Parent process [17864] was terminated
    28.  
     
  16. threedots1

    threedots1

    Joined:
    Oct 9, 2014
    Posts:
    26
    I've tried to upload a bug report three times now and just as it has finished I've gotten a bad gateway error. I'll try again this afternoon but it looks like the bug reporting system is down.
     
  17. MostHated

    MostHated

    Joined:
    Nov 29, 2015
    Posts:
    809
    I keep getting that with the forum, too. I keep having to refresh a few times.

    (I think I accidentally hit report on your post instead of reply! D : ! If an admin/mod reads this and I did report him, that was my bad! I just hit the button and started typing my reply without paying much attention)
     
  18. tim_jones

    tim_jones

    Unity Technologies

    Joined:
    May 2, 2019
    Posts:
    19
    @threedots1 that doesn't sound right - we'll have a look in the bug reporter logs and see if we can track down what happened. Please could you DM me with whatever details you can about the bug report, such as email address you used, report title, exact time, etc.
     
  19. tim_jones

    tim_jones

    Unity Technologies

    Joined:
    May 2, 2019
    Posts:
    19
    @MostHated do you mean you're getting an error with Report a Bug, or with the forum?
     
  20. MostHated

    MostHated

    Joined:
    Nov 29, 2015
    Posts:
    809
    No, that was just running my project with burst 1.2-preview.1. I was going to submit a bug report when I get home.
     
  21. threedots1

    threedots1

    Joined:
    Oct 9, 2014
    Posts:
    26
    I finally got the bug report uploaded @tim_jones

    Kept getting gateway errors, until yesterday where it finally uploaded properly but I never got a confirmation email... it looks like it just disappeared.

    Tried again this morning and worked fine, got the email in 1 min.

    Case number is 1184669.
     
    Lee_Hammerton likes this.