Search Unity

Resolved (Case 1333309) [Netcode][Entities][IL2CPP] IL2CPP Android build failed to spawn ghost and client dc

Discussion in 'NetCode for ECS' started by optimise, May 2, 2021.

  1. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    2,129
    Hi @timjohansson, @CMarastoni. This is shipstopper bug I found for netcode when rolling out Android il2cpp build. I hope next version of Netcode package will fix the bug.

    When roll out Android build with IL2CPP, it will fail to spawn ghost properly and client disconnect but mono build is working properly.

    Currently due to case 1329969 [IL2CPP][Entities] Android build crash (https://issuetracker.unity3d.com/is...68.1431120914.1604680124-350356135.1604680124), only Android il2cpp develop build able to run the game.
     
    optimise, May 2, 2021
    #1

  2. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    894
    Hi optimise, thanks for reporting.
    What NetCode version are you using? 0.6?
     
    CMarastoni, May 3, 2021
    #2

  3. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    2,129
    @CMarastoni Yes. 0.6.0-preview.7.
     
    optimise, May 3, 2021
    #3

  4. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    2,129
    Hi @CMarastoni. Any new progress for this issue? I would like to get unofficial fix first since the release date of next dots packages still unknown and my project deadline is until end of May.
     
    optimise, May 7, 2021
    #4

  5. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    894
    Hi Optimise, not yet unfortunately. We will dig into it as soon as possible, but nothing has been planned at the moment
     
    CMarastoni, May 7, 2021
    #5
    optimise likes this.

  6. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    2,129
    @CMarastoni. Any new updates?
     
    optimise, May 10, 2021
    #6

  7. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    894
    For what I know, 1329969 is still open and a fix working in progress. A part of the problem has been fixed with Burst 1.5.1 but there is another one that is still open:
    https://issuetracker.unity3d.com/is...brid-renderer-v2-and-building-a-release-build
    I don't know honestly when it will be fixed. So it will still be develop build

    For your specific issue I need a little more context in order to be able to understand the problem.
    Do you mean the client fail to resolve the GhostCollection and so it auto-disconnect from the game?
    Or that you got some Invalid RPC error?
    Do you have a sample that give the same problem ?

    That may speed up a little the investigation.
     
    CMarastoni, May 10, 2021
    #7

  8. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    2,129
    @CMarastoni. When press button to spawn hero ghost, it will spawn properly at both Android Mono build and editor but fail at Android il2cpp develop build. For Android il2cpp develop build, I see server is successfully receive rpc to spawn hero ghost but seems like it's just immediately despawn and log Client Disconnected reason from NetworkStreamDisconnected component. I'm not understand why.
     
    Last edited: May 10, 2021
    optimise, May 10, 2021
    #8

  9. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    2,129
    Hi @CMarastoni what the current progress for this case? Did someone look at the bug report? There is zero update since 2 May the date I submit bug report.
     
    optimise, May 12, 2021
    #9

  10. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    894
    I started looking at that problem this week, but without using your project at first. I was trying to repro the issue using IL2CPP and a very simple sample at the beginning, in order to isolate the issue.
    I will try with your project too of course.
     
    CMarastoni, May 13, 2021
    #10
    optimise likes this.

  11. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    2,129
    @CMarastoni I found that issue is caused by il2cpp client connects to mono server or vice versa. But before that it's working properly. I also test it with Netcube sample and it's working properly. Maybe il2cpp generates slightly different code than mono or Burst generates different native at both il2cpp and mono I dunno. Currently I need to have il2cpp Android client able connects to mono linux server and work properly since il2cpp linux server failed to work properly.
     
    optimise, May 16, 2021
    #11

  12. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    894
    I tried myself with netcube and another sample with the lates netcode (just to verify the status there) using IL2CPP and trying to connect it to the server running in the editor (so it was mono) and I had no issue.
    I also tried with a standalone server build (mono) and mono client using asteroids. I will try also the a IL2CPP client and standalone server.

    The first thing that comes to my head now: in 0.6-preview.7 there was an issue with IL2CPP due to the fact that when code stripping is enabled, the ClientServerBootstrap or any inherited class, is stripped from the executable, causing various issues at runtime.
    Did you enable code stripping in your builds ?
    Looking at your project I saw code stripping is enabled in general. I also saw you have a custom bootstrap there (that it is actually empty). Could you please try to add a [UnityEngine.Scripting.Preserve] attribute to it ? So we can removed that issue from the possible causes.
     
    CMarastoni, May 18, 2021
    #12

  13. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    2,129
    I tried code stripping disable before but still get the same result. Actually I put custom empty bootstrap just to avoid being stripped from il2cpp build suggested by @timjohansson. Do u mean put Preserve attribute at custom empty bootstrap? If yes, I will try that.

    But still before that somehow il2cpp Android build is able to connect to Editor(mono) as server successfully. I'm not sure now why cannot. Seems like il2cpp build is not deterministic. And also il2cpp Android build definitely able to connect to il2cpp server successfully. I wonder whether it's really has something being strip that it's still able to connect to il2cpp server successfully.

    Edit:
    Add Preserve attribute at custom empty bootstrap still get the result that il2cpp Android client failed to connect to Editor server. I even untick strip engine code and set code stripping level to disabled still get the same result.

    Edit:
    Today I get something like that. Client il2cpp android connect to Editor as server. Any idea? I guess it's caused by il2cpp not really good at converting generic method.

    New Crash Or Exception Received Native Crash - EntityQueryImpl_GetSingleton_TisCommandTargetComponent_t44163459D9F8B74B20537C43399BD913F707DACD_m96730BAE3E81B16EB4A1954D0970FA614884C313_gshared (C:\Users\xx\Documents\UnityTest2\GameProject\Library\Il2cppBuildCache\Android\arm64-v8a\il2cppOutput\GenericMethods37.cpp:41787)
     
    Last edited: May 19, 2021
    optimise, May 18, 2021
    #13

  14. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    894
    Ok, so you got IL2CPP connect to the server in the editor. Did you do something particular or just added the preserve attribute or you disabled stripping? (or anything of these)
    The crash is actually pretty bad. Question being: why? It might be just an exception being thrown (so there is not singleton and you are trying to to get one)
     
    CMarastoni, May 19, 2021
    #14

  15. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    2,129
    No. I mean I must use mono android build only can connect to editor server or mono server build. Il2cpp android only can connect to il2cpp server build.
     
    optimise, May 19, 2021
    #15

  16. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    894
    Ah ok, I misunderstood your previous message.
     
    CMarastoni, May 19, 2021
    #16

  17. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    2,129
    Hi @CMarastoni. I think I found the root cause. It's caused by different hash. Seems like when u build il2cpp client/server and mono client/server u will get different hash. Can u give me unofficial solution fix for this issue that will generate same hash for both il2cpp and mono build?

    Edit: Now seems like getting even worst. Mono client failed to connect to Mono Server.


    Received a ghost from the server which has a different hash on the client
    UnityEngine.Debug:LogError (object)
    Unity.NetCode.GhostCollectionSystem:OnUpdate () (at Library/PackageCache/com.unity.netcode@0.6.0-preview.7/Runtime/Snapshot/GhostCollectionSystem.cs:336)
    Unity.Entities.SystemBase:Update () (at Library/PackageCache/com.unity.entities@97663dff0d/Unity.Entities/SystemBase.cs:400)
    Unity.Entities.ComponentSystemGroup:UpdateAllSystems () (at Library/PackageCache/com.unity.entities@97663dff0d/Unity.Entities/ComponentSystemGroup.cs:472)
    Unity.Entities.ComponentSystemGroup:OnUpdate () (at Library/PackageCache/com.unity.entities@97663dff0d/Unity.Entities/ComponentSystemGroup.cs:417)
    Unity.Entities.ComponentSystem:Update () (at Library/PackageCache/com.unity.entities@97663dff0d/Unity.Entities/ComponentSystem.cs:114)
    Unity.Entities.ComponentSystemGroup:UpdateAllSystems () (at Library/PackageCache/com.unity.entities@97663dff0d/Unity.Entities/ComponentSystemGroup.cs:472)
    Unity.Entities.ComponentSystemGroup:OnUpdate () (at Library/PackageCache/com.unity.entities@97663dff0d/Unity.Entities/ComponentSystemGroup.cs:417)
    Unity.NetCode.ClientSimulationSystemGroup:OnUpdate () (at Library/PackageCache/com.unity.netcode@0.6.0-preview.7/Runtime/ClientServerWorld/ClientSimulationSystemGroup.cs:90)
    Unity.Entities.ComponentSystem:Update () (at Library/PackageCache/com.unity.entities@97663dff0d/Unity.Entities/ComponentSystem.cs:114)
    Unity.Entities.ComponentSystemGroup:UpdateAllSystems () (at Library/PackageCache/com.unity.entities@97663dff0d/Unity.Entities/ComponentSystemGroup.cs:472)
    Unity.Entities.ComponentSystemGroup:OnUpdate () (at Library/PackageCache/com.unity.entities@97663dff0d/Unity.Entities/ComponentSystemGroup.cs:417)
    Unity.Entities.ComponentSystem:Update () (at Library/PackageCache/com.unity.entities@97663dff0d/Unity.Entities/ComponentSystem.cs:114)
    Unity.Entities.ComponentSystemGroup:UpdateAllSystems () (at Library/PackageCache/com.unity.entities@97663dff0d/Unity.Entities/ComponentSystemGroup.cs:472)
    Unity.Entities.ComponentSystemGroup:OnUpdate () (at Library/PackageCache/com.unity.entities@97663dff0d/Unity.Entities/ComponentSystemGroup.cs:417)
    Unity.Entities.ComponentSystem:Update () (at Library/PackageCache/com.unity.entities@97663dff0d/Unity.Entities/ComponentSystem.cs:114)
    Unity.Entities.ScriptBehaviourUpdateOrder/DummyDelegateWrapper:TriggerUpdate () (at Library/PackageCache/com.unity.entities@97663dff0d/Unity.Entities/ScriptBehaviourUpdateOrder.cs:333)
     
    Last edited: May 22, 2021
    optimise, May 21, 2021
    #17

  18. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,759
    On type hashes? Yeah we have the same experience (we're not using Netcode, just entities).

    We just build IL2CPP all platforms and hide editor games from builds and vice versa to avoid accidental connections.
     
    tertle, May 21, 2021
    #18

  19. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    894
    Hi Optimise,
    I was going to write that to you, since this is what I founded as well. We recently fixed a problem in that regards, that was causing non deterministic order or components, but it hasn't been back ported to 0.6-preview.7 for what I saw (it is very recent).
    I attached a file you need to change in the netcode package. You will need to copy the package in the Packages folder for that.
    Please replace the com.unity.netcode/Runtime/Authoring/Hybrid/GhostAuthoringComponent.cs with the one attached here.
    I wil test as well if that fix the problem or if there is something else.
     

    Attached Files:

    CMarastoni, May 23, 2021
    #19
    optimise likes this.

  20. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    2,129
    @CMarastoni Awesome. I will try it now. Btw can u help me ask Entities DOTS developer about the new patch fix for Entities 0.17 for my case 1279727? The current patch fix is not working properly and now enter play mode I see nothing. Sorry for asking at here but recently I can't find any way to contact them.

    upload_2021-5-23_16-51-20.png
     
    Last edited: May 23, 2021
    optimise, May 23, 2021
    #20

  21. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    894
    I can confirm that with that fix the game now correctly connect and sync with the server and don't get disconnected anymore.
    For the DOTS issue you need to talk with them. I can't help on that.
     
    CMarastoni, May 23, 2021
    #21
    optimise likes this.

  22. Occuros

    Occuros

    Joined:
    Sep 4, 2018
    Posts:
    300
    We are using Netcode 0.51, and just now we started to see this error again:

    Received a ghost from server which has a different hash on the client.

    It started happening when we defined a different default variant for a build-in component (PhysicsVelocity).

    Is this a new bug (which we should report) or has the fix not been implemented in 0.51?
     
    Occuros, Jul 15, 2022
    #22

  23. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    894
    The last fix we did for hash calculation was in 0.50. So looks like a new bug, I need more information to understand what it is going on.

    Did you just created the variant in your project or did you assign some ghosts using this variant?

    Also, because now you have two variants in the project, I'm expecting a warning or a log that says you need to assign the one to use as default. Did you also get that?

    But forgetting about the warning, in 0.51 we didn't print what ghost is and what hashes we expected. Can you try to change the line in the GhostCollectionSystem that print the error with something like:

    Code (csharp):
    1.  
    2. $"Received a ghost - {ghostMetaData.Name.ToString()} - from the server which has a different hash on the client (got {ghost.Hash} but expected {hash})"
    3.  
    That at least will give you an hint witch ghost it is. Then from here, the things to understand is what makes the calculation mismatch.
    And for that you need to dig a little bit in the GhostCollectionSystem.AddComponents, at the end, we calculate the ghost type hash by adding together a lot of small pieces.
    If you know the ghost prefab that have the problem, you can print (only for that one) all the partial hash computation and checking which components serializers has been assigned, along with the used variants.

    In any case, please report a bug and if you can, please provide a repro for that on smaller sample for 0.51. Be very exhaustive about what are you doing in order to see the issue, in case a small sample is not enough to repro it.
     
    CMarastoni, Jul 16, 2022
    #23

  24. Occuros

    Occuros

    Joined:
    Sep 4, 2018
    Posts:
    300
    Thank you, we managed to track it down.

    It seems that didn't have anything to do with NetCode directly, but rather with the build system. We saw that the ghosts which were problematic shouldn't even exist in the scene we were building.

    We used the Build Current Scene and it seems the wrong scene was then built on the server (despite the scene being open during building).

    Specifically defining which scenes to build (with Scene Infos and adding them as elements) has resolved the issue.
     
    Occuros, Jul 16, 2022
    #24

  25. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    894
    Oh, great news. If it something generic enough, would you mind share more about what the problem was with the build configuration so if other users jump in the same problem, they will know how to solve it? (if it make sense of course).
     
    CMarastoni, Jul 17, 2022
    #25

  26. Occuros

    Occuros

    Joined:
    Sep 4, 2018
    Posts:
    300
    We had some libraries which seemed problematic for the Linux builds, after we added them to the Filter Settings, the builds didn't hang anymore in the Incremental stage.

    The build process has been a little unstable for us up to now, but if we discover any more hints of what might cause issues I will post it!.
     
    Occuros, Jul 18, 2022
    #26
    CMarastoni likes this.

  27. Occuros

    Occuros

    Joined:
    Sep 4, 2018
    Posts:
    300
    After a week of testing with different configurations and countless builds, we decided to downgrade from the latest LTS (2021.3.5/6) to version 2020.3.36.

    2021.3.x produces inconsistent build results (issues streaming the scene on the server or client), with the exact same configuration and which produces a valid build on 2020.3.x, 2020.3.x has up to now never produced an invalid build.

    So we would recommend now to NOT use 2021.3.x and stay on 2020.3.x at the current time.
     
    Occuros, Jul 23, 2022
    #27

  28. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    894
    This is really important. I appreciate the suggestion, and please file a bug report with as much details as possible.
     
    CMarastoni, Jul 24, 2022
    #28