Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question Failed to decode ghost X of type Y, got A bits, expected B bits

Discussion in 'NetCode for ECS' started by tertle, Jan 2, 2023.

  1. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,732
    I'm trying to write a custom serializer for a 512 byte binary data blob to send via a ghost component but it's causing a spam of unexpected errors. I have this working fine if I just use a DynamicBuffer<byte> of 512 size but I want to make the serialization/deserialization much more efficient.

    If I don't use a custom serializer and just let Netcode set it up on a per field basis everything works fine generating this monstrosity.

    However when I write a custom serializer to do this properly all my ghost packets just break.

    etc

    The code
    I've attached my full serializer (with .cs added to end to allow forum attaching) but the only codegen difference between this and using a FixedString512Bytes is this

    upload_2023-1-2_11-22-1.png

    (The blob is already compressed and is nearly always static so doing a delta compression isn't going to be that beneficial.)

    Ghost stuff looks like this

    Code (CSharp):
    1. [GhostComponent]
    2. internal struct Packet : IComponentData
    3. {
    4.     [GhostField]
    5.     public FixedBytes512 Buffer;
    6. }
    7.  
    8. [Serializable]
    9. public struct FixedBytes512 : IEquatable<FixedBytes512>
    10. {
    11.     public FixedBytes16 offset0000;
    12.     public FixedBytes16 offset0016;
    13.     public FixedBytes16 offset0032;
    14.     public FixedBytes16 offset0048;
    15.     public FixedBytes16 offset0064;
    16.     public FixedBytes16 offset0080;
    17.     // etc
    18.  
    19.     public unsafe bool Equals(FixedBytes512 other)
    20.    {
    21.        fixed (void* ptr = &this)
    22.        {
    23.            return UnsafeUtility.MemCmp(ptr, &other, sizeof(FixedBytes512)) == 0;
    24.        }
    25.    }
    26. }
    27.  
    28. static partial void RegisterTemplates(List<TypeRegistryEntry> templates, string defaultRootPath)
    29. {
    30.     templates.AddRange(new[]
    31.     {
    32.         new TypeRegistryEntry
    33.         {
    34.             Type = "BovineLabs.Netcode.Payload.FixedBytes512",
    35.             Quantized = false,
    36.             Smoothing = SmoothingAction.Clamp,
    37.             SupportCommand = false,
    38.             Composite = false,
    39.             Template = "FixedBytes512Template",
    40.         },
    41.     });
    42. }
    43.  
    Can anyone see some obvious mistake I've made? Any clue what I'm missing here?
    I've tried smaller sizes (128, 256) but same result.
     

    Attached Files:

    Last edited: Jan 2, 2023
  2. NikiWalker

    NikiWalker

    Unity Technologies

    Joined:
    May 18, 2021
    Posts:
    286
    Hey tertle,

    I have a ticket to take a look at support for FixedList "soon", so I'll be able to dig into this then. At first glance, it looks correct. Ideas:

    • You don't write anything in the PredictDelta, which looks like it shouldn't be causing this problem, but I've been bitten too many times by assumptions.
    • Maybe try removing the changeMask logic, and seeing if it works without that.
    • Add a NetCodeDebugConfig component to your sub-scene and enable packet dumps. There is a small chance those give you more context.
    • Does the NetDbg profiler show you what you'd expect?
     
  3. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,732
    Hey Niki,

    I never did solve this so for now I've just let netcode handle it for me however I would like to figure out what's going on so so when I find a moment I'll take a look at your suggestions and see if I can figure it out, thanks!
     
    Opeth001 and NikiWalker like this.
  4. Opeth001

    Opeth001

    Joined:
    Jan 28, 2017
    Posts:
    1,112
    Did you find a solution to this?
    for me this freezes the Unity editor on Mac OS.
    Unity 2022.3.13f1
    Netcode 1.2.0-exp.3

    Code (CSharp):
    1. Failed to decode ghost 7 of type Ability Instance Prefab(1), got 13 bits, expected 11 bits
    2. UnityEngine.Debug:LogError (object)
    3. Unity.NetCode.NetDebug:LogError (Unity.Collections.FixedString512Bytes&) (at ./Library/PackageCache/com.unity.netcode@1.2.0-exp.3/Runtime/Debug/NetDebug.cs:284)
    4. Unity.NetCode.GhostReceiveSystem/ReadStreamJob:DeserializeEntity (Unity.NetCode.NetworkTick,Unity.Collections.DataStreamReader&,Unity.NetCode.GhostReceiveSystem/ReadStreamJob/DeserializeData&) (at ./Library/PackageCache/com.unity.netcode@1.2.0-exp.3/Runtime/Snapshot/GhostReceiveSystem.cs:1262)
    5. Unity.NetCode.GhostReceiveSystem/ReadStreamJob:Execute () (at ./Library/PackageCache/com.unity.netcode@1.2.0-exp.3/Runtime/Snapshot/GhostReceiveSystem.cs:533)
    6. Unity.Jobs.IJobExtensions/JobStruct`1<Unity.NetCode.GhostReceiveSystem/ReadStreamJob>:Execute (Unity.NetCode.GhostReceiveSystem/ReadStreamJob&,intptr,intptr,Unity.Jobs.LowLevel.Unsafe.JobRanges&,int) (at /Users/bokken/build/output/unity/unity/Runtime/Jobs/Managed/IJob.cs:58)
    7.  
     
  5. Opeth001

    Opeth001

    Joined:
    Jan 28, 2017
    Posts:
    1,112
    Additional information: This issue occurs exclusively when the server writes to a dynamically buffer of a ghost owned by a connection, and it happens in the client world only. Btw, the dynamic buffer comprises only 2 elements, each approximately 32 bytes. I hope this issue will be fixed soon as it causes the Editor to freeze, necessitating a restart.
     
  6. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    873
    Just to clarify:
    - It is not mandatory to write thet__GHOST_PREDICT__ region
    - It it not mandator to write the __GHOST_REPORT_PREDICTION_ERROR__ region
    - It it not mandator to write the __GHOST_GET_PREDICTION_ERROR_NAME__ region

    The template code requires (to be able to support ghosts and ony ghosts) the following regions (and only the regions, the other code is completely useless, it there only for indentation and reference)
    - __GHOST_WRITE__
    - __GHOST_READ__
    - __GHOST_COPY_TO_SNAPSHOT__
    - __GHOST_COPY_FROM_SNAPSHOT__
    - __GHOST_RESTORE_FROM_BACKUP__
    - __GHOST_CALCULATE_CHANGE_MASK_ZERO__
    - __GHOST_CALCULATE_CHANGE_MASK__

    The registration of the template itself if correct. It must not support commands, should always use clamp and not quantisation.

    As a reference, in general for writing string-like template, I would check out the GhostValueFixedStringe32Bytes template in general for these and copy the approach (that is pretty much identical). And indeed they looks pretty much identical or very close.

    The template code to me looks actually correct.

    The generated code is in-fact what I would had expected it to be: only these two pieces (read and write) pretty much changed. As well as the SnapshotData struct of course (that is not in the diff but should be alright).

    Now.. I would do an insanely dumb test and verify that using a FixedString512 work. That may be there is just a bug on the generated code (or somewhere else) that is causing the problem in first place. I know there are a bunch of bugs in relation to change masks, if you have > 32 change mask bits, but this seems not be the case. But that should have been already addressed.

    Given also what OpenTh001 said, It could be another issue that is then causing this as side-effect.

    So for debugging: I would start enable a packet dumping and check what the server send (in term of bits) and compare against what the client decode.

    Secondary, step: with a small test (one single ghost):
    - Drag the generated serializer in the assembly so you can modify and debug the generated code on the fly.
    - putting a breakpoint in the GhostReceiveSystem and check what happen when decoding.

    I will try to debug this ASAP for you guys. I smell a regression somewhere. I will comeback asap Turtle. If in the meanwhile you can open a case with a small repro, that would help track the issue correctly.
     
    Opeth001 likes this.
  7. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,732
    That was 10 months ago so would have been 1.0.0-exp.8

    I haven't tried to do this again since the original post.
     
  8. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    873
    No worries, I repro it right now.
     
  9. Opeth001

    Opeth001

    Joined:
    Jan 28, 2017
    Posts:
    1,112
    I identified the issue in my case. The problem was associated with a component within a GhostPrefab. This component had the attributes GhostEnabledBit and GhostComponent, but it did not define any GhostField.
    Thanks and GoodLuck :)
     
  10. CMarastoni

    CMarastoni

    Unity Technologies

    Joined:
    Mar 18, 2020
    Posts:
    873
    So, this is for everyone trying to do the same. The problem is a "bug" in the way the ghost are serialized to the data stream when intermixing value compressed in delta and some not.

    This has to due with the fact the DataStreamReader.WriteBytes DataStreamReader.ReadBytes expect the data aligned to 8 bits (because the memcpy). For such a reason before writing and before reading the bit pointer is aligned to 8 bit boundary (by fusing and skipping data).

    Now, because the ghost are serialised into a temporary stream first, and each component serialised data aligned to 32 bits (for sake of speed etc etc), the result is that the WriteBytes is invoked the data is aligned and no flush happen.

    However, after that temp write, the ghost data that fit into the snapshot are copied bits by bits.. But that break the alignment constraint. Result: the poor decoder fail to read.

    This is why (well actually is a side effect..) we use the WritePackedFixedStringDelta and ReadPackedFixedStringData for that purpose. It is not doing a memcpy (when reading or writing) but it encode/decodes bytes by bytes the stream by comparing it to the baseline.

    Unfortunately the internal method WritePackedFixedStringDelta is not accessible and this one is a generic buffer utility (encode a length and the bytes).

    So for sake: the correct way to do it is to do this loop
    Code (csharp):
    1.  
    2. for (uint i = 0; i < length; ++i)
    3.     didFailWrite |= !WritePackedUIntDelta(data[i], baseData[I] (or 0), model);
    4. [code]
     
    Kmsxkuse likes this.