DOTS Multiplayer discussion

Good to know, Ill start making multiplayer and once it's ready reconsider switching to Chilli. I'll have few options:
- Playfab + Playfab Servers (Azure)
- Playfab + Multiplay (GCloud or AWS)
- ChilliConnect + Multiplay (GCloud or AWS)

That's the DOTSSample, right?

 

I'm not sure if this has been discussed before, but I'm curious what techniques you're going to utilize to save bandwidth.

I'm planning to use DOTS Multiplayer for large scale battles with several hundred players per server, in order to archive this there must be some optimization going on, like;

Only send data that has changed since the last tick, e.g. when a player presses a button send that info to the server once and once if he releases the button. So only state changes should be synced instead of a constant stream sending the same data over and over again. This way you can get rid of a lot redundant data.

Distance related tick-rate and data compression. Players that are far away from you don't necessarily need to be updated that often and accurate as those who are close up to you. So it's basically a LOD system to save bandwidth.

I haven't really touched the NetCode package yet, so I'm asking if those systems can be easily implemented, or are ideally on the ToDo list.

 

I’ve noticed when playing with the multiplayer cube example and my own implementation that there are some simulation inaccuracies between the client and server due to the server not receiving client input packets in time. This happens when running the example with even a moderate latency 60+. Does anyone else get this issue or am I doing something wrong? I would think the example should accurately out the box. Are there still impromemts to be made to command buffering/transmission to improve this? At the moment you can notice the client predicted cube jittering back and forth when changing input directions even at low latency simulations. At higher latency/packet loss I could understand this but at this level it seems an unacceptable level of misprediction?

 

I also recognized this behaviour. Already wondered why nobody else mentioned this.

Logging the Ticks on the Server when the CommandData is received shows that the CommandData is sometimes too late.
Here is some log at 100ms recv/send delay.


The "Applied tick" is the tick which was last received and should match to the server tick for appropriate simulation.

The NetDbg delivered by Unity shows the CommandAge (oranage Line). The CommandAge appears to go up and down like a wave form. This can be explained by the continous adjustment of the estimated ServerTick on the client.

White line = No delay of command data
Orange line below white line = CommandData received earlier than needed
Orange line above white line = CommandData received too late

At 0 ms send/recv delay; commandData always on time:



At 100 ms send/recv delay; commandData sometimes to Late


At 300 ms send/recv delay; gets worse



Which CommandData is applied on which serverTick is calculated in the client. I had no closer look at the calculation but it seems wrong for larger latency.

 

Any ETA on this feature?
Actually, this could save bandwidth for cases where something changes extremely rare. Like a door -- it won't get it's open/close state switch multiple times per second. Why would we need delta compression for something that doesn't really change? Does this also mean it'll make the ghosts possible to despawn/spawn based on distance? It would've been awesome to have not just LOD for ghosts updates, but also for their presence.

 

Hm, this sounds like the bug I mentioned: https://forum.unity.com/threads/dots-multiplayer-synchronization-lags.757415/#post-5048408

That was with the old multiplayer asteroids sample, I haven't seen this in the new one but I haven't done any accurate testing. Could still be the same issue, just moved to a different place.
Sadly, I don't have the time right now to investigate.

@Flipps Debug.Logs are really bad for network testing. The delay it takes for logging distorts every measure.
Write the log data you need to a List struct and save it to disk when quitting.

 

@Enzi I will do that, but i think the results won't be different, because the CommandAge log in the NetDb shows the same results. Or maybe i am completly wrong.

The sluggery movement is really easy to reproduce. Either with the NetCube Sample or the Tutorial on the NetCode Manual page (with a send recv delay of 200+ms)

 

I had some strange behaviour where the animation of an predicated ability got fired twice.

After a little bit of research i found out that this was caused by the Clients calculated ServerTick.
At 200ms+ send/recv it could happen that the Clients ServerTick could jump back in Ticks (see Screenshot).
So the predicted Entities jump back in time because the prediction always runs until the ServerTick.

I know that the ServerTick is variable adjusted, but is it intended to be less than the time before?
I think this could unintentionally trigger Systems which run after the GhostPredictionSystemGroup twice.

 

Thanks for the really fast and helpful answer - as always
This behaviours seems to be triggered alot when above 150+ Send/recv delay or when the client has a low framerate.

I am aware of this and printed the log outside of the GhostPredictionSystemGroup of the client.

 

In the DOTSSample we can see that some of the data of Ghosts are grouped as "PredictedData" "InterpolatedData" "ReplicatedData" which doesn't seem to have cohesive data at my first glance of it. Is this approach more performant than having it separated in many smaller components? Which is the recommended approach?

 

Any ETA for NetCode upgrade to the new version of Entities? Currently, it depends on 0.3.0 and - understandably- fails to compile with the newest preview package (0.6.0).

I was searching for the NetCode repository but couldn't find it. Is the source code up somewhere? (I know it's in my library folder I was just wondering if it's on GitHub.)

 

Yes please, really difficult at the moment to get things across the DOTS ecosystem to play nicely together when the supported versions are all over the place. The fact that the ECS package has already deprecated AND removed API functionality before NetCode has even manage to release an update to remove these dependencies is a bit disappointing. I understand this is all preview tech, would just be nice if supporting system updates could follow the core tech releases a little closer.

 

Could be a good news:

https://schedule.gdconf.com/session...k-dots-presented-by-unity-technologies/874091

DOTS Sample has to be updated for this event. The only question remains if it's going to be public on that day or just a private unfinished showcase version.

 

Lets not forget that Netcode is still in preview and not production ready. I think a good mindset right now when using Netcode is that we bind ourselves to the DOTS stack version(s) its latest package supports, and should be prepared that each new version can change radically. Its still open source, so can copy Netcode to Packages and update it manually if we must stay up to date with the rest of the DOTS stack. Alternatively base a custom solution on Transport.

 

This is my take.

Lets not forget that Unity itself is still in preview and not production ready. I think a good mindset right now when using Unity is that we workaround ourselves to all version(s) its latest package supports, and should be prepared that each new version can change radically. Its still closed source, so can't fix Unity code and update it manually even if we must stay up to date with the rest of the Unity stack. Alternatively base a custom solution on your own.

Pun intended but I hope someone finds some truth in it. ^^

 

After fixing the compile errors (around 10 or so) and following the Getting Started page of the documentation I successfully built a server and a client and it looks like it's working. Obviously I can't verify if every little thing still works, it just looks like it does, I could move around and a friend of mine could connect to the server and he could move around also. I only had time to test the code that comes with the Getting Started page.

Mainly the

World.Active

is what changed to

World.DefaultGameObjectInjectionWorld

, and maybe one or more other things. If there is no other (functional) change and it turns out it works, it's really a trivial 10-minute upgrade process.

 

Do you use ConvertAndDestroy or ConvertAndInject? I use the latter but my cube doesnt move. with the first one (convertanddestroy) SharedData (cube etc) didnt really render in editor. could you upload your test project? thanks

 

I followed this Getting Started guide: https://docs.unity3d.com/Packages/com.unity.netcode@0.0/manual/getting-started.html , maybe look again if you missed a setting or package somewhere (note: you need hybrid renderer package to see the cube render with convertanddestroy, because it's not automatically installed with dots!)

I'm not sure if I can upload the project as it has the updated NetCode in it which I'm not sure I'm allowed to upload.

 

Netcode 0.1 was released few days ago. https://docs.unity3d.com/Packages/com.unity.netcode@0.1/changelog/CHANGELOG.html

https://github.com/Unity-Technologies/multiplayer also got updated.

https://github.com/Unity-Technologies/DOTSSample not yet but there's new commit 2 days ago so I guess soon.

Can't wait to see epic use case in the upcoming online GDC presentation . Hope it won't be canceled or postponed to much further date than planned. Link I've posted before doesn't work anymore but here's screenshot:


Would be good to pin this thread.

 

Thank you! Good change!

 

I thought I would make my life easier when having multiple similar ghosts to make prefab variants, and bumped into this.

Is this something that might change? Or do I need to have top level prefab for all ghosts.

 

Thanks for all the helpful discussion and replies! Simple question: how do I set up a server with Unity Transport? I understand this is all in preview mode right now, and maybe I missed something simple, but the example documentation I saw all had a "ServerBehaviour" that ran locally in the editor. Is it even possible to use Unity Transport right now? Above I saw mention of a 50 player game being used, and surely that would require a server for all the players to connect to, so I'm guessing this is possible? I currently am using Websockets with a Python server and have basic multiplayer syncing working, but I'd like to upgrade to the new Unity Multiplayer code. I'm open to switching to a C# backend server for the multiplayer networking code, I just haven't been able to find any examples of how to do so. Is it just building the server code in the Unity Editor and then doing a Linux Server Build? If so, how do I actually run that on my Ubuntu server (e.g. with Python I use aiohttp, in node I use express for the web server - is there a recommended C# web server to handle the incoming connections? Or do you just include Unity on the server too and that handles everything for you?) Is there any way to just code the C# directly in a github repo, outside of the Unity Editor? Are there any examples of an actual server running Unity Transport? Sorry if these are newbie questions - I've been coding full stack for 13 years but am new to C# and while C# within Unity for the client made sense, I'm a bit confused on how to make the jump to the server.
Thanks in advance!

 

+1. I am particularly interesting is there any way to host server with .NET Core and use Unity Transport and/or DOTS Netcode to produce as pure ECS code as it possible on the Unity client side?

 

I've noticed that if I use trigger input (e.g. key pressed down) in command struct, with a laggy client that doesn't produce input for all server ticks, the fetches of commad buffer will get the closest older tick (happens both when sending and when you're fetching yourself). This causes laggy clients to trigger key downs multiple frames if used out of the box. Are there any thought of solutions for this?

My current workarounds is to also add the generated tick in the command, and remove triggers if it doesn't matches the tick I'm fetching for.

 

Also I seem to not get the inputs to time in a good manner either with FixedTime or not in the editor when I only have some 10 fps. The command with trigger input is either not sent (non fixed time) in time or has already been computed (FixedTime).

 

I have been working through the Asteroids sample to understand how Unity thinks about multiplayer architecture in the new DOTS stack. I understand the server being authoritative is key, but that clients still have to run certain systems to provide the immediate responses users would expect (can't wait for the server to respond for every single thing or the lag delay will be unacceptable). I am trying to develop a better sense of how to split systems to run on either client, or server, or both. This answer is probably super obvious to anyone that knows the pitfalls of multiplayer video game development but is lost on me.

In Asteroids the systems are held in 3 folders:
/client (for systems run only on the client, like the InputSystem because only clients have input)
/server (for systems run only on the server, like the CollisionSystem because server is authoritative on collision)
/mixed (for systems that run on both client and server, like SteeringSystem because client needs instant feedback but server ultimately decides positions)

I am trying to understand why the BulletSystem is run on both client/server (/mixed) but the AsteroidSystem is run only on the server (/server).

They both seem to do the same thing: make the bullet/asteroid continue traveling the direction it was traveling by updating its position using its velocity. And they both seem to have the same level of importance for the user, knowing exactly where they are. If I shoot a bullet I would expect it to travel in a way that makes sense regardless of server behavior. I would expect Asteroids to behave the same way, in a consistent pattern because I am trying to avoid them/shoot them.

I understand the system to spawn the bullet needs to be both on the client and server (which it is, SteeringSystem is in /mixed) but once it has been created and traveling, it seems to be of the same importance of an Asteroid, no? Either both AsteroidSystem and BulletSystem should be in /mixed, or both of them should be in /server (I am using the folder location as a proxy for meaning whether the client/server should be running it).

BulletSystem

Code (CSharp):

1.             public void Execute([ReadOnly] ref Velocity velocity, ref Translation position, [ReadOnly] ref PredictedGhostComponent prediction)
2.             {
3.                 if (!GhostPredictionSystemGroup.ShouldPredict(tick, prediction))
4.                     return;
5.                 position.Value.xy += velocity.Value * deltaTime;
6.             }

AsteroidSystem

Code (CSharp):

1.             public void Execute([ReadOnly] ref Velocity velocity, ref Translation position, ref Rotation rotation)
2.             {
3.                 position.Value.xy += velocity.Value * deltaTime;
4.                 rotation.Value = math.mul(rotation.Value, quaternion.RotateZ(math.radians(100 * deltaTime)));
5.             }

As with all Ghosts the server provides SnapShot updates to all clients of both the bullet and the asteroid. Why would receiving snapshot updates from the server for Asteroids be acceptable but not Bullets? I am sure there is something very obvious I am missing.

@Jawsarn @Enzi @Kamyker @timjohansson (sorry for calling you out specifically but you guys seem to be my best hope) please save me from this mental pain!

 

Great q

 

Simple advice: don't use it.

 

On your asteroid and bullet ghosts, there are 3 tick boxes Server, Interpolated and Predicted.

without looking my guess is that the asteroid has the first 2 ticked, the server creates and moves them around, the client has no influence on them.

where as the bullet has predicted ticked as it will be doing the same as the player movement, and predicting the journey from its spawn point to its target, so the player does not get lag waiting for the server to reply that its a go, it predicts, does its thing to make the players visual look right, then will readjust if necessary dependent on the servers response.

its early, still on first coffee....

 

Hi @timjohansson. Will new Netcode package version release anytime soon? From what I understand from internal production team, there is a lot of changes to Netcode and I guess there will be big breaking change. I hope I can get the new version to upgrade my project as early as possible.

 

While I don't have any netcode experience myself, this is probably the thing that's cooked my noodle the most. I'm also not sure that I quite like the way it has been implemented in the NetCode package, but I would also not be able to build these things myself. The way I look at the architecture of everything is as follows:

You have two types of entities, predicted entities and interpolated entities. My understanding is that predicted entities would be ones that originate from the current player. So the entity that visually represents the player in the game world and bullets and/or weapons that spawn as a result of player action. The key thing here is that while the player's client would predict these entity positions and data the server is still the authoritative actor and will notify the player's client when desyncs occur, after which the client will update to the correct state and continue running its predictions.

Interpolated entities would be any entity that originates from other players, again consisting of the entity that visually represents the player, bullets and/or weapons. What's important here is that these entities have their data synced through the server. From your player client's perspective, it doesn't care where the initial state originates from, but it will receive this state from the server and buffer it a little. This allows the representations to be nicely interpolated and not look have noticeable "jumps" in position. The core idea here is that the client's buffer is also serving the purpose of correctly ordering some of the packets coming in which then allows for smoother animation. It's a fine edge to ride, but it ends up much more pleasant to look at.

This then means that you could have multiple systems interacting with the same data depending on what context it's running in, a local client, on the server or on a remote client. You might find that a system that the server runs for gameplay code would also be run as a prediction system on the local client, but very likely not on a remote client. It does depend on your game though because you might be doing some fancy compression to sync a physics simulation. I also haven't even touched how certain clients would end up predicting entities when the buffer gets starved and there's also a very deep discussion to have around how to correctly compress the data you're sending, but I feel getting a basic understanding of prediction vs interpolation will help to further understand how systems interact with the data.

 

As @TZ- already said, the bullets are predicted on the client. So they will be spawned and moved before the client receives a tick that they were created from server. The GhostPredictionSystemGroup runs on both server and client, both wants to move the bullet. If I'm not wrong the client runs this one multiple times for all predicted ticks. The asteroids are however only interpolated. I agree that one would actually maybe instead send the velocity when spawned, and have the AsteroidSystem run on both client & server instead of interpolating, but this is small sample after all.

 

If you want to implement a FPS shooter today, what would you recommend if you are frustrated waiting for UNITY to get their fingers out with DOTS?

 

Thank you @TZ- , @LazyGameDevZA and @Jawsarn !!! Seriously!! So helpful to get these points articulated (it is difficult to infer these insights from samples/documentation/talks) you guys have saved me a lot of headache. (and no thank you @Kamyker lol)

I have watched each ECS/Netcode talk multiple times, gone through the ECS/Netcode documentation too many times, and researched general networking principles to get an idea of problems these configurations/models are meant to solve but sometimes I get the feeling that the questions I still have are so obvious to seasoned multiplayer engineers that they aren't mentioned anywhere (or I am so out of the loop I can't see the answers right in front of me). Hopefully you guys are still feeling generous with your knowledge because this opens up a few more questions...

From Asteroids sample netcode project

Asteroid asset (default client instantiation type = interpolated, Velocity component only created on Server)


Bullet asset (default client instantiation type = interpolated, Velocity component on Server and Predicted Client)

Questions

1. Why is the bullet default instantiation interpolated? (I thought it would be predicted because the client creates it?) Is this something that is overwritten in the actual instantiation code? The bullet instantiation code in /mixed:

Code (CSharp):

1. if (bulletPrefab != Entity.Null)
2.                     {
3.                         var e = commandBuffer.Instantiate(index, bulletPrefab);
4.  
5.                         commandBuffer.SetComponent(index, e, position);
6.                         commandBuffer.SetComponent(index, e, rotation);
7.  
8.                         var vel = new Velocity
9.                             {Value = math.mul(rotation.Value, new float3(0, bulletVelocity, 0)).xy};
10.  
11.                         commandBuffer.SetComponent(index, e,
12.                             new PlayerIdComponentData {PlayerId = playerIdData.PlayerId});
13.                         commandBuffer.SetComponent(index, e, vel);
14.                     }
15.                     else
16.                     {
17.                         var e = commandBuffer.CreateEntity(index, bulletSpawnArchetype);
18.                         var bulletData = default(BulletSnapshotData);
19.                         bulletData.tick = currentTick;
20.                         bulletData.SetRotationValue(rotation.Value);
21.                         bulletData.SetTranslationValue(position.Value);
22.                         // Offset bullets for debugging spawn prediction
23.                         //bulletData.SetTranslationValue(position.Value + new float3(0,10,0));
24.                         bulletData.SetPlayerIdComponentDataPlayerId(playerIdData.PlayerId, default(GhostSerializerState));
25.                         var bulletSnapshots = commandBuffer.SetBuffer<BulletSnapshotData>(index, e);
26.                         bulletSnapshots.Add(bulletData);
27.                     }

Does that dropdown in the editor (Default Client Instantiation Type) not mean what I think it means? Or is the issue that remote clients probably aren't going to instantiate other ships' bullets?

2. How can the client have "smoothly" appearing asteroids, when the velocity component is only created on the server?

I understand that "interpolated" means that the server is running the state of that object, but the client has to "smooth" it out. If I don't have any velocity data, and the asteroid system exists only on the server, how are my asteroids not completely stuttering based on data transmission? How can the client "interpolate" where it should be if I don't have any systems to do the math and no components that contain data to run math on? Which brings me to the next question...

3. What is a good mental model of the how interpolated vs. predicted "works" and when to use one over the other?

@TZ- you mentioned

"...the server creates and moves them [asteroids] around, the client has no influence on them."
"so the player does not get lag waiting for the server to reply that its a go, it predicts, does its thing to make the players visual look right, then will readjust if necessary dependent on the servers response."

Which I infer to mean that if a client has no influence on a networked (ghost) entity, then it should be interpolated. If the client does have influence on a networked (ghost) entity, like itself or things it spawns, then it should be predicted. But how does that interpolation work if I don't have any data about the asset or any systems to run math on that data?

@Jawsarn you mention that maybe the asteroids should be spawned with a velocity component on the client and the system run on both server and client. Was I trying to infer insight of what components/systems should be on the server vs. clients from a sample project that wasn't "fully baked" in that regard?

4. What is the difference between Netcode.CurrentSimulatedPosition/Netcode.CurrentSimulatedRotation and Transforms.Rotation/Transforms.Translation and how do they interact?




The CurrentSimulatedPosition and CurrentSimulationRotation is marked for only interpolated client and predicted client. How does this data interact with Transforms.Rotation Transforms.Translation?

Is there a netcode process that works behind the scenes to match these two components? Is this the interpolated vs. predicted?


I know it is way too much to ask of strangers on the internet to help decipher a very much in-preview package and a sample project that might not have had every part of its design vetted to fully make sense with Unity's networking philosophy so I appreciate you guys taking the time regardless! @timjohansson @Joachim_Ante not sure if my deconstruction of the asteroids sample project is trying to read tea leaves or if the project was meant to be interpreted as a guide on Unity's networking philosophy/approach in game design.

 

Yes! Seconding this question plz

 

@adammpolak
1. I'm not sure why they are doing it this way, haven't digged into it. But form SteeringSystem you can see that they are using the predicted when spawning

Code (CSharp):

1.             if (bulletPrefab == Entity.Null && bulletSpawnArchetype == default)
2.             {
3.                 if (World.GetExistingSystem<ServerSimulationSystemGroup>() != null)
4.                 {
5.                     var prefabs = GetSingleton<GhostPrefabCollectionComponent>();
6.                     var serverPrefabs = EntityManager.GetBuffer<GhostPrefabBuffer>(prefabs.serverPrefabs);
7.                     for (int i = 0; i < serverPrefabs.Length; ++i)
8.                     {
9.                         if (EntityManager.HasComponent<BulletTagComponent>(serverPrefabs[i].Value))
10.                             bulletPrefab = serverPrefabs[i].Value;
11.                     }
12.                 }
13.                 else
14.                 {
15.                     bulletSpawnArchetype = EntityManager.CreateArchetype(
16.                         ComponentType.ReadWrite<PredictedGhostSpawnRequestComponent>(),
17.                         ComponentType.ReadWrite<BulletSnapshotData>());
18.                 }
19.             }

2. / 4. The CurrentSimulationPosition/Rotation is queried for in RenderInterpolation systems. Basically it saves the simulated states and interpolates between them in the "ClientPresentationSystemGroup". There is a "BeforeSimulationInterpolationSystem" which by name is ordered in beginning of simulation that moves values from the CurrentSimulationPosition/Rotation => Translation/Rotation components. Similarly there is one named "AfterSimulationInterpolationSystem", which moves Translation/Rotation into PreviousSimulatedPosition/Rotation componenets. These systems are located in the NetCode/Runtime/RenderInterpolation folder.

3. Predicted vs Interpolated. I would say use predicted when you want to avoid the delay between input => action for the player. For asteroids sample you could wait for server to say OK to move your player left or right. You can also add to predict the spawning of things, which the asteroid demo does with it's bullets. To extend the sample, you could ask if you want to predict the asteroids as well to die, or wait for the server to tell that they died? You will need to ask what would happen if the prediction fails and how will this feel for the player, and base your prediction and interpolation around that as well.

It's a sample, I guess they want to show the interpolation as well as prediction, hence it makes sense that the asteroids are interpolated and not locally simulated. But for my own games I would simulate predictable behavior, as the linear non collision objects, locally on the client.

 

you rock thank you so much!!!

 

Hey, I have a question about RpcLevelLoaded under the Mixed builds folder in the Astroids sample project. What's up with the "Rpc" prefix in "RpcLevelLoaded" - does it make it special in any way? I noticed that other RPCs don't have a prefix like that in the program
Is it because it is self-contained in regards to data?
And is it related to the fact that there isn't a system that runs on the server that runs an EntityQuery on entities with RpcLevelLoaded?

 

Hello there. Looks like RPC's hadn't been covered a lot in this thread.

@timjohansson maybe you have some information regarding RPC reliability, maximum size, and how to handle really big RPC responses because that is the only way right now to set up a request/response flow.
Also, I've found that using struct only RPC it's not really useful due to restriction on using arrays, strings, objects, etc. Maybe I'm using them wrong?

Found a thread without any answer:
https://forum.unity.com/threads/netcode-rpc-reliability.860941/

 

Question about LoadLevelSystem on the server. Why would we want to call the ServerSettings for the LevelComponent on each OnUpdate? Why does this not happen in OnCreate?
Is it because the game settings could potentially change during an OnUpdate, so the ServerSettings should constantly be called just in case the data changed?

cc @timjohansson