UDP communication can not run faster than Update()

Hi Community!

I have a problem concerning performance and network communication. For some reason I can't get the communication to run faster than the Update() callback is called. I'll try to describe the setup more in-depth below:

Background

I'm trying to make a Simulator in Unity which should be connected to an external control system. Eventually I'll be using ROS (Robot Operating System) to facilitate the communication, but in this example I have simplified the system to use UDP instead. I see the same problems using both protocols.

The simulation/control loop should look something like this:

1. [Unity Simulator] Receive control signal (e.g. a force to be applied to a body) from the controller via UDP, and apply it to a GameObject via the Unity API.
2. [Unity Simulator] Do simulation (physics) step with a fixed timestep.
3. [Unity Simulator] Read sensor data (e.g. position of a body) from a GameObject via the Unity API, and send it to the controller via UDP.
4. [External Controller] Receive sensor data from the UDP connection and set the controller input.
5. [External Controller] Step controller (calculate control signals from input).
6. [External Controller] Send control signal to simulator via UDP.
7. Go to step 1.

Ideally the rendering should update at the display refresh rate (e.g. 60 Hz), while the simulation loop would run as fast as possible, even if that would make the simulation look sped up or slowed down (compared to real-time). Since the Unity API can only be called from the main thread, steps 1-3 must be performed in a Unity callback. To me it would seem that the best approach would be to do this inside the FixedUpdate() callback, and set the physics update frequency very high.

Example setup

I have made a dummy script which should only receive a signal, and the next time FixedUpdate() is called, a reply is sent. On the other end I have a pure .NET application that does the same thing, which will act as the "controller". The Unity Update() callback will write a message to the log with the rate of received messages since the last time it was called. The Unity C# script looks like this:

Code (CSharp):

1.  
2. using System.Net.Sockets;
3. using System.Text;
4. using UnityEngine;
5.  
6. public class UnityUdpFixedUpdate : MonoBehaviour
7. {
8.     UdpClient udpClient;
9.     public bool sendNext;
10.     private int numreceived;
11.  
12.     // Create server on port 6463 and client on server 6462
13.     void Start()
14.     {
15.         sendNext = false;
16.         numreceived = 0;
17.  
18.         udpClient = new UdpClient(6463);
19.         udpClient.Connect("localhost", 6462);
20.  
21.         StartListen();
22.     }
23.  
24.     // Listen for a message
25.     private async void StartListen()
26.     {
27.         UdpReceiveResult result = await udpClient.ReceiveAsync();
28.         string resultMessage = Encoding.ASCII.GetString(result.Buffer);
29.         sendNext = true;
30.         numreceived++;
31.     }
32.  
33.     // Send a message
34.     private void Send(string message = "Hello from Unity!!")
35.     {
36.         byte[] sendBytes = Encoding.ASCII.GetBytes(message);
37.         udpClient.Send(sendBytes, sendBytes.Length);
38.     }
39.  
40.     // Send a message when sendNext is true and immediately start listening
41.     // for a new one
42.     private void FixedUpdate()
43.     {
44.         if (sendNext)
45.         {
46.             Send();
47.             sendNext = false;
48.             StartListen();
49.         }
50.     }
51.  
52.     // Write the rate of received messages
53.     private void Update()
54.     {
55.         Debug.Log("Receive rate: " + numreceived / Time.deltaTime);
56.         numreceived = 0;
57.     }
58.  
59.     // Close the server
60.     private void OnDestroy()
61.     {
62.         udpClient.Close();
63.     }
64. }

and the "controller", which is compiled and run as a pure .NET application, looks like this:

Code (CSharp):

1. using System.Net;
2. using System.Net.Sockets;
3. using System.Text;
4.  
5. namespace UnityCommunicationTest
6. {
7.     class Program
8.     {
9.         static void Main(string[] args)
10.         {
11.             // Declaration of messages and results
12.             string stringMessage = "Hello from Visual Studio!!";
13.             byte[] byteMessage = Encoding.ASCII.GetBytes(stringMessage);
14.             byte[] byteResult;
15.             string stringResult;
16.  
17.             // Create server on port 6462 and client on port 6463
18.             UdpClient udpClient = new UdpClient(6462);
19.             udpClient.Connect("localhost", 6463);
20.             IPEndPoint RemoteIpEndPoint = new IPEndPoint(IPAddress.Any, 0);
21.  
22.             do
23.             {
24.                 // After a message has been sent, the loop will wait until
25.                 // a new message has been received
26.                 udpClient.Send(byteMessage, byteMessage.Length);
27.                 byteResult = udpClient.Receive(ref RemoteIpEndPoint);
28.                 stringResult = Encoding.ASCII.GetString(byteResult);
29.                 //Console.WriteLine("Received: " + stringResult);
30.  
31.             } while (stringMessage != string.Empty);
32.             udpClient.Close();
33.         }
34.     }
35. }

Expected result

I expect it to run almost as fast as the frequency of FixedUpdate, since when I hook up two "controllers" to each other, they can run at several thousand Hz.

Observed result

The observed result is that the rate of published messages is very close to the frequency of the Update() callback.


Conclusion

I have a vague suspicion that what I'm observing has something to do with task scheduling in Unity, but I know too little about such things to be able to say anything more. Does anyone have any ideas on what this could be about? Or what I should do to investigate it further?

Thanks!

PS. I'm attaching the scripts to this post as well.

 

Yeah, don't try and handle UDP within Unity's update loop; that way lies madness. You'll want to start a thread. Here's an example from a recent project of mine where I did this exact thing:

Code (csharp):

1. UdpClient udpClient;
2.         public void Connect(string ip, System.Action callback)
3.         {
4.             Thread connectThread = new Thread(() => Connect_Thread(ip, callback) );
5.             connectThread.Start();
6.         }
7.         private void Connect_Thread(string ip, System.Action callback) {
8.             udpClient = new UdpClient();
9.             udpClient.Client.ReceiveTimeout = 10000;
10.             if (useLocalhost)
11.                 udpClient.Connect(IPAddress.Loopback, port);
12.             else
13.                 udpClient.Connect(ip, port);
14.        
15.             Debug.Log($"Established UDP connection (port {port})");
16.             callback?.Invoke();
17.         }
18. //elsewhere
19. Connect("192.168.0.8", () => {
20.      Debug.Log("Any code written here will execute after the connection has been established");
21. });
22.  

You can call udpClient.Receive() easily in a thread and just wait for its response, whereas if you run it in the main thread, Unity will freeze until it gets a response. Using Receive() in a thread is much more straightforward than trying to use ReceiveAsync().

The main caveat to using threads, as you've discovered, is that sometimes in the callback, you'll need to execute code on the main thread. This seems difficult but is actually pretty easy to handle if you just pile up callbacks and then execute them in Update using an event:

Code (csharp):

1.  
2.  
3.         private event Action mainThreadQueuedCallbacks;
4.  
5.         private event Action eventsClone; // Use this when executing the queued callbacks to prevent race conditions from queueing callbacks while existing callbacks are being executed
6.         public void ExecuteMainThreadQueuedCallbacks() //call this every Update()
7.         {
8.             if (mainThreadQueuedCallbacks != null)
9.             {
10.                 eventsClone = mainThreadQueuedCallbacks;
11.                 mainThreadQueuedCallbacks = null;
12.                 eventsClone.Invoke();
13.                 eventsClone = null;
14.             }
15.         }
16.  
17. //in your threaded code:
18. mainThreadQueuedCallbacks += callback;
19.  
20. //or, for an example if your callback needs to use parameters:
21. public delegate void CallbackWithString(string s);
22. private void SomeThreadedFunction(CallbackWithString callback) {
23. // blah blah network code that can take any amount of time
24. string foo = "foo";
25. mainThreadQueuedCallbacks += () => {
26. callback?.Invoke(foo);
27. };
28. }

I know I threw a lot of stuff at you in this comment - feel free to ask if you need more clarification as you go through it.

 

@StarManta Lotta good stuff there for sure...

However, I note you use a pair of events to handle race conditions. I've never done that. I guess C# specifies the += and the = operators are atomic, but what about a new thread-driven event arriving between line 10 and line 11 above?? It is a teeny-tiny window of time, but technically I think you have a multi-task hole there where an event could get dropped. And you know what Murphys Law says about that ...

I've always used a List<Action> object and actually lock() it before adding and before removing.

At the remove side I lock() the shared list, and if there is anything in it, I move all elements to another list (just .AddRange() to the other list), then .Clear() the shared list and exit the lock block before processing the copied list on the main thread.

 

Thanks for the reply!

I think I understand what you're saying, but I'm not sure if it's applicable in my case. After each UDP message is received I have to step the physics simulation before sending the reply, so then I don't have time to pile anything up before having to go into Unity's main thread again (since the physics can only be done on the main thread I think).

I don't really have a problem being limited by the main thread. My problem is that the Update() callback seems to be throttling how fast the UDP communication can be done, even though all interaction is done using the FixedUpdate() callback.

Sorry that my original message was a bit messy!

 

Ah, I understand. It looks like you can turn off Physics.autoSimulation and call Physics.Simulate() each time you receive data from UDP. (I didn't realize that that existed until I Googled for it just now, so I can't offer additional advice beyond what it says on the doc page)

 

There can be multiple reasons why it doesn't work as expected.

Besides the things already mentioned, you also need to be aware how Unity's game loop is run and you need to take that into account.
One tiny detail to keep in mind is that Unity's FixedUpdate does not run at those distinct points in time that you might expect it to run, but instead, it simulates physics as if they had happened at these points in time. That's an important difference.

For simulations that only rely on the time provided by the app, this can be used to run deterministic simulations, it's enough because you'll usually only be able to see visual results when rendering has taken place. For a pure simulation, it's often irrelevant how fast it's computed, because often you're interested in the results in some sort of visualization or other kinds of playback.

When you want to communicate with things like controllers, you're now adding a system that's actually using real time and both might not be as "synchronized" as you expect.

For the sake of direct comparison, suppose your controller would run at a low frequency of 100 Hertz. It would tick every 1/100 of a second , i.e. every 10ms in real time.

In contrast, when you tell Unity's physics system to run 100 Updates / second, it does not run at distinct time points 10 ms apart from each other, but it runs as fast as possible and allows to query simulated time values. That's a simulation decoupled from real time. The physics loop needs to catch up to the point at which you could say "it's nearly up to date".

Taking the Unity example, if the application's time (not the real time, they can drastically run out of sync if frames take too long) was 25 ms ahead, FixedUpdate would run 2-3 times (depending on whether it's implemented to overshoot & extrapolate or not).
Those 2-3 times are not run at X + 0.01s, X + 0.02s and X+0.03s and thus do not necessarily take ~ 30ms (let's take away the actual overhead for the cycle). Instead, it can run all cycles in a few ms, pretending they took place at points in time that were 10 ms apart.

That is, in your scenario you could simulate hundreds of physics cycles before the data is even received by your controller, and thus before it responses. So the flag might be false during most or all of these simulated cycles. When the controllers responds, simulation might have already been finished and the next time it runs is when it needs to catch up again = after the next Update cycle.

The solution is basically what StarManta has suggested: Move the communication off the main thread and queue the commands.

 

Thanks for your answers! And sorry I haven't answered in a while.

I still don't think I've managed to explain the problem I'm having.

1. I can run Physics.Simulate(), but it still requires me to be on the main thread.
   
2. There is nothing to queue. The moment I receive a signal (through UDP), I want to run Physics.Simulate(), which requires being on the main thread.
3. I don't care about real-time, I just want to do it as quickly as possible. Be it 0.5 times real time or 100 times real time.
   
4. All three parts which should take time (Physics.Simulate(), stepping of the external controller, and the communication) are really fast, below 1 ms. The loop instead seems to be limited to almost exactly 60Hz by the frequency of the Update() callback, even though that callback is not involved at all in the setup.

Have you tried running the scripts from the first post? I think it shows pretty well the problem. In that case there isn't even any physics or controller to step, and the communication is almost instant since the applications (Unity and controller) are on the same computer. Yet you will only get frequencies of 60Hz (or whatever the frame rate of your monitor is) from the loop.

 

There's not a framerate cap, but in the Editor it does aim for certain framerates. Try this in a standalone - you'll see that it'll run the game literally as fast as it's able, and if you have a really lightweight scene, that could be easily >150fps, meaning the added latency from waiting until the next Update() will be minimal.

 

I guess we already understood the problem.

The "problem" is how Unity's game loop works though.

As long as you use FixedUpdate as shown in the original post, you'll be limited by the workings of the engine. FixedUpdate does not update at fixed intervals, but it simulates that instead. 100 ms of physics (for example 10x 10ms) might be simulated in 1ms, in 3ms or whatsoever. That depends on how much has to be simulated, which callbacks are triggered etc.

Once it caught up, it'll continue running other parts of the game loop. It may simulate those steps while your controller hasn't even stepped once, if the physics only take a very small amount of time.

In practice, that means that while data is still on its way to the controller, or on its way back, Unity might run multiple fixed updates, of which many might see your send-flag still as "false".

Also, another thing that's likely limiting here is the synchronization context which is used when you await something on Unity's main thread. That one is likely going to be picked up on the next frame. So the continuation of the async caller might not even have the chance to run on the same frame, if Unity's sync context is sort of implemented with some "wait at least one frame if it cannot finish synchronously" bevahiour.

As for 2) Unity has its update loops. So that sort of requires queuing even if that's just for a short amount of time. Anything else would require blocking/stalling the main thread until you can "step forward", i.e. "react" to incoming data.
That's not what Unity is made for.

As for 3) that's already clear. I wasn't trying to talk about making a realtime communcation work.
But rather that you're dealing with real-time stepping on your controller's side, vs simulated stepping on the engine's side.
The controller might wait for the next step after it's done with one step, it executes at distinct points in time. Unity doesn't and runs the required steps as fast as it can simulate them. That's a huge difference.

4) it is involved as it needs to run once per frame. It'll always be there and it'll take a certain amount of time. You can try to reduce the time it takes by running Unity in headless mode though.

 

Physics.autoSimulation = false.

Receive UDP messages in a separate thread and enqueue them (in a ConcurrentQueue, which you may steal from the dotnet or mono repos) as tuples with a performance-sensitive timestamp (not Time.time). You may want to instead timestamp in the controller code. Discard times that go backwards.

In a Unity Update() (not FixedUpdate), dequeue the messages until they are empty. Use the timestep and process physics however you like. Physics.Simulate() however much you want, using the timestamps to determine whatever time you'd like.

Since Physics.Simulate may itself may take longer than the time between the timestamps of two UDP messages, it should be obvious to you now that you can't do what you want in the strictest sense, you will have to drop UDP messages that are already "too late." You can do that in your update loop.

And now, it should be clear why "simulating realtime faster than rendering" is sort of an oxymoron. I mean I get what you want to do, but you should try this framework and then interpret your queue differently to get the appropriate result.