Discussion Using DOTS ECS in robotic arm simulator

Hello everyone!

I hope I'm not too daring in making this post and not many of you will get angry at me. The reason for creating this discussion is that I am in a bit of a pickle with a research topic for my master's thesis. I recently encountered some limitations with Unity API and the regular physics, and using DOTS seems like the only way forward with the project. Unfortunately, I am not very experienced with DOTS, and some things have changed since I last used it, and I am running out of time to finalize my thesis assignment. So I'm hoping to get some more information that would help me steer clear of some future problems that I would burn a lot of time figuring out later on. I don't have high hopes that people will swarm this thread to help me, I'm just putting it out there just in case some people were solving similar problems in their project or they know exactly what resources will give me the most helpful information.

The goal of this discussion
I want to tell you more about the project and my expectations of using DOTS. Hopefully, some more experienced users from the community will be able to catch some red flags immediately or tell me that it is not even possible with the current state of DOTS. Perhaps someone else worked on something similar and would be able to explain what to be careful about and direct me to some relevant resources and demos. Any ideas or guidance will be very appreciated.

About the project
I am working on a robotic arms simulator/visualization in Unity that has these features:

• allows users to import their robotic arms using the URDF Importer;
  
• supports integration with ROS (because it is the standard and most users will probably use the simulator that way);
• but also has an integrated online solver (as in 'constructing the movement plan at the same time as the robotic arm is executing the plan') solver based on the Monte Carlo Tree Search (MCTS) algorithm; and
  
• supports VR to view the simulation but perhaps also to interact with the arm and objects in the scene.

The project was 'bootstrapped' by Unity's Pick-and-Place tutorial, which provides a guide for setting up the whole Unity-ROS integration, importing a robotic arm Niryo One from a URDF, and running a solver in ROS and simulating the result movement trajectory in Unity. So I was mainly playing with this demo and with the provided Niryo One model, but we will be adding our own custom-built robotic arm later on. The robotic arm is imported as a physics articulation (a chain of articulation body components), and it provides a realistic physics simulation for the arm. I implemented the VR support and some visualizations and started tackling the integrated solver, where I crashed into a wall.

Problems without DOTS
In short, the power of MCTS (or Monte Carlo methods in general) is in the ability to perform a massive amount of random simulations in the environment in order to empirically estimate which of the next moves is most likely to lead to some goal. So the plan was to create a copy of the articulation and while the physical arm is moving to its next position (e.g. rotating a subset of joints by a few degrees), the MCTS simulations would rapidly reposition the articulation into different configurations, in each configuration compute the position of joins in 3D space, and perform series of sphere casts between the neighboring joins in order to detect potential collisions with the objects in the scene. And there are several issues that I have encountered:

• I cannot perform the simulations in the main thread;
• I cannot use Unity's API from different threads - no manipulation with articulation, no sphere casts;
• anything else would most likely be so ineffective that it would be pointless to use MCTS.

DOTS to the rescue?
So, as far as I understand, these issues might be overcome with DOTS. I want to design and implement the MCTS in a data-oriented way and use Unity Physics to perform the sphere casts to detect potential collisions efficiently. I was wondering if it was possible to have a mixed setting where I use the game objects and physics articulation for simulating the physical robotic arm and use entities and Unity Physics only for the simulations. I also looked at Havok, but it seems to me that Unity Physics is a better fit. Or maybe Havok would perform a large amount of sphere cast queries more efficiently? Also, I don't think I will be able to convert a physics articulation into entities, I might need to construct the robotic arm representation from some joint components.

Specific questions
Here is a set of more specific questions that are easier to grab and answer than extracting the questions from the paragraph above.

• Can I use both regular 3D physics and Unity Physics for DOTS in mixed setting? Using DOTS for MCTS simulations (collision queries) only and having the physical robotic arm simulated by the regular physics?
• Should I consider Havok instead of Unity Physics? I don't have a complex scene, but maybe the amount of sphere-cast queries will be more efficient in Havok. Weren't there some recent issues with Havok's licensing?
• How would you bring the physics articulation into entities? Is the only or preferred way to rebuild it manually using the Joint components in Unity Physics?
• Am I missing something obvious, or is this even doable?

That's a lot of text. I'm sorry.
Thanks to anyone who spent time even reading this. And have yourselves a great holiday!

 

I'll cover that Havok part and let Unity folks jump in around the rest (I don't want to commit them to anything ).

If I understand correctly, Unity.Physics or Havok Physics for Unity would be used just for queries while the whole simulation part will be handled by your code. In that particular case Havok cannot be of any benefit for you. Actually, even when project uses Havok for simulation the queries itself are always done through Unity.Physics.

 

Thanks, that's good to know. I will stick to Unity.Physics.

Yes, Unity.Physics would be used for queries, but the simulation would be done by regular 3D physics with rigid bodies (and my code). But that means I need to have some kind of a hybrid scene so that the robotic arm can bump into things in the game objects world and I can do queries in the entities world. But something sounds weird to me about that. It feels like I am trying to do something unusual, or I will sacrifice some performance.

 

Your colliders / rigid bodies need to be entities.

 

That's clear to me. But since the Runtime Conversion seems to have been removed, is there any other way to do this instead of having all 'obstacles' in the scene duplicated in both the sub-scene (converted to entities) and the main scene (as game objects) and then somehow synchronize their transforms in a system?

I would prefer to have some runtime authoring that would take all the game objects I want to manage in the entities world as well, take their colliders, create new entities in runtime and add a PhysicsCollider (I think) component to them.

 

Different ways exist. A good idea is to separate the physics world from the graphics world so you can support re-simulation. The graphics can either be in the main default ECS world or if you find you need stuff like particle effects and skinned meshes then you can use GameObjects with pooling. In any case your physics world can contain entites that have no graphics components.

The code below uses the old foreach.

Code (CSharp):

1. public partial class moveGraphicsSystem : SystemBase
2. {
3.     private graphicsManager GXpool;
4.     private bool monoRunning = false;
5.  
6.     protected override void OnUpdate()
7.     {
8.         if (monoRunning == false && GameObject.FindObjectOfType<graphicsManager>())
9.         {
10.             GXpool = GameObject.FindObjectOfType<graphicsManager>();
11.             monoRunning = true;
12.         }
13.  
14.         if (monoRunning)
15.         {
16.             //Debug.Log("OnUpdate:  "  + "  :  " + Time.DeltaTime + "  :  " + Time.ElapsedTime);
17.  
18.             Entities.WithoutBurst()
19.             .WithAny<unitTypeComponent>()
20.             .ForEach((in WorldTransform pos, in rigidbodyDOTS rBody) =>
21.             {
22.                 //need to get data for transform
23.  
24.                 GXpool.upDateGx(pos.Position, pos.Rotation, rBody.ID);
25.             }).Run();
26.         }
27.     }
28. }

 

Sorry, I probably didn't explain what I need clearly enough, or I don't understand what are you trying to answer. I'm not trying to do anything with graphics in ECS. I only need ECS for physics, specifically only for colliders, joints, and static queries on those colliders. No simulation of physical bodies, and no graphics.

Let's say I have a cube in the scene (not the sub-scene). It is rendered, it has a rigid body and it is simulated in physics. There is also a robotic arm in the scene that can bump into the cube. So it's some kind of obstacle for the arm. There can be more of these obstacles of various shapes, for now let's imagine a game object that's a cube. Still no ECS until now.

Now, I wanted to create a 'companion' entity for that cube, that has only a PhysicsCollider of the same shape as the cube. No mesh to be rendered, no physics body to be simulated. Just a collider and a LocalTransform. And then I need a system that would synchronize the companion entities' position and rotation based on the reference game object, because that one might be moving in the scene.

When I have this, I can do static queries in the ECS world where the colliders are synchronized with the non-ECS world. In ECS, I will then plan a path for the robotic arm so it will avoid all the obstacles, and then just apply forces on the robotic joints that are simulated in the non-ECS world.

 

Like I said earlier you cannot mix PhysX with Unity Physics. You need to pick one or the other.

See the code I posted above.