Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question Displaying Point Cloud from a Realsense D435 in Unity through ROS

Discussion in 'Robotics' started by GerenMeric, Jun 1, 2023.

  1. GerenMeric

    GerenMeric

    Joined:
    May 15, 2023
    Posts:
    47
    Hello there,

    I have two pcs and i am trying to get live feed from Realsense D435 camera, publish it on ROS from one of the pcs with roslaunch realsene2_camera rs_camera.launch filters:=pointcloud, then, on the other computer using ROS TCP Connector and ROS TCP Endpoint, get ROS messages coming from the other pc into Unity, in Unity with a C# script which subscribes to "camera/depth/color/points" topic and displays the pointcloud in Unity. I tried to write the script. At the beginning of the script, i defined a MeshFilter object and a string which holds the value of the topic we are going to subscribe which is camera/depth/color/points, then in the start function, i assigned GetComponent<MeshFilter>() to the MeshFilter object. Then, i subscribed to the topic with ROSConnection.GetOrCreateInstance().Subscribe<PointCloud2Msg>(topic, UpdatePointCloud) then, in UpdatePointCloud function, i got a object named message of PointCloud2Msg class as the arguement of the function. Then, i tried to get number of points by doing this: (int)(message.data.Length / message.point_step) Then, i created a Vector3 and a Color32 array with size set to number of points i found. Then, i defined two integers vector_index and color_index. Then, looped through all the points and in each step tried to extract color and vectors from message.data using:

    Code (CSharp):
    1. vectors[] = new Vector3(System.BitConverter.ToSingle(message.data, vector_index), System.BitConverter.ToSingle(message.data, vector_index+4), System.BitConverter.ToSingle(message.data, vector_index+8))
    for getting vectors
    and
    Code (CSharp):
    1. colors[] = new Color32(message.data[color_index + 16], message.data[color_index + 17], message.data[color_index + 18], 255)
    for getting the colors.

    Also in each iteration i did += (int)message.point_step for both vector_index and color_index.

    Then, i defined a Mesh object and assigned sharedMesh of the MeshFilter object. Then if mesh is not null, i assign new Mesh() to the Mesh object Then, i set indexFormat of that Mesh object to UnityEngine.Rendering.IndexFormat.UInt32 then assigned sharedMesh of MeshFilter object to Mesh object. Then, finally, i did:

    [/I]
    Code (CSharp):
    1. meshObject.Clear();
    2. meshObject.SetVertices(vectors);
    3. meshObject.SetColors(colors);
    4. meshObject.SetIndices(New Int[0], MeshTopology.Points, 0)

    After writing the code, i created 2 empty GameObjects. To one of them i added ROSConnection script came with ROS TCP Connector package from Unity Robotics Hub. At the other game object i added the ROS subscriber script, a mesh filter and a mesh renderer component. I did't configure both mesh filter and mesh renderer components. So, both mesh field of mesh filter and materials field of mesh renderer is set to none.

    When i press that play button at the top in Unity, i see nothing displayed. But, when i move the D435 camera, i can see that GameObject with ROS subscriber script, mesh filter and mesh renderer component is moving around randomly. Also when i try to print number of points or colors by adding prints and press the play button, Unity just freezes. I wonder if what i did in the code was wrong. Could you help me on this issue? Thanks in advance.

    With kindest regards,

    Meriç Geren
     
    Last edited: Jun 2, 2023
  2. alessandro7111

    alessandro7111

    Joined:
    Jul 2, 2020
    Posts:
    5
  3. GerenMeric

    GerenMeric

    Joined:
    May 15, 2023
    Posts:
    47
    alessandro7111 likes this.