Resolved Use Unity to mirror my robotic application

Hello,

I am currently trying to move a robotic arm (kinova j2n4s300 is a 4 Dof 4r robotic arm) in real time through ROS (Noetic) and show the movement with Unity (2020.3.28f1), URDF Importer (0.5.2), ROS TCP Connector (0.7.0).

To do that I used the getting started tutorials available on the Web that I tried to modify.

In the first place, I modified the pick and place tutorial. I imported the URDF model of my robot and setup my ROS and Unity environments.
I then modified the launch files, the config files, the msgs files and the srv files.

When I launch the part_2.launch file with ROS on my terminal and press play on Unity, the two connect.

When I connect ros and the niryo and publish the joint state, the terminal shows the first picture (picture1).
But when I try the same thing with my rendition of the tutorial (picture2) it shows that there is a missing publisher and that the joint state cannot be published.
When I try to publish the joint state of the robot in Unity, there is an error stating that the line 56 in the SourceDestinationPublisher.cs is not defined (see below).

sourceDestinationMessage.joints = m_JointArticulationBodies.GetPosition();

When I give sourceDestinationMessage.joints hard values for every joints(expl 0), the joint states are published with the value I entered.
Finally, the rqt_graphs and the rostopic lists of the two situations are exactly the same (picture 3 and 4).

Thanks for taking the time to read this.
Do you have any idea of what blocs the program from acceding to the current states of the robot ?
Is there anything I forgot to modify from the nyrio tutorial ?

 

I managed to solve my problem. For those with the same problem as mine, you will find it's origin and how to solve it below.

When you convert your URDF file with the URDF converter, the joints of your robot are assigned different types of joints in Unity (Prismatic, Revolute, Fixed ...), the same as in the URDF file.

In the SourceDestinationPublisher.cs file line 46 you use a UnityEngine method to get the joints value.
This part of the line GetComponent<UrdfJointRevolute>(); only allows you to get a component with a revolute joint defined in your URDF. To check the type of your joints go to the URDF file or check the URDF Joint type_of_joint (Script) in your inspector column.

In this script (SourceDestinationPublisher.cs) the type expected is revolute. You can get the position of joints of different types in the same array, you just need to modify your program, get out of the loop and define a new type of joint and assign them the part of the robot to which it corresponds (UrdfJointRevolute/UrdfJointFixed/UrdfJointContinuous).

I hope this was helpfull.

 

How did you manage to solve your problem? I get the Null Reference exception Error on the same line ,trying to implement the tutorial for a ur10 robot. I followed your advice and changed the joint types from the inspector and nothing the error is still there. This is a project for my diploma thesis, please help!!@

 

On which part of the tutorial are you stuck ? Which line gives you the error ? Can you post it ?

 

I am stuck on part 2,ros-unity Intergration.
i have created a moveit config package for my ur10 robot ,with a robotic gripper which works fine.The communication through unity and ros works fine as well. Though console doesn't print ,because of this error i guess the expected joint poses.

The problem
I get a null reference exception error ,(see the photo),in the source destination publisher code.
unity project

source destination publisher visual studio


ros console

 

The error means your JointArticulateBodies is not getting filled by the linknames. Try to see if your linknames are full or not. If they are full, it means that your assignation to JointArticulateBodies is not working. Otherwise if they are emptyYou need to go up further. You may have filled the LinkNames wrong or not gave them the joints the appropriate type.

 

But how can i see if my Linknames are full or not?

 

Inside of you for loop, type the following line to see if your ArticulationBody m_JointArticulationBodies was filled with your joint names or not.

Debug.Log(m_JointArticulationBodies);

It will print in your Unity console your Linknames one by one.

 

You were absolutely right ,the problem was the linknames , thanks a lot. Now i am running into another issue in part 3 of the tutorial. Even though ,moveit calculates the trajectory and robot in unity moves ,it goes up from the target and then considering the grabbing was done goes to the place position (see the photos).
Also my loaded urdf file in Unity has only dae files and shows 2 base links(i think there must be the issue but i am not sure and not know how to fix that).(photo). Of course after i manually delete from the hierachy window the base everything works fine ,but i don't know about the original urdf moveit model utilizes.
Any help about that would be useful.

 

There can be multiple errors.
For the meshes and the URDF that is not correct, the only thing you can do is check whether the URDF file that you used to generate your robot model is correct. To do that, open the URDF file and check the different robotic parts you have, see that there is not the same joints or links twice. Also you can check that each robotic part is linked to the appropriate mesh file.

For the target offset, check when your scene is not in play, the position of your target and cube. With the gravity on in play, they can be in the air and have the coordinates locked in the air. You may also have to modify the pickplace offset. This offset allows you to approach the target from afar. If you have a high offset, it may cause this error.

 

Thanks a lot for you immediate reply ,i will check everything you said and try to solve the issue.

 

I finally managed to solve that problem too. The urdf was responsible for that strange behaviour .Now my robot meets the target very closely ,a bit to the left but the gripper doesn't open or close. Also i get a warning in ros
Warning :Could not identify parent group for end-effector 'gripper_tool'
My gripper is robotiq_arg2f.
I think the project is close to the end but some final debugging advice would be useful please.

 

The problem was resolved completely ,thank you !!