Resolved How to save 3D object's position and rotation and load them at the exact place using ARkit in unity?

Hi All,

I am trying to figure out a simple thing to do using ARkit and Unity and I need help with it. I would like to place 3D objects in my room on detected planes and save the 3D objects position and rotation along with the detected feature points on to a file. Then I want to be able to load the saved session again by opening the app and see the 3D objects right where I kept them in AR. Now I have been able to save and load the world map using ARkit plugin in unity and I have managed to place objects where I want to place them using a Tap on screen. But I am not able figure out how to save the position and orientation of the 3D objects placed in my room and load them along with the world map. I tried saving the world position of the 3D objects and reload them after loading world map, but it is highly inaccurate.

The key part is I want the load the 3D objects at the whatever places I kept them before saving the world map. I keep reading about AR Anchors or World Anchors in blogs but I don't see it with a specific implementation in unity. Could anybody please guide me on how to efficiently and accurately save the position and rotation of the 3D objects in a session so that they could be loaded back later at the same place?

I am building this app with unity 2019.2.0 and ARkit XR plugin 2.1.1. Look forward to suggestions and advises.

Regards,
Sanket.

 

Did you tried Map Sdks? To store the locations of the objects?

 

Actually the 3D objects are going to be inside my bedroom. They might be on the top of a table or side of the bed, etc. I am not sure if saving GPS locations would help in this case as this is going to be entirely indoors.

Now as seen in some videos, Placenote SDK allows to do something like this, but I am trying to figure out if this could be possible only using Anchors in ARkit XR plugin in Unity.

Again, I have not found any concrete documentation or tutorial to use AR Anchors in Unity. Is it possible to use Anchors for this in some way along with AR world maps? If yes, how to use anchors with unity ARkit plugin?

 

Sorry, I just started developing using Unity recently. Still don't have vast knowledge yet. I just said that because I thought it might help.

 

Hey everyone, just a heads up, I cracked it and leaving it here for someone trying to accomplish this.

If you want to save 3D objects at their positions in AR and spawn them right at the correct position and orientation every time, then you can just place the object on a detected plane using ARkit with AR Foundation and keep the object as its child and then save the plane's trackable ID along with the 3D object's position and rotation vector may be in a persistent way like a serialized DTO object or cloud database.

Just save the world map to save all the detected feature points along with the position, rotation vector and plane trackable id on which the 3D object is placed. For spawning them right where you saved them, just download both (world map and serialized object) and apply the world map to the AR session and simultaneously start a coroutine to keep checking if the desired plane with the saved trackable id is detected in the session. Once found, just place the 3D object as its child and at the saved local position and rotation. And Voila, the 3D objects will be where you saved them previously irrespective of where you start the app from.

Some notes after working hours on it:
1) Make sure to place the 3D object as the child of the plane to have more accurate placement instead of storing its world position. You should get the transform of the plane by looping through the ARPlaneManager's trackables. You can place this (https://github.com/Unity-Technologi...er/Assets/Scripts/PlaneDetectionController.cs) script on Gameobject where ARPlaneManager is placed.
2) Saving world map is an important aspect of this along with plane detection. Saving world map will save the feature points that are detected which will instantaneously spawn the detected planes on verifying if the save feature points are valid for the current environment (like a re-localization). The instantaneous detection of the saved planes in the previous session will ensure quick respawning of your 3D objects where they were saved.
3) This s right now just possible in ARkit due to its world map feature and not in ARCore. So it will only work in iOS devices with ARkit support and not in android. The world map feature is pretty cool and I hope and wish Google rolls it out for ARcore as well.

 

Hi, @thesanketkale
I am also trying to save game object with AR world map data.
If you already accomplish it, can u please share the code that place AR objects as a child of the AR plane and save/load their info with worldmap?

I appreciate your help!

 

I'll explain some bits, while I would highly suggest understanding theories of AR Session Management, AR Plane Detection, Relocalization, and Serialization first to fully understand the implementation workflow. Decent background of how stuff works in Computer Vision algorithms will take you a long way, otherwise, lot of these things won't make any sense.

Now, this post is about implementing placing 3D objects in physical space and then seeing them exactly where they were on restarting the app. It could be achieved simply using the Placenote SDK, but if you wish to understand how it would work natively on iOS, then read on.

I am skipping the implementation of the AR parts like adding ARSession, ARSessionOrigin, ARPointCloud, ARPlaneManager objects in Unity as that's pretty standard. Now, the first step is going to be scanning a room/someplace and detect as many feature points as possible. Once enough feature points are available, ARPlaneManager will start creating planes on the linear surfaces. Make sure plane detection is on as it is required here to anchor the 3D objects. Now, once a plane is detected, place your 3D objects as a child of the plane. Placing a 3D object as a child of a plane is in fact called anchoring to a plane.

If you are wondering how to implement tap to place on an AR plane, here is a good tutorial:

Now, you need to anchor to a plane because each plane has a TrackableId which is useful to identify the ARPlane object. The moment you place your 3D object on a plane, you should save the trackable_subId1, trackable_subId2 of the ARPlane, and localposition and localrotation of the 3D gameobject along with some identification of the 3D object (like prefab name or some id) in a serializable object. You need to persistently save this information along with the trackable ids of each plane where you have anchored your 3D objects so that you are able to spawn the corresponding prefab back at the same position and orientation later.

Now once the 3D objects are placed and their corresponding plane's trackable ids and object's pose info captured, simply save the world map data from the ARSession object on the device or on a server using something like below.

Code (CSharp):

1.     public ARSession m_ARSession;
2.  
3.     private IEnumerator SaveWorldMapData()
4.     {
5. #if UNITY_IOS
6.         if (isWorldMapSupported)
7.         {
8.             var sessionSubsystem = (ARKitSessionSubsystem)m_ARSession.subsystem;
9.             if (sessionSubsystem == null)
10.             {
11.                 Debug.Log("No session subsystem available. Could not save.");
12.                 yield break;
13.             }
14.  
15.             var request = sessionSubsystem.GetARWorldMapAsync();
16.  
17.             while (!request.status.IsDone())
18.                 yield return null;
19.  
20.             if (request.status.IsError())
21.             {
22.                 Debug.Log(string.Format("Session serialization failed with status {0}", request.status));
23.                 yield break;
24.             }
25.  
26.             var worldMap = request.GetWorldMap();
27.             request.Dispose();
28.  
29.             Debug.Log("Serializing world map to byte array...");
30.             var data = worldMap.Serialize(Allocator.Temp);
31.             Debug.Log(string.Format("World map has {0} bytes.", data.Length));
32.             fileByteArray = data.ToArray();
33.             data.Dispose();
34.             worldMap.Dispose();
35.          
36.             //Here fileByteArray will contain all the feature points detected in the current session and will be available in a serialized format for storing and retrieving
37.             //TODO: Save fileByteArray to local file or upload to a server using UnityWebRequest
38.         }
39. #endif
40.     }

On saving the world map data, save the serializable object containing the trackable ids and corresponding 3D object's information to another serialized file or database. Both, world map data and the serializable object containing the info of trackable ids are required for placing them where they were in the saved session.

Now, its time to load the world map data from the serialized object to ARSession object. This will be used in a new ARSession. You can have fancy UI, but the point is to check if any world map is available on the device or server and if yes, you can get the saved world map data from the local file or using UnityWebRequest from a server. Once you have the world map data, use something like below to apply it to the new session.

Code (CSharp):

1.     private void LoadWorldMapData(byte[] allBytes)
2.     {
3.         m_ARSession.enabled = true;
4. #if UNITY_IOS
5.         if (isWorldMapSupported)
6.         {
7.             var sessionSubsystem = (ARKitSessionSubsystem)m_ARSession.subsystem;
8.             if (sessionSubsystem == null)
9.             {
10.                 Debug.LogError("No session subsystem available. Could not load.");
11.                 return;
12.             }
13.  
14.             var data = new NativeArray<byte>(allBytes.Length, Allocator.Temp);
15.             data.CopyFrom(allBytes);
16.             Debug.Log("Deserializing to ARWorldMap...");
17.             ARWorldMap worldMap;
18.             if (ARWorldMap.TryDeserialize(data, out worldMap))
19.                 data.Dispose();
20.  
21.             if (worldMap.valid)
22.             {
23.                 Debug.Log("Deserialized successfully.");
24.                 Debug.Log("Applying save world map to current session.");
25.                 sessionSubsystem.ApplyWorldMap(worldMap);
26.             }
27.             else
28.             {
29.                 Debug.LogError("Data is not a valid world map.");
30.                 return;
31.             }
32.         }
33.         else
34.         {
35.             Debug.Log("AR World point map feature not available in unity editor and android devices...");
36.         }
37. #endif
38.     }

Once the world map data is applied to the ARSession object, it will validate the feature points detected in the current session to the ones supplied by you. This is basically identifying if you are in the same room whose world map data you supplied. This validation takes some time, based on your device, the number of feature points detected while saving the world map, and the number of feature points detected in the new session.

Once the world map data you supplied is successfully validated, the ARSession will spawn all the feature points and the corresponding ARPlanes at their corresponding positions and orientations automatically. Something like relocalization from on a saved session state.

Now, in a coroutine, keep checking if the planes on which you had placed the 3D objects are spawned in the session using the TrackableIds. You can do it using something like below:

Code (CSharp):

1.     ARPlaneManager m_ARPlaneManager;
2.  
3.     public ARPlane GetDetectedPlaneObj(TrackableId trackableId)
4.     {
5.         foreach (var plane in m_ARPlaneManager.trackables)
6.         {
7.             if (plane.trackableId.subId1.Equals(trackableId.subId1) &&
8.                 plane.trackableId.subId2.Equals(trackableId.subId2))
9.             {
10.                 return plane;
11.             }
12.         }
13.         return null;
14.     }
15.  
16.     public ARPlane GetDetectedPlaneObj(string trackable_plane_id, string trackable_subId1, string trackable_subId2)
17.     {
18.         var inputTrackableId = new TrackableId(Convert.ToUInt64(trackable_subId1), Convert.ToUInt64(trackable_subId2));
19.  
20.         //Debug.LogError("Input trackable_subId_1: " + trackable_subId1);
21.         //Debug.LogError("Input trackable_subId_2: " + trackable_subId2);
22.         //Debug.LogError("Input trackable ID: " + inputTrackableId);
23.  
24.         foreach (var plane in m_ARPlaneManager.trackables)
25.         {
26.             //Debug.LogError("Detected plane trackable ID: " + plane.trackableId);
27.             //Debug.LogError("Detected plane trackable_subId1: " + plane.trackableId.subId1);
28.             //Debug.LogError("Detected plane trackable_subId2: " + plane.trackableId.subId2);
29.             if (plane.trackableId.Equals(inputTrackableId) || (plane.trackableId.subId1.Equals(
30.                 Convert.ToUInt64(trackable_subId1)) && plane.trackableId.subId2.Equals(
31.                 Convert.ToUInt64(trackable_subId2))))
32.             {
33.                 return plane;
34.             }
35.         }
36.         return null;
37.     }

Once you have the ARPlane object in the current session, simply get the corresponding 3D object prefab and spawn it as its child at the saved local position and with saved local rotation. Keep something like a while loop to iterate through the list, if you have more than one 3D objects placed, and do this for all of them.

Well, though this is not an extensive tutorial, hopefully, it gives enough idea to the people in need.

 

I had thought that ARAnchors should be used and not the trackableIDs of the planes ...

Also, I'm testing in AR Foundation Remote and can't seem to have the same plane trackableIDs from the previous saved session occur even after successful serialization / ApplyWorldMap @KirillKuzyk

 

You can use either ARAnchor or ARPlane, depending on your use case.
But as an answer to the original question, I would recommend using anchors.

Just tested it once more, ARPlane trackable IDs stay the same after serialization/ApplyWorldMap. Could you please double-check?

 

Can I save a Gameobject with ARAnchor component attached to it, on a worldmap?