Search Unity

  1. Read here for Unity's latest plans on OpenXR.
    Dismiss Notice

Bug ARFoundation: ImageTracking with WorldMap

Discussion in 'AR' started by kenjiro_yamasaki, Feb 2, 2021.

  1. kenjiro_yamasaki

    kenjiro_yamasaki

    Joined:
    Feb 1, 2021
    Posts:
    11
    Hi,
    I'm writing an AR program that uses world map and image tracking.
    Then I ran into a problem.

    What I want to do is simple.
    1. Load the world map.
    2. Add the image to the image reference library. (at runtime)

    Then image tracking does not work properly.
    The "trackedImagesChanged" event of "ARTrackedImageManager" is fired, but the ARTrackedImage passed as an argument is messed up. "ARTrackedImage.referenceImage.name" should normally be set to the name you specified during registration, but in the above situation it is set to "null". It is not possible to determine which image was detected.

    Is there a way to fix the problem?
    Best Regard,

    [Environment]
    - Unity 2019.4.16f1
    - ARFoundation 4.1.1
    - ARKit XR Plugin 4.1.1
    - iPad Pro 2020
    - iOS 14.3

    Code (CSharp):
    1.  
    2. using System.Collections;
    3. using System.IO;
    4. using TMPro;
    5. using Unity.Collections;
    6. using UnityEngine;
    7. using UnityEngine.UI;
    8. using UnityEngine.XR.ARFoundation;
    9. using UnityEngine.XR.ARKit;
    10. using UnityEngine.XR.ARSubsystems;
    11.  
    12. namespace SoftCube.Domain
    13. {
    14.     public class Manager : MonoBehaviour
    15.     {
    16.         [SerializeField]
    17.         private ARSession arSession = default;
    18.  
    19.         [SerializeField]
    20.         private ARTrackedImageManager arTrackedImageManager = default;
    21.  
    22.         [SerializeField]
    23.         private Button worldMapSaveButton = default;
    24.  
    25.         [SerializeField]
    26.         private TMP_Text messageText = default;
    27.  
    28.         private IEnumerator Start()
    29.         {
    30.             arTrackedImageManager.trackedImagesChanged += ArTrackedImageManager_trackedImagesChanged;
    31.             worldMapSaveButton.onClick.AddListener(OnWorldMapSaveButtonClick);
    32.  
    33.             var worldMapFilePath = Path.Combine(Application.persistentDataPath, "ImageTracking.worldmap");
    34.             if (File.Exists(worldMapFilePath))
    35.             {
    36.                 yield return StartCoroutine(LoadWorldMapCoroutine(worldMapFilePath));
    37.             }
    38.  
    39.             LoadImage();
    40.         }
    41.  
    42.         private void ArTrackedImageManager_trackedImagesChanged(ARTrackedImagesChangedEventArgs e)
    43.         {
    44.             foreach (var image in e.added)
    45.             {
    46.                 Debug.Log(image.referenceImage.name);
    47.             }
    48.             foreach (var image in e.updated)
    49.             {
    50.                 Debug.Log(image.referenceImage.name);
    51.             }
    52.         }
    53.  
    54.         private void OnWorldMapSaveButtonClick()
    55.         {
    56.             var worldMapFilePath = Path.Combine(Application.persistentDataPath, "ImageTracking.worldmap");
    57.             StartCoroutine(SaveWorldMapCoroutine(worldMapFilePath));
    58.         }
    59.  
    60.         public IEnumerator LoadWorldMapCoroutine(string worldMapFilePath)
    61.         {
    62.             messageText.text = $"Load WorldMap.";
    63.  
    64.             var arSessionSubsystem = (ARKitSessionSubsystem)arSession.subsystem;
    65.             if (arSessionSubsystem == null)
    66.             {
    67.                 messageText.text = "No session subsystem available. Could not load.";
    68.                 yield return new WaitForSeconds(5.0f);
    69.                 yield break;
    70.             }
    71.             if (!File.Exists(worldMapFilePath))
    72.             {
    73.                 messageText.text = $"File {worldMapFilePath} does not exist.";
    74.                 yield return new WaitForSeconds(5.0f);
    75.                 yield break;
    76.             }
    77.  
    78.             var allBytes = File.ReadAllBytes(worldMapFilePath);
    79.             var data = new NativeArray<byte>(allBytes.Length, Allocator.Temp);
    80.             data.CopyFrom(allBytes);
    81.  
    82.             ARWorldMap worldMap;
    83.             if (ARWorldMap.TryDeserialize(data, out worldMap))
    84.             {
    85.                 data.Dispose();
    86.             }
    87.  
    88.             if (!worldMap.valid)
    89.             {
    90.                 messageText.text = "Data is not a valid ARWorldMap.";
    91.                 yield return new WaitForSeconds(5.0f);
    92.                 yield break;
    93.             }
    94.  
    95.             messageText.text = $"Apply WorldMap.";
    96.             arSessionSubsystem.ApplyWorldMap(worldMap);
    97.  
    98.             yield return new WaitForSeconds(1.0f);
    99.             while (ARSession.state != ARSessionState.SessionTracking)
    100.             {
    101.                 yield return null;
    102.             }
    103.             messageText.text = string.Empty;
    104.         }
    105.  
    106.         public IEnumerator SaveWorldMapCoroutine(string worldMapFilePath)
    107.         {
    108.             messageText.text = $"Save WorldMap.";
    109.  
    110.             var arSessionSubsystem = (ARKitSessionSubsystem)arSession.subsystem;
    111.             if (arSessionSubsystem == null)
    112.             {
    113.                 messageText.text = "No session subsystem available. Could not save.";
    114.                 yield return new WaitForSeconds(5.0f);
    115.                 yield break;
    116.             }
    117.  
    118.             var request = arSessionSubsystem.GetARWorldMapAsync();
    119.             while (!request.status.IsDone())
    120.             {
    121.                 yield return null;
    122.             }
    123.             if (request.status.IsError())
    124.             {
    125.                 messageText.text = $"Session serialization failed with status {request.status}";
    126.                 yield return new WaitForSeconds(5.0f);
    127.                 yield break;
    128.             }
    129.  
    130.             var worldMap = request.GetWorldMap();
    131.             request.Dispose();
    132.  
    133.             var data = worldMap.Serialize(Allocator.Temp);
    134.             var file = File.Open(worldMapFilePath, FileMode.Create);
    135.             var writer = new BinaryWriter(file);
    136.             writer.Write(data.ToArray());
    137.             writer.Close();
    138.             data.Dispose();
    139.             worldMap.Dispose();
    140.  
    141.             yield return new WaitForSeconds(3.0f);
    142.             messageText.text = string.Empty;
    143.         }
    144.  
    145.         private void LoadImage()
    146.         {
    147.             if (arTrackedImageManager.CreateRuntimeLibrary() is MutableRuntimeReferenceImageLibrary referenceLibrary)
    148.             {
    149.                 arTrackedImageManager.referenceLibrary = referenceLibrary;
    150.  
    151.                 foreach (var imageFilePath in Directory.GetFiles(Application.persistentDataPath, "*.png"))
    152.                 {
    153.                     var texture = new Texture2D(1, 1);
    154.                     texture.LoadImage(File.ReadAllBytes(imageFilePath));
    155.  
    156.                     var job = referenceLibrary.ScheduleAddImageWithValidationJob(texture, Path.GetFileName(imageFilePath), 0.1f);
    157.                     job.jobHandle.Complete();
    158.                     Debug.Log(job.status);
    159.                 }
    160.                 arTrackedImageManager.enabled = true;
    161.             }
    162.         }
    163.     }
    164. }
    165.  
     
  2. FrankvHoof

    FrankvHoof

    Joined:
    Nov 3, 2014
    Posts:
    233
    Does the textureGuid in the referenceImage match?
     
  3. kenjiro_yamasaki

    kenjiro_yamasaki

    Joined:
    Feb 1, 2021
    Posts:
    11
    Thank you for reply.

    referenceImage.name is null.
    referenceImage.guid is 00000000-0000-0000-0000-000000000000.
    referenceImage.textureGuid is 00000000-0000-0000-0000-000000000000.
    referenceImage.size is (0.0, 0.0).
    referenceImage.texture is null.

    I think there are three possibilities.
    1. There is a mistake in my usage of the API.

    2. Bug of ARFoundation/ARKit

    3. Specification of ARFoundation/ARKit.
      In other words, WorldMap is not designed to be used with image tracking.
    Which case is this time?
     
  4. FrankvHoof

    FrankvHoof

    Joined:
    Nov 3, 2014
    Posts:
    233
    I can tell you for certain that it's not ARFoundation. ARFoundation is simply a wrapper around ARKit.
    MutableRuntimeReferenceImageLibrary is an abstract class in ARFoundation, that is implemented by the Provider.

    ARKit SHOULD support mutable referencelibraries (it would return null for that library if it didn't).
    Your usage also seems to be correct.
    My guess is a bug in the ARKit XRPlugin, but I'm not sure.

    Have you tried using one of the other Add-Methods on the MutableRuntimeReferenceImageLibrary?
     
  5. kenjiro_yamasaki

    kenjiro_yamasaki

    Joined:
    Feb 1, 2021
    Posts:
    11
    I tried the following code. This is what you call "the other Add-Methods", right?
    The result did not change (it does not work).

    Code (CSharp):
    1.  
    2. //var job = referenceLibrary.ScheduleAddImageWithValidationJob(texture, Path.GetFileName(imageFilePath), 0.05f);
    3. var guid = [URL='http://www.google.com/search?q=new+msdn.microsoft.com']new[/URL] SerializableGuid();
    4. var textureGuid = [URL='http://www.google.com/search?q=new+msdn.microsoft.com']new[/URL] SerializableGuid();
    5. var size = [URL='http://www.google.com/search?q=new+msdn.microsoft.com']new[/URL] Vector2(0.05f, 0.05f);
    6. var name = Path.GetFileName(imageFilePath);
    7. var referenceImage = [URL='http://www.google.com/search?q=new+msdn.microsoft.com']new[/URL] XRReferenceImage(guid, textureGuid, size, name, texture);
    8. var job = referenceLibrary.ScheduleAddImageWithValidationJob(texture.GetRawTextureData<byte>(), [URL='http://www.google.com/search?q=new+msdn.microsoft.com']new[/URL] Vector2Int(texture.width, texture.height), texture.format, referenceImage);
    9. job.jobHandle.Complete();
    10.  
    I think this is a common use case.
    If it's a bug in the ARKit XR Plugin, it should be investigated and fix in a future release.
     
  6. TreyK-47

    TreyK-47

    Unity Technologies

    Joined:
    Oct 22, 2019
    Posts:
    1,650
    After connecting with the team, it was advised that you put in a bug report for us.
     
  7. kenjiro_yamasaki

    kenjiro_yamasaki

    Joined:
    Feb 1, 2021
    Posts:
    11
    Thank you for reply.
    I will make a report.
     
  8. kenjiro_yamasaki

    kenjiro_yamasaki

    Joined:
    Feb 1, 2021
    Posts:
    11
    I received the following reply from Unity's QA team regarding this matter.
    ```
    I think I found the cause of this issue: It looks like you haven't created an Image Tracking subsystem.
    Can you verify that creating one fixes the problem that you're experiencing?
    ```
    But I didn't know how to verify it.
    Is there anyone who knows how to create an Image Tracking subsystem?

    2A2B3E8A-57C5-4E52-B982-890420573EC5.jpg
     
  9. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    705
    Adding ARTrackedImageManager to the ARSessionOrigin game object should create the Image Tracking subsystem automatically.
    Please ensure that you've also enabled the ARKit provider in the 'XR Plugin Management' window.
    upload_2021-4-27_10-34-48.png


    Does this scene from the official AR Foundation Samples work for you?
    https://github.com/Unity-Technologi...g/BasicImageTracking/BasicImageTracking.unity
     
  10. kenjiro_yamasaki

    kenjiro_yamasaki

    Joined:
    Feb 1, 2021
    Posts:
    11
    Thank you for reply.

    I checked the ARKit provider in the XR Plugin Management window, and it was enabled.
    And AR Foundation Samples work for me.

    My perception is the same as yours. In other words, I believe that the Image Tracking subsystem is generated automatically.
    So, I am having a hard time understanding how to fix the bug regarding the reply from QA team.
     
  11. unity03_unity133

    unity03_unity133

    Joined:
    May 26, 2021
    Posts:
    3
    hello there. Do you have any update on this regard? I tried to write this exact same code, but my mutableLibrary.count is equal to 0 (so no images have been loaded in the mutable library). I have this problem with both ARCore and ARKiit. I'm sure that the application with a "standard" image reference library build in edit mode in Unity works fine.
     
  12. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    705
    Please check the AddReferenceImageJobState.status after AddReferenceImageJobState is finished (only available in AR Foudatuon >= 4.1). If the status is ErrorInvalidImage, it means your image is not suitable for Image Tracking.
     
  13. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    705
unityunity