AR Foundation Remote | Test and debug your AR project in the Editor

Thank you for reporting the bug!
Please update to version 2.0.4 for the fix.

 

When the texture is unpacked to the memory, its size can easily increase tenfold.
I've just increased the max network message size. Please update to version 2.0.4 and tell me if it works for you.

 

@BinaryBanana @shiquhudong
The newest version of the plugin now supports Occlusion CPU images.
This feature is available in both AR Foundation Editor Remote and AR Foundation Remote 2.0.
Your feedback is very appreciated

 

I have confirmed that the bug has been fixed in version 2.0.4.
Thank you for quick response!

 

Hey , thanks alot for the Session Recording feature. Is it possible to record a session on the phone without being connected to the Unity Editor? Let's say I go to a specific place, record the session on my phone with the corresponding file and play it back in the Unity Editor.
Best,

Ribin

 

hi there,

i have just installed the ar remote asset. but i have a problem with location servies example. although i enabled location in my device and also added ENABLE_AR_FOUNDATION_REMOTE_LOCATION_SERVICES
to the scripting define symbols i get this error message:

 

this error:

 

No, recording sessions "in the field" is not supported yet because I currently don't know how to make it right.

For example, ARKit and ARCore can record sessions without being connected to a computer. But their recording implementation covers only the simplest case: the session configuration is fixed (only planes and cloud points) and can't be changed during the recording process.

My session recording implementation is far more versatile. With my plugin, you can record all supported features and you can change session configuration during the recording. For example, you can start with plane tracking on a world-facing camera, then switch to face tracking on a user-facing camera, then switch back to the work-facing camera and enable image tracking. To support this, two-way communication is happening between the Unity Editor and the AR Companion app while you record a session. So to support the "in the field" recording, I have to simulate the behavior of the target AR app and the sequence of AR Foundation API calls it produces. There are two possible ways to achieve this:

1. First, make a recording of a user's app with an AR device connected to the computer to capture its behavior and session configuration changes. Then, use the recorded behavior "in the field". This method is the most flexible, but it would be very hard to explain to users.
2. Or let the users choose from the pre-defined list of configurations and sacrifice the flexibility of the previous method.

I'm sorry if this explanation was overloaded with technical details, I just wanted to say that implementing in-the-field recordings is harder than one might think.

 

Please make sure you made a new AR Companion build after adding the ENABLE_AR_FOUNDATION_REMOTE_LOCATION_SERVICES define.
I had to cover the Location Services feature under this define because this feature requires additional app permissions. And this is not suitable for all of my customers because not all apps use GPS.

 

works after new build!! thanks!

 

Ok, so I filed a bug report and the Unity team has confirmed the issue with BinaryFormatter in Unity 2021.2.
This issue prevents the plugin from working on iOS in Unity 2021.2 and it's possible to track its status on the issue tracker.

 

Hi @KirillKuzyk,
Thanks for the tip, in some case I have some drift with object position like you se below. (after scan large zone and back to )
How can I prevent drift like this ?



Best,

 

Do you encounter this behavior in a real build or in the Unity Editor?
Currently, my plugin doesn't synchronize the camera video with the camera pose. And because the camera pose is updated more frequently than the camera video, you may see the discrepancy. I'll soon add a setting so users can sacrifice the camera pose update frequency and keep it in sync with the video.

 

Hi, in real build

 

Please check that you place your augmented objects under the ARSessionOrigin object.
But it also may be the limitation of ARKit. It can lose precision if there are not enough feature points or bad lighting.

 

What I would like to do is use the AR Companion app with the device GPS location. I'm not even sure if this is possible TBH, but looks like it is?

I have added the ENABLE_AR_FOUNDATION_REMOTE_LOCATION_SERVICES define symbol in both the Standalone and iOS (building on a Mac to iPhone) player settings and rebuilt the companion app but when running my scene a location of 0,0 is reported as my GPS coordinates. Also, permission to use location when first launching the companion app does not show which means to me that location services are not a part of the companion app build.

Is it possible to get the GPS location of the phone using the companion app? If so, what am I doing wrong here?

 

Does the LocationServicesExample.unity scene work for you?
The GPS will ask permission and will start to return the correct values only after the call to Input.location.Start().
Please also make sure you've added this using statement on top of your script:

Code (CSharp):

1. using Input = ARFoundationRemote.Input;

 

It does work. I didn't see that example scene there. Doh!

Thanks for the help!

 

I followed the steps in tutorial video "Unity AR Foundation Editor Remote Plugin Now Available !".
But the pose of my sloth is frozen in BlendshapeExampe scene view.
And then I switch to game view again it looks like it's motion will speed up for a while.

Could anybody tell me how to remain tracing in scene view? just like the tutorial video

 

Please align the Scene and Game windows side-by-side, so both windows are visible. The plugin can only send updates if the Game window is visible.

 

Hi @KirillKuzyk

Which version of ARFoundation Remote would work with this constellation of Packages in Unity 2019.4:

Is this possible at all?
I would like to try rebuilding a copy of my project in 2019.4 in pursuit of a solution for my segmentation fault issue and I am bound to these early Versions by another package. I cannot update AR Froundation an ARCore 3.0.1 as demanded by the latest version of AR Foundation Remote.

 

Hi,
Unfortunately, the 3.0.1 requirement was always there, even in the first version of the plugin. There were so many breaking changes between AR Foundation 2.1 and 3.0, so I decided from day one it's not worth it to support AR Foundation older than 3.0. I would like to add that 3.0.1 is only a minimum requirement and any newer version will work with the plugin.

 

Regarding GitHub - googlesamples/arcore-depth-lab: ARCore Depth Lab is a set of Depth API samples that provides assets using depth for advanced geometry-aware features in AR interaction and rendering. (UIST 2020)
Does the remoting tool support the full ARCore depth feature set / samples listed in the repo, incl. those not yet available with the AR Foundation SDK? Thanks!

 

UPDATE:
Please join the GitHub discussion if you have more questions about the ARCore Depth Lab.

ORIGINAL RESPONSE:
A more correct question would be "can googlesamples/arcore-depth-lab repo work in the Editor?" And the answer is "partially".

My plugin does support the Depth API provided by the AR Foundation, including Depth CPU images as of version 1.4.12-release.0. But the arcore-depth-lab repo uses Android-specific shaders that can't work in the Editor. So only OrientedReticle and AvatarLocomotion scenes work in the Editor with the help of AR Foundation Remote.

Also, there are few minor fixes you need to apply for the arcore-depth-lab to work in the Editor:

1. Remove the

   #if !UNITY_EDITOR

   from the DepthDataSourceConfig.cs.
2. Delete or comment out the whole Recorder.cs file because ARCore Recording API is not supported by my plugin.

 

Thanks!

Two additional questions:

1. Why does the AR Foundation Remote restrict the number of XRCpuImage instances existing at any given time to only 1? I can build and run a project with several XRCpuImages living at the same time. So where does the limitation in the Remote come from?
2. Is there a way to wait for the arrival of an XRCpuImage from the companion app before calling code that relies on the image to be ready? I.e. some bool that I can use for WaitUntil in a coroutine – or something along those lines? Higher resolution images (720p and up) take too long to arrive, which blocks me from testing with them in the Remote.

 

1. This limitation comes from the network delay and the fact that the plugin acquires the CPU image lazily. The first time you request a CPU image by calling a TryAcquireCpuImage(), the plugin tells the AR Companion app to start sending CPU images. Only after the roundtrip to the companion app, the TryAcquireCpuImage() method will start to return true.
To address this limitation, I have to block the Editor main thread and wait for the CPU image to be downloaded, but this would be a horrible user experience and will make Unity Editor's framerate unbearable. So I decided to introduce the limitation of one XRCpuImage at a time.
2. I assume you're speaking about the XRCpuImage.Convert(), right? There is currently no way to check if the plugin finished downloading the CPU image before making synchronous image conversion. But in a real build, it's also not possible to tell in advance with 100% confidence that a call to TryAcquireCpuImage() followed by XRCpuImage.Convert() will succeed. In a real build, TryAcquireCpuImage() can return false at any time, and XRCpuImage.Convert() can throw an exception at any time.
One possible solution is to use XRCpuImage.ConvertAsync() method (coroutine or callback version). By using the async conversion method, you'll get notified as soon as the image is ready.

 

After installing all stuff i got an errror

Code (CSharp):

1. AssertionException: xrGeneralSettings != null
2. Assertion failure. Value was Null
3. Expected: Value was not Null
4. UnityEngine.Assertions.Assert.Fail (System.String message, System.String userMessage) (at /Users/bokken/buildslave/unity/build/Runtime/Export/Assertions/Assert/AssertBase.cs:29)
5. UnityEngine.Assertions.Assert.IsNotNull (UnityEngine.Object value, System.String message) (at /Users/bokken/buildslave/unity/build/Runtime/Export/Assertions/Assert/AssertNull.cs:58)
6. UnityEngine.Assertions.Assert.IsNotNull[T] (T value, System.String message) (at /Users/bokken/buildslave/unity/build/Runtime/Export/Assertions/Assert/AssertNull.cs:46)
7. ARFoundationRemote.Runtime.Global.isPluginEnabledInXRManagementWindow () (at Library/PackageCache/com.kyrylokuzyk.arfoundationremote@20c19dfb4070-1631522281000/Runtime/Utils/Global.cs:32)
8. ARFoundationRemote.Runtime.Global.IsPluginEnabled () (at Library/PackageCache/com.kyrylokuzyk.arfoundationremote@20c19dfb4070-1631522281000/Runtime/Utils/Global.cs:25)
9. ARFoundationRemote.Editor.AnchorSubsystem.RegisterDescriptor () (at Library/PackageCache/com.kyrylokuzyk.arfoundationremote@20c19dfb4070-1631522281000/Editor/Subsystems/AnchorSubsystem.cs:12)
10.  

 

Thanks! The plugin is really awesome!

 

This error can appear if the 'XR Plugin Management' window has never been opened.
Please make sure to enable the 'AR Foundation Remote' provider:


I'll fix the error in the next version and will display a tutorial message. Thank you for reporting the bug!

 

Hi, I have a problem. I run 1.4.13-release.3, but had an error below.
Any suggestion to solve this?

Exception: AR Foundation Remote: BlockUntilReceive() timeout.
ARFoundationRemote.RuntimeEditor.TelepathyReceiverConnection.ARFoundationRemote.Runtime.IReceiverConnection.BlockUntilReceive[ResponseType] (ARFoundationRemote.Runtime.BlockingMessage payload) (at Library/PackageCache/com.kyrylokuzyk.arfoundationremote@82381a011c5f-1631783674000/Runtime/Connection/TelepathyReceiverConnection.cs:72)
ARFoundationRemote.Runtime.Connection.BlockUntilReceive[ResponseType] (ARFoundationRemote.Runtime.BlockingMessage payload) (at Library/PackageCache/com.kyrylokuzyk.arfoundationremote@82381a011c5f-1631783674000/Runtime/Connection/Connection.cs:61)
ARFoundationRemote.Runtime.SessionSubsystem+ARemoteSessionSubsystemProvider.GetConfigurationDescriptors (Unity.Collections.Allocator allocator) (at Library/PackageCache/com.kyrylokuzyk.arfoundationremote@82381a011c5f-1631783674000/Runtime/SessionSubsystem.cs:147)
UnityEngine.XR.ARSubsystems.XRSessionSubsystem.GetConfigurationDescriptors (Unity.Collections.Allocator allocator) (at Library/PackageCache/com.unity.xr.arsubsystems@4.2.0/Runtime/SessionSubsystem/XRSessionSubsystem.cs:194)
UnityEngine.XR.ARSubsystems.XRSessionSubsystem.DetermineConfiguration (UnityEngine.XR.ARSubsystems.Feature features) (at Library/PackageCache/com.unity.xr.arsubsystems@4.2.0/Runtime/SessionSubsystem/XRSessionSubsystem.cs:92)
UnityEngine.XR.ARSubsystems.XRSessionSubsystem.Update (UnityEngine.XR.ARSubsystems.XRSessionUpdateParams updateParams) (at Library/PackageCache/com.unity.xr.arsubsystems@4.2.0/Runtime/SessionSubsystem/XRSessionSubsystem.cs:124)
UnityEngine.XR.ARFoundation.ARSession.Update () (at Packages/com.unity.xr.arfoundation@4.2.0-pre.12/Runtime/AR/ARSession.cs:415)

 

Is there a chance you had an older plugin version and recently updated it to a newer one? Please make sure you made a new AR Companion app build after the update. Session Configurations is a new feature and the old version of the AR Companion app doesn't send an expected response, resulting in this error.

 

I moved to the async callback version. Works. Thx!

 

Hi KirillKuzyk,I encountered the issue like https://answers.unity.com/questions/1861923/call-inputlocationstart-but-status-keeps-to-be-sto.html
The issue is calling Input.location.Start seems not to work properly.
If I removed custom AR Foundation package and custom AR Foundation Editor Remote package, the issue goes away.

Instead of removing AR Foundation Editor Remote, is there any way to improve this issue?
I tried to update AR Foundation Editor Remote but I failed( some errors happen when building).

 

I'm using this version.
Unity ver is 2020.3.4f1.

 

I tried to add the line 'using Input = ARFoundationRemote.Input;'. Some build errors happens( there is no declaration for Input.gyro and Input.compass and status type changed to LocationServiceStatusDummy). I changed 'Input' to 'UnityEngine.Input' and added Scripting Define Symbol 'ENABLE_AR_FOUNDATION_REMOTE_LOCATION_SERVICES'.

Now, I can run app on real device but result are same. I didn't read whole source code but using ARFoundationRemote.Input is just hooking, therefore result will be same, right?

 

I'm using a little bit old AR Foundation Editor Remote. If there is a correct way to update plug in, please tell me.

 

Sorry, my conclusion above may be wrong. I thought issue has gone after removing AR FoudnationEditor Remote but it may not be true.
I didn't specify exact cause. If I found it I'll report it.

 

Hi @tw-unityasset2,

Input.gyro and Input.compass are not currently supported by the plugin. I have these features on my roadmap.
Apart from that, you're doing everything right. The Input.location should work exactly the same way as it works on real devices.

 

Thank for your reply. Still I'm investigating the issue. If this issue seems to be related to your product, I'll come back.

 

I could make repro without your product. Sorry for my misinformation.
https://github.com/omanuke/TestARCoreLocation

 

Hello, does ARFoundation Remote 2.0 work with Unity 2021.2 beta now?

Update:
Remote2.0 can be successfully imported, installed in Unity 2021 2.14 beta. And can also build ARCampnion app on Ipad devices. But throws errors as the post below

 

Exception: AR Foundation Remote: BlockUntilReceive() timeout error tested in multiple sample scenes.
I'm on AR Foundation Remote 2.0 with Unity 2021.2 beta14. I tested to run multiple example scenes come with ARkit/AR Foundation.

The feed from ipad pro will somewhat still update in editor even with the errors, but extremely lagging.

 

In addition, if try to retrieve depth textures from AR Foundation, besides the BlockUntilReceive() timeout error, it will also throw XRCpuImage.Convert() failed errors as the attached screenshot:

 

Unfortunately, no. This issue is preventing the plugin from working correctly in Unity beta 2021.2. Please follow the status of the issue until it will say "Fixed in...".

 

Thank you. This issue has been around for 1 month, really hope they can fix it soon -_-

In the meanwhile, do you have local workaround to temporarily fix it just so i can at least keep the flow going?

 

While a workaround is technically possible, it doesn't worth the pursuit in my opinion. It would require rewriting a big part of the plugin's data structures, and the risk of breaking things for existing customers is huge.
I just received an update regarding the bug report and a fix coming soon to Unity 2022.1.0a12. They're also considering backporting the fix to Unity 2021.2.

 

Code (CSharp):

1.             if (m_arAnchorManager.subsystem != null)
2.             {
3.                
4.                 m_anchorStore = await m_arAnchorManager.subsystem.LoadAnchorStoreAsync();
5.  
6.                 // Request all persisted anchors be loaded once the anchor store is loaded.
7.                 foreach (string name in m_anchorStore.PersistedAnchorNames)
8.                 {
9.                    
10.                     TrackableId trackableId = m_anchorStore.LoadAnchor(name);
11.                     m_incomingPersistedAnchors.Add(trackableId, name);
12.                 }
13.             }

So I have this code in my start function and this script gets instantiated every scene switch, And destroyed before every scene switch. For some reason. When I start the game the first time and start my xr, The trackable ID is a random GUID number. When I then switch from that scene to another scene, the trackableID is 00000000000-00000000000. It has the same name in the loadAnchor function so I guess the trackable id should be the same as the scene before. Any clue what could cause this issue? Because of this I lose tracking after scene switching

 

Big fan of this plugin, been using the 1.0 for weeks and just upgraded to 2.0 today. Speaking of which, since I've upgraded I have the configuration descriptors logging to the console whenever I launch the Unity Editor.

Is there a way to disable this? I've seen in the settings there are toggles to disable startup errors and telepathy warnings, but neither of those control it.

 

Could you please provide more context of what you're trying to do?
I found this question in the OpenXR-Unity-MixedReality-Samples and I believe your question is related to it, right?
The thing is my plugin doesn't officially support HoloLens because I don't have this device. I would be very grateful if you can post your feedback in the beta test thread and describe your experience. What's working and what's not, how does the plugin compares to the MRTK Remoting Player?