AR Foundation Remote | Test and debug your AR project in the Editor

It seems to me I have found the issue with Image Tracking on Android. When I send reference images from Editor to companion app, I compress them. And compression makes image tracking not reliable. I’ll release a fix soon.

As for ‘connection lost’ error, could you please describe the situation when it occurs?

 

I’m afraid, the plugin is not suitable for recording video. The video has a low resolution, low fps, and there is no audio.

If you’re using an iOS device, you can record your screen with the help of built-in recording tool. Android OS should have similar tool too.

 

PLUGIN UPDATE POST #10 (version 4.5.0)
​

ARKit Object Tracking support added.
​

This major update brings support for ARKit Object Tracking. Here is how it works:

1. You first scan your 3D object with a scanning app from Apple.
2. Then, you transfer scanned *.arobject file to Unity.
3. Now AR Foundation can recognize scanned real-world objects.

Unfortunately, ARKit Object Tracking has a critical bug that sometimes causes a crash. It seems the crash is happening due to some race condition inside ARKit and, in my experience, is only happening after arSession.Reset() followed by arSession.enabled = false. So avoid resetting and disabling your ARSession when you're using Object Tracking.

ARKit World Map support added.​

So you've spent several hours decorating your room in AR and now you wish to save the positions of your AR objects relative to real-world objects. ARKit World Map serves exactly this purpose.

The World Map feature allows you to save currently detected planes and placed anchors into a file. You can save this file locally or transfer it via a network. This allows you to save and re-create your AR scene in the future.

Imagine doing everything shown in this video, but with the ability to debug your AR scene right in the Editor:

ARKit World Map is very similar to ARCore Cloud Anchors. The main difference lies in the way these systems handle the world map file:

• ARKit World Map lets you control the world map yourself. You can save it locally or send it via a network.
• ARCore Cloud Anchors lets you only save the world map to Google Cloud.

By the way, ARCore Cloud Anchors feature is on my roadmap so stay tuned for future updates.

To test ARKit World Map in Editor, please add these lines to your scripts that handle world map:

Code (CSharp):

1. #if UNITY_IOS && UNITY_EDITOR
2.     using ARKitSessionSubsystem = ARFoundationRemote.Runtime.SessionSubsystem;
3.     using ARWorldMap = ARFoundationRemote.Runtime.ARWorldMapRemote;
4. #endif

Unity 2020.2 support added.
​

Unity 2020.2 is expected to be released somewhere at the end of this year, so it's a good time to prepare the plugin for the official release.

Improvements and bug fixes.​

• The connection to AR Companion app is now persisted across multiple scenes. Thanks to @Duckocide for reporting the issue.
• The ARSession.Reset() does not reset all AR subsystems correctly on Android and on iOS. For example, ARPointCloudManager, ARMeshManager, and ARKit World Map would not be reset properly after calling this method. To overcome this limitation, you should call XRManagerSettings.DeinitializeLoader(), restart current scene, and then call XRManagerSettings.InitializeLoader(). This workaround required me to refactor the AR Companion app to survive the scene reload. This refactoring also increased the overall quality and stability of the companion app.

My roadmap.
​

Thanks to all people who supported me for all this time and thanks to every positive review on the Asset Store and here on the forum. With your help, I can work full-time on the plugin, and hopefully, one day it will become the standard for AR development in Unity.

Here is the list of things I plan to do next. Please vote for a feature you would like to see first, so I can adapt my priorities:

• Support for ARCore Cloud Anchors.
• Support for accessing camera images on CPU.
• Session recordings.
• ... your suggestion.

PLUGIN UPDATE POST #11​

 

Hi,

Thanks for this amazing plugin. The plugin already makes the testing and debugging iterations very fast as compared to builds. It would be great to have recordings of the sensor data and then replaying it. It will make the iterations more fast and will also allow to reproduce results/bugs...

 

The error comes from that issue with Reference Images Library. Really appreciate for your effort on it!

 

@KirillKuzyk I am getting errors when switching between branches of my project relating to the ARMeshManager class, here is the code which is throwing the errors:

Code (CSharp):

1.     // AR_FOUNDATION_EDITOR_REMOTE: delegate mesh subsystem to the plugin
2.     #if UNITY_EDITOR || AR_COMPANION
3.     using XRMeshSubsystem = ARFoundationRemote.Runtime.IXRMeshSubsystem;
4.     using XRGeneralSettings = ARFoundationRemote.Runtime.XRGeneralSettingsRemote;
5.     using SubsystemManager = ARFoundationRemote.Runtime.SubsystemManagerRemote;
6.     using XRMeshSubsystemDescriptor = ARFoundationRemote.Runtime.XRMeshSubsystemDescriptorRemote;
7.     #endif
8.     // AR_FOUNDATION_EDITOR_REMOTE***
9.  

The package is installed, but for some reason it is unable to reference the runtime classes defined above. Any solution to fix this issue? It is very frustrating and every time I try to fix it the process is quite long (usually involved removing it from the project and installing again).

 

Been using this quite a bit, much appreciated for your work here.

One question, with regard to ARKit I need to use the legacy TrackedPoseDriver with a custom pose provider to reflect the camera rotation (to flip the mirrored tracked user-facing camera to behave like it's world-facing). My custom pose provider seems unable to get the camera pose or node pose data and is returning null -- assumably since this is streamed from the companion app. Anything I need to do here to correctly pull this data?

Thanks in advance!

 

I'm sorry for the late reply.

Could you please tell me whether AR Foundation version stays the same across your branches? In my experience, this issue only appears when AR Foundation version changes.

This issue is related to package caching in Unity and I hope I can address it in the next version. Unfortunately, the current hacky workaround is to open the manifest.json file, remove my plugin, go back to Unity, then add the plugin again to manifest.json.

 

Could you please tell me what API you use to get the camera pose? Do you use InputTracking.GetNodeStates()?
I can implement this API in the next version, but you'll have to add this line on top of your custom script:

Code (CSharp):

1. using InputTracking = ARFoundationRemote.InputTracking;

Will it be useful for you?

 

@KirillKuzyk This plugin is so useful and worked very well out of the box! Thank you.

I ran into a problem, however, with meshing. Not sure if this is a bug in the plugin or ARFoundation or something else, but perhaps you have an idea.



What happens:
• meshing instantiates trackables and looks great (left screen) — you can also check in the inspector and see that the mesh is assigned to the MeshCollider component (if present)
• HOWEVER, only SOME of the MeshColliders are correctly initialised (see middle screenshot)
• if you simply deactivate and reactivate the trackables in the scene hierarchy the colliders will then correctly init (right screenshot)
• AFAIK the colliders work fine on the iPhone (I don't have a physics debug display but am not having issues when raycasting against them)

Found this while making a simple navmesh test and finding my raycasts failing even though I could see that the mesh was there.

 

Oops. Forgot system specs:

Unity 2020.1.11f1
AR Foundation 4.1.0-preview.13
AR Foundation Remote 4.5.0
iOS 14.2.1
macOS 10.15.7

 

Any suggestion with errors when trying to connect to the companion app?
Iphone 11 - iOS 14, Unity 2019.4.15f1, Arkit 4.1.1, ArFoundation 4.1.1
Error Log:

Client Recv: failed to connect to ip=30.45.38.93 port=44819 reason=System.Net.Sockets.SocketException (0x80004005): The socket is not connected

  at System.Net.Sockets.TcpClient.Connect (System.String hostname, System.Int32 port) [0x001a9] in <ef151b6abb5d474cb2c1cb8906a8b5a4>:0

  at Telepathy.Client.ReceiveThreadFunction (System.String ip, System.Int32 port) [0x00002] in /Users/plugins/ARFoundationRemoteSource/Runtime/Telepathy/Client.cs:54

UnityEngine.Debug:Log(Object)

Telepathy.Client:ReceiveThreadFunction(String, Int32) (at plugins/ARFoundationRemoteSource/Runtime/Telepathy/Client.cs:75)

Telepathy.<>c__DisplayClass11_0:<Connect>b__0() (at plugins/ARFoundationRemoteSource/Runtime/Telepathy/Client.cs:156)

System.Threading.ThreadHelper:ThreadStart()

 

Thanks for reporting the bug!
Please send me a direct message and I'll send you a new version with a fix.

 

1. Please check that your computer and iPhone are on the same Wi-Fi network.
2. Could you please go to iPhone Wi-Fi settings and check that your IP is indeed 30.45.38.93?

 

hi there, trying to run this on my Pixel which is classed as being AR capable, when running the companion app I get an error, reloadSceneCor() failed because coroutines were running: initSessionCorARFoundationRemote.Runtime ??? any help would be great as it works great on my iPhone 11 Pro

 

I have deleted the companion app from my phone, rebuilt and reinstalled. It now is working correctly.

 

Thanks for the reply, yes through InputTracking.GetNodeStates(), and adding the using statement is not a problem.

Any chance this is something I can patch myself? Or if not trivial, any sense of your ETA for the release that would include it? Bit of a blocker for us. Thanks again, appreciate your help.

 

I've noticed my mobile device (xz3) gets warm when the AR Remote app is an "waiting" state. Battery drains as well (even with USB on charge). Perhaps some tweaks to make the app less power intensive when not in use. Guessing there must be some processing (for connections or perhaps some other processing in the app xxxUpdate() methods?) - Anyway, perhaps a "Sleep" button on the wait screen to shut things down in the app when not needed?

I usually end up closing the app to get some charge back in eventually, so not a big thing.

 

1. Please check that ARCore XR Plugin is installed and enabled in XR Plug-in Management window.
2. If the previous step doesn't help, please connect your Pixel phone to the adb debugger and send me logs to the email: kuzykkirill@gmail.com

 

I added the support for GetNodeStates(). Please tell me your Invoice Number in PM so I can send you a new version before the official Asset Store release.
Please add this line on top of your script after the update:

Code (CSharp):

1. using InputTracking = ARFoundationRemote.Runtime.InputTracking;

I would also recommend to create an *.asmdef file in your project and add a version define for AR Foundation Editor Remote. This way your project will be decoupled from my plugin.
Defining symbols based on project packages.


For example, after adding this version define


You will be able to write the previous line of code like this:

Code (CSharp):

1. #if AR_FOUNDATION_REMOTE_INSTALLED
2. using InputTracking = ARFoundationRemote.Runtime.InputTracking;
3. #endif

 

Thanks for pointing out the issue!
The current implementation sends the AR data to Editor even in waiting mode. I'll release an update soon that will not send the data in waiting mode and it should make the companion app less battery draining.

 

Yup, this is what I have to do as well.

I will check to ensure the versions are the same, they could be different. If they are I will update.

 

Hi,
I'm trying to build the companion App on iOS.
The app only works if I'm debugging from xCode connected with usbc, if I try to start the app from the iPad without xCode, it's just a black screen and it crashes. What should I specify for signing and build settings, like the Code Signing Identity.
Thanks

 

Hi,
Is it possible to install ARFoundationRemoteSource under Packages to avoid modifying Packages/manifest.json?
Modifications to manifest.json can cause problems if you share git repository with other members who have not purchased a license or if you re-clone it.

 

Your issue may be related to this one.
Could you please try to build any iOS project without my plugin and check if the issue persists?
Could you please also try to update your xCode and iOS?

 

Yes! This is exactly what I'm working on right now

 

I tried the fix in the issue "* Go to Unity-iPhone / Build Phases / Link Binary with Libraries build section and add UnityFramework.framework.
UnityFramework will be loaded at the same time as the main executable."

And it works fine now. Thank you.

 

Hi, I am trying to get this plugin running in an existing AR app.

It works in the sample scenes, but not in our own AR scene. Some differences:

* In our app, AR is not initialized on app start, because there's an onboarding process that doesn't need AR. So the app first checks if the onboarding process has been finished, and if so, it activates the AR part by enabling a game object that holds the whole AR structure (AR Session Origin etc.). I placed the setup script there as well.

* When the setup script initializes, it tries to setup an "ARCameraManager" and an "ARCameraBackground" on our AR camera without checking if these components are already there. However, we have these scripts on the camera, so the init() methods runs into an NullPointerReference when trying to enable auto-focus. I removed our own camera manager and background scripts temporarily. Now the setup runs through without any error, but the editor still shows an empty skybox scene instead of the camera feed of the device (the connection seems to work, because the companion app activates the camera).

* I tried to enable the setup script after some milliseconds, to ensure the AR camera object has initialized first, but this still won't work.

What might break the camera feed connection in such a use-case, where the AR camera hasn't been active at app start?

 

1. Does your app work properly if you run it on a real AR device?
2. Could you please create a small reproducible project for me so I can replicate your issue on my end?

 

ad 1: Yes, it does. It's a production app that you can download from https://marblear.com
ad 2: That's tough, the app is actually quite complex.

 

@waldgeist is there a chance you can share the script you've mentioned before? I'll try to replicate your initialization steps in a clean project.

 

Exception: AR Foundation Remote: BlockUntilReceive timeout
ARFoundationRemote.RuntimeEditor.TelepathyReceiverConnection.BlockUntilReceive (System.Guid guid, System.Double timeout) (at plugins/ARFoundationRemoteSource/Runtime/Connection/Telepathy/TelepathyReceiverConnection.cs:93)

Every time I try to ARRaycast this happens - Latest ARFoundationRemote. Unity 2020.1.4f1ARF4.1.1,

 

AR Foundation 4.1.1 destroys anchors in OnDestroy(), but the connection to AR Companion app may already be closed. I'll fix this error in the next version.

 

Great thanks, appreciate getting that in. Will PM my invoice # now

 

Hi!
I have problems with last version [4.6.8]

 

I'm very sorry for this, the version 4.6.8 was corrupted.
Please update to the version 4.6.9, it's already available for download on the Asset Store.

 

Do you have an ETA (sorry, that sounds like nagging, and I don't mean it to be) for support for ARKit depth/confidence textures? Meshing remote support is awesome, would be great to have the same for depthmaps, even if downsampled.

 

Environment Occlusion is already supported, please look at the EnvironmentOcclusionExample scene that comes with the plugin. Keep in mind that not all AR devices support this feature.

 

Hi, I recently download the version 4.6.9 and I'm trying install
But return a error saying: Unable to add package.

AR Foundation Editor Remote installation failed: Unable to add package [https://...]:

  Error when executing git command. remote: HTTP Basic: Access denied

  fatal: Authentication failed for 'https://...'

Is there something I could be doing wrong?

 

It seems like you're using the old version. Starting from version 4.4.2 I removed the dependency on git and the plugin will install itself from a local folder.
Could you please try to uninstall the plugin completely and install it again?
Please also check that the version in changelog and Packages/AR Foundation Remote/package.json is indeed 4.6.9.

 

Hi,
Does AR Foundation Remote support 2020.2.xx version? I failed to install the version 4.6.9 and found inside the code that 2020.2 is not supported. Have you any plan to support 2020.2?
Thanks in advance.

 

The support for Unity 2020.2 was added in version 4.5.0.
Could you please double-check that you have version 4.6.9 installed?

 

Using URP as well, everything works well except for the enviro/human segmentation/occlusion.

I'm getting these errors on editor&companion connect:
Background camera material doesn't contain property with name _HumanStencil.
Background camera material doesn't contain property with name _EnvironmentDepth.

Occurs once the editor connects to Companion.

 

I managed to reproduce the issue with missing textures. The issue only appears in Windows Unity Editor with iOS device, am I right?
Please send me your Invoice Number and I'll send you a version with a fix.

 

Well, I see the items in asset store:

And look in my project, is missing "com.kyrylokuzyk.arfoundationremote" folder:


And I open changelog and the last version is: ## [4.2.3] - 2020-09-30
I think is a AssetStore issue... I dont know what to do...

 

Please uninstall the plugin completely first:
Press 'Installer/Un-install Plugin' button, delete the folder 'Assets/Plugins/ARFoundationRemoteInstaller'

Then re-download the latest version from the Asset Store.

 

@KirillKuzyk what the button "Apply AR Foundation Fixes" does?
I updated to the latest version from 4.4.2 and I found this button.

Cheers!

 

Yeah, this button indeed needs clarification, I should add it to the Documentation.

Not all AR Foundation scripts are designed to work in the Editor, so my plugin applies several fixes to
'Packages/AR Foundation' package.

To my best understanding Unity tries to cache all folders inside the Packages folder to speed-up compilation. But in some cases (like updating the AR Foundation package), this caching mechanism will prevent Unity from taking the most recent version of modified scripts (ARMeshManager.cs in particular).
Pressing the 'Apply AR Foundation Fixes' button will force Unity to re-import modified scripts and take the most recent versions.

So if you have a compilation error related to these lines in ARMeshManager, you should press the 'Apply AR Foundation Fixes' button:

Code (CSharp):

1. // AR_FOUNDATION_EDITOR_REMOTE: delegate mesh subsystem to the plugin
2. #if UNITY_EDITOR || AR_COMPANION
3. using XRMeshSubsystem = ARFoundationRemote.Runtime.IXRMeshSubsystem;
4. using XRGeneralSettings = ARFoundationRemote.Runtime.XRGeneralSettingsRemote;
5. using SubsystemManager = ARFoundationRemote.Runtime.SubsystemManagerRemote;
6. using XRMeshSubsystemDescriptor = ARFoundationRemote.Runtime.XRMeshSubsystemDescriptorRemote;
7. #endif
8. // AR_FOUNDATION_EDITOR_REMOTE***

 

Hi @KirillKuzyk

I think I might have uncovered a bug. I've been looking to move over to the new unity Input package and noted the following error in the console.

InvalidOperationException: You are trying to read Input using the UnityEngine.Input class, but you have switched active Input handling to Input System package in Player Settings.
UnityEngine.Input.get_mousePosition () (at <345a1d9b2aab40b5b6362d7fa1668943>:0)
ARFoundationRemote.Runtime.SimulateTouchWithMouse.Update () (at plugins/ARFoundationRemoteSource/Runtime/TouchInputRemote/SimulateTouchWithMouse.cs:27)
ARFoundationRemote.Runtime.TouchInputReceiver+TouchSimulationUpdate.Update () (at plugins/ARFoundationRemoteSource/Runtime/TouchInputRemote/TouchInputReceiver.cs:114)

The error disappears if you have input set to "both" (old and new available). I tried to import AR Foundation Remote again against the "new" to see if your doing some macro compilation (i.e. selectively using the new input system) - But that didn't work for me.

 

Another quick question for @KirillKuzyk,
there is a way get remote touches (from the phone) with Input.getTouches ?
As for I can see I get only emulate touches with mouse clicks on the editor, but would be great if I could use the phone touch input for drags, pinchs and other gestures on AR Objects.

Cheers! (latest release rocks!)