AR Foundation Remote | Test and debug your AR project in the Editor

The project compiles and works. But there seems to be a bug with the plane API. What's the best way to send you the bug report?

 

I'm trying out some of the face tracking samples on iOS, but the camera remains world facing and no faces are tracked. Am I missing some step? Thanks!

Unity 2020.1

ARFoundation 4.1.0-preview.7

iOS 14 beta

 

Hello.
I am a student studying AR FOUNDATION. While searching, you viewed and purchased your assets. However, I looked at the document and did it, but the editor's execution screen does not appear on my phone. Only a white screen appears in the editor's game scene. I wish there was a short tutorial guide video running on Android mobile.
-From students who started studying AR-

 

Please send an email: kuzykkirill@gmail.com

Could you please tell me what samples are you trying? Those that come with the plugin?
Do you have an opportunity to test it on iOS13?

The Editor's screen should not appear on your phone. Your phone is only used to feed the data back to Editor.
As for the white screen in Editor, it seems like a bug. Could you please describe your setup so I can fix it more quickly? Unity version, AR Foundation version, Windows/Mac, your AR device, render pipeline you use, etc.

 

I keep getting errors like this.
unity: 2019.4.9f1
ARFoundation: 4.0.8
os: windows
AR device: SM-G970N(Galaxy S10e)
I hope you can tell me how to fix it. As I asked at the beginning, I wish there was a short guide video about the setting procedure.

 

Yes, I'm preparing a new video for the next release.

Here is a quote from Documentation that describes how to set the IP:
'3. Run ARCompanion app on your AR device.
4. Enter AR Companion app IP in Assets/Plugins/ARFoundationRemoteInstaller/Resources/Settings.asset.'

 

Number 4 doesn't make sense. When I open the AR Companion app, nothing like IP comes out. Look at the thumbnail image. This is the screen that comes out when you open the AR Companion app. If you do No. 2 in the document, the image of thread #203 comes out.

 

This is not the correct AR Companion app.
Please refer to the number 2 in the Documentation on how to install the AR Companion app:
'2. Switch your project to the appropriate build target and press 'Assets/Plugins/ARFoundationRemoteInstaller/Installer-Install AR Companion app' button.'

 

I did according to the document.
If you do 2, the thumbnail image of thread #203 appears on your phone, and if you do 3, the thumbnail image of thread #207 appears and you need to enter the IP 4 times.
It was frustrating because it didn't proceed. I just think it's right to wait for the video.

 

The image 203 is the correct AR Companion app screenshot. It asks to install the XR Plugin provider (AR Core in your case) and enable it in the XR Management settings.


The easiest way to get running with the plugin is to ensure that your AR project works before installing the plugin.

 

I was able to replicate the issue. Please ensure that the ARKit Face Tracking plugin is installed.

 

I set it up as I told you, and I did it again, but this time, this screen pops up in the AR Companion app. Is it a problem with the Wi-Fi Internet?

 

Please make sure your phone is connected to a Wi-Fi network. If it is, please go to Wi-Fi settings on your phone, take the IP from there and enter into plugin's settings.

 

I tried the one that came with the plugin (FaceIOSBlendshapesExample), as well as a handful of ARFoundation samples
I haven't gotten to test it on iOS13, no, though the samples work normally on device without remoting

ARKit Face Tracking is installed, version 4.0.8

 

Could you please try to make a new build of AR Companion app. Maybe the face tracking plugin wasn't installed before you installed the app for the first time.

 

Email sent. Let me know if it makes sense, and any plans to address the issue.
Thanks!

 

Thanks for the detailed bug report. Yes, you've provided more than enough information, I hope I can fix the issue tomorrow.

 

Is there anything else I need to set up to apply the remote to the AR project I'm working on, not your example scene?
Remote operation in your asset example scene was successful. In the AR project that I am developing, I imported the remote asset and then did it in the order, but the screen does not appear in the AR companion app.

 

It turned out I was sending a plane center in world coordinates instead of local.
Thanks for the detailed bug report, it helped me a lot. Please check your email for the fix.
This fix will be included in the next release.

Could you please tell me, are there any errors in Console? Did you enable the 'AR Fondation Remote' provider in Project Settings -> XR Plug-in Management -> PC, Mac & Linux Standalone?

 

Thank you, the fix worked

 

This tool looks very useful, but we're using object tracking in our project. Is it still not supported?

 

Still not. I'm implementing new features in order of the number of feature-requests.
Currently, I'm working on Meshing (ARMeshManager) because it is a very requested feature, and then will decide what feature to add next.
Your voice for object tracking has been counted

 

Hi, I'm getting the black screen with the message "Can't start sender. Please connect AR device to private network".

I verified that I'm connected to Wifi, even turned off mobile data.
Also verified the IP Address on the phone is correct and I can ping it from the PC.

I also uninstalled and reinstalled the companion app several times. But I always get the same message.
The app used to work, but after the latest update, it started failing with that message.

Are there any other settings I should change to fix this?
Thanks

 

Could you please go to Wi-Fi settings on your phone and take the IP from there? Then enter this IP in plugin's settings.

 

PLUGIN UPDATE POST #7 (plugin version 4.0.19)​

Huge thanks for all supporters, the plugin is evolving because of you!

Meshing support (ARMeshManager).​

Meshing was a highly requested feature for a long time and now it's here! Currently, the iPad 2020 with LiDAR is the only mobile device that supports meshing. But, theoretically, the plugin should support meshing on Magic Leap and Holo Lens too. I have no way to prove it without the actual devices, so if you have one, your help would be very appreciated (Holo Lens, Magic Leap).

Also, the ARKit Mesh Classification is supported. Here is a video that shows the classification in action:

Anchors support (ARAnchorManager).
​

Previously, it was impossible to support ARAnchorManager in Editor, because its API is synchronous. This means that a call to ARAnchorManager.AddAnchor() should return instantly: and I can't make an instant network request to the companion app using built-in Unity connection.

Happy news! The transition from Unity built-in connection to Telepathy TCP gave me the ability to fully control the connection between Editor and AR device. Now I can send a message from Editor to the companion app, then block the Editor's main thread while waiting for the response in background thread. This results in small hiccups (10-20ms) when you use ARAnchorManager in the Editor, but brings full support of ARAnchorManager.

Changes in ARKit Blendshapes.​

I refined the way ARKit Blenshapes are received in Editor. Previously, you were required to write a lot of #defines and get Blendhshapes in Editor from BlendShapeCoefficientsContainer.cs. Now, the only change you need to make to your scripts that deal with Blenshapes is this one:

Code (CSharp):

1. #if UNITY_EDITOR
2. using ARKitFaceSubsystem = ARFoundationRemote.Runtime.FaceSubsystem;
3. #endif

Bug fixes:​

- Disable camera video in Windows Unity Editor 2019.2 to prevent crash.
- Fixed: UI Event System is not working after loading new AR scene at runtime.
- Fixed: BoundedPlane.center should be in local space coordinates instead of the world (thanks @Anastasia-Devana for bug report).
- Fixed: sometimes the IP will not be shown in AR Companion app on Android.


Asset Store:
https://assetstore.unity.com/packages/tools/utilities/ar-foundation-editor-remote-168773

Website:
https://stampedegames.wixsite.com/ar-foundation-remote

itch.io:
https://kyrylokuzyk.itch.io/ar-foundation-remote

PLUGIN UPDATE POST #8​

 

Is there any way to tell in a script if the remote app is currently running (like a global / static variable, etc?)

 

@sskenth

Yes, you can use Receiver.Instance.state to know the current state of the connection to AR device.

 

@KirillKuzyk Hello! Your tool seems to be exactly what I need! I just purchased on your itch.io because i had some trouble processing in Unity Assets store. So I wonder for the future updates, will you sync the changes on itch.io with unity store? and do I need to pay again for updates?

 

Yes, all future updates will be free regardless of where you've made a purchase.
The itch.io has even a small advantage because I can publish a new version there without the Asset Store validation process.

 

Thank you so much! I need to use the AR depth/occlusion manager in AR https://blogs.unity3d.com/2020/06/2...undation-now-supporting-the-arcore-depth-api/ So I updated ARcore, ARFoundation and ARKit from version 3 to the latest 4 using git url in package manager (I can't only update one otherwise there'll be bug, had to update all 3). After all updated to version 4, AR Foundation Remote plugin can't work properly anymore and throwing NullReference error:

 

This is pasted from the editor log, might be cleaner to see:

NullReferenceException: Object reference not set to an instance of an object
at ARFoundationRemote.Runtime.XRCameraFrameSerializable.get_Value () [0x00171] in D:\Projects\FragmentFreeze\Library\PackageCache\com.kyrylokuzyk.arfoundationremote@c85da0b0dd\Runtime\Sender\CameraSubsystemSender.cs:280
at ARFoundationRemote.Editor.CameraSubsystem+CameraSubsystemProvider.TryGetFrame (UnityEngine.XR.ARSubsystems.XRCameraParams cameraParams, UnityEngine.XR.ARSubsystems.XRCameraFrame& cameraFrame) [0x00019] in D:\Projects\FragmentFreeze\Library\PackageCache\com.kyrylokuzyk.arfoundationremote@c85da0b0dd\Editor\Subsystems\CameraSubsystem.cs:108
at UnityEngine.XR.ARSubsystems.XRCameraSubsystem.TryGetLatestFrame (UnityEngine.XR.ARSubsystems.XRCameraParams cameraParams, UnityEngine.XR.ARSubsystems.XRCameraFrame& frame) [0x00000] in D:\Projects\FragmentFreeze\Library\PackageCache\com.unity.xr.arsubsystems@4.0.8\Runtime\CameraSubsystem\XRCameraSubsystem.cs:481
at UnityEngine.XR.ARFoundation.ARCameraManager.Update () [0x0009a] in D:\Projects\FragmentFreeze\Library\PackageCache\com.unity.xr.arfoundation@4.0.8\Runtime\AR\ARCameraManager.cs:327

(Filename: Library/PackageCache/com.kyrylokuzyk.arfoundationremote@c85da0b0dd/Runtime/Sender/CameraSubsystemSender.cs Line: 280)

I noticed in example scenes, it seems there's no depth/occlusion information, virtual objects and real objects don't have front and back position block (forgive me for my poor English). How to get access of that in the meanwhile still be able to use remote view plugin? Thank you again!

 

Please make a new build of AR Companion app. It is required after you change the version of AR Foundation.

'I noticed in example scenes, it seems there's no depth/occlusion information, virtual objects and real objects don't have front and back position block'
Unfortunately, these features are not currently supported. I have them on my roadmap.

 

Thank you! Rebuilt fixed the error!

May I ask when would the depth map feature be available in your remote plugin? \@_@/

Another feature would be super convenient is supporting ios devices streaming to windows Unity editor. Keijiro san's project uses NDI stream ipad video to windows machine in real-time https://github.com/keijiro/Rcam2

 

I hope I'll add occlusion support in the next 3-4 weeks.

'Another feature would be super convenient is supporting ios devices streaming to windows Unity editor. Keijiro san's project uses NDI stream ipad video to windows machine in real-time https://github.com/keijiro/Rcam2'
Yes, I have this on my roadmap too. I will not use NDI though because it's a proprietary technology. I'll add video support on Windows + iOS, but most likely, the occlusion will only work with macOS Unity Editor (Windows Editor doesn't support Metal shaders).

 

There is a great video review came out recently. Thanks, Dilmer!

 

Hello, this plugin seems very promising and useful.

I'm doing a project where the user localizes within a room with the ARWorldMap subsystem, drops some objects and saves the map in order to be localised later again with the objects appearing in the same place they were left.
I would like to use this plugin to save me some time testing and debugging, is it possible to use with the ARWorldMap feature?

Thanks

 

Hi, Guys!

Check this new article with Comparison Table of AR Testing Tools (Top 3 Unity Assets) for AR Foundation & Unity including Asset of this Thread.

The table conveniently gathers all AR features (Face Tracking, Plane Tracking...) so you can pick the tool that better suits the needs of your AR project.

 

Unfortunately, ARWorldMap is not currently supported. But I have it on my roadmap and I'll notify you when it's ready.

 

Heyho, I sadly have a similar problem. So the Game View is white or black in Play Mode.
I'm working with Unity 2019.4.0f1, AR Foundation 4.0.8, the URP and ARCore.
Is there a way to fix this?

 

Please check that you've installed and enabled ARCore XR Plugin. Here is an installation video for iOS, the process for Android is similar:

 

Thank you for your response. ARCore is in the project and activated. And the remote functionality of your Asset works. I see for instance the found planes in the Game View. It's just the missing image from the camera.

Is this maybe an URP problem? Because I tested the asset in another project without URP and there everything works fine.

 

Yes, it seems like the issue with URP. There are several bug reports related to URP and I'm currently working on it.

Could you please tell me your setup?
1. Unity version. Windows or macOS?
2. Packages/manifest.json file.
3. Your AR device and its OS version.

 

Unity 2019.4.0f1, Windows 10, Samsung Galaxy S10 with Android 10. The manifest file you can find under the uploaded files.

Thank you for your effort!

 

Hello! How do you configure the latest 4.3.2 version with URP 7.4.1 and Unity 2019.4.10 with the ARKitHumanSegmentationExample scene? I only get a pink screen. Worked well without URP. Please advice. Many thanks

Some additional questions:
1. Please also explain if the Ipad Pro Lidar tracking improves the tracking and human segmentation?
2. Can you use the selfie front camera for the Human Segmentation?
3. How do you output the depth data so you can make VFX shader around the character?

 

Hi again, how do you put the occuluded human character in a 3D scene? I have now black background behind the human segments. Thanks

 

@LeoThePhoenix @Scherabon @daapage0
Thank you for reporting the bug with URP. I've broken the URP support in version 3.11.3, sorry for that.
I fixed the issue in version 4.2.3 and it's already available on the Asset Store. Please update and tell me if URP is working again.

 

There was a bug with URP, please update to version 4.2.3 to fix the problem with the white background.
The only example scene that supports URP is URP_SwitchBetweenPlaneAndFaceTracking. Please upgrade materials in other example scenes to URP material so they can work properly.

'1. Please also explain if the Ipad Pro Lidar tracking improves the tracking and human segmentation?'
I believe yes. But I can't say for sure because my iPhone doesn't support human segmentation. It would be great if you could test it yourself and share the results.

'2. Can you use the selfie front camera for the Human Segmentation?'
Just tested it on my iPad with LiDAR, and, unfortunately, this is not supported by ARKit.
I also tested environment depth on the front camera, and it also doesn't work (both in ARKit and ARCore).

'3. How do you output the depth data so you can make VFX shader around the character?'
Please look at AR Foundation Samples repository for examples. Here is an example scene that shows how to display the depth data:
https://github.com/Unity-Technologi...preview/Assets/Scenes/Depth/DepthImages.unity

 

PLUGIN UPDATE POST #8 (version 4.2.3)​

ARKit Human Segmentation
​

The plugin keeps evolving and I'm slowly but surely running out of unsupported features. Today, I added the ARKit Human Segmentation support. And it even works in Windows Unity Editor!
Please check that your iOS device has A12 chip or newer and iOS 13+.
A video is below, after the environment occlusion section.

Environment Occlusion (preview)
​

This feature was added in AR Foundation 4.1 (currently in preview). In ARKit this feature requires a LiDAR (iPad 2020) and iOS 14.
But, to my surprise, on Android, it just works with a regular camera! It works slower than LiDAR, but the quality is great, good job Google!

Camera video with iOS device + Windows Editor
​

It turned out that a big number of people are developing iOS apps on Windows PCs. So this update brings the iOS camera background video to Windows Unity Editor.

Universal Render Pipeline support
​

I've broken the support for URP in version 3.11.3, sorry for that. Thanks to @LeoThePhoenix @Scherabon @daapage0 for reporting the bug, this helped me to fix it

PLUGIN UPDATE POST #9​

 

Hello, thanks:
It doesn't seem as the materials in the ARKitHumanSegmentationExample gets updated to URP properly.
Please see the image here end error message. Please advice how to resolve? Thanks

 

ARKitHumanSegmentationExample doesn't support URP out of the box. The only URP-ready scene is URP_SwitchBetweenPlaneAndFaceTracking.
To update ARKitHumanSegmentationExample to URP, please assign a URP material to the occlusion_plane game object.
I'll update all example scenes to URP in the next version, thanks for your feedback!