AR Foundation Remote | Test and debug your AR project in the Editor

Did you manage to solve the issue? If not, please try to make AR Companion build manually and select your AR device explicitly via Console. Here is how to make a manual build:
1. Add all scenes from Scenes/Senders to Build Settings.
2. Enable 'Development Build' option.
3. Press 'Build and Run'.
4. Connect Editor to AR device via Console -> 'Editor↓' -> AndroidPlayer(ADB@127.0.0.1:34999)/iOS device name.

Ideally, yes, I want to support all AR features. I'm planning to add new features as the plugin becomes more popular.

Unfortunately, no. I would love to support all features, but I have only this much time and I need to prioritize one features over the others.
Could you please explain what value does ARAnchorManager add? How it's different from using ARRaycastManager and simply saving a hit position? Is it similar to ARKit Tracked Raycasts?

 

hi! Great plugin! At first I had issues using the companion app, but after manually compiling via Unity + Xcode I was able to make it work with no issues.
I have a question regarding "Camera Background". In the release notes it says this feature is limited to AR Foundation version <= 4.0.0-preview.3. Is this something you plan to upgrade to work with the latest AR Foundation?

 

Yes, I'm trying to fix this for AR Foundation > 4.0.0-preview.3. But it's hard to even debug the issue because Unity is freezing and I can't get any errors or crash logs. It seems like the issue is deep inside AR Foundation source code.

 

Hello,

Just a little question: Any plan to support ARAnchor?

Best,
Alexis

 

Hello!


I'm using that plugin well!

ARAnchor Manager
Do you have a support plan?
More than a simple raycast hit point
Because of the good tracking performance of AR Anchor Manager, it is often necessary to use AR Anchor.

 

There are a lot of requests to add ARAnchorManager support so I'll consider adding it next.
Could you please provide some examples of why is it needed? Because I don't fully understand its advantage over simple ARRaycastManager.Raycast().

 

I use it to have more stabilize objects (I follow ARCore explanations).

Best,
Alexis

 

Now I get it, thanks for the article!

 

Hello,

is it normal to have to recompile companion App each time I restart the computer?

Best,
Alexis

 

The companion app should connect to the Editor after restart. Could you please send me an email describing your setup so I can find and fix the issue quicker?
kuzykkirill@gmail.com

 

Hey @KirillKuzyk -
Been trying to get it to work and am having issues. I simply get the error "Failed to connect to player"

My Unity Editor is a Windows machine running 2020.1.0b14
My iphone (11 Pro iOS 13.5) has the instructions "Waiting for Editor connection....." on the screen
I use a wired connection
I have AR Remote in it's own project. I've tried building on Unity Cloud Build and manually using Build & Run on a Mac.
I've imported the plugin into my main project.
In my main project under project settings --> XR Plugin Management --> AR Foundation Remote is enabled
Both projects use ARFoundation 4.0.2 and ARKit Face Tracking 4.0.2
I've tried rebooting phone/machine as well

Is there something I missed? In the meantime - I'll attempt manually building on a mac from the main project and see if that works.

Any help would be appreciated! Thanks

 

Hello!

My project is using AR Foundation 4.0.2 and there is a new feature to get depth information using AROcclusionManager.
Is there any plan to support AROcclusionManager?

Thanks!

 

EDIT: please see the detailed instructions here: How to connect to the AR Companion app by wire?

I reproduced the issue. Restarting your PC indeed breaks the wired connection to AR Companion app, only Wi-Fi will work.
I'm currently working on a custom network solution to replace Unity's built-in connection and this limitation will go away.

UPDATE 2020-09-28
This limitation was fixed in 3.10.8. Now it's possible to connect iOS device to Windows Unity Editor via Wi-Fi (wired connection works only on macOS + iOS).

Spoiler: OUTDATED ANSWER

I managed to reproduce the issue. It turns out, that you can connect your AR device to Editor only if the app was built on the same computer. This means that it's impossible to connect an iOS device to a Windows PC.
I'm currently working on a custom network solution to replace Unity's built-in connection and this limitation will go away.

I can't tell the exact time frame, but yes, I want to support all AR features in the future.

 

Thanks for the heads up. Any ETA on it? Weeks? Months?

 

I hope two weeks would be enough.

 

Hello,

I will give you work again!

Will you support new Input system?

Best,
Alexis

 

This looks great. We built something similar in-house before ARFoundation came along.

Would you consider adding the ability to record the input and replay it? We've found this invaluable for recording test cases and for recreating situations. We do this by serializing out the data stream and recording the image data to files and replaying it. The performance isn't necessarily great (due to the loading of the images, primarily) but it has been really helpful.

 

I'm not sure if this feature is critical for AR development right now. After all, the new Input System is mainly targeted on gamepads and other controllers.

I don't have record/replay feature on a roadmap currently. But I will consider adding it if there will be many requests.

Please update to the latest version. I managed to make a transition away from PlayerConnection/EditorConnection to a custom TCP/IP solution. It will fix all previous connection limitations.

 

PLUGIN UPDATE POST #6​

A lot of things have changed since the previous update post.

Several last weeks I dedicated to bug fixes and improvements. New features are coming in the future, but for now, I wanted to focus on the experience of existing users.

Touch Input remoting and simulation.
​

I've found a way to simulate touch input in the Editor. Unity 2019.3 introduced a new private Input.SimulateTouch() method that gives the ability to simulate touch in Editor. After a lot of trial-and-error, I came up with a solution that inserts a custom

PlayerLoopSystem

into

UnityEngine.LowLevel.PlayerLoop

to simulate touch. Also, it was required to read a mouse position from Event.current.mousePosition instead of Input.mousePosition because calling SimulateTouch is overriding Input.mousePosition. The solution turned out to be very hacky, but on the upside...

What has changed for users?
Previously it was required to add this line on top of every script that used

UnityEngine.Input

:

using Input = ARFoundationRemote.Input;

Now, this line is not needed anymore. Moreover, touch input remoting now works with UI systems (old, new, you name it).

Connecting AR device to Editor.
​

There were a lot of problems with Unity's built-in PlayerConnection/EditorConnection. It's great when it works, but there are many edge cases that break device-Editor connection. Some of them are:
- Connecting Android adb tool will break the connection between Editor and AR device;
- Wired connection will not work after restarting your PC, only Wi-Fi connection will work;
- It's impossible to build AR Companion app on one computer and connect it to another one;

Connecting the AR device to Editor is the main feature of the plugin, but until now it was the weakest point. EditorConnection suited well as a jumpstart, but the time has come to make a transition to a custom networking solution.

I'm very grateful to guys at vis2k for developing the Telepathy TCP library. With its help, I was able to replace EditorConnection in a matter of hours.
https://github.com/vis2k/Telepathy

What has changed for users?
It is now needed to enter device IP in plugin settings after making AR Companion app build. The app will display its IP so you don’t need to go to Wi-Fi settings to know it.
Now it’s possible to build AR Companion app on macOS and continue development on Windows machine.

Other improvements:​

- Universal Render Pipeline support.
- Light Estimation ambient spherical harmonics support.
- Liner color space support.
- AR camera background video now works with all versions of AR Foundation.

Bug fixes:​

- Fixed mirrored camera in Face Tracking on Android.
- Null reference exception when creating an empty runtime image library.
- Fixed Unity Editor crash when running on OpenGLES3 with AR Foundation < 4.0.2.

PLUGIN UPDATE POST #7​

 

Actually it's not critical, but may be in the future.

 

Wow! Impressively quick! Glad you were able to find a solution so quickly. I hope it ends up being more stable and capable than the editor connection.

I took a stab at the update. Updating over my old installation yielded some issues the new TCP lib coming in (said that ARFoundationRemote.Runtime namespace didn't exist). Deleted the folder and reinstalled. Now these two errors are popping up:

Library\PackageCache\com.kyrylokuzyk.arfoundationremote@95bfad42ff\Editor\Subsystems\CameraSubsystem.cs(113,30): error CS0104: 'Object' is an ambiguous reference between 'UnityEngine.Object' and 'object'

Library\PackageCache\com.kyrylokuzyk.arfoundationremote@95bfad42ff\Editor\Subsystems\CameraSubsystem.cs(113,54): error CS0246: The type or namespace name 'ARCameraBackground' could not be found (are you missing a using directive or an assembly reference?)

 

I'm sorry about that. I already submitted a new version to the Asset Store, but it's going through the validation process.
Could you please contact me via email so I can send the latest version with a fix?

 

I cannot get this to work. The package is installed, the app built and installed on my iPhone no problem. I've got a sample scene up but it just won't connect. At first Remote had black screen listing IP address which I put in settings. When then didn't work I switched methods to Unity built in connection and tried selecting that from the Editor menu. Still no luck. No the message on the Remote app always says Select your device in Editor message. I tried inputting directly the IP address in the editor menu and that didn't work either. Nothing is working...

 

Are there any errors in Console? Does you iPhone is connected to the same network as your computer?
Could you please contact me via email describing your setup? Please tell me your Unity version, ARKit XR Plugin version and send me Packages/manifest.json file.
email: kuzykkirill@gmail.com

 

For those who are having problems updating to a new Telepathy version.

I also had some issues when trying to upgrade, mainly issues in the console about some absent namespaces or redeclared ones.

My solution was to complete remove the plugin from the project (and associated package pulled from github) and reinstall them all.
Same goes for the companion app: remove the old one and compile the new one.
And don't forget to specify the IP address in the settings tab (not the console).

Hope that helps.

 

on the latest, I'm getting the error message "Can't start sender. Please connect AR device to private network." In the settings, I put in the IP address of the laptop running my Unity editor. Is this correct? Firewall is off and this is on Android 9 (Samsung Galaxy5, SM-G950U1). Thanks!

 

I believe you need to put the ip address of the phone. At least that's what I'm doing and it's working well.

 

I receive this error message when my phone is not connected to Wi-Fi. Is there a chance this is your case?
Could you please try to set ConnectionType to UnityBuiltinConnection and re-build Companion app and then connect to your device via Unity Console? If this will not help, please contact me via email describing your Unity version and your OS.

 

Hello, and thanks for your work it seems to be awsome !

I have questions about your AR Foundation Editor Remote.I’m interested by buying it, but I need to understand the constraints :

On this video, I see you specify an IP adress to connect your phone to the editor.Do we need to be on the same network or can we do this with an usb connection ?
I don't have any wifi connection to connect my device and my computer on the same network.

Is there any evaluation version to check compatibility with my computer configuration ?

 

This video is recorded with Unity built-in connection. Currently, the main connection method is Telepathy TCP library. It works by connecting Editor to your AR device via private network.
But the old built-in connection is still there, you can switch to it in plugin settings. It will enable you to connect Editor to your iOS device by wire (unfortunately, there is no way to connect Android devices by wire using built-in connection).

I had an evaluation version for an old version of the plugin, but it is very outdated now and doesn't reflect the current state of the project. I can make a refund if the connection between Editor and device will not work for your setup.

 

Thanks for your fast answer !
So if I work on windows (which means android ), there is no way to use your plugin without wireless network ?

 

Any idea why I'm getting this error after git pull?

Library\PackageCache\com.kyrylokuzyk.arfoundationremote@f6561ed6eb\Editor\Subsystems\PlaneSubsystem.cs(57,52): error CS0115: 'PlaneSubsystem.ARemotePlaneSubsystemProvider.planeDetectionMode': no suitable method found to override

 

It seems like you're running the old version of the plugin. In the old versions, it was required to add the ARFOUNDATION_4_0_OR_NEWER define in Scripting symbols.
Please update the plugin to the latest 3.10.12 version, now the installation process is automatic.

Here how to update:
"Q: How to update to a newer version?
A: Delete the folder 'Assets/Plugins/ARFoundationRemoteInstaller', re-import the plugin, and make a new build of AR Companion app by pressing Installer-BuildAndRunCompanionApp button."

 

Unfortunately, no. I was unable to connect an Android phone to a PC by wire with Unity's built-in EditorConnection. The Telepathy TCP is also not able to solve this connection problem.

 

But if I manage to get a wireless connection, I can connect my android device with Unity editor on windows, right ?

 

Yes, that's correct. Moreover, you'll be able to connect an iOS device to Windows PC (providing that you have a macOS device to build the companion app once).

 

I've just tried your asset and have a few questions:
1. Do I understand correctly that the use case for it is to write some code to work with AR data and then use your device to streamline it to editor?
2. It doesn't seem to work "out of the box" as you need to set up a code to send and receive data. Maybe it's stupid but I was under assumption that it will connect automatically and send data using the one and same scene. Additionally, you don't have any such information in documentation and / or on your website. The only way is to use examples which is not self-evident. It would be great to add a small paragraph of text detailing the data exchange process.
Thanks!

 

No, this would undermine the whole 'fast iterations' idea. The main idea of the plugin is to build AR Companion app once (please refer to Documentation for more details) and then connect Unity Editor to it. AR Companion will then send AR data back to the Editor.

Here is a quote from Asset Store description:

Please feel free to contact me via email if you have any other issues.

 

Thanks Kirill! Phone was connected to WiFi, but it turns out the error message is benign. I still see "can't start sender" on the phone but when I press play in the editor everything works great, I see the camera background and everything. This is on Unity 2020.1, Windows, talking to a Samsung Galaxy SM-GM950U1 (Galaxy S8).

One new question though - my unity scene has both face tracking and image tracking; in code, I toggle between them by enabling or disabling the AR Tracked Image Manager or the AR Face Manager. I have a flip camera button that enables or disables these as needed. This works on an actual build for the phone:

Code (CSharp):

1. IEnumerator DoFlipAsync()
2.     {
3.         ARCameraManager.requestedLightEstimation = LightEstimation.None;
4.         ARSession.enabled = false;
5.  
6.         if (IsUsingRearCamera)
7.         {
8.             TrackedImageManager.enabled = false;
9.             FaceManager.enabled = true;
10.             ARCameraManager.requestedFacingDirection = CameraFacingDirection.User;
11.             ARSession.requestedTrackingMode = TrackingMode.RotationOnly;
12.         }
13.         else
14.         {
15.             TrackedImageManager.enabled = true;
16.             FaceManager.enabled = false;
17.             ARCameraManager.requestedFacingDirection = CameraFacingDirection.World;
18.             ARSession.requestedTrackingMode = TrackingMode.PositionAndRotation;
19.         }
20.  
21.         ARSession.enabled = true;
22.         yield break;
23.     }

But when using your plugin, it seems to only show the user-facing camera, and the flip camera button doesn't work.

I guess my question is, how does your plugin determine which camera and does it "listen" for changes to requestedFacingDirection?

 

Does this asset support body tracking?

 

Currently, Body Tracking is not supported. I have to buy new device to add this feature.

 

The plugin is trying to find a local network IP that starts from 192.168. If such IP is not found, I display the "can't start sender" message suggesting to connect to a Wi-FI network. Could you please tell me where did you find the phone's IP and how does it start? Maybe my assumption that a local network IP always starts from 192.168 is wrong.

The camera facing direction feature is not currently supported. Please give me a few days to add this support.

 

Thanks Kirill! You're right, that's what's happening. I'm on a 10.0.x.x subnet. My Unity PC is 10.0.0.133 and my phone is 10.0.0.165 or something, both hitting a gateway at 10.0.0.1.

Thank you for the support and I look forward to the new release, it will be super helpful to be able to swap cameras!

 

This is a truly great plugin with fantastic support. My application was tracking both planes and images at the same time and the plugin didn't support this, so I emailed Kirill and within 1 day he added this to the plugin and sent me a new version.

This plugin was extremely valuable in helping me see my ARTrackedImage generated prefab in scene view to debug an issue with the mesh it was displaying.

Highly recommended for anyone using ARFoundation.

 

Hi,
I'm having problems with image recognition using the remote. It only works sometimes and seems to have a hard time tracking. Building seems to work much better. Any thing I can do to work around this?

I'm using:
Remote 3.10.8
Unity 2019.4
AR foundation 3.1.3
AR kit 3.1.3
AR subsys 3.1.3
XR management 3.2.13


Remote test (Skip the first 10 seconds):

Build Test:

I'm using the example scene with one more image added to the reference library set up the same way as the other images.

 

I'm starting to have connection problems again after upgrading to the version 3.10.12 and also upgrading Unity to latest stable version 2020.1.1.f1

Using Telepathy, the only error message I'm getting is:

Code (CSharp):

1. Client.Send: message too big: 19398645. Limit: 10485760
2. UnityEngine.Debug:LogError(Object)
3. ARFoundationRemote.Runtime.<>c:<Awake>b__7_1(String) (at Library/PackageCache/com.kyrylokuzyk.arfoundationremote@4b6b22e108/Runtime/Connection/Telepathy/TelepathyConnection.cs:39)
4. Telepathy.Client:Send(Byte[]) (at Library/PackageCache/com.kyrylokuzyk.arfoundationremote@4b6b22e108/Runtime/Telepathy/Client.cs:203)
5. ARFoundationRemote.RuntimeEditor.TelepathyReceiverConnection:send_internal(Byte[]) (at Library/PackageCache/com.kyrylokuzyk.arfoundationremote@4b6b22e108/Runtime/Connection/Telepathy/TelepathyReceiverConnection.cs:64)
6. ARFoundationRemote.RuntimeEditor.TelepathyReceiverConnection:send(Byte[]) (at Library/PackageCache/com.kyrylokuzyk.arfoundationremote@4b6b22e108/Runtime/Connection/Telepathy/TelepathyReceiverConnection.cs:49)
7. ARFoundationRemote.Runtime.TelepathyConnection:Send(Guid, Byte[]) (at Library/PackageCache/com.kyrylokuzyk.arfoundationremote@4b6b22e108/Runtime/Connection/Telepathy/TelepathyConnection.cs:105)
8. ARFoundationRemote.RuntimeEditor.EditorToPlayerMessageExtensions:Send(EditorToPlayerMessage) (at Library/PackageCache/com.kyrylokuzyk.arfoundationremote@4b6b22e108/RuntimeEditor/Receiver/Receiver.cs:307)
9. ARFoundationRemote.Editor.ImageTrackingSubsystemProvider:set_imageLibrary(RuntimeReferenceImageLibrary) (at Library/PackageCache/com.kyrylokuzyk.arfoundationremote@4b6b22e108/Editor/Subsystems/ImageTrackingSubsystem.cs:52)
10. UnityEngine.XR.ARFoundation.SubsystemLifecycleManager`2:OnEnable() (at Library/PackageCache/com.unity.xr.arfoundation@4.0.2/Runtime/AR/SubsystemLifecycleManager.cs:97)

FYI: I removed the old version, reinstalled the package and reinstalled the client on the iPhone.

 

Hi, I'm trying to make a Windows build but it nothing shows in the built exe file, it works well in Unity Editor. Could you please help?

 

It seems like it may be a performance issue. AR Companion app is busy with sending data back to Editor so there is less CPU left for ARKit. Could you please try to disable 'Enable Background Video' in plugin's settings and make a new build of AR Companion?

The image library you're trying to send to AR Companion exceeds 10MB limit. I sent you an updated version with increased limit, please check your PM.

The plugin supports only targets supported by AR Foundation. Could you please provide more details of what you're trying to achieve?

 

Something similar to Face Cap where I can do real time animation. Is there any chance?

 

Correct me if I'm wrong, are you trying to stream face animations from your phone to Standalone Windows build? While this is technically possible, this is not the main purpose of the plugin and is not currently supported out of the box.