AR Foundation Remote | Test and debug your AR project in the Editor

Hi,
I want to use mesh classification. Is there any tutorial for that?
Thanks in advance.

 

My plugin doesn't support the new Input System, only the old one.
As far as I know, the new Input System adds not much value for AR apps. It is primarily designed for handling input from gamepads and other input sources.
I'll consider adding support for the new Input System in the future. Or at least, my plugin shouldn't throw any exceptions with it.

 

Multi-touch remoting is already supported
There is an example scene called PlaneAndCloudPointsExample where you can try it.

 

Not a tutorial per se, but there is an example scene that comes with the plugin called 'ARKit MeshClassificationExample'. The script that deals with classification is called MeshClassificationFrackingFixed.cs (it's a script from AR Foundation Samples repo, but adapted to work in the Editor).

 

For some reason on the Companion App I built in the last days touch input didn't work. I reimported the plugin in a fresh project on a fresh Unity release, rebuilt it and now it works again as expected.
Thanks for putting me in the right direction.

 

Thanks for the great plugin but for now I am struggling to use it because Camera passthrough over lapping the 3D game objects. UI is rendered but camera passthrough coming from the device to editor blocks 3D rendering in the game view. Both in my own scene and the example scene.

 

Checking back... It is still an issue with 4.5.0

 

Already answered you in email
Will duplicate my answer here:

 

Please update to the version 4.7.4, I fixed the warning there.

 

Still getting the error in your AnchorsExample in 4.7.4

Exception: AR Foundation Remote: BlockUntilReceive timeout

Code (CSharp):

1. Exception: AR Foundation Remote: BlockUntilReceive timeout
2. ARFoundationRemote.RuntimeEditor.TelepathyReceiverConnection.BlockUntilReceive (System.Guid guid, System.Double timeout) (at plugins/ARFoundationRemoteSource/Runtime/Connection/Telepathy/TelepathyReceiverConnection.cs:100)
3. ARFoundationRemote.RuntimeEditor.TelepathyReceiverConnection.BlockUntilReceive (ARFoundationRemote.Runtime.EditorToPlayerMessage msg, System.Double timeout) (at plugins/ARFoundationRemoteSource/Runtime/Connection/Telepathy/TelepathyReceiverConnection.cs:75)
4. ARFoundationRemote.Editor.AnchorSubsystem+AnchorSubsystemProvider.sendBlocking (ARFoundationRemote.Runtime.AnchorDataEditor anchorSubsystemMethodsData) (at plugins/ARFoundationRemoteSource/Editor/Subsystems/AnchorSubsystem.cs:96)
5. ARFoundationRemote.Editor.AnchorSubsystem+AnchorSubsystemProvider.TryAddAnchor (UnityEngine.Pose pose, UnityEngine.XR.ARSubsystems.XRAnchor& anchor) (at plugins/ARFoundationRemoteSource/Editor/Subsystems/AnchorSubsystem.cs:52)
6. UnityEngine.XR.ARSubsystems.XRAnchorSubsystem.TryAddAnchor (UnityEngine.Pose pose, UnityEngine.XR.ARSubsystems.XRAnchor& anchor) (at Library/PackageCache/com.unity.xr.arsubsystems@4.1.1/Runtime/AnchorSubsystem/XRAnchorSubsystem.cs:83)
7. UnityEngine.XR.ARFoundation.ARAnchorManager.AddAnchor (UnityEngine.Pose pose) (at Library/PackageCache/com.unity.xr.arfoundation@4.1.1/Runtime/AR/ARAnchorManager.cs:84)
8. ARFoundationRemoteExample.Runtime.AnchorsExample.Update () (at plugins/ARFoundationRemoteSource/Runtime/Examples/AnchorsExample.cs:52)
9.  

 

It seems like on slow connections the timeout of 1 second is being reached. As a temporary workaround, please go to TelepathyReceiverConnection.cs and increase the defaultTimeout field, 10000 should be enough for any connection:

Code (CSharp):

1. const double defaultTimeout = 10000;

 

Hey there. I just got the plugin. At first, it was working great, then I started experiencing some crashes. I think I realized that having my iOS device plugged into the computer was causing the weirdness. Once I unplugged it, it seemed to work again.

I'd like to be able to keep the phone plugged into the computer though. Are there any settings I should be aware of to make this work?

 

By 'plugged into the computer' do you mean running with xCode debugger attached? I'm also experiencing rare crashes with xCode debugger attached, but just having the device connected to the computer shouldn't make a difference.
Could you please download send me the crash logs? You can download them via xCode even if the crash has happed without the attached debugger.

 

@KirillKuzyk I just bought the asset and set it up for Unity 2019.4 with AR foundation 4.11 for Android/Arcore.
I can't seem to connect to my Samsung S8 (it's connected via USB to the PC). The same setup works fine with ArCore SDKs remote.

Here's the error:
Client Recv: failed to connect to ip=100.85.x.x port=44819 reason=System.Net.Sockets.SocketException (0x80004005): A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied.
(also tried the phone's wifi network IP of 192.x.x.x. Same error.) I've allowed Unity app to send on the port 44819 in avast firewall

Any advice?

 

A wired connection is only supported with iOS + macOS. To connect Windows PC to Android phone, they should be on the same Wi-Fi network.
Could you please check your router's firewall allows the connection on port 44819?

 

Hi @KirillKuzyk it's only later I realized that it said wifi needed, and only ios supports a cable connection.

My laptop was on a different ethernet wired connection.
When putting it on the same wifi connection it works.

May I ask why USB does not work for Android, seeing how Arcore sdk works flawlessly over usb?

 

I think the ARCore team included an Android USB driver inside their Instant Preview solution.
In the case of my plugin, I needed a universal solution that will work with any combination of devices (Android + macOS, iOS + Windows, etc.). Wi-Fi connection turned out to be a good platform-agnostic option.

You may wonder why then iOS works with macOS by wire?
macOS comes already with a USB driver for iOS devices and creates a private network automatically. And AR Companion will display two IPs: one for wired connection, one for Wi-Fi. So no additional work was needed from my side

 

Hello @KirillKuzyk, I sterted using your plugin and it is awesome! However, I am having problems with the Image Tracking functionality.

Can a XRReferenceImageLibrary be added to the project after the companion app has been built and installed or does it have to be present before?

Is adding images at runtime supported (Mutable Reference Image Library)?

I am working on Mac Osx, tesing on Android Pixel 4 and I am using AR remote version 4.7.4.

I am having issues with Unity version 2020.1 and ArFoundation 3.1.8. The Ar Companion app throws an error (not the first time it is open though, from the second time onwards). Error: skip sending message because !Sender.isConnected, but the app seems to actually work. See here.



I have also tried ArFoundation 4.0.9 on Unity 2020.2.1 but I may be experiencing the image tracking limitation regarding AR foundation 4.0.x, as I am really struggling to get the companion app (revuilt for AR Foundation 4.0.9) to see custom images I have printed on paper. However, if I export the app on the device (Pixel 4) then the tracking works fine. It would seem a problem with the companion app instead of an AR Foundation issue.

The companion app seems to work fine in this case though and doesn't throw any errors, apart from a ScheduleAddImageJob failed when permission to access the camera is asked. See here



After the app is restarted it works fine, but doesn't track the image targets properly (doesn't track the print version, only occasionally a square version on the screen).

What Am I doing wrong?

 

EDIT: Resolved by updating AR Foundation Remote to 4.7.11. This update wasn't visible in the package manager. Apparently, sometimes you need to manually delete the downloaded asset from Library/Unity/Asset Store-5.x for the latest version to show in the package manager.

Hi, I've used this great plugin a lot over the past few months. Today I added it to a new project and I suddenly get these console errors:

Assets/Plugins/ARFoundationRemoteInstaller/ARFoundationRemoteSource/Runtime/FaceSubsystem.cs(26,37): error CS0115: 'FaceSubsystem.CreateProvider()': no suitable method found to override

Assets/Plugins/ARFoundationRemoteInstaller/ARFoundationRemoteSource/Runtime/FaceSubsystem.cs(40,15): error CS0534: 'FaceSubsystem.FaceSubsystemProvider' does not implement inherited abstract member 'SubsystemProvider<XRFaceSubsystem>.Destroy()'

Any idea of what might cause it? Because of these errors, I can't install the plugin correctly.

Unity 2020.2.0f1

 

Hey there.

Great plugin! It worked out of the box, thanks to your detailed documentation.
But one thing bohers me a bit.

When I connect my mobile device with the editor I still get the text "No AR_COMPANION define!" on the mobile device as you can see at the attached file (Tested with an iPhone SE and iPad 12 both IOS 14 and Unity 2019.4.16f1 with your AnchorsExample.scene).

Did I miss something?

Best Regrads

 

Hi! im trying to get it working on URP
it trows me an assertionexception
here is a screenshot of my phone

I allready check for graphics posibilities but ar foundation background only work wiht certains graphics settings
maybe Im overlooked at some thing?

 

Feature request: Environment Probes

Thanks for your great work!

 

Hi @KirillKuzyk could you please let me know your thoughts about what I mentioned re image tracking working when the scene is built on device (Android 11, Pixel 4) but not using AR Remote? Unity 2020.2.1f1, AR Remote 4.7.4, AR Foundation 4.0.9.

Many thanks

 

Hi @KirillKuzyk

I have not played too much with AR, but i am looking for required tools to capture facial animation data from the phone into the editor into animation clips (blendshapes i guess?), that is my very specific objective.

I'm trying to figure out what my workflow will be and where your asset will come in handy.

Your asset page states:
AR Foundation Editor Remote is an Editor extension that allows you to transmit AR data from AR device to Unity Editor. Test AR projects right in the Editor!

This blog post on the Windup project seemed to imply that ARKIT was used to capture facial animation from the iphone into the editor, and made it sound like transmitting data from the phone to editor was already possible.
https://blogs.unity3d.com/2018/08/13/facial-ar-remote-animating-with-ar/

This other video from Unite Seoul 2018 seemed to imply the same.

End to end guides are however lacking, so it is unclear to me however how many layers of custom script or development had to be done, just to get this ARKIT Face Tracking to work and record facial animations in editor for the above 2 projects. But i understand somewhere along the line your asset might be able to help me save some time, I am just not sure which gaps it is filling.

Could you shed some light on what u think my tools and workflow should be and where your asset will help me? I have a mac and an iphone 11.

Cheers
Kevin

 

Hi everyone!
I'm sorry for the late reply. I just returned from vacation and will continue to work on the plugin with even more energy and determination!

These two questions are related and the answer to both of them is yes
When you start an Image Tracking scene in the Editor, the plugin sends the image reference library to the companion app. Then companion app adds all images via the ScheduleAddImageJob.

Although, on Android with AR Foundation 4.0.x, images added via the ScheduleAddImageJob can be tracked less reliable compared to the static image library (link1, link2). In this case, there is an option in version 4.7.11 to embed the image reference library inside the companion app via 'Settings/Embedded Image Libraries' (a new build of the companion app is required).

Thanks for reporting the issue! This error should not break anything, I added the isConnected check as an extra consistency check.
Could you please update to version 4.7.11 and tell me the issue still persists?

Please update to version 4.7.11, add your image reference libraries to 'Settings/Embedded Image Libraries' and make a new build of the companion app. This should fix the issue and image tracking will work the same way as it works in real build.

 

Yes, unfortunately, sometimes Package Manager will not display the latest version available version. The Asset Store web version always shows the correct one though.
Your issue was related to Unity 2020.2, I added the support for it in plugin version 4.5.0 a month ago.

 

I assume you built the companion app manually via the Build Settings window. In this case, the needed AR_COMPANION define will not be added to the companion app and Meshing will not work.
Please use the 'Installer/Install AR Companion app' button and make a new companion app build.

 

Thanks for the feature request!
I added the Environment Probes to my roadmap

 

Back in 2018, there was an ARKit Remote solution made by Unity team. They used it to make the presentations you've mention before.
ARKit Remote was working only with the standalone version of ARKit SDK and when the transition to AR Foundation was made, the solution became outdated.

Yes, it's possible using the build-in EditorConnection/PlayerConnection classes. Although I use a more versatile Telepathy TCP solution in my plugin.


My plugin also supports ARKit Blendshapes and you can make them work in the Editor in a matter of minutes. Then, you can devise your own recording solution.

If you just need to store the recorded amination inside Unity, then this task is relatively easy. All you need is to store the Blendshapes coefficients (they are just enum/float pairs) into an array, save the data somewhere (ScriptableObject, for example), then apply these coefficients to your model when you need to play the animation. This example shows how to apply the coefficients to your model.

If you want to store your animation in some 3D modeling tool file format, then the task becomes harder: you need to have a full understanding of the format to be able to write coefficients into it.

 

I am having issues building the ar remote. Every time I try it says:
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
CommandInvokationFailure: Gradle build failed.
One of the "deprecated features" was this:
BuildPipeline.BuildPlayer(scenes, getInstallDirectory() + EditorUserBuildSettings.activeBuildTarget + getExtension(optionalCompanionAppExtension), EditorUserBuildSettings.activeBuildTarget, options);
in line 142 of CompanionAppInstaller.cs
Do I need to install a different version of gradle?

 

Some versions of AR Foundation require specific version of Unity. Can you successfully build your AR app, or the problem occurs only when you're trying to build the companion app?

Could you please also tell me your setup?
1. Unity version. Windows or macOS?
2. Packages/manifest.json file.
3. Your AR device and its OS version.
4. Plugin version. You can find it in the Packages/AR Foundation Remote/package.json file.

 

I am having almost the same image tracking issue with a similar set-up on Android - Pixel 4 device

Android Version 11
AR Foundation Remote: 4.7.11
Ar Foundation 4.1.1
ARCore XR Plugin: 4.1.1

Image tracking works fine in the compiled AR app, but not through the remote.

Square images seem to work more reliably than non-square images with the remote.
For example, the QR code provided in the samples seems to be recognized and tracked most of the time, whereas the Rafflesia image in the samples almost never seems to be recognized (it has happened once, but the tracking was way off)

My custom marker image that I am using cannot unfortunately be made square. Again, tracking of my custom marker image works fine in my compiled and installed AR app, but not when using the remote (it seems to recognize the image about 1 out of 8 times, and once again the tracking and positioning is way off).

Is there a fix coming or any workarounds for this?

Otherwise, a great tool!

Thanks.

 

@KirillKuzyk
I apologize that I did not see you addressed this some above.

I backed everything out, uninstalled the plugin, and tried everything again. This time by adding my ReferenceImageLibrary to the Remote through the plugins settings.

It seems to work much better now. In the SceneView in Unity I can see the image is tracked and my prefabs instantiate properly.

However, I also now see my source image on the device laid over the marker image. Is that Intentional?

Best,
Garrett

 

I'm glad that adding the image reference library to the 'Embedded Image Libraries' helped.

Yes, this is intentional. AR Companion currently is not capable of showing the Editor View, so I decided to show debug info there (in case of image tracking it is an overlay that shows the original image). I understand that this behavior may confuse users, so I'm currently working on the solution that will show Editor View in the companion app.

 

Hiya @KirillKuzyk, is it possible that only one instance of ARPoseDriver is supported? I was using multiple (for layering camera effects) on a project which seemed to work fine on device. But after integrating the editor remote it seems like it only finds and updates one instance in the editor (i.e. my other camera layers are frozen in place and do not track with device movements).

Work-around is to just parent my other camera layers to the ARCameraBackground object and have only one ARPoseDriver attached to that. For other reasons I should probably have done this anyway, but it seems like a difference between on-device and in-editor/remote behaviour. It's also very possible I just set up something poorly, so I'm hoping to confirm this limitation with you.

 

Hi again @KirillKuzyk,
I am also experiencing this same issue with URP and the remote.

I see in the changelog that URP broke once before but was fixed.

However, I am seeing the same issue @AkamaruDesign reported above.
MacOS 11.1
Unity 2020.2.0f1
AR Foundation Editor Remote 4.7.11
ARFoundation 4.1.1
ARCore XR Plugin 4.1.1
ARKit XR Plugin 4.1.1
Android 11
Pixel 4 device

Thanks again for the great tool!

Best,
Garrett

Screenshot of the error:

 

Yes, you're absolutely right, my plugin updates only one ARPoseDriver. To be specific, the one that is sitting on ARCamera.
The XRInputSubsystem is not implemented by the plugin because it requires to write a native plugin. Of course, this is possible and maybe I'll do it one day. But for now, I decided to allocate my resources to the most requested features.

 

Hello.
1) I have build the ARCompanion app on my android with INSTALL AR COMPANION APP button
2) I open the ImageTrackingExample scene
3) I open the companion app on my phone
4) I enter the IP in the Settings scriptable object
5) I press play in the editor
6) see the picture to see what happens

 

I'm sorry, I missed your message.

@garrettstrobel @Fangh
I fixed the issue internally already, but haven't released an official update yet.
Please send me your Invoice Number in PM or by email and I'll send you a version with a fix.

 

Hi @KirillKuzyk thank you so much for your reply. I hop eyou had anice time away!

Great thanks for claryfying. I haven't tested this yet but I will asap.

I had seen these two links from your asset description. However I had the issue not only with images added via the ScheduleAddImageJob but also those in the XRReferenceImageLibrary from start.

Version 4.7.11 with the option to embed the image library seems to have solved the tracking issue for me at the moment!! Great work

However I have a different problem now. There is a huge amount of lag between the phone camera and what I see in the editor. I am talking 15 seconds lag after less than a minute of use.

I tested again with Unity 2020.1.17f and AR Foundation 3.1.6 and the issue seems fixed.

As I said above, the tracking issue seems fixed but I now have the lag issue. However, this is happening only in the project I am working on (4x XRReferenceImageLibrary assets, each with 5, 3, 15, 1 images). Unity 2020.2.1f1, AR Remote 4.7.11 (updated) I have embedded the XRReferenceImageLibrary assets in the settings. AR Foundation 4.0.9.

I am on Mac OSX.

I tried a new project and imported the AR Remote asset then installed Ar foundation 4.0.9 and ArCore Xr plugin 4.0.9. In this case, I don't experience the lag.

I also wanted to flag a couple more issues:

1) I couldn't install the latest version of the plugin automatically from the package manager. It wouldn't let me update and download the verison on disk altough it did ackowledge a new version (4.7.11). I had to manually delete the asset from the unity assets folder on my mac before it let me download the updated version.

2) Every time I try and install the asset from package manager I get this error:

Code (CSharp):

1. AR Foundation Editor Remote: please re-import the plugin or buy the additional license if you're trying to install the plugin on different development machine.
2. UnityEngine.Debug:LogError (object)
3. ARFoundationRemote.Editor.ARFoundationRemoteInstaller/<>c__DisplayClass22_0:<InstallPlugin>b__0 (bool) (at Assets/Plugins/ARFoundationRemoteInstaller/Scripts/Editor/ARFoundationRemoteInstaller.cs:152)
4. ARFoundationRemote.Editor.ARFoundationRemoteInstaller/<>c__DisplayClass14_0:<checkDependencies>b__0 () (at Assets/Plugins/ARFoundationRemoteInstaller/Scripts/Editor/ARFoundationRemoteInstaller.cs:70)
5. ARFoundationRemote.Editor.ARFoundationRemoteInstaller:editorUpdate () (at Assets/Plugins/ARFoundationRemoteInstaller/Scripts/Editor/ARFoundationRemoteInstaller.cs:138)
6. UnityEditor.EditorApplication:Internal_CallUpdateFunctions () (at /Users/bokken/buildslave/unity/build/Editor/Mono/EditorApplication.cs:327)
7.  

Then after I import a second time it then works. This only seems to happen in Unity 2020.2.1f1. I don't experience this error in Unity 2020.1.17f1

 

Alrighty, thanks! My "work around" is a better structure anyway and I just wanted to make sure I wasn't missing something

 

Hi,
I'm now getting this error almost every time I try to start a scene, usually I can test once and after that I have to restart the editor:

Code (CSharp):

1. AssertionException: Assertion failure. Value was True
2. Expected: False
3. CreateInstanceIfNull() was called while isQuitting
4. UnityEngine.Assertions.Assert.Fail (System.String message, System.String userMessage) (at <bb4ed9929fe04849bbe2b059cba4d31f>:0)
5. UnityEngine.Assertions.Assert.IsFalse (System.Boolean condition, System.String message) (at <bb4ed9929fe04849bbe2b059cba4d31f>:0)
6. ARFoundationRemote.RuntimeEditor.Receiver.CreateInstanceIfNone () (at Packages/com.kyrylokuzyk.arfoundationremote/RuntimeEditor/Receiver/Receiver.cs:42)
7. ARFoundationRemote.Runtime.SessionSubsystem+ARemoteSessionSubsystemProvider.Resume () (at Packages/com.kyrylokuzyk.arfoundationremote/Runtime/SessionSubsystem.cs:70)
8. UnityEngine.XR.ARSubsystems.XRSessionSubsystem.OnStart () (at Library/PackageCache/com.unity.xr.arsubsystems@4.1.1/Runtime/SessionSubsystem/XRSessionSubsystem.cs:76)
9. UnityEngine.XR.ARSubsystems.XRSubsystem`1[TSubsystemDescriptor].Start () (at Library/PackageCache/com.unity.xr.arsubsystems@4.1.1/Runtime/XRSubsystem.cs:69)
10. UnityEngine.XR.ARFoundation.ARSession.StartSubsystem () (at Library/PackageCache/com.unity.xr.arfoundation@4.1.1/Runtime/AR/ARSession.cs:394)
11. UnityEngine.XR.ARFoundation.ARSession+<Initialize>d__36.MoveNext () (at Library/PackageCache/com.unity.xr.arfoundation@4.1.1/Runtime/AR/ARSession.cs:382)
12. UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) (at <bb4ed9929fe04849bbe2b059cba4d31f>:0)
13. UnityEngine.XR.ARFoundation.ARSession:OnEnable() (at Library/PackageCache/com.unity.xr.arfoundation@4.1.1/Runtime/AR/ARSession.cs:348)
14.  

Any idea?
Thanks,

 

This issue should be fixed in the next version (will be available on the Asset Store in a few days). Please check your PM, I'll send you an unreleased version directly.

It's an issue with the Package Manager caching. A lot of other Asset Store customers experience it too and deleting the Asset Store folder cache folder helps.

 

I've found the cause! The 'isQuitting' error is happening if 'Project Settings/Editor/Enter Play Mode Options (Experimental)' is enabled and the 'Reload Domain' is disabled.
I understand that skipping the reload domain step speeds up iterations cycle, but the plugin wasn't designed to work without it.
I'll consider adding the support for disabled 'Reload Domain' in the future, but for now, please enable it for the plugin to work correctly.

 

@KirillKuzyk

Version: 4.8.7

Getting error when pressing Install Companion App


DirectoryNotFoundException: Could not find a part of the path '/Volumes/DevExt/Dev/UnifiAR/Assets/Plugins/ARFoundationRemoteInstaller/Scenes/ARCompanion'.

Problematic code ->

       

static IEnumerable<FileInfo> getSenderScenePaths() {

            return new DirectoryInfo(Application.dataPath + "/Plugins/ARFoundationRemoteInstaller/Scenes/ARCompanion")

                .GetFiles("*.unity");

        }

Fixed by changing path to “/Plugins/ARFoundationRemoteInstaller/Scenes/Senders”

 

It seems like you didn't delete the ARFoundationRemoteInstaller folder before updating to the new version. In recent versions, the 'Senders' folder was renamed to 'ARCompanion'.
Here is the safest and most consistent way to update to the new version (from the documentation):

 

Also receive errors for each image referenced in image ref library.

Destroying assets is not permitted to avoid data loss.
If you really want to remove an asset use DestroyImmediate (theObject, true);

 

Could you please send me the error's stack trace in email or in PM?

 

Hi,
I am using Unity 2020.2.1f1 and ar remote 4.8.7. I want to use AR Pose Driver to rotate other game object. I added AR Pose Driver component to the game object but it didn't work. I tried Tracked Pose Driver, but it also didn't work.
let me know how to use AR Pose Driver in ar remote.
Thanks in advance.

 

Sent.

In packages i see dependency based on
"com.unity.xr.arfoundation": "3.0.1",

We use the latest ARFoundation 4.1.1. Any plans to update dependency versions soon ?