AR Foundation Remote | Test and debug your AR project in the Editor

Ok, so your question wasn't related to the plugin, right?
Please let me elaborate on what this sentence means:

It means that when you call the ARCameraManager.TryAcquireLatestCpuImage(), it will return not the current CPU image, but the image from several frames behind. This is a limitation of my plugin and it comes from the fact that sending a texture over the network takes some time.
AR Foundation running on an actual AR device doesn't have such a limitation and should return the camera image that corresponds to the current camera position.

 

OK, I see. Thanks.
By the way, when I use a new camera to render a cube, and set the camera's projection matrix and pose the same as ARCamera in Update method, the cube drifts too. May you tell me more infos about this, thanks very much.

 

hi, having a hard time getting connected to my remote device (Pixel 3) on Windows 10. Connection fails every time saying I should check that they're on the same wifi network.

Unity 2020.2.3f1
AR Foundation Remote 4.11/0
AR Foundation 4.1.7
ARCore XR Plugin 4.1.7

Wifi and phone are definitely on the same Wifi network, and I tried both listed IPs in the running Android app. Turning off firewall doesn't help. Any idea what could be going on?

 

Do you experience the same thing in a simple AR Foundation example? I mean, without the manipulation with the camera's projection matrix.

 

Could you please try a wired connection method?

 

hi, thanks for the reply. Unfortunately that wired method didn't fix the problem. Though it does make the errors slightly different:


Really no idea what's going on. The XR subsystem seems to be set up fine, and only this adb method causes that error. Any idea what could be causing this?

 

Hmm, this is strange. The 'Editor lost connection' error only displayed when the connection was established and was interrupted after that.
Does the AR Companion app show the IP address on its screen?

 

Hi,

I m tried the ARFoundation remote[4.9.3/4.11.0], It is not working in WIFI，The IP network segment of my device is consistent.

iPhone device IP: 192.168.3.17
Unity Editor IP: 192.168.3.252

NOT ERROR REPORTED.


But sometimes connect to the USB IP it’s working prefect on iPhone6s(iOS 13.x).
Not working with iPhone 12pro[iOS 14.5]. I don’t know why.
Could you help me plz?

--Edit--
Problem:
I found the problem, iOS14.x has `Private WIFI Address`. So I can not connected.
I was disabled it. It's working perfect.

Is there any good way to do this(do not disabled the Private WIFI Address)

--Edit--
A Good way:
Adding the `NSLocalNetworkUsageDescription` to your info.plist.

Thanks

--Update--

AR Companion does not have newwork access permissions on iOS, so we need to add a network request to obtain permissions!

 

Hi,

I am having trouble acquiring the CPU images. I am using the method

Code (CSharp):

1. cameraManager.TryAcquireLatestCpuImage(out XRCpuImage image)

but it return false.

Version information:
AR Foundation Editor Remote: 4.11.0
AR Foundation: Version 4.1.0-preview.12 - November 03, 2020
AR Subsystems: Version 4.1.0-preview.12 - November 03, 2020
ARKit XR Plugin: Version 4.1.0-preview.12 - November 03, 2020

AR Device: Ipad Pro 2020
IOS Version: 14.1

Unity version: 2020.1.8f1
macOS: 10.15.5

Companion App built using AR Foundation Editor Remote version 4.11.0.

 

Huge thanks for finding a proper solution. I’ll add it to the document soon.

 

Hi!
Does the CPUImageExample.scene that comes with the plugin work for you?

 

Hi,

There is no scene by the name CPUImageExample.scene in Assets/Plugins/Scenes/Examples/.

However, there is a scene by the name CpuImages.unity that comes with ARFoundation sample repo (https://github.com/Unity-Technologies/arfoundation-samples/tree/main/Assets/Scenes). This scene does not works when I use it with your plugin. But if I make a build of the same scene and deploy it on my ipad in works there.

 

It seems like you haven’t updated the plugin correctly. Please uninstall the plugin completely and re-install it. To verify that you have the latest version, please go to Packages/AR Foundation Remote/package.json.

 

Yes it is working after the update, thank you very much for the quick reply.

What I am trying to do is record the human tracking data to file along with with camera frames so that I can replay it later to do development is a faster and deterministic way. I am already done with recording and replaying human tracking data, now I am looking for a solution to dump camera frames along side. From the first test, acquiring cpu images looks really slow (i can imagine the reason being the network lag).

Can you suggest a more efficient way to acquire camera images using this plugin?

 

When launching the app, it works but my entire iPhone screen shows this error log

Unable to find internal function......Parallelforjobstruct...

How can I clear it?

 

I'm currently working on the official Session Recording feature and will release it soon. Please stay tuned

As an alternative to CPU camera image API, you can use Graphics.Blit() to copy the texture from GPU to the CPU side. Please see the Texture2DSerializable.cs script for the examples.

 

I'm sorry for the late reply.
Could you please try to update to the latest plugin version and rebuild the AR Companion app?

Could you please also tell me your setup so I can reproduce the issue on my end?
1. Unity version. Windows or macOS?
2. Send me your Packages/manifest.json AND Packages/packages-lock.json files.
3. Your AR device and its OS version.
4. Plugin version. You can find it in the Packages/AR Foundation Remote/package.json file.

 

I just got your plugin. I tried to install it but ran into some errors.

Unity Version: 2020.3.4f1
AR Foundation: 4.0.12
ARCore XR Plugin : 4.0.12

Spoiler: 1

Unity.InteractiveTutorials.BuildStartedCriterion must be instantiated using the ScriptableObject.CreateInstance method instead of new BuildStartedCriterion.
UnityEngine.ScriptableObject:.ctor ()
Unity.InteractiveTutorials.Criterion:.ctor ()
Unity.InteractiveTutorials.PreprocessBuildCriterion:.ctor ()
Unity.InteractiveTutorials.BuildStartedCriterion:.ctor ()
UnityEditor.BuildPipeline:BuildPlayer (string[],string,UnityEditor.BuildTarget,UnityEditor.BuildOptions)
ARFoundationRemote.Editor.CompanionAppInstaller/<>c__DisplayClass20_0:<build>b__0 () (at Assets/Plugins/ARFoundationRemoteInstaller/Scripts/Editor/CompanionAppInstaller.cs:148)
ARFoundationRemote.Editor.ARFoundationRemoteInstaller:editorUpdate () (at Assets/Plugins/ARFoundationRemoteInstaller/Scripts/Editor/ARFoundationRemoteInstaller.cs:150)
UnityEditor.EditorApplication:Internal_CallUpdateFunctions ()

Spoiler: Error 2

Failed to generated ARCore reference image library 'ReferenceImageLibrary'
UnityEditor.BuildPipeline:BuildPlayer (string[],string,UnityEditor.BuildTarget,UnityEditor.BuildOptions)
ARFoundationRemote.Editor.CompanionAppInstaller/<>c__DisplayClass20_0:<build>b__0 () (at Assets/Plugins/ARFoundationRemoteInstaller/Scripts/Editor/CompanionAppInstaller.cs:148)
ARFoundationRemote.Editor.ARFoundationRemoteInstaller:editorUpdate () (at Assets/Plugins/ARFoundationRemoteInstaller/Scripts/Editor/ARFoundationRemoteInstaller.cs:150)
UnityEditor.EditorApplication:Internal_CallUpdateFunctions ()

Spoiler: Error 3

BuildFailedException: Failed to generate image database. Output from arcoreimg:
stdout:
Image database generated at: C:/Users/Tim/Desktop/Personal.imgdb
Image list file generated at: C:/Users/Tim/Desktop/Personal-imglist.txt
====
stderr:
====
UnityEditor.XR.ARCore.ARCorePreprocessBuild.BuildImageTrackingAssets () (at Library/PackageCache/com.unity.xr.arcore@4.0.12/Editor/ARCoreBuildProcessor.cs:333)
UnityEditor.XR.ARCore.ARCorePreprocessBuild.OnPreprocessBuild (UnityEditor.Build.Reporting.BuildReport report) (at Library/PackageCache/com.unity.xr.arcore@4.0.12/Editor/ARCoreBuildProcessor.cs:57)
UnityEditor.Build.BuildPipelineInterfaces+<>c__DisplayClass15_0.<OnBuildPreProcess>b__1 (UnityEditor.Build.IPreprocessBuildWithReport bpp) (at <7587a42c8322471cbc42b27c9b8eab3c>:0)
UnityEditor.Build.BuildPipelineInterfaces.InvokeCallbackInterfacesPair[T1,T2] (System.Collections.Generic.List`1[T] oneInterfaces, System.Action`1[T] invocationOne, System.Collections.Generic.List`1[T] twoInterfaces, System.Action`1[T] invocationTwo, System.Boolean exitOnFailure) (at <7587a42c8322471cbc42b27c9b8eab3c>:0)
UnityEditor.BuildPipeline:BuildPlayer(String[], String, BuildTarget, BuildOptions)
ARFoundationRemote.Editor.<>c__DisplayClass20_0:<build>b__0() (at Assets/Plugins/ARFoundationRemoteInstaller/Scripts/Editor/CompanionAppInstaller.cs:148)
ARFoundationRemote.Editor.ARFoundationRemoteInstaller:editorUpdate() (at Assets/Plugins/ARFoundationRemoteInstaller/Scripts/Editor/ARFoundationRemoteInstaller.cs:150)
UnityEditor.EditorApplication:Internal_CallUpdateFunctions()

----

Error building Player: 2 errors

Can you help me to fix them?

 

The issue with generating reference image library should be fixed in ARCore 4.1.7. Could you please try to upgrade?

https://docs.unity3d.com/Packages/com.unity.xr.arcore@4.1/changelog/CHANGELOG.html#417---2021-03-31

OR

You can just delete the example ReferenceImageLibrary.asset that comes with the plugin and the build should succeed.

 

Thank you very much! It worked now

 

It would be nice to have a way to force enable the AR_COMPANION symbol definition permanently without modifying the build scripts. I have a codebase set aside for the companion app to make life easier when doing stuff without image/object target libraries, and that builds via Azure DevOps... so I don't use the Installer and have to modify the build scripts to not change the product name, always set the `AR_COMPANION` symbol, etc. Of course these changes break on updates. I know it's quite an obscure use case, but I can't imagine it's that obscure to have a generic companion app for use internally, and the build scripts are quite intrusive right now as they change the product name, etc.

Maybe just `FORCE_AR_COMPANION` which is never unset and sets the AR_COMPANION symbol in the build pre-processor, and `LEAVE_PRODUCT_NAME` which means it doesn't modify the product name or app identifier?

(Edited my request slightly as I just realised AR_COMPANION is part of the Unity codebase so not something you can mess with too much)

I should also note, that modifying a developer's .gitignore is incredibly bad practice and something you really should not be doing. Especially as this then breaks the codebase if you're using the usings for meshing, etc in your own code.

 

Any ways to make this works in a project with Vuforia ? I got some Vuforia-related null reference on the companion app while running my project in editor.

Thank you

 

Will this plugin work together with Manomotion? Has anyone tested it and can confirm its working? I'm going to participate in the Manomotion's GameJam and this tool would be a lifesaver (if it works of course)!

 

The Manomotion SDK is not currently working with my plugin. I have a plan to contact the Manomotion team so we can adapt our solutions to work together.

 

Vuforia is not currently supported. But I have an integration with Vuforia on my roadmap.

 

Thanks for the feature request, I'll add these settings to the plugin soon. Please send me your invoice number so I can send you a new version before the official release.

Could you please tell me why do you think the AR_COMPANION is part of the Unity codebase? Have I missed something?

Yes, I agree that modifying the .gitignore is not the best solution. But people were constantly committing the plugin's source code to open-source projects. This still happens even with the .gitinore modification.
Moreover, the plugin has a per-seat license, so adding the plugin's folder to the .gitignore also serves as a reminder to other team members.
Ideally, to decouple your codebase from the plugin, you can add a version define symbol. This way, your codebase will not be dependent on the plugin being installed or not. Here is an example:
https://forum.unity.com/threads/ar-...ject-in-the-editor.898433/page-7#post-6561910

 

Brilliant thanks! I’ll PM you when I’m in the office

I then realised that you have some forked ARFoundation code with the package to enable those changes I think?

Ah yeah that’s always a pain in the rear, happens to a lot of assets it seems…

 

Hello,

Would this work for ARWT, which is the AR.js + Unity project?

https://github.com/ToughNutToCrack/ARWT

Thanks,
jrDev

 

Unfortunately, no. My plugin covers only AR Foundation API.

 

Hi, thanks for building this awesome tool. Do you already have a timeline in mind, when the Session Recording feature will be available?

 

Thank you!

I hope I'll be able to release the Session Recording feature in the next month, I'm currently on the finish line doing minor refinements.

Please keep in mind this feature alongside several other new features will be a paid upgrade. This means existing customers will be able to get the upgrade at a reduced price. I explained this in my last plugin update post here:
https://forum.unity.com/threads/ar-...ect-in-the-editor.898433/page-10#post-7046131

 

Thats unfortuate.

Thanks for reponse,
jrDev

 

When I uncheck "AR Foundation Remote" in XR Plug-In Management window, I get the error:

Code (CSharp):

1. AssertionException: Assertion failure. Value was Null
2. Expected: Value was not Null
3. ...
4. UnityEngine.Assertions.Assert.IsNotNull[T] (T value) (at <a5172b547855499dbc8d9d50552f0797>:0)
5. ARFoundationRemote.RuntimeEditor.Receiver.addSubsystemToReceivers[T] () (at Packages/com.kyrylokuzyk.arfoundationremote/RuntimeEditor/Receiver/Receiver.cs:66)
6. ARFoundationRemote.RuntimeEditor.Receiver.Start () (at Packages/com.kyrylokuzyk.arfoundationremote/RuntimeEditor/Receiver/Receiver.cs:41)

using latest version: 4.11.3

I'm trying to disable the plugin, and just use the Editor like normal..

 

With AR Foundation Editor Remote installed, I notice that my Unity project takes a lot longer to start up.

Looking at the editor logs, it seems like script re-compilation is the culprit:

[ScriptCompilation] Recompiling all scripts because: Assembly Definition File(s) changed

Total time taken on a fresh Unity project (with AR Foundation Editor Remote):

[Project] Loading completed in 62.384 seconds

    Project init time:                 1.099 seconds

        Template init time:         0.000 seconds

        Services packages init time:     0.000 seconds

        Package Manager init time:         0.000 seconds

        Asset Database init time:         0.000 seconds

        Global illumination init time:     0.000 seconds

        Assemblies load time:             0.000 seconds

        Unity extensions init time:     0.001 seconds

        Asset Database refresh time:     0.000 seconds

    Scene opening time:             0.505 seconds

It usually takes <10 seconds on a fresh Unity project (without AR Foundation Editor Remote).

Any ideas here? Not a huge deal considering that AR Foundation Editor Remote saves you far more time, but it's a minor inconvenience at the start of your day.

 

.

 

Hello,
This plugin is awesome, but now I want to work for ar remote in 2 or 3 devices. Is that possible to work for 2 or 3 devices for ar remote from unity.

 

The AR Foundation wasn't meant to work in the Editor, so my plugin modifies the AR Foundation's source code for it to work in the Editor. These modifications are non-invasive and take effect only in the Editor.

But when you re-open the Unity Editor, Package Manager will revert all changes made to any built-in package, including the AR Foundation package modification mentioned above.

So if you want to speed up the Unity Editor launch, please do the following.

1. Press the right mouse button on 'Packages/AR Foundation' folder -> Show the folder in the file manager. It will open a PackageCache folder.
2. Then, COPY the com.unity.xr.arfoundation@x.x.x folder to the ROOT_PROJECT_FOLDER/Packages folder. Please make sure to make a COPY, not move.

This workaround will prevent Unity from resetting the contents of the AR Foundation package and will prevent the re-compilation process you're experiencing.

I'll add this info to documentation soon, thanks for pointing out the issue!

 

Thank you!
It's possible to connect any number of different AR devices to the AR Foundation Remote, but not simultaneously. Just run the AR Companion app on a device you wish to connect and you're good to go.

 

Thanks for pointing out the issue! I'll fix it in the next release soon.

 

Hey there, I just bought the plugin in and everything works great so far. I'm really impressed with how well this works for testing on Windows (a little bummed about the frame rate but this is fine for testing). One issue I do have however is that I can't seem to get meshing working. I read through the forum and saw a few people had similar issues but I didn't really see any steps to fix the problem. So far I have no errors on either the phone screen or console so I think the code is running fine. I am using the latest iPhone and have tested with other LiDAR projects so this shouldn't be an issue either. I also went through the tutorial video posted on the store page and could get every feature except meshing to work just fine. Any idea if I am missing something or if something has changed? I am currently working with ARKit 4.1.5 since I am trying to merge this into an already existing project once I finish testing the base features.

 

OK, so I've figured out the cause of the issue.
Since Unity 2020.3.6, the Package Manager caching mechanism became more aggressive and will revert the fixes applied to the AR Foundation package before the AR Companion app build. These fixes are required for AR Foundation to work in the Editor and to intercept calls to the XRMeshSubsystem in the AR Companion app.

So the current workaround is to embed the AR Foundation package. This will prevent Unity Package Manager from reverting the aforementioned modifications. Unfortunately, this is the cleanest solution I can come up with:
https://forum.unity.com/threads/ar-...ect-in-the-editor.898433/page-11#post-7185178

I'll include this in the documentation soon and will make the AR Foundation embedding a requirement in the next version.

Huge thanks for the bug report!

 

Thanks for the quick reply. Moving the package into a local asset folder seems to work when building the companion app. I hope you can find a more permanent solution soon though because when switching back to my windows PC for development I usually have to reimport the package through package manager. I'm not sure if this is some kind of licensing enforcement thing but it makes initial setup a bit of a pain. Luckily the app was designed for testing new code without building to the iPhone/iPad so it looks like this should be fine since I only need to build the companion app once to get meshing data. Thanks again for the quick response, your turn around time on bug fixes is top notch.

 

my device is iPad Pro 2020, device very hot even editor not run.
make device cooler when editor disconnect,please.

 

Thank you for the feedback!
Will the Screen.sleepTimeout setting be sufficient for you? Your AR device will automatically lock itself after some idle time.

 

I think I came up with a better approach to a manual AR Foundation folder copy. The plugin will embed the AR Foundation package semi-automatically during the plugin installation. And there would be an option to 'un-embed' the AR Foundation with a press of a button. I understand this is not ideal, but it's less error-prone than manually copying folders.

This is not related to licensing, rather than to the fact the AR Foundation wasn't meant to work in the Editor. There is some code that will simply crash the Editor, and some code will throw exceptions. So my plugin applies non-intrusive fixes to the AR Foundation package so it can work safely in the Editor.
But the Unity Package Manager interferes with this process and reverts the AR Foundation package back to the original state. And embedding the package will prevent UPM from reverting packages.

 

no,I don't want to sleep device, because I need edit code and reconnect iPad frequently. If iPad sleep, I need to unlock device each time.
when editor disconnect, device app eat 25% cpu, some code running crazy on idle.
please limit framerate when editor disconnect, and find which part code eat much cpu.
please optimization code.

 

Hi all,
I tested the ARMesh and CpuImage examples, and would like to ask if it is possible to set the CpuImage on the Mesh and generate a Mesh with textures, or it may be implemented on this basis?

 

Okay, I'll lower the framerate when the companion app is not connected to the Editor, thank you for the suggestion!

 

Unfortunately, applying texture to scanned mesh is not a trivial task and not what ARKit/AR Foundation can do out of the box.

 

got it, thank.

and when I install plugins, it show: AR Foundation Editor Remote: please re-import the plugin or buy the additional license if you're trying to install the plugin on different development machine.

how to solve this? i had uninstalled others machine's plugin when install