Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

AR Foundation Remote | Test and debug your AR project in the Editor

Discussion in 'Assets and Asset Store' started by KyryloKuzyk, May 26, 2020.

  1. Saied_R

    Saied_R

    Joined:
    Mar 12, 2019
    Posts:
    2
    Hi,

    thanks for this amazing time saver plugin. Keep going.

    Just one thing, after updating to the latest version (4.9.3), my Game view in Unity Editor freezes suddenly after 1 seconds but my AR device keeps tracking and shows new detected planes.
    My Game view updates every 30 seconds to 1 min in a while and fast forwards (like fast forward of a video) and freezes again. I tried to reduce the video resolution scale and also the quality but nothing changed. I have also a good Wi-Fi connection but everything is suddenly sooo slow after the update.
    Do you have any ideas why this is happening???
     
  2. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Hmm, this is strange.
    Could you please send me an email describing your setup?
     
  3. any_user

    any_user

    Joined:
    Oct 19, 2008
    Posts:
    374
    New user here. Is it normal that it takes around 1-2 minutes to connect every time I enter play mode? As soon as it is connected, everything run smoothly. Tried it on different networks with the example scenes.

    macOS 11.2.1, iOS 14.4
    Unity 2020.2.3
    ARFoundation/ARKit 4.1.1
     
  4. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    No, this is not normal. Could you please tell me what plugin version do you use? I fixed a similar connection issue in version 4.8.7.
    Please also try connecting via a wired connection, here is a quote from the docs:
     
  5. NEEEU

    NEEEU

    Joined:
    Mar 8, 2013
    Posts:
    4
    Having an issue with the combination of AR Foundation Remote 4.9.3, AR Foundation 4.1.1 and Unity 2019.4.4f1.

    InvalidOperationException: Could not find shader named '' required for video overlay on camera subsystem.

    Looks like the shader's name is missing as an argument.

    Rolling back AR Foundation Remote 4.9.3 to 4.8.22 solved it.
     
  6. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    This error typically means that you haven't switched Unity Editor to the appropriate platform.
    Please check that the AR Companion and Editor are running the same build target.
     
  7. NEEEU

    NEEEU

    Joined:
    Mar 8, 2013
    Posts:
    4
    OK, that makes sense.
    Thanks for the quick reply. I wish I had seen this info somewhere before.
    It wasn't clear to me as the Companion Build process seemed to be running pretty autonomously.
     
  8. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    The Editor build target platform should correspond to your target AR Foundation platform.
    For example, if you plan to use the plugin with ARKit, you should switch the Editor to iOS build target and deploy the AR Companion app to your iOS device.
    I'll update the documentation to reflect this more clearly. Thank you for the feedback!
     
  9. Griffo

    Griffo

    Joined:
    Jul 5, 2011
    Posts:
    700
    Hi, do i have to do anything when i build my game ? Like disable the plugin ? Thanks.
     
  10. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    No, the plugin doesn't execute a single line of code and doesn't add any performance overhead in production. It just adds a couple of extra KBs to the resulting builds. You can safely ship your game/app with the plugin enabled in the XR Management window.
     
    Griffo likes this.
  11. linojon

    linojon

    Joined:
    Aug 25, 2014
    Posts:
    118
    +1 for new Input System. I'm anticipating the legacy input manager will be deprecated at some point so we're starting all new projects to use the new Input System
     
    KyryloKuzyk likes this.
  12. any_user

    any_user

    Joined:
    Oct 19, 2008
    Posts:
    374
    Found the issue, it was caused by the reference image library (with around 10 images) of my tracked image manager. It seems to upload the library before starting the camera feed. Not sure why that takes so long, 1 min for 10 images seems a bit excessive.

    Removing the reference „fixed“ it – I anyway needed a mutable reference library i can fill at runtime for my project. When dynamically adding the images at runtime everything works well, the camera preview starts immediately after entering play mode. Adding the same 10 images to the library at runtime only takes a few sec.
     
    KyryloKuzyk likes this.
  13. fanjiang

    fanjiang

    Joined:
    Jan 9, 2020
    Posts:
    3
    Hi, where is this scene? I can't find this face mask and sloth head in the latest package.
    Thx!
     
  14. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Thank you for finding the cause of the issue!
    Yes, you're right, the plugin sends all images from the reference library to the companion app on every scene start. I should make this process non-blocking, or at least log a warning in case of a big image library.
     
  15. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    The scene with ARKit Blendshapes example is included inside the plugin and is called 'ARKit FaceBlendshapesExample.scene'.
     
  16. cecarlsen

    cecarlsen

    Joined:
    Jun 30, 2006
    Posts:
    858
    Hi @KirillKuzyk I just bought your plugin and it seems to work well so far. I just don't understand how UT prioritises their work. AR Foundation makes absolutely no sense without a working AR Remote. I wish they would hand you a bag of gold and a fresh team of developers to make this a native solution.

    Anyway. I ran into this issue with rendering the camera background in URP and wonder if you have a recommendation?
    https://forum.unity.com/threads/war...-inside-the-scope-of-a-scriptableren.1028794/
     
    KyryloKuzyk likes this.
  17. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Thank you for your support! I'm doing my best.

    The "You can only call cameraDepthTarget inside the scope" is not related to the Editor or to the plugin itself, but to the URP 10. Do you have an option to downgrade the URP? If not, I'm afraid the only possible solution is to ignore the warning and wait for the official fix.
     
  18. cecarlsen

    cecarlsen

    Joined:
    Jun 30, 2006
    Posts:
    858
    I ended up downgrading to build-in RP. This seems to happen every time I decide to try out URP =)
     
    KyryloKuzyk likes this.
  19. makaka-org

    makaka-org

    Joined:
    Dec 1, 2013
    Posts:
    1,003
    Hi, Any Plans for Input System Package and Unity 2021?

    Honestly, I use the both input systems in single project for Unity 2021.
     
    KyryloKuzyk likes this.
  20. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Yes, the new Input System is on my roadmap. Could you please tell me what features of the new Input System do you use the most for AR development?
     
  21. makaka-org

    makaka-org

    Joined:
    Dec 1, 2013
    Posts:
    1,003
    screenshots:
    https://ibb.co/KzfcFmK
    https://ibb.co/Vt5B0tM
     
    KyryloKuzyk likes this.
  22. cecarlsen

    cecarlsen

    Joined:
    Jun 30, 2006
    Posts:
    858
    Hi @KirillKuzyk, is there an easy way to temporarily disable the plugin? When I run scenes without the phone connected I get a SocketException. Pretty rough. Perhaps this should be warning. Sometimes I may want to test things without plugging the phone ... and without uninstalling your plugin.

    If I simply disable the plugin in XR Management, I'm faced with another error:
    "Please enable "AR Foundation Remote" provider in Project Settings".
     
    Last edited: Mar 23, 2021
    KyryloKuzyk likes this.
  23. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    You're almost there :)
    Please also disable the 'Settings/Debug Settings/Log Startup Errors'.
     
    cecarlsen likes this.
  24. Aaron-Meyers

    Aaron-Meyers

    Joined:
    Dec 8, 2009
    Posts:
    305
    Hey there... I'm trying to start a new project with the plugin and I'm having a problem where it doesn't show up as an option under XR Plug-in Management. I also noticed that there is no "AR Foundation Remote Loader.asset" in the Assets/XR/Loaders/ folder. It's been a few months since I've used the plugin so I might be missing something obvious?

    I tried using the Unity AR Project template as well as just making a new project and manually adding the ARFoundation and ARKit packages through the package manager, but in both cases, I can't seem to enable the AR Foundation Remote Plug-in provider (because its just not showing up at all).

    upload_2021-3-24_14-59-6.png
     
    Last edited: Mar 24, 2021
  25. Aaron-Meyers

    Aaron-Meyers

    Joined:
    Dec 8, 2009
    Posts:
    305
    Just an update: I've tried to setup a new project with the plugin in three different versions of Unity: 2020.3 (latest LTS release), 2019.4 (previous LTS release), and 2021.1 (latest official release). My steps in each one:
    1. Create New Project
    2. Switch project build target to iOS
    3. Add ARFoundation and ARKit packages through package manager
    4. configure XR Plugin-Management in Project Settings
    5. Install AR Foundation Remote Plugin
    6. Console says "AR Foundation Remote installed successfully", but there is no check box for it in the Win/Mac/Lin XR Plug-in Management section as seen in the screenshot in my previous post.
    I got this to work when I was last using it a few months ago, but for some reason its not working for me anymore!!! I have the latest version updated via package manager. Please let me know if I missed a critical detail. Thanks.
     
  26. NSWell

    NSWell

    Joined:
    Jul 30, 2017
    Posts:
    89
    emmm.... I can connect to 192.168.0.x , It was working before, but after a few days it was unable to connect. Under the same network. It can be pinged.

    Edit:

    Unity 2020.3.0f1
    AR Remote: 4.9.3
    ARFoundation 4.1.5
     
  27. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Thanks for the bug report!
    I was able to reproduce it. It started to happen in Unity 2020.2 and newer, and I was able to fix it internally. Please contact me in direct messages or via email so I can send you a pre-release version with a fix.
    The current workaround is to unfocus Unity and focus it again. This will force the Package Manager to update and will fix the issue.
     
  28. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Could you please tell me what's changed since the last time the plugin was working? Did you update Unity or the plugin itself?
     
  29. NSWell

    NSWell

    Joined:
    Jul 30, 2017
    Posts:
    89
    Thank you for your reply!
    > what's changed since the last time the plugin was working?
    Nothing has changed.

    > Did you update Unity or the plugin itself?
    Yes, But I was import and reinstall companion app

    Test Result:
    I just import the plugin to new project(Not rebuild remote app)
    But I was tried connect using USB(set the USB-IP[169.xx.xx]) it's working on Mac Big sur, Not working on the Windows 10.

    ---
    Unity 2020.3.0f1
    AR Remote: 4.9.3
    ARFoundation 4.1.5
     
  30. Frankster

    Frankster

    Joined:
    May 6, 2013
    Posts:
    12
    Hey Kirill this is a GREAT lifesaver-tool!!! Is it possible to temporarily disable it without uninstalling? When i uncheck it in the project settings and hit play in the editor it gives me an error. Thank you so much!
     
    Griffo likes this.
  31. Marks4

    Marks4

    Joined:
    Feb 25, 2018
    Posts:
    521
    KyryloKuzyk likes this.
  32. Bersaelor

    Bersaelor

    Joined:
    Oct 8, 2016
    Posts:
    110
    I hadn't used the remote for a while, but when I recently needed it again, it doesn't work anymore. The companion app builds successfully after clicking "Install AR Companion App". But then it fails to install and start with the following error:

    Code (CSharp):
    1.  
    2. Installation failed with the following output:
    3. Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
    4. [BT:0.10.3] Error: The archive doesn't seem to be an App Bundle, it is missing required file 'BundleConfig.pb'.
    5. com.android.tools.build.bundletool.model.exceptions.BundleFileTypesException$MandatoryBundleFileMissingException: The archive doesn't seem to be an App Bundle, it is missing required file 'BundleConfig.pb'.
    6.     at com.android.tools.build.bundletool.validation.MandatoryFilesPresenceValidator.checkBundleHasBundleConfig(MandatoryFilesPresenceValidator.java:68)
    7.     at com.android.tools.build.bundletool.validation.MandatoryFilesPresenceValidator.validateBundleZipFile(MandatoryFilesPresenceValidator.java:58)
    I double checked that `buildAppBundle` is set in the build settings. Furthermore, `build & run` for actual main app works fine on the same Android phone.

    Version 4.9.3, running Unity 2021.1.2f1

    PS: If I manually turn `buildAppBundle` in the build settings OFF, the companion app installs again. That just means I have to remember to turn it back ON again before I build our main android app. Maybe you can set
    Code (CSharp):
    1. EditorUserBuildSettings.buildAppBundle = false;
    during the builds of the companion app?
     
    Last edited: Apr 10, 2021
    KyryloKuzyk likes this.
  33. Bersaelor

    Bersaelor

    Joined:
    Oct 8, 2016
    Posts:
    110
    Now, that I got the Remote running again, it starts to look real funky: (I mean it does work, the normal UI is clickable, just somehow the video that shows behind it is real spacy)
     
    KyryloKuzyk likes this.
  34. warrenLD

    warrenLD

    Joined:
    Jul 2, 2019
    Posts:
    9
    Hi could someone elaborate on this? Specifically how to forward the TCP port over USB using that software? I have no idea how to do that
     
    KyryloKuzyk likes this.
  35. mvanniekerk19

    mvanniekerk19

    Joined:
    Oct 18, 2019
    Posts:
    7
    Hi, I'm having an issue with the build for the Companion App.

    Once I have imported the plugin and added all the packages needed, I build the Companion App from the Installer. However, installing the app and opening it on my phone (Samsung S21) i get the Unity splash screen and then it stays blank.

    I have tried restarting the process in a new clean project, which results in either another blank screen as above or once I am able to add the IP and rebuild the Companion App, the camera feed is completely white (Plane and Point tracking is still visible)

    I am using:
    • Unity: 2019.4
    • Remote Editor: 4.9.3
    • ARCore: 4.0.8
    • AR Foundation: 4.0.8
    • XR Management: 3.2.16
     
  36. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Yes, huge thanks for your suggestion! I'll add this in the next version.

    Hmm, this looks strange. Could you please send me your setup in a private message or via email?
    1. Unity version. Windows or macOS?
    2. Send me your Packages/manifest.json and Packages/packages-lock.json files.
    3. Your AR device and its OS version.
     
  37. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    1. Could you please attach the ADB debugger and tell me if there are any errors when you start the companion app?
    2. Do you use URP in your project?
    3. Does your own AR app work fine if you run it on your device?
     
  38. Elliott-Mitchell

    Elliott-Mitchell

    Joined:
    Oct 8, 2015
    Posts:
    88
    Last edited: Apr 14, 2021
    KyryloKuzyk likes this.
  39. Elliott-Mitchell

    Elliott-Mitchell

    Joined:
    Oct 8, 2015
    Posts:
    88
    KyryloKuzyk likes this.
  40. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Not directly related to your question, but I've managed to connect the Android phone by wire with the help of ADB port forwarding! I'll leave the steps here for future visitors:
    1. Launch the AR Companion app with disabled Wi-Fi. The app will display the 'Can't start server...' message. This is OK and actually means that the companion app can't find any externally visible IP address.
    2. Open the adb tool and execute the following command: adb forward tcp:44819 tcp:44819
      Where 44819 is a port from the plugin's setting.
      This command will forward any localhost request to the connected Android device.
    3. Enter this IP in the plugin's settings: 127.0.0.1
     
  41. ferretnt

    ferretnt

    Joined:
    Apr 10, 2012
    Posts:
    412
    On iOS (iPhone 12 with Lidar) we get a Depth Texture over arfoundation remote from m_occlusionManager.environmentDepthTexture, but getting depth confidence with m_OcclusionManager.environmentDepthConfidenceTexture always returns null. Is this expected to work and if not do you have any timeline for it to work?

    Note to be clear we only need a GPU texture - we can handle marshalling to CPU on editor (i.e. equivalent to the *CPUImage* functions) ourselves.

    Thanks for all your work to date!
     
    KyryloKuzyk likes this.
  42. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    The environmentDepthConfidenceTexture was supposed to work, but it turned out there is a bug that prevents it.

    When I serialize textures in the companion app, I check if the camera material has the texture name in its shader. If it doesn't, I skip the serialization. The current implementation of the ARKitBackground.shader doesn't support depth confidence, so the environmentDepthConfidenceTexture will be ignored.

    Please send me your invoice number in a PM, and I'll send you an unreleased version with a fix.

    Huge thanks for reporting the bug!
     
  43. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    PLUGIN UPDATE POST #11 (version 4.11.0)

    Hi, forum!

    It's been a while since I wrote the last plugin update post, but it doesn't mean I stopped working on the plugin. I've been working intensively on the plugin for the previous five months, and let me tell you what's new.


    Wikitude integration.
    upload_2021-4-16_12-30-50.png
    Thanks to our recent collaboration with the Wikitude team, we ensured our solutions work side by side seamlessly.

    Wikitude SDK is a cross-platform SDK designed for Image Tracking, Object Tracking, Scene Reconstruction, and so much more. Thanks to their platform-agnostic approach, their SDK can work literally everywhere, including Unity Editor. Follow this documentation page to know how to use Wikitude SDK in conjunction with AR Foundation Editor Remote and be even more productive!


    Editor Game View is visible in the AR Companion app.
    Now the Editor Gave View is visible in the companion app, so you can test your AR app right on your actual AR device without making a build with full access to scene hierarchy and all debug information available in the Editor.

    This feature is a final step to catch up with the Unity Remote app in terms of user experience. Now I can confidently say that AR Foundation Editor Remote = Unity Remote app + AR Foundation support.


    Camera CPU images.
    If you're developing a computer vision algorithm or wish to process camera images, then I have good news for you!

    ARCameraManager.TryAcquireLatestCpuImage(), XRCpuImage.GetPlane(), all three variants of XRCpuImage.Convert(): all this now works in Unity Editor!

    The main limitation here is network bandwidth, so framerate is not as high as on an actual device. But as our recent integration with Wikitude shows, the framerate is sufficient for their computer vision algorithms to work flawlessly in the Editor.


    Location Services support (GPS) .
    To test GPS with the help of the plugin, you have to complete these steps:
    1. Add the 'ENABLE_AR_FOUNDATION_REMOTE_LOCATION_SERVICES' define to 'Project Settings/Player/Scripting Define Symbols'. Not all projects require location services permission, so this scripting define prevents an unnecessary 'this app wants to know your location' message from popping up.
    2. Make a new build of the AR Companion app. As always, you can build the companion app from a fresh project; just ensure it has the same package manager dependencies as your main project.
    3. Add this line on top of every script that uses UnityEngine.Input: using Input = ARFoundationRemote.Input;


    What I'm working on right now.
    This plugin became my full-time job for the whole previous year, and I genuinely love working on it. Such a remoting tool is an essential part of building high-quality AR apps, and I understand the responsibility I carry.

    To be able to continue the full-time development further, I'm planning to release a paid plugin upgrade soon. All existing customers will be able to pay only the difference in price between the current version and AR Foundation Editor Remote 2.0 (I'm still working on a name here). The current version of the plugin will still receive bug fixes, and existing customers will still receive top-notch support.

    The initial release of AR Foundation Editor Remote 2.0 will include:
    1. Session Recordings and pre-recorded sessions.

      If you can stream the AR session via the network, then it is possible to stream it from a file, right? Exactly!

      This feature will bring the ability to record your AR session to a file. Then, you can play the session back to debug your AR app in a reproducible environment with full access to scene hierarchy and all debug information.

      Also, I will include pre-recorded sessions with the most common testing scenarios, so you can start developing your AR app even before you have access to an actual AR device.

    2. New Input System support with all benefits of input events and enhanced touch functionality. Input recording is also available as a part of the Session Recordings feature.

    3. ARCore Cloud Anchors support.



     
    Last edited: Aug 16, 2021
    ferretnt and hyperbaton like this.
  44. Fooping_

    Fooping_

    Joined:
    Aug 15, 2020
    Posts:
    1
    Hi @KirillKuzyk
    I got an error when installing the AR Companion App. How can I fix it?
    upload_2021-4-17_2-36-35.png
     
  45. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Hi!
    This script is not a part of the plugin. Could you please send me the whole script so I can check it on my side?
     
  46. nucleartide

    nucleartide

    Joined:
    Jun 15, 2012
    Posts:
    16
    This is the best plugin ever. Thanks @KirillKuzyk!

    Note for new users: if you're on an iOS device, disable your Wi-Fi. Then, the AR Companion app will only show 1 IP address (from the wired connection). You want the wired connection so the framerate of the video isn't choppy.
     
    KyryloKuzyk likes this.
  47. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Thank you!
    Here a quote from the documentation that describes how to connect by wire for other setups:
    Code (CSharp):
    1.     Q: How to connect an AR device to Unity Editor by wire?
    2.     A:  iOS + macOS:
    3.             One of the IP addresses displayed in the AR Companion app is an IP of a wired connection.
    4.             Please disable Wi-Fi, and there should be one IP left that corresponds to the wired connection.
    5.         iOS + Windows:
    6.             https://forum.unity.com/threads/ar-foundation-editor-remote-test-and-debug-your-ar-project-in-the-editor.898433/page-9#post-6849035
    7.         Android + macOS/Windows:
    8.             Launch the AR Companion app on your Android device.
    9.             Open the adb tool and execute the following command: adb forward tcp:44819 tcp:44819
    10.                 Where 44819 is a port from the plugin's setting.
    11.                This command will forward any localhost connection to the connected Android device.
    12.                More info: https://developer.android.com/studio/command-line/adb#forwardports
    13.            Enter the localhost IP (127.0.0.1) in the plugin's settings instead of the Wi-Fi IP displayed in the companion app.
     
  48. songchunyu01

    songchunyu01

    Joined:
    Feb 5, 2021
    Posts:
    3
    Do you have any solutions of this problem?
     
  49. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Currently no, but I'll add this to my roadmap.
    Could you please tell me how critical the synchronization of the camera CPU image and camera position for your current AR project?
     
  50. songchunyu01

    songchunyu01

    Joined:
    Feb 5, 2021
    Posts:
    3