Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

AR Foundation Remote | Test and debug your AR project in the Editor

Discussion in 'Assets and Asset Store' started by KyryloKuzyk, May 26, 2020.

  1. Admin_Friend_Factory

    Admin_Friend_Factory

    Joined:
    Oct 23, 2018
    Posts:
    41
    Hi again! Yes, did - tried all URP materials (Unlit/Simple LIt/Lit) but it still doesn't work? Can you check if it works on your side? I am on URP 7.4.1 and Unity 2019.4.10. Thanks
     
  2. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Just checked, everything is working fine.
    Could you please run this scene on your AR device? If the result is the same, it means you didn't set up URP correctly.
    Here is my post describing the setup:
    https://forum.unity.com/threads/arf...k-screen-and-no-tracking.915527/#post-6374280
     
  3. Admin_Friend_Factory

    Admin_Friend_Factory

    Joined:
    Oct 23, 2018
    Posts:
    41
    Many thanks, it was the missing NewARBackgroundRendererFeature. Works fine now. Could you please advice how to get transparent background on the ARKitHumanSegmentationExample scene so I can put the human in a 3D environment?
     
  4. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Just disable the occlusion_plane GameObject and you'll see the camera video background.
     
  5. Admin_Friend_Factory

    Admin_Friend_Factory

    Joined:
    Oct 23, 2018
    Posts:
    41
    Thanks, but i only want the occluded human without the background so I can put the human (with transparent background) in an 3D environment. How do you configure that?
     
  6. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    This question is not related to the plugin, but I'll try to address it.
    You can move the occlusion_plane further from the camera and make it bigger. This will block all the camera background but will leave humans on the foreground.
     
  7. Admin_Friend_Factory

    Admin_Friend_Factory

    Joined:
    Oct 23, 2018
    Posts:
    41
    Excellent, thanks - such great asset you have developed for quick live iterations. It was a scaling issue on the occlusion plane that was way too small and the Camera Far Clip view was set too low. Works perfectly now. Do you know how to route the live occluded AR video to a video texture? I would like to make reflections of the real human on a mirror plane in the 3D environment?
     
    Blarp likes this.
  8. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Please search for Texture2DSerializable.cs in Packages/ARFoundationRemote folder for the example on how to copy video texture from GPU to CPU.

    Or you can use this API, but it's currently not supported by the plugin and you will not be able to test it in the Editor:
    https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@4.0/manual/cpu-camera-image.html
     
    Blarp likes this.
  9. Admin_Friend_Factory

    Admin_Friend_Factory

    Joined:
    Oct 23, 2018
    Posts:
    41
    Blarp likes this.
  10. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Yes, I have it on my roadmap. Can't say the exact date though.
     
    Blarp likes this.
  11. Alexis-Dev

    Alexis-Dev

    Joined:
    Apr 16, 2019
    Posts:
    121
    Hello,

    I download the last version today. Did you remove the posibility to select the input system in settings?

    Best,
    Alexis
     
  12. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    The plugin only supports the old Input System and never supported the new one.
    In version 4.0.17 I removed the InputSimulationType setting from the plugin's Settings. This setting was confusing and it didn't add any value. I think it confused you too in believing that there was a setting for an Input System type.
     
  13. Alexis-Dev

    Alexis-Dev

    Joined:
    Apr 16, 2019
    Posts:
    121
    Me mistake,
    I speak about the InputSimulationType setting.
    I use it to prevent error because the plugin doesn't support new Input System.

    But I think I can use the plugin without this settings, an error spam me console but I can interact with my objects...

    Best,
    Alexis
     
  14. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    I see. So it would be best for you to add a setting to disable input simulation, right?
     
  15. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    PLUGIN UPDATE POST #9 (version 4.4.2)


    ARKit 2D and 3D Body Tracking support.

    In this update, I added support for ARKit 2D and 3D Body Tracking. If you have an iOS device with A12 chip or later, you can try the Body Tracking yourself.
    Here is a cringeworthy video I recorded in my room using the plugin :)



    Face Tracking improvements.

    ARKit and ARCore documentation state that face mesh indices and UV coordinates never change after the face was detected for the first time. So this was a great opportunity to reduce the network load by caching this data.

    Also, in this update, you can choose what face data to receive from AR Companion to increase FPS and reduce latency. For example, if you want to record a smooth facial Blendshape animation, you can turn all other options off, and your framerate will go up to 60 FPS. With AR Foundation Editor Remote you can easily devise a custom solution for recording face animations, similar to MocapX, but right in the Unity Editor.

    Camera configurations support.

    Facing performance issues in your AR project? Or want to select a camera video framerate? Now you can do with camera configurations support right in the Editor. Please note that you'll not be able to see a full video resolution in the Editor due to plugin limitations.


    Git dependency is now gone.

    Git is now not required for the plugin installation process. While git is installed on macOS by default, on Windows it is sometimes needed to manually set the correct git environment path, which was not a great user experience.

    Fixes.

    - Fixed mesh flickering while using ARMeshManager.
    - Fixed: ARMeshManager will not work after calling ARSessionOrigin.MakeContentAppearAt(). This method creates a new parent for ARMeshManager, but ARMeshManager requires to be a direct child of ARSessionOrigin. This is a bug of AR Foundation, and I created a temporary workaround by restoring ARMeshManager's parent.
    - Fixed error caused by changing occlusion resolution at runtime.


    You can get the latest version here:
    Asset Store

    itch.io
    Website


     
    Last edited: Nov 15, 2020
    Alexis-Dev likes this.
  16. Alexis-Dev

    Alexis-Dev

    Joined:
    Apr 16, 2019
    Posts:
    121
    Not necessary, don't waste your time with this feature. ;)
     
  17. LeoThePhoenix

    LeoThePhoenix

    Joined:
    Oct 4, 2017
    Posts:
    7
    Hi,
    I opened my project on my MacBook for the first time today (on Windows everything works fine) and I'm not able to tick the "AR Foundation Remote" option in the XR Plug-in Management. I also tried deleting the whole asset and reimporting it again.
    During the tick it shortly writes "Assigning all requested loaders" and stops again.

    Did someone experienced something similar?

    I'm working with Unity 2019.4.0.
     
  18. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    I managed to reproduce the issue. Could you please try these steps and tell me if one of them will help?
    1. Remove the Packages/packages-lock.json file.
    2. If the first step will not help, press Assets/Reimport All.
     
  19. MyPassion

    MyPassion

    Joined:
    Jun 21, 2018
    Posts:
    8
    Hi KirillKuzyk,
    I build the xcode project by using the Installer->Build AR Companion and Show in folder on windows,then I transfer to my mac and build the app successfully,when i start the app and set the ip in settings and run a sample scene,it always connect failed(I want use my iphoneX to remote my windows pc),I'm surely that all the device is under the same wifi.
    thanks alot.
    Unity version 2019.3.9
     
  20. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    1. Have you completed this step?
    Enable AR Foundation Remote provider in Project Settings -> XR Plug-in Management -> PC, Mac & Linux Standalone (skip this step for Unity 2019.2).

    2. Do you have any errors in the console?

    3. Is your phone unlocked and running companion app?
     
  21. MyPassion

    MyPassion

    Joined:
    Jun 21, 2018
    Posts:
    8
    Yes I completed all the step before build Xcode project and no error display in console,the app is running and phone unlocked.
    I'll try different net environment later,maybe the firewall can cause the problem.
     
  22. MyPassion

    MyPassion

    Joined:
    Jun 21, 2018
    Posts:
    8
    there is a warning
    No active UnityEngine.XR.XRInputSubsystem is available. Please ensure that a valid loader configuration exists in the XR project settings.
    UnityEngine.XR.ARFoundation.ARInputManager:OnEnable() (at Library/PackageCache/com.unity.xr.arfoundation@4.0.9/Runtime/AR/ARInputManager.cs:24)
     
  23. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    This warning is harmless and can be ignored.
    Could you please check that the IP in companion app is the same as in phone's WiFi settings?
     
  24. MyPassion

    MyPassion

    Joined:
    Jun 21, 2018
    Posts:
    8
    Yes it is.
     
  25. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    @MyPassion Could you please send me a bug report to this email describing your setup? kuzykkirill@gmail.com
    1. Packages/manifest.json file.
    2. Your AR device model and its OS version.
     
  26. MyPassion

    MyPassion

    Joined:
    Jun 21, 2018
    Posts:
    8
    The mail has been sent,many thanks.
     
  27. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Ok, here is the solution that doesn't require reimporting all assets:
    1. Delete the file "Assets/XR/Loaders/AR Foundation Remote Loader.asset".
    2. Re-enable "AR Fondation Remote" checkbox in XR Plug-in Management window.

    This workaround will also work for other XR providers.
     
    LeoThePhoenix likes this.
  28. LeoThePhoenix

    LeoThePhoenix

    Joined:
    Oct 4, 2017
    Posts:
    7
    Reimport All solved it for me. Thank you!
     
  29. Admin_Friend_Factory

    Admin_Friend_Factory

    Joined:
    Oct 23, 2018
    Posts:
    41
  30. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Yes, it does!

    iOS: all face tracking features are supported: Face Mesh, Blendshapes, etc.
    Android: only Face Mesh is supported. Face regions are not supported. I can add the support for Face Regions in the next version if you need them.
     
  31. Admin_Friend_Factory

    Admin_Friend_Factory

    Joined:
    Oct 23, 2018
    Posts:
    41
    Excellent, thanks! Let me first use the face mesh as it is, I will let you know if we need more regions.
     
    KyryloKuzyk likes this.
  32. MyPassion

    MyPassion

    Joined:
    Jun 21, 2018
    Posts:
    8
    I‘m done,
    I borrowed a Mac mini then build the project on it,all is ok and it can remote the blendshape normally.
    but it still can't work with windows,I don't know why.
     
  33. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    I'm glad you have access to the macOS computer, but I'm not happy about the plugin doesn't work with Windows + iOS.

    It seems like this issue only appears for users from China. One of my Chinese customers was able to connect Windows PC to iOS device by connecting via Personal Hotspot (unfortunately, not all PCs have a Wi-Fi module). I wonder if the issue may be related to a firewall?

    Could you please try to change the port in the plugin's settings and make a new build of companion app?
     
    Last edited: Oct 26, 2020
  34. skoteskote

    skoteskote

    Joined:
    Feb 15, 2017
    Posts:
    87
    Brilliant app, this is gonna save me so much time, really wish I knew about it sooner! Quick q: I suppose it doesn't support ARKit participant manager for shared AR sessions? (I'd have two unity instances running at the same time and play the app separately from each to two different devices each running the remote.)

    Currently working on a shared experience AR thing, and the workflow is a nightmare, having to build to two separate devices for every test.
     
  35. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Thanks!
    No, the collaborative AR sessions feature is currently not supported, but I have it on my roadmap.
     
    skoteskote likes this.
  36. Duckocide

    Duckocide

    Joined:
    Aug 30, 2014
    Posts:
    35
    Hi @KirillKuzyk

    Dropped you an email but suspect an answer to this may be useful on the forums...

    Just love this asset by the way. So so useful and use it constantly.

    When I load a new scene, the AR Foundation Remote loses connection - losing AR control. Basically, I'm using the usual tricks of Don't Destroy and Singleton patterns to manage loading different scenes (exit from one to another).

    I'm a bit stuck.

    I've tried this (below), but I've not found what is triggering the disconnect (reluctant to start hacking through your plugin scripts if indeed they are what is causing it) and wonder if the remote app connection closure is something to do with the old scene closure event?

    Code (CSharp):
    1.             // AR Foundation Remote
    2.             if (Application.isEditor)
    3.             {
    4.                 GameObject go = GameObject.Find("ARFoundationRemote.RuntimeEditor.Receiver");
    5.                 if (go != null) GameObject.DontDestroyOnLoad(go);
    6.             }
    Help !? :)
     
  37. Gatskop_Software

    Gatskop_Software

    Joined:
    Sep 21, 2014
    Posts:
    86
    Hi couple of questions before buy
    1. Will this work with galaxy note 10 plus
    2. Can face tracking be saved as animation and used in unity on other characters
    3. Body track will it work on note 10 plus and can animation be saved as animation to be used on full-body character
     
  38. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    I'm glad to hear that :)

    Yes, indeed, the Receiver is destroyed when AR scene is unloaded. The receiver will be re-created on the next AR session start and a new connection will be established.
    Could you please tell me more about your use-case? I believe you wish to load a new scene without stopping your AR session, right? Is there a chance you can create a small example project for me so I can adapt the plugin for it?
     
  39. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    1. According to this page, yes, ARCore is supported on galaxy note10+
    upload_2020-11-5_9-37-47.png
    To be 100% sure, you can install this samples repo and run it on your phone. This is the best way to test what AR features your device supports.

    2. There is no simple way to achieve this with ARCore: ARCore only supports face mesh, and there is no simple way to transform mesh transformations into reusable animation.
    The ARKit Face Blendshapes are more suitable for this task (iOS device with True Depth camera required). The plugin supports ARKit Blendshapes and you can devise your custom solution to record animation: blendshapes are no more than enum-float pairs that describe animation rig coefficients.

    3. Same as for the previous question, body tracking is only supported in ARKit (iOS device with A12 chip or newer is required). The plugin supports ARKit Body Tracking, but there is also no out-of-the-box support for recording body animation.


    To sum up, the plugin is designed to speed up the development process and adds no extra functionality to what AR Foundation can do. If you can do something with AR Foundation, then you can do the same with my plugin and test it right in the Editor (please mind the Limitations section).
     
    Last edited: Nov 5, 2020
  40. Duckocide

    Duckocide

    Joined:
    Aug 30, 2014
    Posts:
    35
    Hi

    I'll try to pull a project together for you, however the use case is pretty simple. Let me elaborate a little more...

    The AR session (and scene) to begin with is using horizontal plane detection (imagine a table top). All works fine. The scene is placed on a selected plane (and remains there). It's a third person type view. AR Foundation remote happily send the telemetry back.

    In the scene (game), there are various exits (game objects) that the player can interactive with (doorways if you will). You go through a doorway (collision trigger) and a new scene (beyond the door) is loaded. Fairly simple and common in games.

    I've tried using single and additive scene loading (and removing the old scene through unload) but the connection is dropped either way (remote app returning to the connection wait screen).

    If this explains things enough, let me know and I'll top cobbling together an example project ;)

    ps) In the use case, the AR Session items etc are not destroyed scene to scene.
     
    Last edited: Nov 5, 2020
    KyryloKuzyk likes this.
  41. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Thanks!
    I don't think there is a need to make an example project, your explanation is good enough.
    Does your project work correctly if you run it on a real device?
     
  42. Duckocide

    Duckocide

    Joined:
    Aug 30, 2014
    Posts:
    35
    Oh. Thats' a good question. Not tried it for while - Just tried it, game play is bugged although scene loads :( ... I'll comeback to you once working. Will it make that much difference (Are you thinking perhaps the AR Sessions are being trashed - They are currently set to don't destroy).
     
  43. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    I managed to replicate your problem and will release a fix in a few days.
     
    Duckocide likes this.
  44. MolangRabbit

    MolangRabbit

    Joined:
    Feb 5, 2018
    Posts:
    5
    Hi, I purchased this awesome tool today.
    I followed all of your steps except the git things and I can confirm that your example scenes were completely worked via my device and editor, but it was different with the scene I've been creating in my project.

    In the console, it keeps saying 'AR Foundation Remote: please ensure that your AR device is unlocked and running ARCompanion app.' It was succeeded once (maybe I tried more than 30 times to get that), but I couldn't figure how it was. When opening your sample scenes, then it works well. Are the git things necessary, or do I have other issues to deal with?
     
  45. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,128
    Does your AR scene work on a real AR device if you make a regular build?
    The latest 4.4.2 version doesn't require git to be installed.
     
  46. Alexis-Dev

    Alexis-Dev

    Joined:
    Apr 16, 2019
    Posts:
    121
    Hi,

    I install the last version (4.4.2) in project to test image tracking, but I have an error in AR companion app:

    This error appear with example scene and my scene. I try it in existing project and in clean project.

    Version:
    - Unity 2020.1.10
    - AR Foundation 4.0.9
    - AR Core XR Plugin 4.0.9
    - AR Kit XR Plugin 4.0.9

    Best,
    Alexis
     
  47. Duckocide

    Duckocide

    Joined:
    Aug 30, 2014
    Posts:
    35
    Awesome... Thanks! Looking forward to that :)
     
  48. MolangRabbit

    MolangRabbit

    Joined:
    Feb 5, 2018
    Posts:
    5
    Yes, my scene works on Android devices perfectly. I don't know why, but now the plugin begins to work well. There was the error with the input system when I was back to my work and after dealing with it, no problems.

    [UPDATED]
    My project is using AR Tracked Image Manager. When using your Reference Image Library in the sample scene, it works well with the plugin. But not the case of the library I created.

    'Editor to Player connection lost, consistency is not guaranteed. Please restart Editor scene.' keeps coming up.
    I also tried rebuild AR companion with the state but no changes.
     
    Last edited: Nov 12, 2020
  49. Duckocide

    Duckocide

    Joined:
    Aug 30, 2014
    Posts:
    35
    Hi

    Confession time ... I've managed to clean-up all the post additive scene load game logic issues I had. AR Remote seems to be working now without dropping connection? Glad you managed to replicate the issue either way but it looks like the issue for me was triggered (indirectly?) by a few null pointer exceptions following new scene load.
     
  50. ProjectileVomitTV

    ProjectileVomitTV

    Joined:
    Aug 12, 2020
    Posts:
    30
    This sounds like it could potentially be the solution to a problem I've been having for ages. I need to monitor the video and audio of my AR app from a separate device and record with good quality. Is this possible? I know it's designed for debugging, but seems like a possible fringe benefit here.