Search Unity

  1. Are you interested in providing feedback directly to Unity teams? Sign up to become a member of Unity Pulse, our new product feedback and research community.
    Dismiss Notice

AR Foundation Editor Remote | Test and debug your AR project in the Editor

Discussion in 'Assets and Asset Store' started by KirillKuzyk, May 26, 2020.

  1. Anastasia-Devana

    Anastasia-Devana

    Joined:
    Aug 18, 2013
    Posts:
    16
    The project compiles and works. But there seems to be a bug with the plane API. What's the best way to send you the bug report?
     
  2. marnsmith

    marnsmith

    Joined:
    Feb 7, 2013
    Posts:
    22
    I'm trying out some of the face tracking samples on iOS, but the camera remains world facing and no faces are tracked. Am I missing some step? Thanks!

    Unity 2020.1
    ARFoundation 4.1.0-preview.7
    iOS 14 beta
     
  3. dh7300

    dh7300

    Joined:
    Jan 23, 2020
    Posts:
    9
    1746c4b0ebe2563f.jpg


    Hello.
    I am a student studying AR FOUNDATION. While searching, you viewed and purchased your assets. However, I looked at the document and did it, but the editor's execution screen does not appear on my phone. Only a white screen appears in the editor's game scene. I wish there was a short tutorial guide video running on Android mobile.
    -From students who started studying AR-
     
  4. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    Please send an email: kuzykkirill@gmail.com

    Could you please tell me what samples are you trying? Those that come with the plugin?
    Do you have an opportunity to test it on iOS13?

    The Editor's screen should not appear on your phone. Your phone is only used to feed the data back to Editor.
    As for the white screen in Editor, it seems like a bug. Could you please describe your setup so I can fix it more quickly? Unity version, AR Foundation version, Windows/Mac, your AR device, render pipeline you use, etc.
     
  5. dh7300

    dh7300

    Joined:
    Jan 23, 2020
    Posts:
    9
    er.png

    I keep getting errors like this.
    unity: 2019.4.9f1
    ARFoundation: 4.0.8
    os: windows
    AR device: SM-G970N(Galaxy S10e)
    I hope you can tell me how to fix it. As I asked at the beginning, I wish there was a short guide video about the setting procedure.
     
  6. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    Yes, I'm preparing a new video for the next release.

    Here is a quote from Documentation that describes how to set the IP:
    '3. Run ARCompanion app on your AR device.
    4. Enter AR Companion app IP in Assets/Plugins/ARFoundationRemoteInstaller/Resources/Settings.asset.'
     
  7. dh7300

    dh7300

    Joined:
    Jan 23, 2020
    Posts:
    9
    Screenshot_20200908-164743_Urban Gaming AR.jpg


    Number 4 doesn't make sense. When I open the AR Companion app, nothing like IP comes out. Look at the thumbnail image. This is the screen that comes out when you open the AR Companion app. If you do No. 2 in the document, the image of thread #203 comes out.
     
  8. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    This is not the correct AR Companion app.
    Please refer to the number 2 in the Documentation on how to install the AR Companion app:
    '2. Switch your project to the appropriate build target and press 'Assets/Plugins/ARFoundationRemoteInstaller/Installer-Install AR Companion app' button.'
     
  9. dh7300

    dh7300

    Joined:
    Jan 23, 2020
    Posts:
    9
    I did according to the document.
    If you do 2, the thumbnail image of thread #203 appears on your phone, and if you do 3, the thumbnail image of thread #207 appears and you need to enter the IP 4 times.
    It was frustrating because it didn't proceed. I just think it's right to wait for the video.
     
  10. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    The image 203 is the correct AR Companion app screenshot. It asks to install the XR Plugin provider (AR Core in your case) and enable it in the XR Management settings.


    The easiest way to get running with the plugin is to ensure that your AR project works before installing the plugin.
     
  11. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    I was able to replicate the issue. Please ensure that the ARKit Face Tracking plugin is installed.
     
  12. dh7300

    dh7300

    Joined:
    Jan 23, 2020
    Posts:
    9
    Screenshot_20200908-174931_ARCompanionAR_Remote.jpg


    I set it up as I told you, and I did it again, but this time, this screen pops up in the AR Companion app. Is it a problem with the Wi-Fi Internet?
     
  13. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    Please make sure your phone is connected to a Wi-Fi network. If it is, please go to Wi-Fi settings on your phone, take the IP from there and enter into plugin's settings.
     
  14. marnsmith

    marnsmith

    Joined:
    Feb 7, 2013
    Posts:
    22
    I tried the one that came with the plugin (FaceIOSBlendshapesExample), as well as a handful of ARFoundation samples
    I haven't gotten to test it on iOS13, no, though the samples work normally on device without remoting

    ARKit Face Tracking is installed, version 4.0.8
     
    Last edited: Sep 8, 2020
  15. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    Could you please try to make a new build of AR Companion app. Maybe the face tracking plugin wasn't installed before you installed the app for the first time.
     
  16. Anastasia-Devana

    Anastasia-Devana

    Joined:
    Aug 18, 2013
    Posts:
    16
    Email sent. Let me know if it makes sense, and any plans to address the issue.
    Thanks!
     
  17. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    Thanks for the detailed bug report. Yes, you've provided more than enough information, I hope I can fix the issue tomorrow.
     
  18. dh7300

    dh7300

    Joined:
    Jan 23, 2020
    Posts:
    9
    Is there anything else I need to set up to apply the remote to the AR project I'm working on, not your example scene?
    Remote operation in your asset example scene was successful. In the AR project that I am developing, I imported the remote asset and then did it in the order, but the screen does not appear in the AR companion app.
     
    Last edited: Sep 9, 2020
  19. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    It turned out I was sending a plane center in world coordinates instead of local.
    Thanks for the detailed bug report, it helped me a lot. Please check your email for the fix.
    This fix will be included in the next release.

    Could you please tell me, are there any errors in Console? Did you enable the 'AR Fondation Remote' provider in Project Settings -> XR Plug-in Management -> PC, Mac & Linux Standalone?
     
  20. Anastasia-Devana

    Anastasia-Devana

    Joined:
    Aug 18, 2013
    Posts:
    16
    Thank you, the fix worked :)
     
  21. any_user

    any_user

    Joined:
    Oct 19, 2008
    Posts:
    368
    This tool looks very useful, but we're using object tracking in our project. Is it still not supported?
     
  22. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    Still not. I'm implementing new features in order of the number of feature-requests.
    Currently, I'm working on Meshing (ARMeshManager) because it is a very requested feature, and then will decide what feature to add next.
    Your voice for object tracking has been counted :)
     
    any_user likes this.
  23. Inter-Illusion

    Inter-Illusion

    Joined:
    Jan 5, 2014
    Posts:
    540
    Hi, I'm getting the black screen with the message "Can't start sender. Please connect AR device to private network".

    I verified that I'm connected to Wifi, even turned off mobile data.
    Also verified the IP Address on the phone is correct and I can ping it from the PC.

    I also uninstalled and reinstalled the companion app several times. But I always get the same message.
    The app used to work, but after the latest update, it started failing with that message.

    Are there any other settings I should change to fix this?
    Thanks
     
  24. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    Could you please go to Wi-Fi settings on your phone and take the IP from there? Then enter this IP in plugin's settings.
     
  25. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    PLUGIN UPDATE POST #7 (plugin version 4.0.19)​


    Huge thanks for all supporters, the plugin is evolving because of you!


    Meshing support (ARMeshManager).

    Meshing was a highly requested feature for a long time and now it's here! Currently, the iPad 2020 with LiDAR is the only mobile device that supports meshing. But, theoretically, the plugin should support meshing on Magic Leap and Holo Lens too. I have no way to prove it without the actual devices, so if you have one, your help would be very appreciated (Holo Lens, Magic Leap).

    Also, the ARKit Mesh Classification is supported. Here is a video that shows the classification in action:


    Anchors support (ARAnchorManager).
    Previously, it was impossible to support ARAnchorManager in Editor, because its API is synchronous. This means that a call to ARAnchorManager.AddAnchor() should return instantly: and I can't make an instant network request to the companion app using built-in Unity connection.

    Happy news! The transition from Unity built-in connection to Telepathy TCP gave me the ability to fully control the connection between Editor and AR device. Now I can send a message from Editor to the companion app, then block the Editor's main thread while waiting for the response in background thread. This results in small hiccups (10-20ms) when you use ARAnchorManager in the Editor, but brings full support of ARAnchorManager.


    Changes in ARKit Blendshapes.

    I refined the way ARKit Blenshapes are received in Editor. Previously, you were required to write a lot of #defines and get Blendhshapes in Editor from BlendShapeCoefficientsContainer.cs. Now, the only change you need to make to your scripts that deal with Blenshapes is this one:
    Code (CSharp):
    1. #if UNITY_EDITOR
    2. using ARKitFaceSubsystem = ARFoundationRemote.Runtime.FaceSubsystem;
    3. #endif

    Bug fixes:

    - Disable camera video in Windows Unity Editor 2019.2 to prevent crash.
    - Fixed: UI Event System is not working after loading new AR scene at runtime.
    - Fixed: BoundedPlane.center should be in local space coordinates instead of the world (thanks @Anastasia-Devana for bug report).
    - Fixed: sometimes the IP will not be shown in AR Companion app on Android.


    Asset Store:
    https://assetstore.unity.com/packages/tools/utilities/ar-foundation-editor-remote-168773

    Website:
    https://stampedegames.wixsite.com/ar-foundation-remote

    itch.io:
    https://kyrylokuzyk.itch.io/ar-foundation-remote


     
    Last edited: Oct 2, 2020
  26. Anastasia-Devana

    Anastasia-Devana

    Joined:
    Aug 18, 2013
    Posts:
    16
    Is there any way to tell in a script if the remote app is currently running (like a global / static variable, etc?)
     
  27. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    @sskenth
    Yes, you can use Receiver.Instance.state to know the current state of the connection to AR device.
     
    Anastasia-Devana likes this.
  28. theOnlyEyez

    theOnlyEyez

    Joined:
    Feb 6, 2020
    Posts:
    4
    @KirillKuzyk Hello! Your tool seems to be exactly what I need! I just purchased on your itch.io because i had some trouble processing in Unity Assets store. So I wonder for the future updates, will you sync the changes on itch.io with unity store? and do I need to pay again for updates?
     
  29. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    Yes, all future updates will be free regardless of where you've made a purchase.
    The itch.io has even a small advantage because I can publish a new version there without the Asset Store validation process.
     
  30. theOnlyEyez

    theOnlyEyez

    Joined:
    Feb 6, 2020
    Posts:
    4
    Thank you so much! I need to use the AR depth/occlusion manager in AR https://blogs.unity3d.com/2020/06/2...undation-now-supporting-the-arcore-depth-api/ So I updated ARcore, ARFoundation and ARKit from version 3 to the latest 4 using git url in package manager (I can't only update one otherwise there'll be bug, had to update all 3). After all updated to version 4, AR Foundation Remote plugin can't work properly anymore and throwing NullReference error:
    upload_2020-9-19_16-38-47.png
     
  31. theOnlyEyez

    theOnlyEyez

    Joined:
    Feb 6, 2020
    Posts:
    4
    This is pasted from the editor log, might be cleaner to see:

    NullReferenceException: Object reference not set to an instance of an object
    at ARFoundationRemote.Runtime.XRCameraFrameSerializable.get_Value () [0x00171] in D:\Projects\FragmentFreeze\Library\PackageCache\com.kyrylokuzyk.arfoundationremote@c85da0b0dd\Runtime\Sender\CameraSubsystemSender.cs:280
    at ARFoundationRemote.Editor.CameraSubsystem+CameraSubsystemProvider.TryGetFrame (UnityEngine.XR.ARSubsystems.XRCameraParams cameraParams, UnityEngine.XR.ARSubsystems.XRCameraFrame& cameraFrame) [0x00019] in D:\Projects\FragmentFreeze\Library\PackageCache\com.kyrylokuzyk.arfoundationremote@c85da0b0dd\Editor\Subsystems\CameraSubsystem.cs:108
    at UnityEngine.XR.ARSubsystems.XRCameraSubsystem.TryGetLatestFrame (UnityEngine.XR.ARSubsystems.XRCameraParams cameraParams, UnityEngine.XR.ARSubsystems.XRCameraFrame& frame) [0x00000] in D:\Projects\FragmentFreeze\Library\PackageCache\com.unity.xr.arsubsystems@4.0.8\Runtime\CameraSubsystem\XRCameraSubsystem.cs:481
    at UnityEngine.XR.ARFoundation.ARCameraManager.Update () [0x0009a] in D:\Projects\FragmentFreeze\Library\PackageCache\com.unity.xr.arfoundation@4.0.8\Runtime\AR\ARCameraManager.cs:327

    (Filename: Library/PackageCache/com.kyrylokuzyk.arfoundationremote@c85da0b0dd/Runtime/Sender/CameraSubsystemSender.cs Line: 280)

    I noticed in example scenes, it seems there's no depth/occlusion information, virtual objects and real objects don't have front and back position block (forgive me for my poor English). How to get access of that in the meanwhile still be able to use remote view plugin? Thank you again!
     
  32. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    Please make a new build of AR Companion app. It is required after you change the version of AR Foundation.

    'I noticed in example scenes, it seems there's no depth/occlusion information, virtual objects and real objects don't have front and back position block'
    Unfortunately, these features are not currently supported. I have them on my roadmap.
     
    theOnlyEyez likes this.
  33. theOnlyEyez

    theOnlyEyez

    Joined:
    Feb 6, 2020
    Posts:
    4
    Thank you! Rebuilt fixed the error!

    May I ask when would the depth map feature be available in your remote plugin? \@_@/

    Another feature would be super convenient is supporting ios devices streaming to windows Unity editor. Keijiro san's project uses NDI stream ipad video to windows machine in real-time https://github.com/keijiro/Rcam2
     
  34. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    I hope I'll add occlusion support in the next 3-4 weeks.

    'Another feature would be super convenient is supporting ios devices streaming to windows Unity editor. Keijiro san's project uses NDI stream ipad video to windows machine in real-time https://github.com/keijiro/Rcam2'

    Yes, I have this on my roadmap too. I will not use NDI though because it's a proprietary technology. I'll add video support on Windows + iOS, but most likely, the occlusion will only work with macOS Unity Editor (Windows Editor doesn't support Metal shaders).
     
  35. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    There is a great video review came out recently. Thanks, Dilmer!

     
  36. Sue_Gamaxx

    Sue_Gamaxx

    Joined:
    Jan 8, 2019
    Posts:
    1
    Hello, this plugin seems very promising and useful.

    I'm doing a project where the user localizes within a room with the ARWorldMap subsystem, drops some objects and saves the map in order to be localised later again with the objects appearing in the same place they were left.
    I would like to use this plugin to save me some time testing and debugging, is it possible to use with the ARWorldMap feature?

    Thanks :)
     
  37. makaka-org

    makaka-org

    Joined:
    Dec 1, 2013
    Posts:
    490
    KirillKuzyk likes this.
  38. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    Unfortunately, ARWorldMap is not currently supported. But I have it on my roadmap and I'll notify you when it's ready.
     
  39. LeoThePhoenix

    LeoThePhoenix

    Joined:
    Oct 4, 2017
    Posts:
    7
    Heyho, I sadly have a similar problem. So the Game View is white or black in Play Mode.
    I'm working with Unity 2019.4.0f1, AR Foundation 4.0.8, the URP and ARCore.
    Is there a way to fix this?
     
    Last edited: Sep 28, 2020
  40. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    Please check that you've installed and enabled ARCore XR Plugin. Here is an installation video for iOS, the process for Android is similar:
     
  41. LeoThePhoenix

    LeoThePhoenix

    Joined:
    Oct 4, 2017
    Posts:
    7
    Thank you for your response. ARCore is in the project and activated. And the remote functionality of your Asset works. I see for instance the found planes in the Game View. It's just the missing image from the camera.

    Is this maybe an URP problem? Because I tested the asset in another project without URP and there everything works fine.
     
  42. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    Yes, it seems like the issue with URP. There are several bug reports related to URP and I'm currently working on it.

    Could you please tell me your setup?
    1. Unity version. Windows or macOS?
    2. Packages/manifest.json file.
    3. Your AR device and its OS version.
     
  43. LeoThePhoenix

    LeoThePhoenix

    Joined:
    Oct 4, 2017
    Posts:
    7

    Unity 2019.4.0f1, Windows 10, Samsung Galaxy S10 with Android 10. The manifest file you can find under the uploaded files.

    Thank you for your effort!
     

    Attached Files:

  44. Admin_Friend_Factory

    Admin_Friend_Factory

    Joined:
    Oct 23, 2018
    Posts:
    41
    Hello! How do you configure the latest 4.3.2 version with URP 7.4.1 and Unity 2019.4.10 with the ARKitHumanSegmentationExample scene? I only get a pink screen. Worked well without URP. Please advice. Many thanks

    Some additional questions:
    1. Please also explain if the Ipad Pro Lidar tracking improves the tracking and human segmentation?
    2. Can you use the selfie front camera for the Human Segmentation?
    3. How do you output the depth data so you can make VFX shader around the character?
     
  45. Admin_Friend_Factory

    Admin_Friend_Factory

    Joined:
    Oct 23, 2018
    Posts:
    41
    Hi again, how do you put the occuluded human character in a 3D scene? I have now black background behind the human segments. Thanks
     
  46. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    @LeoThePhoenix @Scherabon @daapage0
    Thank you for reporting the bug with URP. I've broken the URP support in version 3.11.3, sorry for that.
    I fixed the issue in version 4.2.3 and it's already available on the Asset Store. Please update and tell me if URP is working again.
     
  47. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    There was a bug with URP, please update to version 4.2.3 to fix the problem with the white background.
    The only example scene that supports URP is URP_SwitchBetweenPlaneAndFaceTracking. Please upgrade materials in other example scenes to URP material so they can work properly.

    '1. Please also explain if the Ipad Pro Lidar tracking improves the tracking and human segmentation?'
    I believe yes. But I can't say for sure because my iPhone doesn't support human segmentation. It would be great if you could test it yourself and share the results.

    '2. Can you use the selfie front camera for the Human Segmentation?'
    Just tested it on my iPad with LiDAR, and, unfortunately, this is not supported by ARKit.
    I also tested environment depth on the front camera, and it also doesn't work (both in ARKit and ARCore).

    '3. How do you output the depth data so you can make VFX shader around the character?'
    Please look at AR Foundation Samples repository for examples. Here is an example scene that shows how to display the depth data:
    https://github.com/Unity-Technologi...preview/Assets/Scenes/Depth/DepthImages.unity
     
  48. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    PLUGIN UPDATE POST #8 (version 4.2.3)


    ARKit Human Segmentation
    The plugin keeps evolving and I'm slowly but surely running out of unsupported features. Today, I added the ARKit Human Segmentation support. And it even works in Windows Unity Editor!
    Please check that your iOS device has A12 chip or newer and iOS 13+.
    A video is below, after the environment occlusion section.


    Environment Occlusion (preview)
    This feature was added in AR Foundation 4.1 (currently in preview). In ARKit this feature requires a LiDAR (iPad 2020) and iOS 14.
    But, to my surprise, on Android, it just works with a regular camera! It works slower than LiDAR, but the quality is great, good job Google!



    Camera video with iOS device + Windows Editor
    It turned out that a big number of people are developing iOS apps on Windows PCs. So this update brings the iOS camera background video to Windows Unity Editor.


    Universal Render Pipeline support
    I've broken the support for URP in version 3.11.3, sorry for that. Thanks to @LeoThePhoenix @Scherabon @daapage0 for reporting the bug, this helped me to fix it :)


     
    Last edited: Oct 15, 2020
    LeoThePhoenix likes this.
  49. Admin_Friend_Factory

    Admin_Friend_Factory

    Joined:
    Oct 23, 2018
    Posts:
    41
    Hello, thanks:
    It doesn't seem as the materials in the ARKitHumanSegmentationExample gets updated to URP properly.
    Please see the image here end error message. Please advice how to resolve? Thanks
     

    Attached Files:

  50. KirillKuzyk

    KirillKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    692
    ARKitHumanSegmentationExample doesn't support URP out of the box. The only URP-ready scene is URP_SwitchBetweenPlaneAndFaceTracking.
    To update ARKitHumanSegmentationExample to URP, please assign a URP material to the occlusion_plane game object.
    I'll update all example scenes to URP in the next version, thanks for your feedback!
     
unityunity