Search Unity

  1. Read here to learn more about our latest preview of the XR Interaction Toolkit.
    Dismiss Notice
  2. Click here to receive a gift with your purchase of Unity Pro or Unity Enterprise.
    Dismiss Notice
  3. Read here for Unity's latest plans on OpenXR.
    Dismiss Notice

ARKit support for iOS via Unity-ARKit-Plugin

Discussion in 'AR' started by jimmya, Jun 5, 2017.

  1. flyingdeutschman

    flyingdeutschman

    Joined:
    Jul 26, 2013
    Posts:
    8
    Hey Jimmya, thanks for answering.
    Yes, that's what I'm using. Are there any options that keep your objects grounded?
     
  2. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    Change the example so that it hit tests only against existing planes.
     
    flyingdeutschman likes this.
  3. KarelA

    KarelA

    Joined:
    Dec 30, 2008
    Posts:
    414

    Hi. haha i am so sorry. I must have had a major brainfart while typing my comment. I meant i attached "UnityARCameraNearFar" on both cameras. I will check my hierarchy later tonight when i get home and post back to you.

    Sorry for confusion :)
     
  4. kennyallau

    kennyallau

    Joined:
    Aug 5, 2014
    Posts:
    8
    hi @jimmya , I've tried to replicate the UnityRemoteVideo.cs to get the raw data from y texture just like you've suggested but I'm still not getting the right results so I might be retrieving the texture wrongly. When you say y texture, do you mean the y texture from the m_textureYBytes or the one from YByteArrayForFrame(1-currentFrameIndex)?
    And also, from this y texture, do I just load them into Texture2D using Texture2D.LoadRawTextureData?
     
    Last edited: Sep 5, 2017
  5. massivebacon

    massivebacon

    Joined:
    Apr 24, 2014
    Posts:
    19
    Just ran some tests with this and it seems like it fixed it! I'm obviously worried it's a hack to fix it but I'm not sure if culling was supposed to be on or if it was intentionally left off. So far it looks good though and my "ghosting" is gone! Can @jimmya provide any guidance here as to why this method works? Feels like it should be pushed to the main branch if it is a true fix.
     
  6. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    I have never had this issue, and don't really understand how it could happen that the quad being drawn is flipped. But the fix does not seem to have any bad effects since we always expect to render the quad, so I will add it to the shader if it works.
     
  7. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    First of all, if OpenCV accepts Metal textures, you already have them - you can pass it _videoTextureY in UnityARVideo - you would NOT need to copy them to main memory and then create new texture out of it. If you have to create your own new texture, you can use the mechanism in UnityRemoteVideo (use YByteArrayForFrame(1-currentFrameIndex) as it double buffers when it copies over to main memory to avoid stalling the GPU), and LoadRawTexture.

    Make sure you create the texture with the right size and format ( TextureFormat.R8 for Y texture).
     
    Last edited: Sep 5, 2017
    kennyallau likes this.
  8. KarelA

    KarelA

    Joined:
    Dec 30, 2008
    Posts:
    414
    petey likes this.
  9. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    A couple more changes on BitBucket:
    - Focus Square UI example (slightly different implementation than shown in ARKit example, but you can customize for your use)
    - YUVShader has turned culling off to fix a bug some people were having with the camera clear skipping frames
     
    christophergoy likes this.
  10. kushG

    kushG

    Joined:
    May 25, 2017
    Posts:
    18
    Has anybody come across the issue when you lock the phone and unlock, the camera orientation (i.e whatever we get from UnityARMatrixOps.GetRotation(matrix)) is no longer correct in terms of Magnetic/Compass North. This is the case when you configure with ARKitWorldTackingSessionConfiguration.
     
  11. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    Whenever your app is backgrounded, you lose your sensor and camera data, and so ARKit cannot maintain its world tracking. You will need to reset your world tracking and try and re-establish where your planes are.
     
    kushG likes this.
  12. Aaron-Meyers

    Aaron-Meyers

    Joined:
    Dec 8, 2009
    Posts:
    290
    I've been developing an ARKit Unity app for a few weeks and today I added shadows on the main directional light. I have a plane using the included MobileARShadow shader to catch shadows on the plane that the scene is placed on and I'm finding that sporadically it kind of freaks out in a way where it looks like the framebuffer isn't getting cleared and i get some trippy smeary trails from all the 3D objects in my scene.

    The camera is set to clear "Depth Only" a-la the included example scenes. Is there anything else I might be missing that could be the cause of this bizarre glitch?
     
  13. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    There were a few people earlier on in this thread who were having this issue. Putting in a Cull Off in the YUVShader seemed to fix it for them. I've added it to the YUVShader on Bitbucket (commit).
     
  14. kushG

    kushG

    Joined:
    May 25, 2017
    Posts:
    18
    Maybe a stupid question...how do I reset the tracking ? or How do I kill a session so that I can create a new session ?
     
  15. yuliwei

    yuliwei

    Joined:
    Aug 19, 2017
    Posts:
    22

    Hi, I stuck at step 6, there is no connected player dropdown menu show up.. Plz Help.
     
  16. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
  17. HuangJianming

    HuangJianming

    Joined:
    Jun 20, 2017
    Posts:
    1
    I found a Strange phenomenon.
    1. Open the AR app, show the object in the ARworld
    2. shake the phone back and forth for some time (shake fast)
    3. The position of the camera will deviate from the original position
    Is it a bug? How to fix it?
     
  18. Terry_Stark

    Terry_Stark

    Joined:
    Jun 16, 2017
    Posts:
    5
    Thanks for your suggestion. It actually works because I can walk close to the virtual object now. However, I encounter two new issues:

    1. I notice there is still drift, potentially due to different FOV of ARKit and cardboard. I am wondering whether I can change the FOV of cardboard?

    2. I am using cardboard 1.7.0 and it appears to me that I can only use distorted view. The video background is totally wrong and causes sickness. Is there a way to put video background really far away, or chose the background not distorted by cardboard sdk?

    Looking forward to any reply!
     
  19. Arvin6

    Arvin6

    Joined:
    Jun 2, 2017
    Posts:
    27
    I was wondering pretty much the same thing. By the way, how did you render stereo? Are you using a rendertexture? Could you post your results, so that we can think of something.The render texture method seems unusable from a customer point of view, we should probably wait until apple adds MR support to ARkit.
     
  20. Arvin6

    Arvin6

    Joined:
    Jun 2, 2017
    Posts:
    27
    You seem to have a better result than mine. Can you tell us how you added stereo?
     
  21. cinematik

    cinematik

    Joined:
    Sep 6, 2017
    Posts:
    1
    Hi all,
    i'm fighting with two ARKit unexpected behaviors...

    1- it's possible to "lock" the object in a fixed scene position? i've build a 3D environment in scale with the real world around me.
    (i've a scene with a cube, the cube are fixed in a real world room, when i walk in another room the cube move a little bit and when come back in the first room with my iPhone, the cube is not in the previous position, as if the cube follow me)

    2- i've an issue when i build for iPhone with Virtual Reality and Cardboard support.
    (the Xcode give me a "Googletoolboxformac" issue)

    I've all iOS and Xcode updated to the latest beta and Unity 2017.1.0f3

    I wish to be helpful for the
    community too.
    Thank you!
     
  22. TECNOLOGIA

    TECNOLOGIA

    Joined:
    Jan 23, 2015
    Posts:
    8
    I get ARKit video too bright when i use Linear workflow, how can i solve this?, thanks.
    Best regards.
    Jesus.
     
  23. artician

    artician

    Joined:
    Nov 27, 2009
    Posts:
    337
    I need some help building to iOS11 for ARKit.

    Has anyone successfully built for this platform using a "Personal Team" provisioning profile?

    Is anyone using the Automatic Signing feature of the latest XCode? (It was recommended I turn this feature *off* when building to iOS11).

    Thank you in advance.
     
  24. 4NDDYYYY

    4NDDYYYY

    Joined:
    Jan 28, 2016
    Posts:
    13

    yes and yes. usually, my first build and any "replace" enabled builds require adjusting the General / Signing properties in the xcodeproj.
     
  25. TECNOLOGIA

    TECNOLOGIA

    Joined:
    Jan 23, 2015
    Posts:
    8
    I use "Personal Team" and Automatic signing with lastest XCode with no problem, just remember to change the boundle com.unity.arkitscene for something else.
     
  26. TECNOLOGIA

    TECNOLOGIA

    Joined:
    Jan 23, 2015
    Posts:
    8
    Nobody working in linear space? need help to make the video looks ok with linear space. Thanks.
    Best regards.
    Jesus.
     
  27. Terry_Stark

    Terry_Stark

    Joined:
    Jun 16, 2017
    Posts:
    5
    I just used the Google Cardboard sdk 1.7.0. The stereo is rendered automatically by Unity native sdk I suppose? As a result I can not change the related setting.
     
  28. Arvin6

    Arvin6

    Joined:
    Jun 2, 2017
    Posts:
    27
    Did you tick VR supported in XR settings? Can you switch back and forth between Stereo and Mono view? What do you mean by "related setting" ?
     
  29. Terry_Stark

    Terry_Stark

    Joined:
    Jun 16, 2017
    Posts:
    5
    Same as the other post, I enabled the native VR support and use code to toggle between stereo mode (cardboard sdk) and mono mode (none):

    Code (CSharp):
    1. IEnumerator LoadDevice(bool isVR)
    2.     {
    3.  
    4.         if(!isVR)
    5.         {
    6.             //none
    7.             newDevice = VRSettings.supportedDevices[0];
    8.         }
    9.         else
    10.         {
    11.             //split or cardboard
    12. #if UNITY_EDITOR
    13.             newDevice = "Split";
    14. #else
    15.             newDevice = VRSettings.supportedDevices[1];
    16. #endif
    17.         }
    18.  
    19.         Debug.Log("initialting " + newDevice + "...");
    20.         VRSettings.LoadDeviceByName(newDevice);
    21.        
    22.         yield return null;
    23.  
    24.         Debug.Log(VRSettings.loadedDeviceName + " is loaded");
    25.  
    26.         if (isVR)
    27.         {
    28.             Debug.Log("Try to disable the native tracking");
    29.             VRDevice.DisableAutoVRCameraTracking(mainCamera, true);
    30.                
    31.         }
    32.  
    33.         VRSettings.enabled = isVR;
    34.  
    35. }
    I don't know where to change the setting as I want to change the FoV and disable the distorted view.
     

    Attached Files:

  30. skdev3

    skdev3

    Joined:
    Jul 15, 2015
    Posts:
    55
    DisableAutoVRCameraTracking - do not completely solve the problem (((
    pls watch video


    My code:
    Code (CSharp):
    1.  
    2.     public override void SetupVR(bool isVR)
    3.     {    
    4.         Camera[] camers = (Camera[])GameObject.FindObjectsOfType<Camera>();
    5.         StartCoroutine(LoadDevice(isVR,  camers));
    6.     }
    7.  
    8.     IEnumerator LoadDevice(bool isVR, Camera[] camers)
    9.     {
    10.         // VRDeviceName none
    11.         string newDevice = VRSettings.supportedDevices.Length>0 ? VRSettings.supportedDevices[0] : "none";
    12.         if (isVR)
    13.         {
    14.             //VRDeviceName Cardboard or Split
    15. #if UNITY_EDITOR
    16.             newDevice = "Split";
    17. #else
    18.             newDevice = VRSettings.supportedDevices.Length > 1 ? VRSettings.supportedDevices[1] : "none";
    19. #endif
    20.         }
    21.         Debug.Log("VRDeviceName " + newDevice);
    22.         VRSettings.LoadDeviceByName(newDevice);
    23.         yield return null;
    24.         Debug.Log(VRSettings.loadedDeviceName + " is loaded");
    25. #if !UNITY_EDITOR
    26.         foreach (Camera cam in camers)
    27.             {
    28.                 if (cam != null)
    29.                 {
    30.                     VRDevice.DisableAutoVRCameraTracking(cam, isVR);
    31.                 }
    32.             }
    33.         if(Camera.main)
    34.         VRDevice.DisableAutoVRCameraTracking(Camera.main, isVR);
    35. #endif
    36.         VRSettings.enabled = isVR;
    37.     }
    38.  
    39.  
     
  31. Faye-Euler

    Faye-Euler

    Joined:
    Mar 22, 2013
    Posts:
    3
    Hi man,Have you solve this problem?I had try so many way,and i find that arkit can not be re init....please help!
     
    sdprasad likes this.
  32. harperAtustwo

    harperAtustwo

    Joined:
    Nov 15, 2016
    Posts:
    21

    If I took a guess here, I think this has to do with your the order of setup. ARKit requires the camera parent to be at vector.zero at the start. The VR part may move the camera before ARKit gets a chance to initialize. Just a guess.

    PS: does anyone know where the change log is for the update that was released today ?
     
  33. Wattosan

    Wattosan

    Joined:
    Mar 22, 2013
    Posts:
    297
    How do I go about writing a second pass for the "Custom/MobileARShadow" shader? I would like to have Point lights work with it as well. Not just the directional light.

    Thank you!
     
  34. Jason-RT-Bond

    Jason-RT-Bond

    Joined:
    Mar 26, 2012
    Posts:
    68
    @jimmya: Will we see this API unified with one for ARCore in the near future? I know some capabilities will vary, but would be wonderful to build once and deploy to multiple AR platforms.
     
    Last edited: Sep 8, 2017
  35. pixelsteam

    pixelsteam

    Joined:
    May 1, 2009
    Posts:
    924
    My arkit application runs fine on my iOS11 beta 6 devices.
    I am using Xcode 9 beta version 5
    ARKit Plugin version is from the app store.

    We are not able to submit to the app store.
    I am getting some error see below. But more importantly I am wanting to deploy to the App store. In addition it says that Apple will not accept apps built with Xcode beta versions, not technical support.
    This is a serious issue for us. Is it possible to use Xcode 8 and Application loader to get our .ipa up to the asset store.
    These are the errors we are having
    ITMS 90060
    ITMS 90534
    ITMS 90704
    We have built a really great thing. But I feel screwed. We were planning to announce the app on Sept 12.
     
  36. MonHao

    MonHao

    Joined:
    Jun 30, 2017
    Posts:
    2
    I Meet the same error.
    {7535BD72-F2F8-4D57-AD15-6A73BE2D90ED}.png
    I use XCode 9 beta 6
    unity 5.6.3p2
    new arKit from store
    I can build from empty project with arkit package
    But after import it to my project i meet this error.
    I check the architecture in projectSetting ,and it is "arm64".
    Please help me.3Q
     
  37. sohamsardar

    sohamsardar

    Joined:
    Sep 8, 2017
    Posts:
    2
    Hi Chris, I am working with iOs11 beta 10, Xcode 9 beta 6 and Unity 2017.1.1f1. I would be little elaborate about my issue. So, I am trying to build an example
    . The issue I am facing now is that the app crashes as soon as it open in my iphone 7. Reason:
    Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[ARFrame displayTransformWithViewportSize:eek:rientation:]: unrecognized selector sent to instance 0x1c42eec00'. Now I realised that displayTransformWithViewportSize deprecated and the new function displaytransformwithviewportsize. Now one solution I tried was to rename this function call in my ARSessionNative.mm file but it was complaining no visible interface. Also when I saw the definition of displayTransformWithViewportSize:eek:rientation, it said
    (CGAffineTransform)displayTransformForOrientation:(UIInterfaceOrientation)orientation viewportSize:(CGSize)viewportSize;
    /** Unavailable */
    - (instancetype)init NS_UNAVAILABLE;
    + (instancetype)new NS_UNAVAILABLE;
    @End
    @interface ARFrame (Deprecated)
    - (CGAffineTransform)displayTransformWithViewportSize:(CGSize)viewportSize orientation:(UIInterfaceOrientation)orientation API_DEPRECATED_WITH_REPLACEMENT("displayTransformForOrientation:viewportSize:", ios(11.0, 11.0));
    Can you please help me out as to what I am missing here?
    Thanks in Advance
    Soham
     
  38. sohamsardar

    sohamsardar

    Joined:
    Sep 8, 2017
    Posts:
    2
    I have xCode 9 beta 6 but even I get the same error. Sharing the definition of the deprecated function.
    (CGAffineTransform)displayTransformForOrientation:(UIInterfaceOrientation)orientation viewportSize:(CGSize)viewportSize;
    /** Unavailable */
    - (instancetype)init NS_UNAVAILABLE;
    + (instancetype)new NS_UNAVAILABLE;
    @End
    @interface ARFrame (Deprecated)
    - (CGAffineTransform)displayTransformWithViewportSize:(CGSize)viewportSize orientation:(UIInterfaceOrientation)orientation API_DEPRECATED_WITH_REPLACEMENT("displayTransformForOrientation:viewportSize:", ios(11.0, 11.0));
     
  39. Terry_Stark

    Terry_Stark

    Joined:
    Jun 16, 2017
    Posts:
    5
    To my understanding, the issue occurs because of different Field of View used. I encountered the same problem when I tried to develop portal effect based on renderTexture. I have two cameras in my scene. When I run on iPhone, it appears to drift because FoV of ARCamera has been changed by sdk but not the renderTexture camera. The drift disappeared only when I changed the FoV of second camera to be 36.5.
     
  40. skdev3

    skdev3

    Joined:
    Jul 15, 2015
    Posts:
    55
    Okey. I understand that This is ARKit Bug. RenderTexture - Bad idea(There is no sensation 3d, at 3d objects).

    Unity team, pls help.

     
  41. MonHao

    MonHao

    Joined:
    Jun 30, 2017
    Posts:
    2
    Never mind.I fixed it. There is a compile error in my Project, then log those error instead. Thanks for YOUR support
     
  42. gate1

    gate1

    Joined:
    May 31, 2013
    Posts:
    22
  43. saikiranunity1

    saikiranunity1

    Joined:
    Jun 1, 2017
    Posts:
    2
    I got an issue here....everything works fine unless the Unity AR hit test script is not working, it's detecting the feature points and planes but not able to place the cube on the plane, but it was mentioned hit test will work with the remote.
    OR Am I doing anything wrong expecting a quick reply
     
  44. kennyallau

    kennyallau

    Joined:
    Aug 5, 2014
    Posts:
    8
    @jimmya It works now, thank you!!!
     
  45. Jason-RT-Bond

    Jason-RT-Bond

    Joined:
    Mar 26, 2012
    Posts:
    68
    @jimmya: I'd like to make scenes where the camera is not assumed to start at 0,0,0. I've managed to do this by parenting it to an offset and hacking some of the provided scripts to work in local space, but this makes updating the SDK difficult as you can imagine. Any chance there is an easier way or that support for this could be added out of the box?
     
    Elliott-Mitchell likes this.
  46. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    which solution did you end up using (pointer to texture or copy raw data)? and would you like to contribute an example to the plugin?
     
  47. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    Did you check out the scaled-content branch? You can use an offset in the ContentCamera instead of a scale and it should work for you?
     
  48. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    567
    Can you give us more output from your errors? I have no idea what ITMS XXXXX is or why it's showing up.
     
    pixelsteam likes this.
  49. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    567
    You need to update your plugin.
     
  50. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    567
    You need to add the ARKit.framework to your target's linked frameworks.
     
unityunity