Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

ARKit support for iOS via Unity-ARKit-Plugin

Discussion in 'AR' started by jimmya, Jun 5, 2017.

  1. Ghanshaam

    Ghanshaam

    Joined:
    Apr 12, 2013
    Posts:
    4
    Hi, I am currently working on incorporating ARKit into my Unity project and generally it works fine. However, in my Unity project the y-axis is used for depth and the z-axis is used for height as opposed to the ARKit asset in which the y-axis represents real-world height and z-axis represents depth.

    Is there a way to flip the y-axis and z-axis in the ARKit asset? Thanks
     
  2. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    898
    What's the recommended way to disable ARKit for re-enabling later, ie., for user switching from rear facing camera to front facing (and back again)

    Also, is there a reason why front facing via unity webcamtexture and UnityARVideo aren't working?
     
  3. mimminito

    mimminito

    Joined:
    Feb 10, 2010
    Posts:
    780
    For question number one, I would either look to see if Apple provide an API to get the battery consumption of the device or the current temperature, and if they do you could use that information to tactically display a warning. Otherwise just have it on a timer which asks for the users to stop playing after a while.

    Shaders wise, I would say on mobile always go for mobile oriented shaders. I believe Unity integrates with Metal quite well, so this is something you wont have to worry about too much. Custom shaders are always an option to squeeze performance out and achieve the look your after.
     
  4. mimminito

    mimminito

    Joined:
    Feb 10, 2010
    Posts:
    780
    Looks like it. All iPad Pro's I think are included (so ensure you have all of those listed) and all 2017 iPad's are included as well I believe.

    You can force the app to only work on supported devices using the following technique:
    https://developer.apple.com/documentation/arkit
     
  5. inyman

    inyman

    Joined:
    Oct 24, 2015
    Posts:
    9
    I was wondering how do i receive ARCamera (not UnityArCamera) information. I need euler angles any idea ?
     
  6. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    898
    has it been updated yet
     
  7. Elliott-Mitchell

    Elliott-Mitchell

    Joined:
    Oct 8, 2015
    Posts:
    58
    Thanks!
     
  8. Grislymanor

    Grislymanor

    Joined:
    Aug 15, 2014
    Posts:
    23
    Thanks for the input!
     
  9. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    There have been other suggestions, but one other thing you could try: you can revert to a lower cost tracking configuration once you have established surface to use e.g. do not detect planes anymore, or use 3DOF tracking. These will use lower amounts of energy.
     
  10. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    I would not rely on listing all the supported devices in your code, but instead use ARKit framework's isSupported call on the worldtracking configuration to figure out whether this device is supported.
     
    DerekLerner likes this.
  11. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    This is going to be a pain - we already had to figure out how to convert from ARKit's right handed coordinate system to Unity's left handed coordinate system, and it was not trivial. I would try and change your project to use Unity's coordinate system.
     
  12. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    Front facing camera is not supported by ARKit at this time.
     
  13. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    inyman likes this.
  14. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    898
    Sorry, let me try to clarify my question.

    ARKit works with rear facing camera. However, does this disable using front facing camera (webcamtexture) in the same ARKit app (of course, not using ARKit but using opencv or other tracking)? This was what I meant about disabling ARKit when user selects front facing and re-enabling it when user selects rear facing
     
  15. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    898
    a great feature request for 2017.2 would be a matrix that seamlessly converts left handed to right and back again, for easy compatibility with this and other SDKs

     
  16. Senamoru

    Senamoru

    Joined:
    Aug 9, 2017
    Posts:
    1
    Hey all this might be a questions that has already been answered in a form or another but i'm getting a bit stumped by it currently. I'm currently working on adding markers into Arkit as a way to get the initial position of the where i specific item would be placed in a given scene and then have Arkit deal with keeping the object in the given location rather than constantly looking for the marker.

    Currently I've got opencv's 2D marker texture tracking successfully getting the coordinates based on the image i'm feeding which is coming from the Arkit's supplied camera feed. When i go to place this an object at said location it pops up into the correct spot in the scene, but unlike the objects that I placed into the scene in the editor which tend to keep their relative position when I move around the objects that i'm placing at on demand tend to shift and drift all over when I walk around.

    Is there something trivial i'm missing and or overlooking?
     
    Aiursrage2k likes this.
  17. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    898
    Or is there a way to pause and later resume the ARKit or Video from ARKit? I notice that OnResume() and such are not provided
     
  18. TimeWalk-org

    TimeWalk-org

    Joined:
    Nov 3, 2014
    Posts:
    31
    I had my Unity ARKit iOS demo working great as of last week, but when I updated Unity to 2017.2.0b6 and Xcode to 9.0 Beta 5 (yesterday, August 8), my app now crashes every time. I did, of course, update the ARKit plugin to the latest.

    I tried building a plain-vanilla version of the scene "UnityARKitScene" and it crashes with the same error (log attached). I also noticed that in Xcode, I'm seeing "libiconv.2.dylib" in RED in my project hierarchy. I don't think that was there before my update. [My Mac is running Sierra 10.12.6]

    Is anyone else building ARKit demos with Unity 2017.2.0b6 with Xcode 9.0 Beta 5 successfully?
     

    Attached Files:

  19. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    567
    Hey all,
    Sorry for the delay. The bitbucket repo has been updated for beta 5. The asset store package will be updated shortly.
    Cheers,
    Chris

    Edit: The asset store package has been submit and is awaiting review.
     
    Last edited: Aug 9, 2017
    phits and vfxdad like this.
  20. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    898
    Thanks for supporting this effort - it's a little better than the days when this kind of stuff would be a volunteer-based community effort. But... a few days of downtime is a long time in arkit land...

    Is there a way that you guys can work closer with Apple to have the Unity plugin be up to speed with their updates, especially when things would break?

    Or, simply, auto-update everyone on the thread not to update to the latest iOS 11 update (yet)?
     
  21. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    567
    Hey,
    I understand your frustration with a few day delay on releases. We are doing our best to keep up to date with Apple releases. ARKit is important to us as it is important to you. Please understand that we also have a lot of other work, and there vacations and other things that come up that might cause us to be a bit out of step with Apple releases. If you read back through the thread, I did post to say that Beta 5 was released, and to hold off on updating until I could get an update out. I apologize for the inconvenience.
    Cheers,
    Chris
     
    KwahuNashoba and JoeBibby like this.
  22. jessevan

    jessevan

    Joined:
    Jan 3, 2017
    Posts:
    35
    This question has been asked a few times on the forum. The short answer is that it isn't possible to relocalize the camera after pausing a session. I felt this was a bug, as the `resetTracking` option makes it sound as if you can resume tracking. @jimmya and now Apple's developer relations have told me this is by design.
     
  23. sama-van

    sama-van

    Joined:
    Jun 2, 2009
    Posts:
    1,643
  24. itskenmoore

    itskenmoore

    Joined:
    Jun 11, 2016
    Posts:
    2
    Thanks for all you're doing Chris, I was also directly affected by the Beta 5 issue (I hadn't seen your note to avoid the upgrade). However, I just updated the plugin and tried building and running the UnityARKitScene scene, but I'm getting several errors in XCode beginning with these few in ARSessionNative.mm:

    211: Unknown type name 'ARWorldTrackingConfiguration'; did you mean 'ARWorldTrackingSessionConfiguration'?
    218: Unknown type name 'ARConfiguration'
    255: No visible @interface for 'ARCamera' declares the selector 'projectionMatrixForOrientation:viewportSize:zNear:zFar:'
    417: No visible @interface for 'ARFrame' declares the selector 'displayTransformForOrientation:viewportSize:'


    Can you advise on whether this looks like something I'm not doing properly, or whether the updated plugin has bugs in it?

    Thanks much
     
    bretthoyer likes this.
  25. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    567
    Did you update to Xcode beta 5? It looks like you might still be using an older beta.
     
    bretthoyer likes this.
  26. lourd

    lourd

    Joined:
    May 20, 2017
    Posts:
    15
    Hey y'all,

    Here's a video I just recorded of what I've made so far. Comments & critique welcome — I'm a Unity beginner, I'll take all the feedback I can get.



    Apologies for the lack of audio. YouTube & Google Photos seem to be stripping the audio from videos made with the new iOS 11 screen recording feature ¯\_(ツ)_/¯. I actually talked through the whole thing explaining what I'm doing, I would have made it shorter if I'd known it would be muted.

    This is about a week's worth of work. I'm really happy with progress so far. I made the models in VR using Google's Blocks program. I'm happily surprised with how good they look with just 1 tweak in Unity. I got the pop particle effect working today, which was a really satisfying milestone.

    The arrows are being shot when swiping the screen. Touching down and swiping "pulls back" the arrow. Releasing your finger releases the arrow. I'm working on the animations for that so it's visual.

    A couple of my biggest questions right now are:
    • How does Unity define y=0 for the AR scene? Is it the origin of the device when the AR world tracking starts? Is it the first anchor detected? I have every balloon being created at y=0, but many of them look like they appear at eye level. I'm misunderstanding something.
    • Should I leave ARKit plane detection on after getting the first anchor? Does leaving it on decrease the chance of scene/localization drift? In the video I'm turning it off after getting the first anchor (using the NativeSessionInterface.runWithConfig, with the plane detection parameter as UnityARPlaneDetection.None). It seems to have a little more drift. I've read Apple's docs through several times but don't understand if the precision of world tracking is coupled with having plane detection on.
    • I want to swap out the debug plane for a custom one. I want to program the plane so that when the arrows hit it, they stick in place, like an actual arrow. I'd like for the plane to receive shadows, appear infinite to the player, and occlude any virtual objects underneath it, i.e. you wouldn't be able to see the tip of an arrow stuck in the ground a few centimeters. How would I go about that? This is kind of related to my first question as well — would I assume that the first anchor is the ground?

    Thank you in advance!
     
    Last edited: Aug 10, 2017
    jimmya and christophergoy like this.
  27. macdude2

    macdude2

    Joined:
    Sep 22, 2010
    Posts:
    685
    I know the machine learning API is outside the scope of AR kit, but it seems to me that the vision APIs don't rely on learning data? I think face and object tracking capabilities in Unity would be extremely useful in building out an AR app...
     
    jessevan likes this.
  28. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    567
    When the session starts, the origin of the world is (0,0,0)

    I don't know the answer to this question. My intuition tells me it shouldn't make a difference, but it's been wrong before :p. The reason you might be getting drift is because when your phone is pointed up, it's not able to detect anything. If you are checking the ARTrackingState, you might be able to switch to 3 degrees of freedom in order to reduce drift if you drop into the Limited tracking state.

    You could check out the debugPlane prefab that exists in the project already and make your own asset that is really large to act as an infinite plane. I believe Jimmy wrote some shaders to help with occlusion. The debugPlane has collision already on it so you should be ok there. You will need to write some custom code to have your arrow "stick" to the plane like it was stuck in the ground. You could probably do this by looking at the physics callbacks that you can receive in scripts. There are plenty examples if you google them like this one.

    Let me know if you have any other questions. Cool demo!

    Cheers,
    Chris
     
    lourd likes this.
  29. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    898
    is there a way to pause and later resume the ARKit or Video from ARKit? I notice that OnResume() and onPause() and such are not provided

    why? one example - to switch to front facing camera via webcamtexture. afterwards, being able to switch back to the arkit video stream.
     
  30. arturmandas

    arturmandas

    Joined:
    Sep 29, 2012
    Posts:
    217
    For remote control: can I use Android device (Windows Unity). If not, can I use an older pre-iOS 11 device like iPad 2 (of course this time on OSX Unity).

    Best!
     
  31. planetfactory

    planetfactory

    Joined:
    May 18, 2016
    Posts:
    28
    Hi,

    I think we found a bug. Last iOS 11 beta, xCode beta 5.
    App Crashes, in device, when we unload or destroy the scene containing the ARKit. xCode log error:

    libc++abi.dylib: terminating with uncaught exception of type Il2CppExceptionWrapper

    How to test:
    Have an ARKit scene (for example, the ARKit "EditorTestScene"), and add a UI button to load another scene (any scene, it can be empty). Run the app on device, the ARKit scene starts, press the button to load the new scene, and the app crashes.
    It also crashes if we manually destroy the gameobjects containing the ARKit stuff.

    See attached project.

    As anyone encountered the same problem? It did not happen with xCode beta 4.

    Thanks!
     

    Attached Files:

  32. planetfactory

    planetfactory

    Joined:
    May 18, 2016
    Posts:
    28
    It seems the problem occurs when de component "UnityARAmbient" is destroyed.
     
  33. Raisanen

    Raisanen

    Joined:
    Jan 18, 2017
    Posts:
    3
    Does anyone have a memory issues with Unity ARPlugin. It seems that the example project (UnityARKitScene) uses more and more memory and never releases it. I've debugged this in Xcode and Unity profiler and both show the same results.

    Any ideas how to fix this?
     
  34. prabhusam

    prabhusam

    Joined:
    Aug 3, 2017
    Posts:
    6
    what is visual inertial odometry ?
    why use visual inertial odometry ?
    how it works?
     
  35. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    567
    I believe this is a bug in the creation of textures for the video. I’ll see if I can get a fix in today.
     
  36. lernerbot

    lernerbot

    Joined:
    Jul 27, 2017
    Posts:
    1
    In the mean time the following will work:
    Code (CSharp):
    1. s_CurAffineTransform = [frame displayTransformForOrientation:[[UIApplication sharedApplication] statusBarOrientation] viewportSize:nativeSize];
    Also you may have to comment out the following:
    Code (CSharp):
    1. [NSException raise:@"UnrecognizedARTrackingStateReason" format:@"Unrecognized ARTrackingStateReason: %ld", (long)trackingReason];
     
  37. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    567
    If you have the latest bitbucket code, these issues should be fixed. The asset store package I waiting for review.
     
  38. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    567
    There is a blog post about debugging Il2cpp issues here. If you encounter this again please let us know what you find. If you want to submit a PR for the fix we are happy to take a look at it.
    Cheers,
    Chris
     
  39. Aiursrage2k

    Aiursrage2k

    Joined:
    Nov 1, 2009
    Posts:
    4,834
    Did anyone figure out how to make the objects always appear in the same spot. it seems odd that this is such so complex, if I wanted to use it do pathfinding, multiplayer or whatever, is it better to use location services?
     
  40. it_ra

    it_ra

    Joined:
    Feb 28, 2015
    Posts:
    5
    Prompt how to start the application without the need for connection to the editor. What wouldn't be at start of the message of "Waiting for editor connection"
     
  41. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    567
    For anyone that is having issues with "Failed to Connect to Player" with the ARKit Remote, please make sure you have "Development Build" checked in your Build Settings window.
     
    macdude2 likes this.
  42. kapuxino

    kapuxino

    Joined:
    Jul 14, 2017
    Posts:
    3
    Is this supposed to work with deferred rendering path?
     
  43. it_ra

    it_ra

    Joined:
    Feb 28, 2015
    Posts:
    5
    There are ideas? That it was possible to depart from the computer far
     
  44. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    567
    I'm not sure I understand the question but I'll take a guess. It sounds like you are running the ARKit Remote scene and maybe you want to run the UnityARKitScene? If so, when you go into your build settings window, make sure you uncheck the ARKit Remote scene, and check the UnityARKitScene.
    Cheers,
    Chris
     
  45. Elliott-Mitchell

    Elliott-Mitchell

    Joined:
    Oct 8, 2015
    Posts:
    58
    Good idea - I guess the only reason I added the list was to support older / other devices with ARKit disabled. The game has AR mode and non-AR mode. Does that make sense? I have yet to test out this on demo devices.
     
  46. HonorableDaniel

    HonorableDaniel

    Joined:
    Feb 28, 2007
    Posts:
    2,970
    I noticed that the Unity ARKit Plugin available on the Asset Store is submitted as a complete project. Why is this? It makes it a little bit more difficult to upgrade the plugin in my existing project. It tries to overwrite my projects settings every time I import it.
     
    phits likes this.
  47. HonorableDaniel

    HonorableDaniel

    Joined:
    Feb 28, 2007
    Posts:
    2,970
    Upgraded to Xcode Beta 5/Unity ARKit Plugin for Beta 5 and now I am getting a crash on device as well as in the editor right when I enable AR. Here is the Xcode log:

    2017-08-10 17:30:24.910858-0700 shotpro[558:114813] -[ARFrame displayTransformWithViewportSize:eek:rientation:]: unrecognized selector sent to instance 0x1c82fed80

    2017-08-10 17:30:24.927467-0700 shotpro[558:114813] Uncaught exception: NSInvalidArgumentException: -[ARFrame displayTransformWithViewportSize:eek:rientation:]: unrecognized selector sent to instance 0x1c82fed80

    (

    0 CoreFoundation 0x0000000183530d74 <redacted> + 148

    1 libobjc.A.dylib 0x0000000182a44528 objc_exception_throw + 56

    2 CoreFoundation 0x000000018353e218 <redacted> + 0

    3 CoreFoundation 0x0000000183536708 <redacted> + 1380

    4 CoreFoundation 0x000000018341c35c _CF_forwarding_prep_0 + 92

    5 shotpro 0x0000000103afa02c -[UnityARSession session:didUpdateFrame:] + 1168

    6 ARKit 0x000000019e073b54 <redacted> + 132

    7 libdispatch.dylib 0x0000000182eb546c <redacted> + 24

    8 libdispatch.dylib 0x0000000182eb542c <redacted> + 16

    9 libdispatch.dylib 0x0000000182ec1e70 <redacted> + 1016

    10 CoreFoundation 0x00000001834d9138 <redacted> + 12

    11 CoreFoundation 0x00000001834d6d14 <redacted> + 2012

    12 CoreFoundation 0x00000001833f7558 CFRunLoopRunSpecific + 436

    13 GraphicsServices 0x000000018527af84 GSEventRunModal + 100

    14 UIKit 0x000000018d0c2984 UIApplicationMain + 208

    15 shotpro 0x00000001028d50b4 main + 228

    16 libdyld.dylib 0x0000000182f19db4 <redacted> + 4

    )
     
  48. mikejohnstn

    mikejohnstn

    Joined:
    Dec 9, 2016
    Posts:
    10
    Navigation (and other systems) start breaking at very small scales. You'll find some discussion earlier in this thread. @jimmya mentioned they may have a way to scale the camera instead of the world so we can preserve 1m = 1 unit while making the world appear much smaller. This is the same trick lots of people use in VR.

    Anyone have any updates on scaling the camera with ARKit?
     
  49. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    567
    We are still waiting for the asset store package to be approved.
     
  50. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    567
    I’ve done some preliminary work, but I have been caught up in another project. Hopefully I can get back to it this weekend sometime.
     
    Robosaru, mikejohnstn and sama-van like this.
unityunity