ARKit 3

Is ARKit 3 fully supported by ARFoundation?

Or is the existing ARKit plugin getting updated soon?

Starting a new Slack for AR Dev for Unity. Feel free to join here:

https://join.slack.com/t/unityardev...Y2OWQzYzA4OTBlZGRhNWZiYTAxOThmYzgxYjJlNWFhMDM

 

ARKit 3 was announced yesterday, and will release in iOS 13, which hasn't been released yet.

 

iOS 13 beta has already been released to developers. And Unity team can work on the integration already. I wonder how long will it take to integrate features like people occlusion and motion capture (and possibly with support for Mecanim). And ETA from the team would be greatly appreciated.

 

I don't expect it to be a small task for the team to "just" implement these things and it's only 2 days old news. However, it would be really nice to be kept informed on when/if and to what degree these tools could be expected. Many ongoing projects could benefit from them and I would rather not optimize for issues that are automatically fixed very soon with the ARkit3

 

Oh! I just realised that the ARkit plugin has been deprecated and we should now use ARFoundation?? Isn't ARFoundation only catering to the features that the platforms have in common? Something like body tracking, body occlusion etc. are currently unique to iOS. Does that mean that Unity will down prioritize those features until ARcore catches on?

 

I hope not, that would be a huge bummer for Unity Devs around the world. My guess is they will implement ARKit3's new features in order to be a state-of-the-art dev platform for which Unity is known for.

 

We seriously hope that’s not the case too. Maybe they would have platform specific features on top of the cross platform functionality. Hoping the devs would clarify more on this matter.

 

No, AR Foundation also features ARKit-only functions (for instance object tracking).
But what worries me is that the ARKit plugin has always been updated very quickly. In 2017, ARKit was available in Unity within days (I suppose they worked on it before the beta release). But with AR Foundation I've seen that things takes much longer time. Only recently image tracking was added, while this has been around for ages in ARKit and ARCore.
edit: Unity support for ARKit3 https://blogs.unity3d.com/2019/06/06/ar-foundation-support-for-arkit-3/

 

We were wondering too if we could get our assets out of Unity into RealityKit through USDZ for example. RealityKit has many cool features that the current Unity struggles with (using image effects like motion blur in AR without affecting up the tracking and severely impacting the performance, accurate camera based depth of field, raytraced soft shadows and many more)

 

There is a USDZ preview package for Unity that allows you to export assets to USDZ I think. Check the package manager
And yes, I'm also going to look into RealityKit, it looks very promising. Motion tracking, depth of field blur, HDR environment maps, camera grain, good looking realtime soft shadows on mobile, recording an AR session for in-editor testing, continues mapping&relocalization across multiple devices, easy multiplayer.

Mostly also possible in Unity, but with RealityKit it's by default it seems

 

PS: Unity ARKit3 announcement
https://blogs.unity3d.com/2019/06/06/ar-foundation-support-for-arkit-3/

 

That was fast! Exciting!
I can’t see a mention of improvements to the post processing workflow in AR to match RealityKit level. I hope the devs address that soon.

 

Wow. "people occlusion features are available only on iOS devices with the A12 Bionic chip and ANE." Anyone know which specific devices these are?

Also, just to confirm

https://github.com/Unity-Technologies/arfoundation-samples

This repo contains latest ARkit 3 features? Not perfectly clear either from the blog post or reading the manifest.json file.

Also am wondering about LWRP + ARkit3 support.

 

@eco_bach A12 chip is available in iPhone Xs

wow, that was really fast, so excited!!

Has anybody tried the samples already?

 

• iPhone XS and XS Max
• iPhone XR
• iPad Mini (2019)
• iPad Air (2019)

edit: whoops also all A12X devices of course:

• 11-inch iPad Pro
• Third-generation 12.9-inch iPad Pro

from https://en.wikipedia.org/wiki/Apple_A12 & https://en.wikipedia.org/wiki/Apple_A12X

 

Thank you for the update!! Looks like I'm getting the following error when trying to deploy the ARFoundation 1.5 LWRP sample to iPhone X running iOS 12.1.2:
dyld: Symbol not found: _OBJC_CLASS_$_ARMatteGenerator

 

Cool. I wonder when ARCore 1.9 will be supported....

 

The device also needs to run iOS13

 

Watching the ARkit3 deep dive am wondering how people occlusion is handled under the hood in ARFoundation.

-Scenekit + ARSCNView
or
- Custom compositon thru ARMattegenerator

 

Does anybody know what kind of impact "Human Segmentation Stencil Mode" and "Human Segmentation Depth Mode" has?

 

Yep that was it, device needed to be on iOS 13. Thank you!

 

Apple mentioned in their WWDC talk they now can also detect white walls (feature-less surfaces) using some AI. Is this also available in ARFoundation? I can't find any documentation on that.

from 41:00: https://developer.apple.com/videos/play/wwdc2019/604/

They call it ML-enhanced plane estimation

 

I am playing around with with the simultaneous world and face tracking feature (RearCameraWithFrontCameraFaceMesh sample). It is implemented as a toggle, toggling between viewing the back- or front-facing camera's feed, while the tracking data from both managers are accessible.

However, what happens exactly to the position and rotation of the face when we are using the front-facing mode?

The face position seems to be relative to the device's starting position (0,0,0) as all other trackables, but what about the face rotation? It seem to be broken, where just certain face rotations seem to be tracked (e.g. only y-axis, while the other two do not change). To what is this rotation relative to?

 

It's described in the Unity Blog post (https://blogs.unity3d.com/2019/06/06/ar-foundation-support-for-arkit-3/). Basically, stencil mode tells you if a pixel contains a human or not, depth mode also estimates the distance of that human pixel from the device. I guess stencil mode is much lighter in terms of performance impact, as it only requires one ML network to run, while depth estimation requires (at least) an additional one. So depending on your use-case, stencil mode might be useful to optimise performance if you only need segmentation, not distance-based occlusion.

 

I'm trying the new ARKit 3. I'm using Unity 2019 with ARFoundation 2.2. I'm working on the new motion capture function. It works fine with 2D body joint data. But there is nothing when using 3D body joint data. Anyone has successfully retrieved the 3D body joint data?

And there is no ARKit Remote tool. Any idea?

Thanks!

 

It works with iOS 13 beta v1 and XCode 11 beta 1! It will not work with beta v2. Unity is working on a fix for that.

 

Big issue is that the iOS 13 public beta is build 2. Hope they fix it soon!

 

Would be neat to just ask questions in realtime, starting a Unity AR Devs slack. Please join here: https://join.slack.com/t/unityardev...Y2OWQzYzA4OTBlZGRhNWZiYTAxOThmYzgxYjJlNWFhMDM

 

@tdmowrer I'm also seeing that ARFoundation 2.2 only runs on devices with the iOS 13 beta. I see this as a bug; shouldn't ARFoundation also be able to support iOS 11 and iOS 12 with reduced ARKit features? I believe there's a bunch of logic built in to the framework to check which subsytems are supported on the device, but this isn't really useful if we can't run the app on older iOS versions anyway.

Are you guys looking at this? Thanks!

 

@kennyd-itp you can use ARFoundation 2.1, I think that contains ARKit 2

 

@InakiAndres Thanks, but I want to make an app with ARKit 3 features that can fall back to ARKit 2 or 1.5 if necessary (it's a demo app that shows the various AR features we have available in ARKit and ARCore).

 

The ARkit 3 features only works in iOS devices. But for example, in human recognition you need an A12 chip and ios 13 (for the occlusion too). I think it's so early to work with this but we are crazy Best luck!

 

Thanks for the information.

 

using ipados 13 beta 3,still the 3D body tracking scene is not working

 

I'm also testing on iOS 13 iPhone XsMax and body tracking is not working.

 

HumanSegmentation 3D has been broken for over a month.

N.B. Isn't it against Apple policy to stick to an old beta iOS? There have now been several iOS 13 beta version releases including two public beta's that have been released... and Human Segmentation 3D is still borked.

 

https://github.com/Unity-Technologies/arfoundation-samples/issues/170