Resolved ARKit support for iOS via Unity-ARKit-Plugin

ARKit only has support for face tracking on iPhone X - it needs a depth camera. https://developer.apple.com/documentation/arkit/creating_face_based_ar_experiences

 

Please read my blog post and ARKit docs - you seem to misunderstand how it works. There is a mesh provided per frame, but that is the mesh of how your face looks like in this frame. It also provides coefficients to say what blendshapes you're using with your current expression. You can use those coefficients to alter a virtual head with a face rig to have the same expression as you. I'm working on providing such a head for an example to use.

 

Thank you jimmya, Yesterday afternoon I have updated the ARKit plugin to the latest version, everything is ok now.

 

i built a face 3d scanner for the iphonex to grab the mesh. will pack it up for release soon - i am surprised one does not exist yet.

Feel free to use this https://gist.github.com/yosun/03dc47a5f8cba094960192cce1dac872

Mesh topology remains the same. Using this mesh, I can then use the Wireframe shaders from the asset store. etc

 

n.b. iphonex face arkit update - i notice that the blendshape key names for the face locations does not match the apple doc documentation?

i.e., eyeBlinkLeft is actually eyeBlink_L

etc

 

Hey thanks a lot! I will def. look into to it!
On the hackathon I had some fun making a snapchat-like app with the standard facemesh and some different textures I'll post a video once I get my hands on an iPhoneX again

 

yup, only X as it uses the depth camera.
(Though I wonder if there isn't a snapchat-like library that does the same without the depth cam?)

 

thanks, i just want to play a face tracking app..
jobs gone, anything gone, sigh..

 

This is probably due to your usage of iVidCapPro - currently the only performant solution to recording video for unity ios apps

The .mm files were fixed for this in earlier builds but for some reason they were not merged for the face update
@jimmya could you please add that?

https://bitbucket.org/Unity-Technol...support-opengl-es-2-as-the-rendering-api/diff

 

Okay, It works fine on iPhone 8 Plus, so probably processing / GPU issue with iPad 2017

 

@jimmya or @christophergoy

Built out the UnityARKitRemote on a mac to my iPhone 6s, and I can connect easily via USB to the Unity Profiler (using the EditorTestScene)
With the remote on the phone, attempts to connect to the profiler via USB on Windows 10 are all met with "failed to connect to player" errors (but no other errors)
Tried with Unity versions 5.6.4f1, 2017.1.1f1, and 2017.2.0f3

Can anyone help me out? Or has anyone got their iPhone hooked up via USB to a Windows PC working properly?
(based on what I've read connecting via WiFi is basically unusable)

I've read through every similar error in this thread and online, and no solutions have worked for me.

 

Has anyone been able to get iVidCapPro working with the latest (post face) ARKit release 2017.2.0f3 (or let me know if I should ve on a different Unity version)

 

Can anyone recommend any tutorials that cover how to replicate the functionality in this video?
Ideally in Unity?
i.e.:

1. activate library
2. choose object to place from library
3. place object (see orientation in video)
4. move / re-orient placed object

 

Yeah this is what is passed up from the SDK - go figure.

 

You mean you're writing out the values that FaceMeshManager gets for the face geometry? Alright, but this mesh geometry does not have any blendshapes associated with it, so you will not be able to deform it using your expression.

 

So far we have been eyeballing the blendshapes from making different faces that get as close to the extremum blendshape floats for each key. It is kind of gruntwork, kind of like using glextract to rip 3D assets from a game

It would be nice if there is a sample 3D model with blendshapes included - even the model that was used to create the documentation pics for arkit face sdk blendlocations

 

i tried pushing (one of) the first unity arkit face app to the app store last week. still waiting for approval. Currently, the functionality basically only works on iPhone X. however, iTunesConnect seems to require functionality for non-X versions even if you are creating primarily an app that uses the iPhone X-only feature.

i have been trying to get OpenCVForUnity working with the ARKit face stuff to be able to support older generations of iPhones. it's essentially possible to do "ok" face puppeteering using dlib/opencv - however wrangling different plugin libraries to work with Unity is nontrivial

advice appreciated!

 

Memory (MTLTexture) leaks anyone?

Hi. Before starting to experiment with ARKit I decided to have a quick look through both native plugin and c# code. I've noticed that AR frames are passed to engine as native texture (id<MTLTexture>) pointers. These are then used with Texture2D.CreateExternalTexture() & Texture2D.UpdateExternalTexture() to keep the material up to date. Clean and efficient.

However, having hit a roadblock with external Metal textures in similar native plugin usage case on MacOS, I'm a bit puzzled and worried about memory leaks.

Can anyone please explain what happens with the previously assigned external MTLTexture texture objects when UnityARVideo.cs does this:

_videoTextureY.UpdateExternalTexture(handles.textureY);
_videoTextureCbCr.UpdateExternalTexture(handles.textureCbCr);

Please note that there is no guaranty that the plugin (CVMetalTextureCacheCreateTextureFromImage()) will re-cycle a small set of metal textures, i.e. chances are you can get a fresh pair of MTLTexture objects on every frame.

Some time ago I discovered that on MacOS Texture2D.UpdateExternalTexture() will add 2 or 3 strong references to MTLTexture texture object which seems to be never released afterwards. So that if the native plugin supplied a new MTLTexture object on every frame the old ones will stuck in memory and get never released.

I.e. something like this would give a funny result:

<code>
IntPtr nativeTexture = NativePlugin_GetNativeTexture();
int retainCountBefore = NativePlugin_GetRetainCount(nativeTexture); // e.g. == 3
myTexture.UpdateExternalTexture(nativeTexture);
int retainCountAfter1 = NativePlugin_GetRetainCount(nativeTexture); // == 5
myTexture.UpdateExternalTexture(someOtherNativeTexture);
int retainCountAfter2 = NativePlugin_GetRetainCount(nativeTexture); // still == 5
</code>

Basically what I'm trying to say here, I believe Texture2D.UpdateExternalTexture() creates a few strong references to MTLTexture and never releases them. So unless the native plugin will cycle through a small set of pre-allocated MTLTexture objects the app will be leaking memory due to unreleased MTLTexture objects left by Texture2D.UpdateExternalTexture().

Can someone please confirm what happens to previously assigned native MTLTexture objects after UnityARVideo.cs assigns new ones via Texture2D.UpdateExternalTexture()? Isn't Texture2D supposed to remove any strong references to old texture so ARC can dispose it? Has anyone tried to monitor memory usage during AR video stream for long enough?

Please note that leaking <id>MTLTexture is not the same as leaking the actual texture buffer as these are actually re-cycled. I'm talking about leaking the the Obj-C objects.

Thanks

 

I was one of the people responsible for this baby:

Just to show the possibilities.

Disclaimer: not me in the video

 

Was this using Unity and this plugin?

 

hmm I was wondering the same. Also t seems impossible to direct apps only to iPhone X. While it was possible to use device required features on plist to address arkit-capable devices, this is not available for the front face camera on the X. Pretty strange choice by apple.

 

Yes, Unity and ARKit plugin

 

Anybody facing this issue? While running Unity ARkit Remote, console says it's not connected but the editor game shows it's connected. When I open the remote app on phone, the console shows me two devices. and none of them works.

 

I've done this, and hit test is passing through our UI and moving objects in our scene when player is merely trying to place "OK" button after setup.

Shouldn't the UI stop from "Hit Test" registering in our room?

 

Hi, I want to move my character only in the seletected plane? do you have any sample for that? im using joystick then I just want to make the character move only from the plane..

 

I have UnityARCameraNearFar attached to my (only) camera in the scene, but the Far clipping plane is not doing anything. Changing Near works as expected. Anyone else having this issue?

Edit: very strange, Far works as well, but only up to a certain distance it seems? I have a big inverted sphere in my scene that shows up on the editor game view when not in play, and also in other cameras if I add them, but not in ARcamera at runtime, no matter how big the Far Clipping Plane value...

 

I have the same issue, how do you fixed it?

 

Ok, I found the problem: Clipping Plane Far only work between 0 and 30. Changing the value beyond 30 keeps it at 30..

 

Check the rendering order (any render layer set?) - remember that the video clears the screen at a certain point in the rendered frame.

 

All these errors usually mean you don't have latest of one of these: the plugin code from bitbucket or asset store, XCode 9 version on your dev machine, or iOS 11 version on your device.

 

Hi guys! I have a problem with ARKit camera on iPhone X https://gyazo.com/930def9cf9f91dc06f7c39f897db51f5
(incorrectly rendering from below and from above ) XCode 9.1 Any idea?

 

update your plugin code (this was fixed more than a month ago)

 

You may have some ghost connections - try closing your editor, close your remote app, and then restart editor and app, and try again.

 

Hey @jimmya, can you expand on this, like others I am trying to grab a texture from the camera feed which I can access from `UnityARVideoScript().m_ClearMaterial.GetTexture ("_textureCbCr")` and `UnityARVideoScript().m_ClearMaterial.GetTexture ("_textureY")`. However, I was unable to combine them to a Texture2D.

 

Hi @jimmya, I found a problem about unity2017.2.0f3, when all the packages are installed and release to the IOS platform will be an error.but when only the following packages are installed is normal.Is there a conflict between the packages? I try to replace the installation order but still not work, it looks like a bug?
I'm using the Windows version of Unity
Another problem,I installed different versions of unity，also release to ios plateform，but will be an error for the first time.The error is "Missing assembly....".I close unity and then deleting the file "C:\ProgramData\Unity\ index-local.xml" and restarting unity and everthing is ok.

My English is poor, I wonder if you can understand my meaning?

 

I have a very weird problem.. I deploy one of the sample scene into my iPhone 6S, at first works perfectly but if I try to open this app after that the app crash.. just start and crash.. I try with differents scenes and always happend the same.

 

Make sure you have updated your phone with latest ios11, and you are building with latest version of XCode.

 

Yes, I have installed the last version... the app crash, doesn't show even the Unity logo..

 

I've got two questions:
1. Any posibility to adjust the brightness or stop the auto-exposure of the camera? The light estimation seems to adjust the virtual light to the real world exposure, but the camera image is too bright and noisy, especcially in darker rooms.

2. Is it possible to exclude the video background from specifiy effects like anti aliasing and image effects (especially motion blur). I've noticed the background is a lot more blurry than the standard camera app, so I would like to increase the quality and performance.

 

Hi Everyone, So I just tried publishing my first Unity ARKit app and it got rejected by apple due to the TrueDepth APIs which I am not using. Is there anyway to easily remove those from the plugin so that it passes? Or is there another workaround anyone is using? Here is my responce from iTunes Connect just incase this might also be happeninh to other people.

"Your app includes TrueDepth APIs, but we could not find the functionality related to this. Please provide information on how we can access and review the TrueDepth functionality in your app."

 

https://forum.unity.com/threads/submitting-arkit-apps-to-appstore-without-face-tracking.504572/

 

You must have changed something? Which version of Unity are you using? And have you looked at the FAQ to make sure you are following all the steps?

Also, have you given your app camera permissions?

 

1) Camera exposure control is not possible right now and I doubt Apple will add it in the future. Automatic exposure helps with camera tracking. You could try lowering the strength of any Unity lights you have in the scene. There also should be a way to control the strength of reflected light estimation under material properties.

2) Are you using a device with a dual camera (7+ 8+ or X)? I've noticed the cameras on these a fixed at a near focus distance. I fear this has something to do with how the devices are factory calibrated, and variable focus may never be possible.

 

I have a similar issue. In earlier versions of my project UnityARKitRemote was working fine. I rebuilt the project with the UnityARKitRemote scene using the latest version of my project, but when I connect to it in the editor I get:

Code (CSharp):

1. SerializationException: Unable to find assembly 'Assembly-CSharp-firstpass, Version=0.0.0.0, Culture=, PublicKeyToken=null'.
2. System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo.GetAssembly () (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binarycommonclasses.cs:408)
3. System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetType (System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo assemblyInfo, System.String name) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryobjectreader.cs:1489)
4. System.Runtime.Serialization.Formatters.Binary.ObjectMap..ctor (System.String objectName, System.String[] memberNames, System.Runtime.Serialization.Formatters.Binary.BinaryTypeEnum[] binaryTypeEnumA, System.Object[] typeInformationA, System.Int32[] memberAssemIds, System.Runtime.Serialization.Formatters.Binary.ObjectReader objectReader, System.Int32 objectId, System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo assemblyInfo, System.Runtime.Serialization.Formatters.Binary.SizedArray assemIdToAssemblyTable) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binarycommonclasses.cs:2153)
5. System.Runtime.Serialization.Formatters.Binary.ObjectMap.Create (System.String name, System.String[] memberNames, System.Runtime.Serialization.Formatters.Binary.BinaryTypeEnum[] binaryTypeEnumA, System.Object[] typeInformationA, System.Int32[] memberAssemIds, System.Runtime.Serialization.Formatters.Binary.ObjectReader objectReader, System.Int32 objectId, System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo assemblyInfo, System.Runtime.Serialization.Formatters.Binary.SizedArray assemIdToAssemblyTable) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binarycommonclasses.cs:2204)
6. System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped (System.Runtime.Serialization.Formatters.Binary.BinaryObjectWithMapTyped record) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryparser.cs:683)
7. System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped (System.Runtime.Serialization.Formatters.Binary.BinaryHeaderEnum binaryHeaderEnum) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryparser.cs:656)
8. System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run () (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryparser.cs:146)
9. System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize (System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Runtime.Serialization.Formatters.Binary.__BinaryParser serParser, System.Boolean fCheck, System.Boolean isCrossAppDomain, System.Runtime.Remoting.Messaging.IMethodCallMessage methodCallMessage) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryobjectreader.cs:174)
10. System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Boolean fCheck, System.Boolean isCrossAppDomain, System.Runtime.Remoting.Messaging.IMethodCallMessage methodCallMessage) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryformatter.cs:197)
11. System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Boolean fCheck, System.Runtime.Remoting.Messaging.IMethodCallMessage methodCallMessage) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryformatter.cs:173)
12. System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Boolean fCheck) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryformatter.cs:118)
13. System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryformatter.cs:149)
14. System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryformatter.cs:111)
15. UnityEngine.XR.iOS.Utils.ObjectSerializationExtension.Deserialize[T] (System.Byte[] byteArray) (at Assets/Plugins/3rdParty/UnityARKitPlugin/ARKitRemote/ObjectSerializationExtension.cs:39)
16. UnityEngine.XR.iOS.ARKitRemoteConnection.UpdateCameraFrame (UnityEngine.Networking.PlayerConnection.MessageEventArgs mea) (at Assets/Plugins/3rdParty/UnityARKitPlugin/ARKitRemote/ARKitRemoteConnection.cs:121)
17. UnityEngine.Events.InvokableCall`1[T1].Invoke (T1 args0) (at /Users/builduser/buildslave/unity/build/Runtime/Export/UnityEvent.cs:206)
18. UnityEngine.Events.UnityEvent`1[T0].Invoke (T0 arg0) (at /Users/builduser/buildslave/unity/build/Runtime/Export/UnityEvent_1.cs:58)
19. UnityEngine.Networking.PlayerConnection.PlayerEditorConnectionEvents.InvokeMessageIdSubscribers (System.Guid messageId, System.Byte[] data, System.Int32 playerId) (at /Users/builduser/buildslave/unity/build/Runtime/Export/Networking/PlayerConnection/PlayerEditorConnectionEvents.cs:66)
20. UnityEditor.Networking.PlayerConnection.EditorConnection.MessageCallbackInternal (System.IntPtr data, System.UInt64 size, System.UInt64 guid, System.String messageId) (at /Users/builduser/buildslave/unity/build/Editor/Mono/Networking/PlayerConnection/EditorConnection.cs:143)
21.  

 

may build was crashing after 20 to 30 seconds? any reason? sorry I cant show logs.. im using windows and only building in unity cloud? is there any common cuase of crashed? also im just using sample AR scene.. then just add 3d object.. the crash happens when I transfer scene and back to ar scene..may there is need to stop ar or something?

 

is this possible
1.if 1 plane show.. I can pause or stop from generating another plane.. ?
2. If I already tap screen and show the object.. the plane disappear ?also that will link to #1

im my case after tap screen and plane disappear and show the 3d object..

 

Hi! I know I'm fairly late to the thread but I have recently just attempted to implement the Unity AR Kit Plugin and I tried to set up an example with a custom 3d model that I imported. However, I have not been able to get it to appear on the blue plane prefab when I tap on the screen. The default UnityARKitScene with the HitCube works perfectly fine but when I replace the HitCube with a custom 3d obj it does not appear.

I just replaced the HitCube with my custom model and made it a child to the HitCubeParent object. Then I attached the AR Hit Test Example script onto the custom object.

Any suggestions on why it isn't working?

 

Ran into the same thing, following the steps now to use the non-face tracking version now. Crossing fingers, hope I get approved!

Nav

 

any? my app crash when chaging scene.. but when in 1st come in ARscene it was working fine..
it crash when
Menu scene ->AR scene -> menu scene -> AR scene..then crash in seconds..

my menu scene was just a simple scene