Resolved ARKit support for iOS via Unity-ARKit-Plugin

Derp, thx Jimmy.

 

Is there a way to map the screen video texture on to a mesh's UVs? Kind of like the "Snap" method in World Lens Studio?

https://lensstudio.snapchat.com/api/classes/PostEffectVisual/

Basically what I want to do is map the video texture on to a mesh so I can do stuff like a plane that breaks away to reveal a portal. But since the video frame is mapped on to the mesh, it looks like the actual table is crumbling. I've seen this effect on some ARKit videos, but not sure how they are doing the projection.

 

Can anyone confirm UnityARAlignmentGravityAndHeading is working? My world is still oriented to the initial camera orientation.

 

Hi,

We start detecting planes like this
UnityARAnchorManager unityARAnchorManager = new UnityARAnchorManager();
UnityARUtility.InitializePlanePrefab (planePrefab);

Now I need to stop generating planes once I have set up my game play with first detected plane.
I don't see any function like this
unityARAnchorManager.StopGeneratingPlanes() //I was expecting this kind of function?

Thanks
Waiting for reply
mobexperts

 

Just got my first Unity AR project approved before the Holidays: https://itunes.apple.com/no/app/jular/id1326501371?mt=8

 

When running the remote app on my device it just says waiting for editor connection. The editor will display the camera from my phone as normal, but I am unable to interact with the scene from the phone because the phone doesn't think its connected. Has anyone seen this or know what to do?

 

Did you get the latest commit on the repo? https://bitbucket.org/Unity-Technol...mits/55f14b4a678ebb34e043ea32ed043f39803b40b0

 

Thanks @jimmya
No I was using the version from the asset store. I updated to this one and now my phone will display the camera as well but I am getting weird behavior in the editor now. It seems to take a still photo when I first start running the remote and uses that as a background to display everything else on top of. I'm attaching a video for clarity.

Another question: Are touches on the iPhone remote supposed to register? It seems to me that touch count is always 0 no matter what I do while using the remote.

 

Hi there, just trying to build an ArKit application for my project. I have a canvas with screen space overlay for main UI buttons. And also have several World Space UI buttons located on the model those opens an image gallery OnClick. Those world space UI buttons falls back when I start the application with an iOS device. They are acting like non-static objects and are falling back after clicking on model (hit test). Any suggestion would be appreciated.

Edit: Just noticed that I have attached the World Space canvases inside the HitCubeParent object. Just fixed the issue by removing those World Space canvases outside of the HitCubeParent.

 

hi guys the AR interface repo is kinda useful for Scaling and other examples, will this repo continue to see updates or should we use the ARkit repo only?

 

This seems to only allow for broadcasts? What about local recording or saving to a local photo gallery file?

 

Tried adding these to Start in ARCameraTracker for the face example... However the screen out of frustum still happens in xcode debug messages


UnityARCamera scamera = new UnityARCamera ();
scamera.worldTransform = new UnityARMatrix4x4 (new Vector4 (1, 0, 0, 0), new Vector4 (0, 1, 0, 0), new Vector4 (0, 0, 1, 0), new Vector4 (0, 0, 0, 1));
Matrix4x4 projMat = Matrix4x4.Perspective (60.0f, 1.33f, 0.1f, 30.0f);
scamera.projectionMatrix = new UnityARMatrix4x4 (projMat.GetColumn(0),projMat.GetColumn(1),projMat.GetColumn(2),projMat.GetColumn(3));

UnityARSessionNativeInterface.SetStaticCamera (scamera);

 

2017.3.0f3 ARKit shader bugs...

WARNING: Shader Unsupported: 'Hidden/BlitToDepth' - Pass '' has no vertex shader

WARNING: Shader Unsupported: 'Hidden/BlitToDepth' - Setting to default shader.

WARNING: Shader Unsupported: 'Hidden/BlitToDepth_MSAA' - Pass '' has no vertex shader

WARNING: Shader Unsupported: 'Hidden/BlitToDepth_MSAA' - Setting to default shader.

 

Camera position tracking with face tracking:

Hi, when face tracking is enabled, it seems to me that only the camera rotation is tracked and not the camera position. I.e., if I add a stationary object in the 3D scene of the face tracking example and move the camera in the room, this stationary object does not move relative to the camera, as it would in a AR scene without face tracking. Is this normal behavior? Is it possible to enable positional tracking for the camera?

 

try parenting it to the head anchor

 

Does ARKitRemote work with/provide face tracking data?

Also, does it work with 2017.3.0f3
I keep getting - Failed to connect to player

 

Hello I have a question that could not find any answer from forums.
I want to place a plane over the generated plane with same size. I tried move my plane inside generated plane or give to plane generatedplane's sizes but evry time my plane look like white and materials gone. How can i put plane with the same size with generated plane . Thanks for your helps

 

It should work for what you require. See StartRecording and StopRecording.

 

I would expect ARInterface to see updates shortly.

 

I tried parenting the camera to the head and also parenting the head to the camera, but both do not work.

To be a bit more specific, the head does indeed move relative to the world, but the camera does not and simply stays at the origin at all times. I confirmed this with a Debug log. Simply parenting the camera to the head makes the camera rotate with the head, and thereby the camera is always pointing towards the head, which is interesting, but does not enable camera movement relative to the world - i.e., walking around in a room does not change the head position relative to the 3D world. For this, true tracking on both the camera position and the head position relative to the camera would be necessary and this is what I am aiming for.

 

I'm using the AR Remote Editor from the latest bitbucket coding as well and am also having this issue on my iPhone 6S. With the Remote Editor, it freezes the first frame and becomes very laggy in the editor. The remaining playback becomes glitchy as shown here in this screenshot of my editor window. Any ideas how to fix this?


Also - how can I get UI elements to appear on my phone using the remote editor? Is this possible?

 

Is there a way to implement two touch input for scaling and rotating the 3D objects?

Ex: using your two fingers to do the traditional method of scaling/zooming on an iPhone. Adding scale and rotation sliders to my UI is not ideal for my particular application. I've looked everywhere and can't find anything that works. I'm not the strongest C# coder. Thanks!

 

Hi, can I use a windows machine to create the unity scene with ARKit plugin and build for iOS, then share and open the build folder on a mac with xcode to run it on ios device?

 

It might be possible, but I don't encourage it - we have not tested this kind of setup.

 

You could look into TouchScript or other Asset Store solutions that will give you cross-platform multitouch gestures.

 

Are you folks using USB connection? I've heard WiFi connection is slower.

 

Thanks for the reply Jimmya. I'm using USB to connect my iPhone to computer.

 

@jimmya any reply for this? I need to stop and generate plans again.

Anyone suggest me plz

 

When I use ReplayKit, i don't see an option to save the recording locally - it prompts Periscope or Youtube... Is there some sort of way to have ReplayKit record locally?

Prior to the last few Unity ARKit's, there was a community supported performant iOS (also locally saves-to-photo-album) video capture solution called iVidCapPro - however it breaks with Metal. On your bitbucket, I notice there is a branch that tries to support OpenGL ES 2, but it never seems to get merged in (and as a result breaks with every update)?

 

Have not been able to get ARKit Remote to work with face tracking... I assume that is not supported yet?

 

For face tracking portrait, xcode spews out these benign errors in 2017.3.0f3 - what's going on? This happens even with a blank project building any of the face tracking samples.

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 0.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

Screen position out of view frustum (screen pos 1125.000000, 0.000000, 30.000000) (Camera rect 0 0 1125 2436)

 

https://docs.unity3d.com/ScriptReference/Apple.ReplayKit.ReplayKit.html

 

I tried to upgrade ARKit plugin version as 1.0.12. I deleted "UnityARKitPlugin" folder and imported plugin from asset store. But when I open project again, there is an Unity editor layout issue and I can't open project again. I found new version including project settings. I'm not sure about it but is it necessary to add project settings in the unity package? Issue window is saying there was a custom layout window or error in project.

[EDIT: I didn't import the project settings. It seems issue is fixed]

 

StartBroadcast() and StopBroadcast() seem to be the only stable supported methods and these only allow for posting to Periscope/Youtube.

StartRecording() and StopRecording() are not reliable because ReplayKit.isRecording does not always return the right value - so there isn't a certain way to check for if a recording is going on. https://forums.developer.apple.com/thread/91475
Just tested StartRecording/StopRecording from a series of 1-20 second clips and reliability fail is less than 50% from the script posted in the link you sent

 

I believe I have fixed this: https://bitbucket.org/Unity-Technol...mits/6563a4df856d5cb91babb13167e5a62c99eb90d6

 

Should be fixed with latest.

 

This keeps happening even after initialization though - have you checked xcode debug log

 

Any guidelines for switching from front face tracking camera to rear facing world tracking (and back) in the same scene?

 

You can do it but you lose your session (plane anchors, etc) when you switch from one to the other.

 

I looked at the debug log and saw it before the fix - and it goes away after the fix - have you tried the fix?

 

is there a way to save the plane anchor somehow from one session to the other?

 

It seems FaceRemoved/ARFaceAnchorRemovedEvent doesn't properly trigger? When my face is visibly not being tracked, that should return right?

ARFaceAnchor.id isn't passed in FaceAdded either.

Is there a way to detect if the user is "too close" (before face is tracked) or "out of bounds"

 

I notice that when you resume the app from suspend, the ARKit starts to drop frames like crazy.

Is the best practice to just restart ARKit after suspend?

I know this would lose the anchor points but right now performance drops with each successive resume until I restart the AR session.

 

Basically, once app is suspended, the tracking will be off anyway (since it doesnt have access to sensors while suspended), so you should reset.

 

Just use basic touch input code as you would for any mobile app, then hook up the slider scale values to that.

For scaling you get the two touch points and measure when their distance has changed, there's a good generic example here: https://unity3d.com/learn/tutorials/topics/mobile-touch/pinch-zoom

 

Any updates on this?

 

Thanks for your reply! I will do that. One other problem I have here is that I can't seem to get the point cloud data to come back after resuming it the first time.

When I resume the session, I start it with the pointcloud enabled. I've tried resetting up the events but maybe I'm missing something.

 

Crash on scene change in Facetracking example:

I can't find a way to switch scenes which use the ARKitFaceTrackingConfiguration. I just downloaded the newest bitbucket repo and added a few lines of code to switch the scene after a few seconds. Everything works fine in the editor, but the device crashes with the following message:

libc++abi.dylib: terminating with uncaught exception of type Il2CppExceptionWrapper

and the following debug trace:

Thread 1 Queue : com.apple.main-thread (serial)
#0 0x000000018212c2e8 in __pthread_kill ()
#1 0x0000000182245748 in pthread_kill$VARIANT$armv81 ()
#2 0x000000018209afbc in abort ()
#3 0x0000000181838068 in abort_message ()
#4 0x0000000181838210 in default_terminate_handler() ()
#5 0x0000000181860820 in _objc_terminate() ()
#6 0x000000018185054c in std::__terminate(void (*)()) ()
#7 0x000000018184fea8 in __cxa_throw ()
#8 0x0000000103d8a974 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > std::__1:perator+<char, std::__1::char_traits<char>, std::__1::allocator<char> >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char const*) [inlined] ()
#9 0x00000001037226a8 in ::RaiseNullExceptionObject() at /Users/builduser/buildslave/unity/build/Runtime/Scripting/Scripting_Il2Cpp.cpp:208
#10 0x000000010398c720 in ::GetReference() at /Users/builduser/buildslave/unity/build/Runtime/Scripting/ScriptingObjectOfType.h:48
#11 0x000000010398c6a4 in ::GameObject_Get_Custom_PropTransform() at /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/GameObjectBindings.gen.cpp:193
#12 0x0000000102e5ba3c in ::UnityARFaceAnchorManager_FaceUpdated_m1801072947(UnityARFaceAnchorManager_t2288420337 *, ARFaceAnchor_t4162019119 *, const RuntimeMethod *) at /Users/xxx/Downloads/Unity-Technologies-unity-arkit-plugin-2bcc7d110304/test/Classes/Native/Bulk_Assembly-CSharp_0.cpp:25278
#13 0x0000000102e66bc0 in ::ARFaceAnchorUpdated_Invoke_m4213012087(ARFaceAnchorUpdated_t3345519745 *, ARFaceAnchor_t4162019119 *, const RuntimeMethod *) at /Users/xxx/Downloads/Unity-Technologies-unity-arkit-plugin-2bcc7d110304/test/Classes/Native/Bulk_Assembly-CSharp_0.cpp:35604
#14 0x0000000102e66bc0 in ::ARFaceAnchorUpdated_Invoke_m4213012087(ARFaceAnchorUpdated_t3345519745 *, ARFaceAnchor_t4162019119 *, const RuntimeMethod *) at /Users/xxx/Downloads/Unity-Technologies-unity-arkit-plugin-2bcc7d110304/test/Classes/Native/Bulk_Assembly-CSharp_0.cpp:35604
#15 0x0000000102e63aac in ::ReversePInvokeWrapper_UnityARSessionNativeInterface__face_anchor_updated_m559944031(UnityARFaceAnchorData_t2991923452) at /Users/xxx/Downloads/Unity-Technologies-unity-arkit-plugin-2bcc7d110304/test/Classes/Native/Bulk_Assembly-CSharp_0.cpp:31792
#16 0x00000001031d878c in ::-[UnityARFaceAnchorCallbackWrapper sendAnchorUpdatedEvent:](ARAnchor *) at /Users/xxx/Downloads/Unity-Technologies-unity-arkit-plugin-2bcc7d110304/test/Libraries/UnityARKitPlugin/Plugins/iOS/UnityARKit/NativeInterface/ARSessionNative.mm:531
#17 0x00000001031d9f4c in ::-[UnityARSession sendAnchorUpdatedEventToUnity:](NSArray *) at /Users/xxx/Downloads/Unity-Technologies-unity-arkit-plugin-2bcc7d110304/test/Libraries/UnityARKitPlugin/Plugins/iOS/UnityARKit/NativeInterface/ARSessionNative.mm:813
#18 0x00000001031d99e4 in ::-[UnityARSession session:didUpdateAnchors:](ARSession *, NSArray *) at /Users/xxx/Downloads/Unity-Technologies-unity-arkit-plugin-2bcc7d110304/test/Libraries/UnityARKitPlugin/Plugins/iOS/UnityARKit/NativeInterface/ARSessionNative.mm:755
#19 0x000000019d148b28 in __38-[ARSession _sessionDidUpdateAnchors:]_block_invoke ()
#20 0x0000000181f96a54 in _dispatch_call_block_and_release ()
#21 0x0000000181f96a14 in _dispatch_client_callout ()
#22 0x0000000181fd7c80 in _dispatch_main_queue_callback_4CF$VARIANT$armv81 ()
#23 0x00000001825c2544 in __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ ()
#24 0x00000001825c0120 in __CFRunLoopRun ()
#25 0x00000001824dfe58 in CFRunLoopRunSpecific ()
#26 0x000000018438cf84 in GSEventRunModal ()
#27 0x000000018bb5f67c in UIApplicationMain ()
#28 0x0000000102e38b80 in main at /Users/xxx/Downloads/Unity-Technologies-unity-arkit-plugin-2bcc7d110304/test/Classes/main.mm:33
#29 0x0000000181ffc56c in start ()

Any help would be highly appreciated!

 

Could this just be ameliorated with
OnDisable() -> Retrack()

 

Is there a way to display the video feed on a UI texture - presumably through renderTexture? It seems that the video feed only shows up on the Main Camera - how do you access the stream?