Resolved ARKit support for iOS via Unity-ARKit-Plugin

There are optimizations coming that will, hopefully, reduce the power usage. This is a pretty computationally heavy operation though, so the heat/power usage of the device is expected to be high.

 

Indoor navigation is a topic that is being heavily researched by large companies and, while AR seems like the most likely medium to visualize this, it is out of the scope of my knowledge. The only location based APIs that are available from Apple at the moment are the Core Location APIs.

This doesn't mean you can't map out a place with your own data and do something that works. I imagine you could use ARKit as your mapping tool. Identify locations, and distances between them, save that data out. Then either in the same app or another, you could read that data, identify where you are, and then say you want to go somewhere else. Then you could draw a virtual line from where you have identified yourself to be, to where you want to go.

This sounds simple, but I imagine it it a lot of work.

 

Hi, without knowing your code, or having a video of reference, it is hard to know what is wrong with what you are doing. Could you provide more information?

 

Hey, which shaders are on the materials that you are using? Are you sure you're changing the color on the correct game object? Can you provide more information and, possibly, code to show us what you are doing? Without any of this information it is hard for us to diagnose the issue and help you.
Cheers,
Chris

 

@christophergoy Hey, not sure if this has been answered already, but it seems with the arkit remote, the touch input does not transfer over to the computer as well. Is this intended behavior or did I mess something up?

 

You haven’t messed anything up the ARKIt Remote doesn’t transfer touches or hit tests at the moment.

 

Hmmm, so then it's recommended to make a PC based input controller to test code? Wait, I also just saw hit test is unsupported as well? So I guess we'll just have raycast on the planes then?

 

Hello guys! Currently I started to use this plug-in and I appreciate this. Well done!!
I have a question. I could not find how I can get the config variable of session.
I understand how to create a config and run with ARSession. However after I run the config, I want to change it later. I couldnt find how I can change it. Basically after some point I want to disable the plane detection but I cannot get the config variable, which holds planeDetection info, from ARSessionNativeInterface. Any help would be appreciated. Thanks a lot!

 

Thanks. This advice enabled progress!
Updated to 2017.2, imported the latest version of plugin. It compiles and runs! We even have a "Start Remote ARSession". But clicking doesn't trigger the real start (no green / life feed from device)...

How to solve the Player Connection error?
Searching by error code mentions fix about editing the manifest but am unable to navigate to those variables through (my understanding of) the unity editor.

Imagine this will make the development cycle much faster for testing things like 'how well a model spawns into the local dev environment.

PS. is anyone good with snow?
Been trying to tweak particle editor so that my snow falls more like the snow here – i.e. directly down in rays rather than flurries in flurries like the vimeo video.

 

Know how to solve this Player Connection issue?
Trying to setup basic 'launch animation into session' app. Seems that's the coolest thing ARKit does – load animations into your scene. In this case testing with trying to load a waterfall (particle effect) onto my coffee table. However, it – does not connect (see vid). How does one go about getting ARKit to load in an animation like a water effect like this?

This is the log:

<i>Autoconnected Player</i> TypeLoadException: A type load exception has occurred.
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetDeserializationType (Int64 assemblyId, System.String className) [0x00000] in <filename unknown>:0
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadTypeMetadata (System.IO.BinaryReader reader, Boolean isRuntimeObject, Boolean hasTypeInfo) [0x00000] in <filename unknown>:0
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadObjectInstance (System.IO.BinaryReader reader, Boolean isRuntimeObject, Boolean hasTypeInfo, Int64& objectId, System.Object& value, System.Runtime.Serialization.SerializationInfo& info) [0x00000] in <filename unknown>:0
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadNextObject (System.IO.BinaryReader reader) [0x00000] in <filename unknown>:0
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadObjectGraph (BinaryElement elem, System.IO.BinaryReader reader, Boolean readHeaders, System.Object& result, System.Runtime.Remoting.Messaging.Header[]& headers) [0x00000] in <filename unknown>:0
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.NoCheckDeserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler) [0x00000] in <filename unknown>:0
at Utils.ObjectSerializationExtension.Deserialize[T] (System.Byte[] byteArray) [0x00000] in <filename unknown>:0
at UnityEngine.XR.iOS.ConnectToEditor.HandleEditorMessage (UnityEngine.Networking.PlayerConnection.MessageEventArgs mea) [0x00000] in <filename unknown>:0
at UnityEngine.Events.InvokableCall`1[T1].Invoke (System.Object[] args) [0x00000] in <filename unknown>:0
at UnityEngine.Events.InvokableCallList.Invoke (System.Object[] parameters) [0x00000] in <filename unknown>:0
at UnityEngine.Networking.PlayerConnection.PlayerEditorConnectionEvents.InvokeMessageIdSubscribers (Guid messageId, System.Byte[] data, Int32 playerId) [0x00000] in <filename unknown>:0
at UnityEngine.Networking.PlayerConnection.PlayerConnection.MessageCallbackInternal (IntPtr data, UInt64 size, UInt64 guid, System.String messageId) [0x00000] in <filename unknown>:0

(Filename: currently not available on il2cpp Line: -1)

 

Did you build the UnityARKitRemote scene to your device from the latest project? It appears that the Remote side is expecting a different format of data from the one you're sending.

 

You can call RunWithConfig (or RunWithConfigAndOptions) again with a new session config and/or runoptions, and it will restart the AR Session with the new config/options.

 

One thing you could do is use a cross-platform touch input framework like TouchScript (written by our very own Valentin Simonov), or use mouse input which iOS simulates via touch. This way your code will work in Editor as well as device.

 

Nope – (or at least don't think so). I built the UnityARKitRemote scene to my device from a prior project. So I go onto my iOS device and launch ARKitRemote app. Then I opened the default waterfall scene (which i labelled ArKit Waterfall 3D) and then imported ARKit plugin to that scene.

 

You misunderstood me. Both the project you are trying to connect to Remote and the Remote itself should be built with the same version of Unity ARKit Plugin. Otherwise the data structures will be out of sync.

 

Hello, thanks for the answer. When I restart AR Session, will there be some information loss? I mean, Anchors, Planes etc..

I have one more question. When I do HitTest, it returns false until first plane created, which is good. But after there are some planes, where-ever I touch, HitTest returns true. Is there a reason for this behaviour?

Thanks a lot!

 

You can control what information is reset or not by using RunOptions. See the description of the kinds of HitTest you can query for (you may be using a test for an infinite plane?)

 

Thanks. I started a new project, imported the latest ARKit plugin from asset store. Built ARKitRemote to my iphone 6s. Created a new scene. Inserted ARKitRemoteConnection into root Assets folder. Imported the snow package and pulled the pre-fab into the scene. Clicked Play... the Start ARKit Session button appears. It asks for camera access on my phone. But then nothing happens. Continues to show 'Waiting for editor connection' on phone.

Understand that my above workflow should avoid any issues around using different plugins. Or am I missing something?
Any ideas why it's not working / how to get it working so that i can tap a plane through the phone to launch a waterfall (using unity dev environment)?

 

Hi

In UnityAppController.h seems to be using UIView instead of ARSCNView.
How to do it
convert RawFeaturePoints to CGPoint = ARSCNView .projectPoint .

Thanks.

 

Hello again! I really appreciate your helps sir! They really helped me.
I have one more question

I use the HitTestExample with the given small cube. It works perfectly. When I walk around that small grey cube, it remains at the same position perfectly.

I add another model, 1 unit left to the this cube. When I touch both are placed to the touched coordination perfectly but when I move around the model that I use is not stable anymore. The cube stills remains perfectly at the touched position. But the model moves when change the direction or when move a bit. Sometimes it is stable, but most of the time is so unstable.

Do you have any idea?

Thanks a lot! Have a nice day!

 

You need to add MainCamera and ARCameraManager gameobjects from EditorTestScene to your new scene.

 

Can you help me with these In Xcode
2017-08-04 20:03:49.815137+0700 xxxxx [1328:263105] [Technique] World tracking performance is being affected by resource constraints [1]

 

Ok so loaded the EditorTestScene then copied MainCamera and ARCameraManager and then loaded up the other scene again, pasted, and tried to get AR Remote working but still not functioning. Any thoughts?

 

^ does the default "UnityARKitScene" test scene work? Have you tried that? Why don't you just add your waterfall to that scene?

 

To project feature points to 2D screen coordinates try https://docs.unity3d.com/ScriptReference/Camera.WorldToScreenPoint.html or https://docs.unity3d.com/ScriptReference/Camera.WorldToViewportPoint.html (see examples on those pages)

 

Please add DontDestroyOnLoad class into namespace. I have conflicts with my own code. Thanks

 

Is it possible to decrease bitrate of captured video in AR Remote mode? iPhone is heating a lot and video is running very slow in remote mode.

 

Yes it works! There's a waterfall in my house! LOL

 

Hi,

I'm trying to get ARKit to work in my project and for some reason perspective is messed up. I run the examples built from my project and they work fine. When I copy the managers over to my own scene, taking care of making sure all the settings on the camera are correct and this this camera is linked where it must be. When running the scene, the objects I track feel like they are being seen from a camera with a really small field of view. I don't get what's happening. I added the box with with the direction gizmos from the hit test example scene to my scene and it isn't showing like in the hit test example. Any ideas? Does a multi-camera setup screw with ARKit?

Cheers,

Paul

 

@jimmya:
I'm having the same problem (and can also confirm the spinning planes) Any news on this?

Thanks!

 

Hey Chris, thanks for the reply. The model is just using a Standard shader. Here's the relevant snippet that isn't working for me:

GameObject obj = Instantiate (modelPrefab, atPosition, Quaternion.identity);
MeshRenderer renderer = obj.GetComponent<MeshRenderer> ();
renderer.material.color = Color.blue;

The object is white, the color set for the Albedo value for the prefab in the inspector, instead of the expected blue.

 

Anyone got hit testing work with the ARKit plugin (for faster iteration)?

Can get remote working but unable to use hit test to drop a plane/object in the iphone (connected via ArKit remote) viewport. Reason this would be valuable is because i'm trying to figure out how to get a snowing cloud to spawn & scale appropriately in the real world environment when the screen is touched on the ARKit device.
THanks!

 

I was planning on jumping into ARKIT sooner or later but a meeting with one of my clients on Friday spurred me on to get some compatible hardware, in the form of a shiny new iPad Pro! I am very excited to start working with this revolutionary AR milestone!

 

Hi

Am trying to orient a compass in AR to the Magnetic North in real life
So far I have this code but is not always working:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4.  
5. public class OrientNorthSimple : MonoBehaviour {
6.  
7.  
8.     void Start(){
9.         Input.compass.enabled = true;
10.         StartCoroutine(Example());
11.     }
12.  
13.     IEnumerator Example()
14.     {
15.        
16.         yield return new WaitForSeconds(3);
17.         transform.rotation = Quaternion.Euler(0, -Input.compass.magneticHeading, 0);
18.         Debug.Log (transform.eulerAngles + " " + Input.compass.magneticHeading);
19.  
20.     }
21. }

Right now it uses one measurement which is tricky with the phone's compass as it could be wrong. Would like to monitor for a while and use the average.
Also, the Camera's Y axis is not reliable when the phone is facing the floor...
Any ideas?

 

Have you tried doing the rotation setting in `Update` instead of in a coroutine? What's your reason for not updating it every frame?

 

So is there a good way to pause or otherwise stop the plane generation but keep tracking the existing ones? I noticed there's a pause method in the plugin, but no resume. And I guess it's suggested here that there's no real concept of pausing ARKit. I'd like to stop plane generation once the game begins--essentially keeping all the existing anchors where they are.

 

That's what I thought first, but actually, you only need to do it once, because afterwards the tracking of ARkit does all the rotation.
It is also possible to do it all the time, but because of the iPhones internal compass it would change a bit all the time, thus move. For my use-case it is better if it is set once -does not need to be super accurate- and then it should not rotate anymore.

My script almost does that, but it is difficult when that one measurement is wrong..

Coroutine is btw because the compass doesn't necessarily return a value directly.. it needs to "warm up". Oh well.. maybe it is good enough now that I think about it

 

I am getting the following error with the EditorTestScene, Failed to connect to player. See attached screenshot. I started with a new project and running Unity 2017.1.0f3 and Xcode 9.0 beta 4. Any suggestions on how to get it to work?

 

I would really like to show (virtual) reflections on the (real) floor. I can't find any way to do this
Something similar to the mobileARshadows shader..

 

I figured it out. I didn't have the remote app running on the iPhone.

 

Personally i think you should avoid reflection due to them being expensive and have a high probability of making things looking a bit more unrealistic than intended since you know nothing about the surface you are hitting. If you always know the type of surface you will be projecting to(for a demo or usage in a controlled environment), then it could be a good idea to implement.a feature like that.

You could start to look at something like this and do the necessary changes:
http://wiki.unity3d.com/index.php/MirrorReflection3

The wiki site is down at the moment though, so use this link in the meanwhile:
http://webcache.googleusercontent.c...p/MirrorReflection3+&cd=1&hl=en&ct=clnk&gl=dk

 

if UnityARVideo can not get picture from device camera
How do I know that?(in coding)

 

I would be cool to build a cube map out of the video feed on the fly and use that for reflections. Won't be super accurate, but if the surface is rough enough it wouldn't really matter. Would be great for lighting as well--for IBL shaders.

 

I have created a thing that uses the camera feed to create a pseudo reflection that integrates with the standard unity shading pipeline:


I still think its a bit too slow for mobile devices and the effect can be a bit harsh on very reflective surfaces (as the reflection per nature is not correct due to it never having the full 360 picture), but it can absolutely be done.

 

Anyone having problems recording in Landscape with the Screen Recorder?? From last update of ios11 the saved video gets frozen. If i record in portrait the screen recorder works fine but my Unity-app is in Landscape so ...

 

I'm looking to switch from Vuforia to ARKit but I'm having problems with the rotations when I'm switching to ARKit. How did you manage to do it ?

 

I think this was answered before: but you can restart via RunWithConfigAndOptions with config not doing plane detection, and runoptions keep existing anchors.

 

You might also want to look into "GravityAndHeading" setting in the startup config. It seems to currently mess up with the planes detected, but once you have established a starting orientation, you may be able to restart with "Gravity" setting in config.

 

hi thanks, While they are expensive indeed, ARKit only runs on high end devices anyway that can handle these kind of reflections quite well, A9 and A10 chip are very powerful. The realism is depending on what the developer does with it.

I am looking for this effect (with the blurry-ness, I know that is really hardcore):


I think Unity should pay attention to materials that are common in CGI packages in order to blend real life footage and virtual scenes.
In 3DS Max there is a material called Matte, that does what mobileshadow and mobileocclusion does in Unity's ARKit plugin, but it also does a whole lot more things.

For developing great ARKit experiences, it would be awesome if Unity had an allround "special effects" material available in an easy-to-use way. It should handle shadows, occlusion, possibly bumpmaps, reflections and maybe some sort of transparency. I think it would even be possible to do refractions of the underlying videofeed and distort the videofeed with a normalmap.

Allthough not really ARKit related, I do experience problems with horizontal video having a wrong aspect ratio. hopefully Apple will fix this later.

 

Same here. Recording a video of my app in landscape creates a portrait video with the wrong aspect. I guess this is related to the latest iOS beta (4 right now), because I can't recall having this problem with beta 3.