Search Unity

ARKit Remote Connection frame freeze

Discussion in 'ARKit' started by aliatsky, Feb 6, 2018.

  1. aliatsky

    aliatsky

    Joined:
    Feb 6, 2018
    Posts:
    1
    Hi,
    First of all, Remote Connection is an amazing tool and I can't wait to get it to work so thank you for that.

    I'm experiencing a problem where upon connecting to the device this situation happens:

    1. On the device, the video stream appears as normal with no lag and fully functioning.
    2.In Unity, the very first frame of the video from the phone appears. But it freezes there.
    3. As I move the phone around, in Unity I can just about see a red/blue halo of images from the camera appearing on top of the frozen frame. These seem to be following the camera smoothly and don't freeze.
    4. If I wait long enough, every now and then the frozen frame changes to a newer frame from the camera but it freezes again.

    Hope I managed to describe it well enough for you to be able to tell me what I'm missing here. I also played around with the editor settings but can't seem to be able to change anything.

    Any help would be greatly appreciated :)
     
  2. digitalHealthcareGuy

    digitalHealthcareGuy

    Joined:
    Sep 6, 2017
    Posts:
    1
    I am experiencing the same issue on MBP Mid 2017, iPhone 6s & Unity 2017.3.
     
  3. Autarkis

    Autarkis

    Joined:
    Oct 10, 2011
    Posts:
    272
    Like you guys, I'm unable to make arkit remote work properly.
    At some point in time I was able to get the device camera feed into it, and was seeing the point cloud particles ( appeared like an overlay mode in photoshop), but now I'm just getting the camera feed, and nothing is happening in editor - cant see debug planes, particle cloud, or any models placed in scene.
     
  4. bluefireseattle

    bluefireseattle

    Joined:
    Nov 22, 2017
    Posts:
    2
    I am seeing the exact same behavior as @aliatsky -once connected you can see the first fame of the phone video in the Game widow but when you move the phone you only see odd red and blue translucent halo images.
    This is with a fresh clone of Arkit-Plugin from today.
    iPhone X
    Xcode 9.2
    OSX 10.13.3
    Unity 2017.3.0f3
     
  5. Coding2Handed

    Coding2Handed

    Joined:
    Jan 14, 2018
    Posts:
    4
    Bump. Seeing the same behavior.
     
  6. MikeGardiner

    MikeGardiner

    Joined:
    Dec 8, 2016
    Posts:
    4
    Same issue here - any word from Unity?
     
  7. bestofluck

    bestofluck

    Joined:
    Feb 28, 2016
    Posts:
    1
    I'm seeing the same behavior, but only over wifi.

    iPhone 8+, iOS 11.3.5 beta
    macOS 10.13.2
    XCode 9.2 (9C40b)
    Unity 2017.3.1f1

    The editor behaves as expected when I connect to my device with a USB cable.
     
    Last edited: Mar 18, 2018
  8. inihility

    inihility

    Joined:
    Oct 17, 2014
    Posts:
    7
    Experiencing the same issue over both WiFi and USB.

    10.13.3 macOS High Sierra
    iPhone X
    iOS 11.3 beta
    Unity 2017.3.1f1
     
  9. sulamatraore

    sulamatraore

    Joined:
    Jul 31, 2017
    Posts:
    1
    iPhone X running iOS 11.3 Beta
    Early 2015 Macbook Pro running OS X High Sierra 10.13.4 Beta
    Unity 2017.3.1f.1
    XCode 9.3 Beta

    Currently having the same issue, regardless of whether or not I use a USB Cable or connect through Wi-Fi. Any word on this issue? I'm using the Spring2018_Update branch of the ARKit Plugin from BitBucket...
     
  10. davisadam10

    davisadam10

    Joined:
    Oct 22, 2016
    Posts:
    2
    Same here
    Iphone 8 running IOS 11.3 beta
    10.13.3 macOS High Sierra
    Unity 2017.3.1f1 & 2018.1b11

    Tested on 2 machines, and connected phone via wifi and via USB cable as stated else where, both report the same issues (wifi is obviously worse and would expect it to struggle and not work)
     
  11. football928

    football928

    Joined:
    Sep 24, 2015
    Posts:
    17
    Same here.
    MacBook 12 inch 2016
    iPhone X iOS 11.3 beta
    Spring update of ar kit plugin
    Unity 2017.3.1f1

    I connected with usb.
     
  12. tibmatt

    tibmatt

    Joined:
    Feb 11, 2013
    Posts:
    2
    Yeah same. USB. I can usually see some shadow image (like an overexposed photo underneath). Very very occasionally the image does catch up but I’ve never been able to consistently work out why it catches up.
     
  13. inihility

    inihility

    Joined:
    Oct 17, 2014
    Posts:
    7
    I managed to get ARKit Remote working on recent iMacs, seems like older devices like the MacBook Air have trouble performing well with ARKit Remote.

    What is everyone's Mac specs/model?
     
  14. GameDevCouple_I

    GameDevCouple_I

    Joined:
    Oct 5, 2013
    Posts:
    1,875
    same, macbook air high seirra ios 11.3 beta
     
  15. tibmatt

    tibmatt

    Joined:
    Feb 11, 2013
    Posts:
    2
    Mine is MBP i7, Late 2017, iPhoneX.
     
  16. FlashyGoblin

    FlashyGoblin

    Joined:
    Apr 1, 2017
    Posts:
    23
    Has anyone found a solution to this yet?
     
  17. davisadam10

    davisadam10

    Joined:
    Oct 22, 2016
    Posts:
    2
    https://zeroichi.hatenablog.jp/entry/2018/04/04/155420

    This works but you will need to be running .Net 4.x or you'll need to manage your own CopyTo for the DeflateStream.

    The problem seems to be the newer models of the ios devices are sending over way more data than they used too, tested this on an iphone 6s and my new iphone 8. By compressing and decompressing it works around the too much data being sent errors
     
  18. fjordmahoney

    fjordmahoney

    Joined:
    Jan 15, 2018
    Posts:
    1
    Yeah, I've seen this in two scenarios: Breaking out from USB-C to regular USB on the new Macbook Pros with a crappy breakout device, and having a poor/saturated wifi connection. In both cases, there was an indication in the XCode debug log that the amount of data being sent by the device could not reliably be handled and I'm guessing that prevents the editor experience from accurately replicating on-device behavior.
     
  19. Go_Sato

    Go_Sato

    Joined:
    May 21, 2017
    Posts:
    4
    seanomlor likes this.
  20. AdyC

    AdyC

    Joined:
    Mar 1, 2018
    Posts:
    1
  21. Unlink

    Unlink

    Joined:
    Jul 23, 2013
    Posts:
    2
  22. JOJOGT

    JOJOGT

    Joined:
    Jan 16, 2015
    Posts:
    2
    Macbook air 2015
    Intel Core i5
    Intel HD Graphic 6000 1536MB
    High Sierra 10.13.4
    iPhoneX 11.3.1
    Hope to get a solution.
     
  23. sambrego

    sambrego

    Joined:
    Nov 5, 2017
    Posts:
    1
    I didn't have time to try Go_Sato's solution, but I have found a workaround:
    For me ARKit-Remote works alright if iOS-side scene is build in Unity version 5.6.5f1(Xcode version doesn't matter).
    So for now, I build UnityARKitRemote.scene in 5.6.5 and connect to it from my scene in Unity Editor 2017.3.1f1.

    Hope it will work for somebody!
     
    Scarlett7 likes this.
  24. james_co

    james_co

    Joined:
    Mar 26, 2018
    Posts:
    2
    Same issue here. Late 2017 Macbook and Iphone 8.
     
  25. Coding2Handed

    Coding2Handed

    Joined:
    Jan 14, 2018
    Posts:
    4
    I tried Go_Satos solution but I get this stack trace after attempting to start the ARKit remote face tracking session.

    Can anyone help with this?

    Code (CSharp):
    1. SerializationException: Unable to find assembly 'Assembly-CSharp, Version=0.0.0.0, Culture=, PublicKeyToken=null'.
    2. System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo.GetAssembly () (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binarycommonclasses.cs:408)
    3. System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetType (System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo assemblyInfo, System.String name) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryobjectreader.cs:1489)
    4. System.Runtime.Serialization.Formatters.Binary.ObjectMap..ctor (System.String objectName, System.String[] memberNames, System.Runtime.Serialization.Formatters.Binary.BinaryTypeEnum[] binaryTypeEnumA, System.Object[] typeInformationA, System.Int32[] memberAssemIds, System.Runtime.Serialization.Formatters.Binary.ObjectReader objectReader, System.Int32 objectId, System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo assemblyInfo, System.Runtime.Serialization.Formatters.Binary.SizedArray assemIdToAssemblyTable) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binarycommonclasses.cs:2153)
    5. System.Runtime.Serialization.Formatters.Binary.ObjectMap.Create (System.String name, System.String[] memberNames, System.Runtime.Serialization.Formatters.Binary.BinaryTypeEnum[] binaryTypeEnumA, System.Object[] typeInformationA, System.Int32[] memberAssemIds, System.Runtime.Serialization.Formatters.Binary.ObjectReader objectReader, System.Int32 objectId, System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo assemblyInfo, System.Runtime.Serialization.Formatters.Binary.SizedArray assemIdToAssemblyTable) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binarycommonclasses.cs:2204)
    6. System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped (System.Runtime.Serialization.Formatters.Binary.BinaryObjectWithMapTyped record) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryparser.cs:683)
    7. System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped (System.Runtime.Serialization.Formatters.Binary.BinaryHeaderEnum binaryHeaderEnum) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryparser.cs:656)
    8. System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run () (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryparser.cs:146)
    9. System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize (System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Runtime.Serialization.Formatters.Binary.__BinaryParser serParser, System.Boolean fCheck, System.Boolean isCrossAppDomain, System.Runtime.Remoting.Messaging.IMethodCallMessage methodCallMessage) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryobjectreader.cs:174)
    10. System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Boolean fCheck, System.Boolean isCrossAppDomain, System.Runtime.Remoting.Messaging.IMethodCallMessage methodCallMessage) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryformatter.cs:197)
    11. System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Boolean fCheck, System.Runtime.Remoting.Messaging.IMethodCallMessage methodCallMessage) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryformatter.cs:173)
    12. System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Boolean fCheck) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryformatter.cs:118)
    13. System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryformatter.cs:149)
    14. System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream) (at /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/runtime/serialization/formatters/binary/binaryformatter.cs:111)
    15. Utils.ObjectSerializationExtension.Deserialize[T] (System.Byte[] byteArray) (at Assets/UnityARKitPlugin/ARKitRemote/ObjectSerializationExtension.cs:39)
    16. UnityEngine.XR.iOS.ARKitFaceTrackingRemoteConnection.UpdateCameraFrame (UnityEngine.Networking.PlayerConnection.MessageEventArgs mea) (at Assets/UnityARKitPlugin/ARKitRemote/ARKitFaceTrackingRemoteConnection.cs:121)
    17. UnityEngine.Events.InvokableCall`1[T1].Invoke (T1 args0) (at /Users/builduser/buildslave/unity/build/Runtime/Export/UnityEvent.cs:206)
    18. UnityEngine.Events.UnityEvent`1[T0].Invoke (T0 arg0) (at /Users/builduser/buildslave/unity/build/Runtime/Export/UnityEvent_1.cs:58)
    19. UnityEngine.Networking.PlayerConnection.PlayerEditorConnectionEvents.InvokeMessageIdSubscribers (System.Guid messageId, System.Byte[] data, System.Int32 playerId) (at /Users/builduser/buildslave/unity/build/Runtime/Export/Networking/PlayerConnection/PlayerEditorConnectionEvents.cs:66)
    20. UnityEditor.Networking.PlayerConnection.EditorConnection.MessageCallbackInternal (System.IntPtr data, System.UInt64 size, System.UInt64 guid, System.String messageId) (at /Users/builduser/buildslave/unity/build/Editor/Mono/Networking/PlayerConnection/EditorConnection.cs:164)
    21.  
     
  26. novaVision

    novaVision

    Joined:
    Nov 9, 2014
    Posts:
    292
    I followed instructions on @Go_Sato web page, but unfortunately, I got ping texture on a screen + cloud points generated from the device. Plane detections seems works pretty fast as it should be without lags, but something wrong with a main camera texture.
    Here is a screenshot of error in Editor.
    I am using Unity 2018.1.3 (4.x .NET), latest ARKit, iPhone 7, MacbookPro 2015.

    Did someone had same case? How did you guys solve it?
     
  27. Go_Sato

    Go_Sato

    Joined:
    May 21, 2017
    Posts:
    4
    Hi, novaVision!

    I want you to confirm.
    Have you changed two scripts on the sending side and the receiving side?
    If one of them is not changed, data exchange will be damaged.
    I wish you good luck!
     
  28. novaVision

    novaVision

    Joined:
    Nov 9, 2014
    Posts:
    292
    Not sure what do you mean under "script on both sides" but once I modified ARKit 2 scripts as it mentioned in your tutorial, I build new ARKit Remote scene to the device, and run EditorTestScene. There are not too many things where to make a mistake :)
    Here is a screencast

    [UPDATED]
    Tried that method using clean project and fresh ARKit plugin - everything works. Still a bit laggy but at least usable.

    Just replaced entire ARKit folder to my current project and everything works fine now. After I start to investigate, what was wrong and found that in
    Code (CSharp):
    1. public static byte[] ConvertByteDecompress(byte[] source)
    2. {
    3.         using (MemoryStream input = new MemoryStream(source))
    4. ...
    5. }
    I had
    Code (CSharp):
    1. new MemoryStream())
    That happened because firstly I found you web page in google (not here in forum) in Japanese, and used Auto-translate to read the topic. Doing this code been formatted as well... That's crazy

    Thanks for support!

    Btw, is there any other options to speedup ARKit remote? Any chances Unity will fix it at all?
     
    Last edited: Jun 12, 2018
  29. yonyonjue

    yonyonjue

    Joined:
    Jun 8, 2017
    Posts:
    1
    Hi I am a new developer trying to use ARKit remote and I need to implement your solution. I am confused where I should include the code for compress and decompress that you wrote for this solution. How/where do I put it in my project to have it function correctly?
     
  30. novaVision

    novaVision

    Joined:
    Nov 9, 2014
    Posts:
    292
    Just create new static class and add these static methods. Put it anywhere you wish. Other code corrections must be modified within existing ARKit scripts
     
  31. seth_hunter

    seth_hunter

    Joined:
    Jun 11, 2018
    Posts:
    1
    I was able to use noraVision's compression and decompression but it took me a while to figure out how. Some tips:

    a) Create a static class for the functions with includes like so:
    Code (CSharp):
    1. using UnityEngine;
    2. //these included need to be added for compression an decompression.
    3. using System.IO;
    4. using System.IO.Compression;
    5. public static class CompressDecompress
    6. {
    7. // add functions here.
    8. }
    b) Edit >> Project Settings >> Player >> Other Settings contains the options for .Net Configuration. Change the "runtime version" to 3.5x Equivalent when building the iOS app. After this is on your phone, change the Runtime version to .NET 4.x Equivalent. Unity will ask you restart - do so and you should be able to run some of the other apps.

    c) There are two files used to decompress textures. "ARKitFaceTrackingRemoteCamera.cs" and "ARKitRemoteConnection" (the one referenced in the blog post). Be sure that you change the
    Code (CSharp):
    1. remoteScreenYTex.LoadRawTextureData(CompressDecompress.ConvertByteDecompress(mea.data));
    line in both files, depending on which example you are running. FaceAnchor for example uses ARKitFaceTrackingRemoteCamera.cs.

    The result is pleasing because it fixes all the streaming errors but it does seem like its running at 10fps.
     
  32. Cascho01

    Cascho01

    Joined:
    Mar 19, 2010
    Posts:
    1,107
    Any official support of Unity-Guys would be nice!
     
  33. seanomlor

    seanomlor

    Joined:
    Aug 10, 2018
    Posts:
    1
    Kubic75 likes this.
  34. Kubic75

    Kubic75

    Joined:
    Jan 2, 2017
    Posts:
    83
    Got the remote working with the hack from Go_Sato.
    But it´s very laggy (~2 FPS), any idea to improve this?
     
  35. sunu-pipi

    sunu-pipi

    Joined:
    Jan 11, 2018
    Posts:
    1
    Me too. iPhone camera and camera of Mac book are freezing. I don’t know what happened.
     
  36. qijie9532

    qijie9532

    Joined:
    Oct 17, 2018
    Posts:
    1
  37. Kubic75

    Kubic75

    Joined:
    Jan 2, 2017
    Posts:
    83