Search Unity

Submitting ARKit apps to AppStore without Face Tracking

Discussion in 'ARKit' started by jimmya, Nov 15, 2017.

  1. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    The current latest version of the Unity ARKit plugin includes some native calls to Face Tracking Configuration that will trigger an App Review failure on non-Face Tracking apps because they use TruDepth API without providing a privacy policy.

    We're in the process of fixing this problem for the latest plugin code, but in the meantime, you have two options for submitting your app:

    1. You can submit your apps with an older version of the plugin that does not contain the face-tracking code: you can get the version from bitbucket here: https://bitbucket.org/Unity-Technol...mits/6253989dc33a7341d0fd0ec3b05a69a404a520e9

    [edit: zip file here: https://bitbucket.org/Unity-Technologies/unity-arkit-plugin/get/1.0.10.zip]

    2. Even if your app does not use face tracking, you can submit a privacy policy with the App Store and reference it within your app according section 3.3.10 of https://developer.apple.com/programs/information/Apple_Developer_Program_Information_8_12_15.pdf

    We apologize for this inconvenience.
     
    Last edited: Nov 20, 2017
    christophergoy likes this.
  2. sd_trent

    sd_trent

    Joined:
    Jul 1, 2017
    Posts:
    6
    Thank you! I just ran into this problem.
     
  3. gamescorpion

    gamescorpion

    Joined:
    Feb 15, 2014
    Posts:
    131
    Just ran into the same thing, doing the first step now (recoding everything with the non-face tracking version)...
     
  4. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    You can try the latest version of the plugin on BitBucket that should fix the problem, but I haven't tested it yet, so it may still be broken. Let me know if it works for you!

    Shout out to @christophergoy who helped me out with the solution as usual :)
     
    gamescorpion and sd_trent like this.
  5. gamescorpion

    gamescorpion

    Joined:
    Feb 15, 2014
    Posts:
    131
    Yes I used the bitbucket solution. Still awaiting response from Apple, has been in review for 3 days now :( So hopefully when I do get a response back I'll know if the bitbucket solution works or not.

    Will keep you posted :)

    Nav
     
  6. gamescorpion

    gamescorpion

    Joined:
    Feb 15, 2014
    Posts:
    131
    Still the same issue, I just got off the phone with Apple and they said that the TrueDepth is still enabled (Even though I redid the entire app with the old sdk).

    Several of my apps are sitting awaiting approval, seems I'll have to figure this one out, please advise...

    Nav
     
  7. gamescorpion

    gamescorpion

    Joined:
    Feb 15, 2014
    Posts:
    131
    I just downloaded the zip file and realized that the pull I did from BITBUCKET and the zip file you outlined with the edit ([edit: zip file here: https://bitbucket.org/Unity-Technologies/unity-arkit-plugin/get/1.0.10.zip]) were different. I'm now going to redo the entire thing with the version in the zip file. Will then resubmit my apps and see if that fixes the issue. Pulling an all nighter again to get this thing working...will let you know the results for the zip this time around in the next days ahead.

    Nav
     
  8. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    Sorry about that - hope it works better this time around.
     
  9. gamescorpion

    gamescorpion

    Joined:
    Feb 15, 2014
    Posts:
    131
    jimmya and sd_trent like this.
  10. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    PROBLEM

    Recent versions of the Unity ARKit Plugin have had an issue that would cause apps that included it but was not using face tracking to fail App Review with the message:
    "We noticed your app contains the TrueDepth APIs but we were unable to locate these features in your app. Please provide information about how your app uses the TrueDepth APIs."

    This was because the plugin used the Face Tracking API in its native code for supporting ARKit face tracking.

    SOLUTION

    With the latest version of the Unity ARKit plugin, we introduced a settings file where you can specify how you want to use ARKit - with face tracking or without.

    If using face tracking, you may also have to provide a Privacy Policy and disclosures for your app along with the submission or you will get a failure as above.

    If you do NOT use face tracking, all the native code that has to do with face tracking is stripped out when being built on XCode. This means that if you do attempt to use face tracking in your app with this option, you will get an Objective-C exception called UnityARKitPluginFaceTrackingNotEnabled.

    The settings file for doing this is located at "Assets/UnityARKitPlugin/Resources/UnityARKiPlugin/ARKitSettings.asset"
    Screen Shot 2017-11-21 at 6.36.11 PM.png

    Selecting that file will allow you to select whether to use face tracking or not in the Inspector:
    Screen Shot 2017-11-21 at 6.36.29 PM.png

    If the ARKitSettings asset is missing, you can create a new one from the "Assets/Create/UnityARKitPlugin/Settings" menu:
    Screen Shot 2017-11-22 at 10.36.26 AM.png

    [Update: You can download from https://bitbucket.org/Unity-Technologies/unity-arkit-plugin/get/1.0.12.zip ]
    [Update: the Unity Asset Store also has this version now]
     
    Last edited: Nov 28, 2017
  11. gamescorpion

    gamescorpion

    Joined:
    Feb 15, 2014
    Posts:
    131
    Thanks for the update! To confirm about the previous version, Apple approved my second app with the old zip file (https://itunes.apple.com/us/app/augmented-ar-christmas-tree/id1315453691?ls=1&mt=8) as well, so that gives a full confirmation about the old code working for sure 100% (Now with 2 Live apps). I will attempt the new arkit plugin on my next several ARKit apps over the next weeks and let you know if I get approved/rejected so you guys have some live data to work with and if there are any errors I'll let you guys know :)

    Nav
     
    jimmya likes this.
  12. atomarcade

    atomarcade

    Joined:
    Sep 23, 2014
    Posts:
    1
    Last edited: Nov 26, 2017
    gamescorpion likes this.
  13. gamescorpion

    gamescorpion

    Joined:
    Feb 15, 2014
    Posts:
    131
    @jimmya now submitting my third ARKit app to Apple this time using your latest 1.0.12.zip file. I will let you know the result once I get more information. Crossing my fingers hoping it gets approved with the latest zip update :)

    Nav
     
    jimmya likes this.
  14. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    Did you make sure the "ARKit uses Face Tracking" checkbox was off in the settings?
     
    gamescorpion likes this.
  15. gamescorpion

    gamescorpion

    Joined:
    Feb 15, 2014
    Posts:
    131
    Yes I followed your images exactly and made sure to check only the ‘App Requires ARKit’ box and left the other one unchecked.
     
  16. joshpetersonhush

    joshpetersonhush

    Joined:
    Nov 9, 2017
    Posts:
    12
  17. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    Have not tried it, but the ARKit plugin is a separate drop-in project for that repo - so you can try deleting the UnityARKitPlugin folder and updating it with latest. It should just work.
     
  18. ifeltdave

    ifeltdave

    Joined:
    Aug 28, 2012
    Posts:
    16
    If we go the privacy policy route (avoiding import of the new plugin at the last minute) is it just a matter of defining the url in the app store parameters pointing to our companies privacy policy page? Do you know if it has to be linked from within the app too or is the itunes connect URL reference sufficient?
     
  19. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    I'm not sure they will let that pass - from new message someone else was getting, they would like you to disclose what you're using the TrueDepth API for as well. If you're not actually using it, they would like you to remove it.
     
  20. ifeltdave

    ifeltdave

    Joined:
    Aug 28, 2012
    Posts:
    16
    Our privacy policy specifically mentions the use of the camera and anything collected by the TrueDepth API.. wondering if that will help us?

    Our issue with upgrading the plugin is with the way that MultiARManager (a unity asset using an old version of Unity's plugin). MultiAR displays a point cloud on tracked points / surfaces during the initial phase of our app. These are invisible or do not render at all when using the updated ARKit plugin. While we track down what is going on we are trying to push a working version to the store.
     
  21. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
  22. ifeltdave

    ifeltdave

    Joined:
    Aug 28, 2012
    Posts:
    16
    We did not try with that version, so we must be using 1.0.11. Would a rollback to that version cause any headaches you can immediately think of? Good to know it might be a solution.
     
  23. TomKhan3

    TomKhan3

    Joined:
    Jul 21, 2017
    Posts:
    15
    Having issue with Apple rejection due to Facetracking as well. Updated ARKit Plugin to 1.0.12 made a new build and there are 1600+ search results of "ARFace" in the XCode (and yes, the box is unchecked). Same number of search results as the 1.0.11 build. I spoke to two people in Apple App review due to multiple rejections because of this and they mentioned that all mention of TrueDepth features should be removed from code (not just commented) and if it is included at all, they want to know exactly how it is being used. No mention of privacy policy. Probably being extra cautious due to face recognition tech
     
  24. sd_trent

    sd_trent

    Joined:
    Jul 1, 2017
    Posts:
    6
    I was able to get my app approved with 1.0.12! Thanks guys!
     
  25. TomKhan3

    TomKhan3

    Joined:
    Jul 21, 2017
    Posts:
    15
    I was wondering if your XCode also included references to "ARKitFace" or "ARFace"?
     
  26. ifeltdave

    ifeltdave

    Joined:
    Aug 28, 2012
    Posts:
    16
    Our app made it into beta testing (was previously rejected due to the TrueDepth stuff) using 1.0.11 and a privacy policy linked from the app store page.

    No idea if the actual app submission will get rejected. We are going to resubmit with 1.0.12 with the face stuff stripped for good measure.

    Edit: I can confirm TomKhan's results that searching my XCode project for "ARFace" returns many things that are not commented / still referenced. We haven't resubmitted yet so I don't yet know if this is an issue.
     
    Last edited: Dec 1, 2017
  27. TomKhan3

    TomKhan3

    Joined:
    Jul 21, 2017
    Posts:
    15
    Perhaps an insight into the issue: "App Requires AR Kit" is checked but when making a build the "arkit" text that is supposed to be added to the XCode > Info > Required device capabilities when that box is checked (at least I think so), is not in fact added. This points to possibility that the settings file is not being used and a default one is being generated by UnityARBuildPostprocessor.cs

    Even if this was the case, where a default ScriptableObject was being created (line 26 of UnityARBuildPostprocessor.cs), ARKitUsesFacetracking would be set to false. So perhaps the UnityARBuildPostprocessor is not working...

    Also worthy of note: When updating I had to move the UnityARKitPlugin folder from Assets to Assets\Plugins\iOS\UnityARKit\Editor folder so that line 24 of UnityARBuildPostprocessor.cs would work
     
  28. ifeltdave

    ifeltdave

    Joined:
    Aug 28, 2012
    Posts:
    16
    Tom, I'm curious. When you brough the 1.0.12 into your project, did you have an ARKitSettings object in your UnityARKitPlugin folder? Initially I didn't have one of these, and was about to use the manual creation method (Assets > Create > UnityARKitPlugin) before finding it buried in the Resources folder.

    I'm wondering if you might have two instances of the ARKitSettings object in your project?
     
  29. gamescorpion

    gamescorpion

    Joined:
    Feb 15, 2014
    Posts:
    131
    jimmya likes this.
  30. TomKhan3

    TomKhan3

    Joined:
    Jul 21, 2017
    Posts:
    15
    There was indeed a ARKitSettings file created initially so I do not have a duplicate of that file.

    To the people that have an app accepted using 1.0.12: does your XCode have "ARFace..." search results (Cmd + shift + f to find)? I feel like if I get rejected again I will lose my mind so I'm hesitant to submit to Apple.
     
  31. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    Yes, the file is not changed - there is a #define that is used to remove the code.

    Check that your Preprocessor.h has this line

    #define ARKIT_USES_FACETRACKING 0

    In the code, all face tracking code is surrounded by
    #if ARKIT_USES_FACETRACKING
    ...
    #endif

    This means, when compiling, this code is not used. If you
    #define ARKIT_USES_FACETRACKING 1
    then the face tracking code will be used.

    The UnityARBuildPostprocessor.cs file is supposed to do this for you by reading the settings file - but if you move the plugin files around, it could be problematic.
     
  32. TomKhan3

    TomKhan3

    Joined:
    Jul 21, 2017
    Posts:
    15
    Thank you so much for the reply Jimmy! I do have #define ARKIT_USES_FACETRACKING 0 in my Preprocessor.h file (UnityARBuildPostprocessor.cs did it for me) so I feel confident to submit now and not get rejected for the face tracking stuff.

    P.S. I checked out the Austin Unite video "How to Leverage ARKit and ARCore with Unity" and it was really helpful and entertaining. Cross platform AR multiplayer has huge potential
     
  33. TomKhan3

    TomKhan3

    Joined:
    Jul 21, 2017
    Posts:
    15
    Oh, and for everyone who wants to check, the Preprocessor.h file is in XCode project/Classes folder.
     
  34. domdev

    domdev

    Joined:
    Feb 2, 2015
    Posts:
    300
    Hi, I'm trying to reimport the latest build in ARkit is it okay to ignore the part of ProjectSetting?
     
    Last edited: Dec 5, 2017
  35. eyalfx

    eyalfx

    Joined:
    Oct 8, 2010
    Posts:
    98
    Are the blendshapes KeyPairValues that we are getting from the device ( anchorData.blendShapes) sorted in the same order?
    eyeBlink_R, eyeWide_R, mouthLowerDown_L, eyeLookDown_R, cheekSquint_L, etc...
    Can I expect them to always be sorted the same way ?
    Thanks
     
  36. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    They are not guaranteed to be - that's why its a dictionary - use the key to get a specific value.
     
  37. eyalfx

    eyalfx

    Joined:
    Oct 8, 2010
    Posts:
    98
    Great. thanks.
    The iPhone X uses depth camera right? is there a way to get depth information per pixel of the image / video ?
     
  38. gaozy124

    gaozy124

    Joined:
    Jan 20, 2018
    Posts:
    7
    Hi,i want to build a game in that face tracking is a very important function,so the game can only setup on iphoneX,i wonder if the game can pass app review, anyone know it? great thanks!
     
  39. eyalfx

    eyalfx

    Joined:
    Oct 8, 2010
    Posts:
    98
  40. Batman1208

    Batman1208

    Joined:
    Mar 9, 2018
    Posts:
    1
    Hi,@jimmya
    Can you give a DEMO of face tracking and gesture recognition?
     
  41. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
  42. danbfx

    danbfx

    Joined:
    Feb 22, 2011
    Posts:
    40
    Hey Jimmya

    We still get a hard crash on the face anchor test scene as is even though the rest of arkit works. We checked the uses fave tracking box in settings. What could be causing this?

    Thx
     
    lildub likes this.
  43. BinaryBanana

    BinaryBanana

    Joined:
    Mar 17, 2014
    Posts:
    64
    Hi,
    I am on the latest version 1.5 (https://bitbucket.org/Unity-Technologies/unity-arkit-plugin), I have a setting file set and I confirmed that ARKIT_USES_FACETRACKING is set to 0 in the Preprocessor.h but my submit to the store is still rejected. They told me they still see the TruDepth API.

    1. Is there a way to test it locally? Do you know how Apple is checking it? Maybe there is a way to find all references in XCode and make sure it's not the case before publishing as it takes days.
    2. Did anyone else submitted the app on ARKit 1.5 recently and didn't have any other issues?

    I have some open cv logic there and ARCore code. If you search by name "face" in the code, there are things like remote player, examples etc.., that are using "face" word but it shouldn't cause any issues.

    I am removing everything manually under ARKIT_USES_FACETRACKING flag and retrying today. Hopefully that will resolve the issue.


    --
    Smart AR Home - A gateway into augmented reality in your home
    http://smartarhome.com
     
    Last edited: May 21, 2018
  44. BinaryBanana

    BinaryBanana

    Joined:
    Mar 17, 2014
    Posts:
    64
    Rejected again by TrueDepth APIs. I am on latest 1.5 AR Kit plugin, I unchecked the flag in the settings and then even removed the stuff manually. This is so frustrating. Apple is not providing any info where they see APIs.

    I added as well stuff to privacy policy but it seems to not be enough.

    I also have AR Core code in solution, is there a chance AR Core shows up somehow as rueDepth APIs? Are they looking by "face" keyword?


    --
    Smart AR Home - A gateway into augmented reality in your home
    http://smartarhome.com
     
    Last edited: May 21, 2018
  45. BinaryBanana

    BinaryBanana

    Joined:
    Mar 17, 2014
    Posts:
    64
    From my last investigations I started to think that AR Core 1.2 might be the reason why app submission to iOS is failing with True Depth APIs.

    It seems that for cloud anchor sharing they are doing a check on anchor type and if they are comparing ARFaceAnchor, there is a chance they will create a reference to ARFaceAnchor selector and cause the app submission failure.

    Issue filed for investigation: https://github.com/google-ar/arcore-unity-sdk/issues/223

    Hopefully they can confirm soon if that's the case and advise a workaround.

    --
    Smart AR Home - A gateway into augmented reality in your home
    http://smartarhome.com
     
    Last edited: May 21, 2018
  46. jimmya

    jimmya

    Unity Technologies

    Joined:
    Nov 15, 2016
    Posts:
    793
    Yes if they have a reference to ARFaceAnchor in their code, that would do it.
     
  47. josselinco

    josselinco

    Joined:
    May 21, 2018
    Posts:
    4
    Any solution found on your side ?

    I just had my app rejected even with ARKit’s Face Tracking unchecked.
    We use ARCore 1.2 :-/

    I am wrong in saying that the only solution for now, is to include a TrueDepth function inside my app ? Would it pass iOS Review ?
     
  48. BinaryBanana

    BinaryBanana

    Joined:
    Mar 17, 2014
    Posts:
    64
    I added in privacy policy info what happens with face recognition data, even though I am not using it, but it didn't help, they want to know how it's used and I don't want to build features that are not needed.. So eventually what I did yesterday, only for iOS build, I removed AR Core. My code is structured the way that only one file is shared so a little cleanup was enough to remove it from iOS completely.

    The app is in the review now so I will update the topic if it goes through.

    --
    Smart AR Home - A gateway into augmented reality in your home
    http://smartarhome.com
     
    josselinco likes this.
  49. josselinco

    josselinco

    Joined:
    May 21, 2018
    Posts:
    4
    Alright thanks for the news. Good luck !
     
  50. BinaryBanana

    BinaryBanana

    Joined:
    Mar 17, 2014
    Posts:
    64
    Just an update on the thread. It seems that Google confirmed the issue I pointed out. Which means all apps that are using AR Core 1.2 are blocked from publishing on iOS! :D

    upload_2018-5-21_21-27-31.png
    More: https://github.com/google-ar/arcore-unity-sdk/issues/223#issuecomment-390815707

    Either start using facetracking feature or remove the AR Core from iOS app version before submitting. I don't believe there is a way to explain that in Apple review process, they are always very strict. If you are using Cloud Anchor and you want to publish to iOS, you are blocked.

    --
    Smart AR Home - A gateway into augmented reality in your home
    http://smartarhome.com
     
    studio-kura and josselinco like this.