Search Unity

  1. Unity 2020.2 has been released.
    Dismiss Notice
  2. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

UnityPlayer.displayChanged() method to add a Surface as an additional display

Discussion in 'Android' started by Konaju-Games, Dec 16, 2020.

  1. Konaju-Games

    Konaju-Games

    Joined:
    Jul 11, 2012
    Posts:
    17
    Has anyone successfully used the displayChanged() method in the UnityPlayer Java class to add a Surface as an additional display that can be rendered to? I'm trying to use this with a Surface created for input to a MediaCodec encoder. The only reference I can find is a single forum post that references a single dot point in Unity 5.x release notes, but this looks like the perfect feature I need to get a rendered image from Unity into the MediaCodec encoder.

    When the app starts, Display.displays returns one display (the phone screen). I call UnityPlayer.displayChanged(1, encoderSurface), and a short time later Display.displays will return two displays. The Display.onDisplaysUpdated event never got triggered however, so I check each update for a change to the length of the Display.displays array. Then I enable a camera and set its targetDisplay to the index of the new display. I don't appear to be getting anything though.

    I'm still working on it, but if anyone has used this successfully it would be great to know I'm not running up a dead-end alleyway. And Unity, please document this stuff. Please.
     
    makaka-org likes this.
  2. Konaju-Games

    Konaju-Games

    Joined:
    Jul 11, 2012
    Posts:
    17
    Odd behaviour found. The input surface created by the MediaCodec encoder is 720x1280, as expected because that is the size we requested for the encoder. When the surface is added to Unity's displays list, the Display object returns a rendering and system size of 1080x2057. The main display has a rendering size of 1080x2204 and system size of 1080x2340, but this second display should be 720x1280, not the oddball size it is returning.
     
  3. florianpenzkofer

    florianpenzkofer

    Unity Technologies

    Joined:
    Sep 2, 2014
    Posts:
    325
    'targetDisplay' is only supported on Android since Unity 2020.2. On older Unity versions you would have to use 'Camera.SetTargetBuffers'
     
  4. Konaju-Games

    Konaju-Games

    Joined:
    Jul 11, 2012
    Posts:
    17
    Thanks for that. For reference, we are using 2019.4.11f1. I changed the code to use Camera.SetTargetBuffers, but I haven't been able to verify it's working yet. The MediaCodec encoder that created the surface still doesn't appear to be producing any output yet.

    The odd rendering and system sizes returned for the new display are still there.
     
  5. Konaju-Games

    Konaju-Games

    Joined:
    Jul 11, 2012
    Posts:
    17
    The MediaCodec encoder does not produce any output when the surface has not been rendered to, and that is what appears to be the case here. Is UnityPlayer.displayChanged() verified to work and able to be rendered to?
     
  6. Konaju-Games

    Konaju-Games

    Joined:
    Jul 11, 2012
    Posts:
    17
    I decided to try the project in Unity 2020.2 to see if it worked better. No luck since it breaks on deterministic compilation (had to turn that off) and a very unhelpful error for our custom Gradle build template.
    "mainTemplate.gradle file is using the old aaptOptions noCompress property definition which does not include types defined by unityStreamingAssets constant."
    but nowhere does the editor or the docs say what the fix is or how to fix it. Every avenue I take I find a dead-end.
     
    makaka-org and Hyperiment like this.
  7. Hyperiment

    Hyperiment

    Joined:
    Jul 5, 2020
    Posts:
    3
    Hi.
    Unity 2020.2
    We cant build on Android too with the same error: "mainTemplate.gradle file is using the old aaptOptions noCompress property definition which does not include types defined by unityStreamingAssets constant." Could someone from Unity help us?
     
  8. Josha

    Josha

    Joined:
    Jan 8, 2013
    Posts:
    3
    With regards to the mainTemplate.gradle error. I had the same thing, so I moved the file and let Unity3D regenerate the default one.

    It added a new noCompress line to the aaptOptions section:

    Code (JavaScript):
    1.  
    2. aaptOptions {
    3.         noCompress = ['.ress', '.resource', '.obb'] + unityStreamingAssets.tokenize(', ')
    4.         ignoreAssetsPattern = "!.svn:!.git:!.ds_store:!*.scc:.*:!CVS:!thumbs.db:!picasa.ini:!*~"
    5.     }**PACKAGING_OPTIONS**
    6.  
    I copied the noCompress line to my original mainTemplate.gradle file and I no longer had this error (I did however get other errors).
     
    makaka-org and TerraUnity like this.
  9. Hyperiment

    Hyperiment

    Joined:
    Jul 5, 2020
    Posts:
    3
    I did the same, but in the console you can see that the error regarding unityStreamingAssets, so not sure that this could help =( I think we need to wait for answer from someone from Unity.
     
  10. Tomas1856

    Tomas1856

    Unity Technologies

    Joined:
    Sep 21, 2012
    Posts:
    2,412
    Can you show your gradle files?
     
  11. Konaju-Games

    Konaju-Games

    Joined:
    Jul 11, 2012
    Posts:
    17
    Old (generated from a 2019.4.11f1 project)
    In the
    aaptOptions
    section, the
    noCompress
    line is either missing or contains
    Code (CSharp):
    1. noCompress = ['.unity3d', '.ress', '.resource', '.obb'**STREAMING_ASSETS**]
    New (freshly generated from a new 2020.2.0f1 project)
    The
    aaptOptions
    section contains
    Code (CSharp):
    1. noCompress = ['.ress', '.resource', '.obb'] + unityStreamingAssets.tokenize(', ')
    Notice the change to how the Streaming Assets extensions are added?

    The error that 2020.2.0f1 gives when confronted with the earlier
    aaptOptions
    section (missing the
    noCompress
    line or the older format) is

    upload_2020-12-23_1-13-46.png

    No hint as to how to fix the problem or any link to documentation describing what is expected, not even in the Console error output. Perhaps it could take a hint from a later different error, also related to Gradle files, that describes what to do and offers to fix the problem.

    upload_2020-12-23_1-14-50.png

    It then repeats the earlier
    aaptOptions
    error for the launcherTemplate.gradle file.

    That should be all of the information you need. I do not know what else to give.
     
    Last edited: Dec 22, 2020
  12. ChaosHelme

    ChaosHelme

    Joined:
    Nov 22, 2016
    Posts:
    4
    Same here. Would be nice to have some documentation or instructions on what to do.
     
  13. Konaju-Games

    Konaju-Games

    Joined:
    Jul 11, 2012
    Posts:
    17
    Back to the original issue (adding an Android surface as an additional display), after making the change to the gradle files to get the project to build in 2020.2.0f1, the rendering to the extra display appears to work. The MediaCodec encoder is seeing updates to the input surface and producing encoded h264 output.

    The summary for those just catching up and for future developers searching for these errors:

    1. The feature to add an Android surface as an additional display appears to require Unity 2020.2.0f1. It does not appear to work correctly in Unity 2019.4.11f1 (I tried for days to get it working), even though the feature was added back in the Unity 5.x series.

    2. If upgrading to Unity 2020.2.0f1 from an earlier Unity project that has a custom Android gradle file, change the existing
    noCompress
    line in the
    aaptOptions
    section (or add it if it is missing) to the following

    Code (CSharp):
    1. noCompress = ['.ress', '.resource', '.obb'] + unityStreamingAssets.tokenize(', ')
     
    makaka-org, TerraUnity and Hyperiment like this.
  14. MaurreKiTu

    MaurreKiTu

    Joined:
    Apr 12, 2017
    Posts:
    1
    Thanks, that's exactly what I needed!
     
    Konaju-Games likes this.
  15. Hyperiment

    Hyperiment

    Joined:
    Jul 5, 2020
    Posts:
    3
    Hi!

    So, have you found the way to resolve the issue after adding this string? Could you share it please? )
     
    nikich0601 likes this.
  16. ChaosHelme

    ChaosHelme

    Joined:
    Nov 22, 2016
    Posts:
    4
    Thanks for your help!
     
  17. chetan312

    chetan312

    Joined:
    Mar 12, 2017
    Posts:
    6
    same error ."mainTemplate.gradle file is using the old aaptOptions noCompress property definition which does not include types defined by unityStreamingAssets constant." Is anyone found a solution.
     
  18. chetan312

    chetan312

    Joined:
    Mar 12, 2017
    Posts:
    6
    when trying to build an apk . Android gradle file changes to its previous version.
     
  19. Konaju-Games

    Konaju-Games

    Joined:
    Jul 11, 2012
    Posts:
    17
    If you have custom gradle files, you need to make the change listed above in both launcherTemplate.gradle and mainTemplate.gradle.
     
  20. chetan312

    chetan312

    Joined:
    Mar 12, 2017
    Posts:
    6
    yeah i have custom gradle files and i changed both launcherTemplate.gradle and mainTemplate.gradle but when i tried to make a build then the same error occurred and mainTemplate.gradle changed to their previous version.
    i dont know the exact reason but this issue starts when i ugraded my unity editor to latest
     
  21. Konaju-Games

    Konaju-Games

    Joined:
    Jul 11, 2012
    Posts:
    17
    So, as for the original issue of adding a Surface as an additional display on Android, I do now have it working. But it was not without a lot of frustration and cursing along the way. And as it seems this will be the only source of documentation of this feature for the near future, I'll try to make it decent.

    Firstly, you do need to be using Unity 2020.2.0f1 or later. It just does not work in earlier versions. If upgrading from earlier versions and you have custom gradle files, make sure you apply the fix listed above.

    I'm using C++ for my MediaCodec implementation, but you should be able to do the same if you are using Java plugins.

    I originally implemented this using a custom Activity derived from UnityPlayerActivity solely to get access to the protected mUnityPlayer field. That is ugly and messy, so I have since worked out how to use JNI to avoid any custom Activity class. This code is currently assuming the additional display will be at index 1 (the second display). This may cause an issue if an Android device already has more than one display. Who knows what an Android device will have these days. It would make more sense to pass in the current length of the Display.displays array property and use that as the index of the new display.

    Code (csharp):
    1. static void displayChanged(jobject surface)
    2. {
    3.     // Get the current activity from the static field in the UnityPlayer class.
    4.     const jclass playerClass = jniEnv->FindClass("com/unity3d/player/UnityPlayer");
    5.     const jfieldID currentActivityID = jniEnv->GetStaticFieldID(playerClass, "currentActivity", "Landroid/app/Activity;");
    6.     jobject currentActivity = jniEnv->GetStaticObjectField(playerClass, currentActivityID);
    7.  
    8.     // Get the current UnityPlayer instance from the current activity in the mUnityPlayer field.
    9.     // This field is protected in Java, but apparently everything is accessible via JNI.
    10.     const jclass activityClass = jniEnv->GetObjectClass(currentActivity);
    11.     const jfieldID unityPlayerID = jniEnv->GetFieldID(activityClass, "mUnityPlayer", "Lcom/unity3d/player/UnityPlayer;");
    12.     jobject unityPlayer = jniEnv->GetObjectField(currentActivity, unityPlayerID);
    13.  
    14.     // Call the displayChanged method on the UnityPlayer instance
    15.     const jmethodID displayChangedID = jniEnv->GetMethodID(playerClass, "displayChanged", "(ILandroid/view/Surface;)Z");
    16.     jboolean result = jniEnv->CallBooleanMethod(unityPlayer, displayChangedID, 1, surface);
    17. }
    18.  
    I call this after I create the encoder input surface.

    Code (csharp):
    1.     // Create the surface that will feed the encoder
    2.     error = AMediaCodec_createInputSurface(encoder_, &inputSurface_);
    3.     if (error != AMEDIA_OK)
    4.     {
    5.         stopEncoder();
    6.         return -1;
    7.     }
    8.    
    9.     // Get the Java Surface object handle to the native window
    10.     jobjectSurface_ = ANativeWindow_toSurface(jniEnv, inputSurface_);
    11.     // Add the encoder input Surface as an additional display for Unity
    12.     displayChanged(jobjectSurface_);
    13.  
    This will start the asynchronous process of adding the Surface as an additional display. Now, you would expect the Display.onDisplaysUpdated event to be triggered when the additional display is added or removed. No, it does not. You must regularly check the length of the Display.displays array property and respond appropriately when the length changes from the previous check.

    Code (CSharp):
    1.     void Update()
    2.     {
    3.         // Detect when encoder input display is added or removed
    4.         if (_displayCount != Display.displays.Length)
    5.         {
    6.             _displayCount = Display.displays.Length;
    7.             Display_onDisplaysUpdated();
    8.         }
    9.     }
    Now we need to tell the relevant camera which display to render on. The call to display.SetRenderingResolution(w, h) may or may not be required. I was seeing some odd display resolutions being reported by Unity for this additional display, even though the surface it was given was of the correct dimensions.

    Code (CSharp):
    1.     void Display_onDisplaysUpdated()
    2.     {
    3.         if (_displayCount > 1)
    4.         {
    5.             int index = _displayCount - 1;
    6.             Debug.Log("Activating additional display");
    7.             var display = Display.displays[index];
    8.             // We are rendering to a portrait screen, hence taller than wide
    9.             display.SetRenderingResolution(720, 1280);
    10.             // The display is apparently already activated, but we call this anyway
    11.             display.Activate();
    12.             _encoderInputCamera.targetDisplay = index;
    13.             _encoderInputCamera.gameObject.SetActive(true);
    14.         }
    15.         else
    16.         {
    17.             _encoderInputCamera.gameObject.SetActive(false);
    18.         }
    19.     }
    The MediaCodec encoder will now start seeing input and producing h.264 packets of the rendered output. Yay!

    But the fun is not over yet. No, there are other gotchas in store when encoding video. Despite what the documentation states, setting Application.targetFrameRate had no effect on my Google Pixel 4a. The app will always try to render at the refresh rate of the screen (Screen.currentResolution.refreshRate, though what does Screen and currentResolution mean when you have multiple displays?). In my case, it always renders at 60fps regardless of what I set Application.targetFrameRate to. When creating a MediaCodec encoder, you tell it what framerate you want the video stream to play at. This will have to match the framerate of the Unity app because the rendering of the additional display is feeding the encoder with video frames. To get different framerates, I needed to set QualitySettings.vSyncCount. By default, this is 0 which means "no sync", but on mobile devices it is always synced to the refresh rate, so 0 is effectively the same as 1. Set QualitySettings.vSyncCount to 2 to render at half the refresh rate (30fps in my case), 3 to render at one third (20fps), 4 for a quarter (15fps) and so on. It is not the cleanest solution (it also blocks updates), so I'm still working on that.

    I think that is everything. It has been incredibly frustrating over the past several weeks but it is finally working. @florianpenzkofer does this look like how it should be working?
     
    Last edited: Jan 6, 2021
  22. Konaju-Games

    Konaju-Games

    Joined:
    Jul 11, 2012
    Posts:
    17
    This gradle upgrade issue probably deserves its own thread.

    It should not be changing or reverting that line in the gradle file while attempting to build the project. @Tomas1856 do you have any additional thoughts on this for @chetan312 ?
     
  23. chetan312

    chetan312

    Joined:
    Mar 12, 2017
    Posts:
    6
    i dont know what to say but this is happening and this issue is very frustrating and you cant downgrade your project from unity latest to previous version . If you do so , there will be a lot of other errors. Fortunately i made a copy my project few days ago.
     
  24. Konaju-Games

    Konaju-Games

    Joined:
    Jul 11, 2012
    Posts:
    17
    I can definitely understand the frustration. Unity do need to make their error reporting and handling more human-friendly. Programmers are people too. :)
     
  25. edvinas-mandravickas

    edvinas-mandravickas

    Unity Technologies

    Joined:
    Aug 8, 2019
    Posts:
    3
    Hi guys. I'm sorry to hear you are having this issue. Perhaps I'll be able to help. Could you check and make sure that these custom gradle files (if you are using them) has these bits of code:

    gradleTemplate.properties
    (could be added anywhere in the file)
    Code (CSharp):
    1.  
    2. unityStreamingAssets=.unity3d**STREAMING_ASSETS**
    launcherTemplate.gradle and mainTemplate.gradle
    (both files has to have this bit of code inside android object)
    Code (CSharp):
    1. aaptOptions {
    2.         noCompress = ['.ress', '.resource', '.obb'] + unityStreamingAssets.tokenize(', ')
    3.         ignoreAssetsPattern = "!.svn:!.git:!.ds_store:!*.scc:.*:!CVS:!thumbs.db:!picasa.ini:!*~"
    4.     }
    If you have any additional file types which should be included into noCompress list, add the to the array iside launcherTemplate.gradle like this:
    Code (CSharp):
    1. aaptOptions {
    2.         noCompress = ['.ress', '.resource', '.obb', '.yourExt1', '.yourExt2'] + unityStreamingAssets.tokenize(', ')
    3.         ignoreAssetsPattern = "!.svn:!.git:!.ds_store:!*.scc:.*:!CVS:!thumbs.db:!picasa.ini:!*~"
    4.     }
    This should be enough to make your custom gradle files up-to-date and skip upgrade procedure. Hope it will help :)
     
    DoBrain likes this.
  26. Konaju-Games

    Konaju-Games

    Joined:
    Jul 11, 2012
    Posts:
    17
    Thanks for responding @edvinas-mandravickas. Hopefully you are part of the team that are able to improve this part of the Unity build process to make it friendlier and ideally have the editor make the relevant change since the change is so specific (but undocumented!). That is exactly what I worked out above a few weeks ago, and what fixed the issue for me. @chetan312 said he is having the same issue, and has made the change you describe, but his gradle templates keep getting changed back.
     
  27. Qhuhuit

    Qhuhuit

    Joined:
    Feb 17, 2018
    Posts:
    13
    Hello, I did your steps and still cannot build, with this error :
    Code (JavaScript):
    1. FAILURE: Build failed with an exception.
    2.  
    3. * Where:
    4. Build file 'C:\xxx\Temp\gradleOut\launcher\build.gradle' line: 98
    5.  
    6. * What went wrong:
    7. A problem occurred evaluating project ':launcher'.
    8. > Could not get unknown property 'unityStreamingAssets' for object of type com.android.build.gradle.internal.dsl.AaptOptions.
     
  28. Konaju-Games

    Konaju-Games

    Joined:
    Jul 11, 2012
    Posts:
    17
    This is what my
    aaptOptions
    section in
    launcherTemplate.gradle
    looks like.

    Code (JavaScript):
    1.     aaptOptions {
    2.         noCompress = ['.ress', '.resource', '.obb'] + unityStreamingAssets.tokenize(', ')
    3.         ignoreAssetsPattern = "!.svn:!.git:!.ds_store:!*.scc:.*:!CVS:!thumbs.db:!picasa.ini:!*~"
    4.     }**SIGN**
    5.  
     
  29. Qhuhuit

    Qhuhuit

    Joined:
    Feb 17, 2018
    Posts:
    13
    I only need to use a custom main gradle template, but yes, I have exactly the same aapt Options and yet a build failure as mentioned above.
    I have tried to build a blank new project with nothing but the asset that needs this custom gradle template and it failed the same way. So I'm rolling back to 2020.1. until either Unity or this asset developer fix the issue.
     
  30. Rabadash8820

    Rabadash8820

    Joined:
    Aug 20, 2015
    Posts:
    25
    @edvinas-mandravickas Thanks for your summary of steps that need to be taken in Unity 2020.2.

    Unfortunately, all of this has reinforced the fear that I had when I enabled custom Gradle templates back in Unity 2019.4 (for use with the Appodeal SDK); namely, that my custom templates would not stay up-to-date with the requirements of later Unity versions. Generally speaking, is there some forum thread, manual page, or other resources that we can check for changes to these Gradle files between Unity versions? Neither the release notes nor upgrade steps in the manual have this information.
     
unityunity