UnityPlayer.displayChanged() method to add a Surface as an additional display

Has anyone successfully used the displayChanged() method in the UnityPlayer Java class to add a Surface as an additional display that can be rendered to? I'm trying to use this with a Surface created for input to a MediaCodec encoder. The only reference I can find is a single forum post that references a single dot point in Unity 5.x release notes, but this looks like the perfect feature I need to get a rendered image from Unity into the MediaCodec encoder.

When the app starts, Display.displays returns one display (the phone screen). I call UnityPlayer.displayChanged(1, encoderSurface), and a short time later Display.displays will return two displays. The Display.onDisplaysUpdated event never got triggered however, so I check each update for a change to the length of the Display.displays array. Then I enable a camera and set its targetDisplay to the index of the new display. I don't appear to be getting anything though.

I'm still working on it, but if anyone has used this successfully it would be great to know I'm not running up a dead-end alleyway. And Unity, please document this stuff. Please.

 

Odd behaviour found. The input surface created by the MediaCodec encoder is 720x1280, as expected because that is the size we requested for the encoder. When the surface is added to Unity's displays list, the Display object returns a rendering and system size of 1080x2057. The main display has a rendering size of 1080x2204 and system size of 1080x2340, but this second display should be 720x1280, not the oddball size it is returning.

 

Thanks for that. For reference, we are using 2019.4.11f1. I changed the code to use Camera.SetTargetBuffers, but I haven't been able to verify it's working yet. The MediaCodec encoder that created the surface still doesn't appear to be producing any output yet.

The odd rendering and system sizes returned for the new display are still there.

 

The MediaCodec encoder does not produce any output when the surface has not been rendered to, and that is what appears to be the case here. Is UnityPlayer.displayChanged() verified to work and able to be rendered to?

 

I decided to try the project in Unity 2020.2 to see if it worked better. No luck since it breaks on deterministic compilation (had to turn that off) and a very unhelpful error for our custom Gradle build template.
"mainTemplate.gradle file is using the old aaptOptions noCompress property definition which does not include types defined by unityStreamingAssets constant."
but nowhere does the editor or the docs say what the fix is or how to fix it. Every avenue I take I find a dead-end.

 

Hi.
Unity 2020.2
We cant build on Android too with the same error: "mainTemplate.gradle file is using the old aaptOptions noCompress property definition which does not include types defined by unityStreamingAssets constant." Could someone from Unity help us?

 

With regards to the mainTemplate.gradle error. I had the same thing, so I moved the file and let Unity3D regenerate the default one.

It added a new noCompress line to the aaptOptions section:

Code (JavaScript):

1.  
2. aaptOptions {
3.         noCompress = ['.ress', '.resource', '.obb'] + unityStreamingAssets.tokenize(', ')
4.         ignoreAssetsPattern = "!.svn:!.git:!.ds_store:!*.scc:.*:!CVS:!thumbs.db:!picasa.ini:!*~"
5.     }**PACKAGING_OPTIONS**
6.  

I copied the noCompress line to my original mainTemplate.gradle file and I no longer had this error (I did however get other errors).

 

I did the same, but in the console you can see that the error regarding unityStreamingAssets, so not sure that this could help =( I think we need to wait for answer from someone from Unity.

 

Old (generated from a 2019.4.11f1 project)
In the

aaptOptions

section, the

noCompress

line is either missing or contains

Code (CSharp):

1. noCompress = ['.unity3d', '.ress', '.resource', '.obb'**STREAMING_ASSETS**]

New (freshly generated from a new 2020.2.0f1 project)
The

aaptOptions

section contains

Code (CSharp):

1. noCompress = ['.ress', '.resource', '.obb'] + unityStreamingAssets.tokenize(', ')

Notice the change to how the Streaming Assets extensions are added?

The error that 2020.2.0f1 gives when confronted with the earlier

aaptOptions

section (missing the

noCompress

line or the older format) is



No hint as to how to fix the problem or any link to documentation describing what is expected, not even in the Console error output. Perhaps it could take a hint from a later different error, also related to Gradle files, that describes what to do and offers to fix the problem.



It then repeats the earlier

aaptOptions

error for the launcherTemplate.gradle file.

That should be all of the information you need. I do not know what else to give.

 

Same here. Would be nice to have some documentation or instructions on what to do.

 

Back to the original issue (adding an Android surface as an additional display), after making the change to the gradle files to get the project to build in 2020.2.0f1, the rendering to the extra display appears to work. The MediaCodec encoder is seeing updates to the input surface and producing encoded h264 output.

The summary for those just catching up and for future developers searching for these errors:

1. The feature to add an Android surface as an additional display appears to require Unity 2020.2.0f1. It does not appear to work correctly in Unity 2019.4.11f1 (I tried for days to get it working), even though the feature was added back in the Unity 5.x series.

2. If upgrading to Unity 2020.2.0f1 from an earlier Unity project that has a custom Android gradle file, change the existing

noCompress

line in the

aaptOptions

section (or add it if it is missing) to the following

Code (CSharp):

1. noCompress = ['.ress', '.resource', '.obb'] + unityStreamingAssets.tokenize(', ')

 

Thanks, that's exactly what I needed!

 

Hi!

So, have you found the way to resolve the issue after adding this string? Could you share it please? )

 

Thanks for your help!

 

same error ."mainTemplate.gradle file is using the old aaptOptions noCompress property definition which does not include types defined by unityStreamingAssets constant." Is anyone found a solution.

 

when trying to build an apk . Android gradle file changes to its previous version.

 

If you have custom gradle files, you need to make the change listed above in both launcherTemplate.gradle and mainTemplate.gradle.

 

yeah i have custom gradle files and i changed both launcherTemplate.gradle and mainTemplate.gradle but when i tried to make a build then the same error occurred and mainTemplate.gradle changed to their previous version.
i dont know the exact reason but this issue starts when i ugraded my unity editor to latest

 

So, as for the original issue of adding a Surface as an additional display on Android, I do now have it working. But it was not without a lot of frustration and cursing along the way. And as it seems this will be the only source of documentation of this feature for the near future, I'll try to make it decent.

Firstly, you do need to be using Unity 2020.2.0f1 or later. It just does not work in earlier versions. If upgrading from earlier versions and you have custom gradle files, make sure you apply the fix listed above.

I'm using C++ for my MediaCodec implementation, but you should be able to do the same if you are using Java plugins.

I originally implemented this using a custom Activity derived from UnityPlayerActivity solely to get access to the protected mUnityPlayer field. That is ugly and messy, so I have since worked out how to use JNI to avoid any custom Activity class. This code is currently assuming the additional display will be at index 1 (the second display). This may cause an issue if an Android device already has more than one display. Who knows what an Android device will have these days. It would make more sense to pass in the current length of the Display.displays array property and use that as the index of the new display.

Code (csharp):

1. static void displayChanged(jobject surface)
2. {
3.     // Get the current activity from the static field in the UnityPlayer class.
4.     const jclass playerClass = jniEnv->FindClass("com/unity3d/player/UnityPlayer");
5.     const jfieldID currentActivityID = jniEnv->GetStaticFieldID(playerClass, "currentActivity", "Landroid/app/Activity;");
6.     jobject currentActivity = jniEnv->GetStaticObjectField(playerClass, currentActivityID);
7.  
8.     // Get the current UnityPlayer instance from the current activity in the mUnityPlayer field.
9.     // This field is protected in Java, but apparently everything is accessible via JNI.
10.     const jclass activityClass = jniEnv->GetObjectClass(currentActivity);
11.     const jfieldID unityPlayerID = jniEnv->GetFieldID(activityClass, "mUnityPlayer", "Lcom/unity3d/player/UnityPlayer;");
12.     jobject unityPlayer = jniEnv->GetObjectField(currentActivity, unityPlayerID);
13.  
14.     // Call the displayChanged method on the UnityPlayer instance
15.     const jmethodID displayChangedID = jniEnv->GetMethodID(playerClass, "displayChanged", "(ILandroid/view/Surface;)Z");
16.     jboolean result = jniEnv->CallBooleanMethod(unityPlayer, displayChangedID, 1, surface);
17. }
18.  

I call this after I create the encoder input surface.

Code (csharp):

1.     // Create the surface that will feed the encoder
2.     error = AMediaCodec_createInputSurface(encoder_, &inputSurface_);
3.     if (error != AMEDIA_OK)
4.     {
5.         stopEncoder();
6.         return -1;
7.     }
8.    
9.     // Get the Java Surface object handle to the native window
10.     jobjectSurface_ = ANativeWindow_toSurface(jniEnv, inputSurface_);
11.     // Add the encoder input Surface as an additional display for Unity
12.     displayChanged(jobjectSurface_);
13.  

This will start the asynchronous process of adding the Surface as an additional display. Now, you would expect the Display.onDisplaysUpdated event to be triggered when the additional display is added or removed. No, it does not. You must regularly check the length of the Display.displays array property and respond appropriately when the length changes from the previous check.

Code (CSharp):

1.     void Update()
2.     {
3.         // Detect when encoder input display is added or removed
4.         if (_displayCount != Display.displays.Length)
5.         {
6.             _displayCount = Display.displays.Length;
7.             Display_onDisplaysUpdated();
8.         }
9.     }

Now we need to tell the relevant camera which display to render on. The call to display.SetRenderingResolution(w, h) may or may not be required. I was seeing some odd display resolutions being reported by Unity for this additional display, even though the surface it was given was of the correct dimensions.

Code (CSharp):

1.     void Display_onDisplaysUpdated()
2.     {
3.         if (_displayCount > 1)
4.         {
5.             int index = _displayCount - 1;
6.             Debug.Log("Activating additional display");
7.             var display = Display.displays[index];
8.             // We are rendering to a portrait screen, hence taller than wide
9.             display.SetRenderingResolution(720, 1280);
10.             // The display is apparently already activated, but we call this anyway
11.             display.Activate();
12.             _encoderInputCamera.targetDisplay = index;
13.             _encoderInputCamera.gameObject.SetActive(true);
14.         }
15.         else
16.         {
17.             _encoderInputCamera.gameObject.SetActive(false);
18.         }
19.     }

The MediaCodec encoder will now start seeing input and producing h.264 packets of the rendered output. Yay!

But the fun is not over yet. No, there are other gotchas in store when encoding video. Despite what the documentation states, setting Application.targetFrameRate had no effect on my Google Pixel 4a. The app will always try to render at the refresh rate of the screen (Screen.currentResolution.refreshRate, though what does Screen and currentResolution mean when you have multiple displays?). In my case, it always renders at 60fps regardless of what I set Application.targetFrameRate to. When creating a MediaCodec encoder, you tell it what framerate you want the video stream to play at. This will have to match the framerate of the Unity app because the rendering of the additional display is feeding the encoder with video frames. To get different framerates, I needed to set QualitySettings.vSyncCount. By default, this is 0 which means "no sync", but on mobile devices it is always synced to the refresh rate, so 0 is effectively the same as 1. Set QualitySettings.vSyncCount to 2 to render at half the refresh rate (30fps in my case), 3 to render at one third (20fps), 4 for a quarter (15fps) and so on. It is not the cleanest solution (it also blocks updates), so I'm still working on that.

I think that is everything. It has been incredibly frustrating over the past several weeks but it is finally working. @florianpenzkofer does this look like how it should be working?

 

This gradle upgrade issue probably deserves its own thread.

It should not be changing or reverting that line in the gradle file while attempting to build the project. @Tomas1856 do you have any additional thoughts on this for @chetan312 ?

 

i dont know what to say but this is happening and this issue is very frustrating and you cant downgrade your project from unity latest to previous version . If you do so , there will be a lot of other errors. Fortunately i made a copy my project few days ago.

 

I can definitely understand the frustration. Unity do need to make their error reporting and handling more human-friendly. Programmers are people too.

 

Thanks for responding @edvinas-mandravickas. Hopefully you are part of the team that are able to improve this part of the Unity build process to make it friendlier and ideally have the editor make the relevant change since the change is so specific (but undocumented!). That is exactly what I worked out above a few weeks ago, and what fixed the issue for me. @chetan312 said he is having the same issue, and has made the change you describe, but his gradle templates keep getting changed back.

 

Hello, I did your steps and still cannot build, with this error :

Code (JavaScript):

1. FAILURE: Build failed with an exception.
2.  
3. * Where:
4. Build file 'C:\xxx\Temp\gradleOut\launcher\build.gradle' line: 98
5.  
6. * What went wrong:
7. A problem occurred evaluating project ':launcher'.
8. > Could not get unknown property 'unityStreamingAssets' for object of type com.android.build.gradle.internal.dsl.AaptOptions.

 

This is what my

aaptOptions

section in

launcherTemplate.gradle

looks like.

Code (JavaScript):

1.     aaptOptions {
2.         noCompress = ['.ress', '.resource', '.obb'] + unityStreamingAssets.tokenize(', ')
3.         ignoreAssetsPattern = "!.svn:!.git:!.ds_store:!*.scc:.*:!CVS:!thumbs.db:!picasa.ini:!*~"
4.     }**SIGN**
5.  

 

I only need to use a custom main gradle template, but yes, I have exactly the same aapt Options and yet a build failure as mentioned above.
I have tried to build a blank new project with nothing but the asset that needs this custom gradle template and it failed the same way. So I'm rolling back to 2020.1. until either Unity or this asset developer fix the issue.

 

@edvinas-mandravickas Thanks for your summary of steps that need to be taken in Unity 2020.2.

Unfortunately, all of this has reinforced the fear that I had when I enabled custom Gradle templates back in Unity 2019.4 (for use with the Appodeal SDK); namely, that my custom templates would not stay up-to-date with the requirements of later Unity versions. Generally speaking, is there some forum thread, manual page, or other resources that we can check for changes to these Gradle files between Unity versions? Neither the release notes nor upgrade steps in the manual have this information.

 

Hello, I did your steps and still cannot build, with this error :

Code (CSharp):

1. Note:...\Temp\gradleOut\unityLibrary\src\main\java\com\unity3d\player\UnityPlayerActivity.java uses or overrides a deprecated API.
2. Note: Recompile with -Xlint:deprecation for details.
3.  
4. FAILURE: Build failed with an exception.
5.  
6. * What went wrong:
7. Execution failed for task ':launcher:checkReleaseDuplicateClasses'.
8. > 1 exception was raised by workers:
9.   java.lang.RuntimeException: Duplicate class com.unity3d.ads.BuildConfig found in modules jetified-UnityAds-runtime.jar (:UnityAds:) and jetified-com.unity3d.ads.unity-ads-3.4.8-runtime.jar (:com.unity3d.ads.unity-ads-3.4.8:)
10.   Duplicate class com.unity3d.ads.IUnityAdsListener found in modules jetified-UnityAds-runtime.jar (:UnityAds:) and jetified-com.unity3d.ads.unity-ads-3.4.8-runtime.jar (:com.unity3d.ads.unity-ads-3.4.8:)
11.   Duplicate class com.unity3d.ads.UnityAds found in modules jetified-UnityAds-runtime.jar (:UnityAds:) and jetified-com.unity3d.ads.unity-ads-3.4.8-runtime.jar (:com.unity3d.ads.unity-ads-3.4.8:)
12.  ...