[Released] AVPro Video - complete video playback solution

Thanks for the reply. I will change our implementation accordinlgy. We are building an AR experience. It tells a story about a certain exhibition piece. The story is told via video; playing in multiple 2D planes in the 3D space; creating a paralax efect.

I'll make sure to send some links when the exhibition opens

 

Do I have the opportunity to get the wav law data before outputting them to Unity or Sound Device?
Than I can modify them to be louder...

 

Is it possible to reload MediaPlayer options like "Android API" and "use TBE audio" on runtime?

 

Hey guys, thanks for your pointers for this. Unfortunately your suggestions 2) 3) and 4) don't help with the problem, so we are now upgrading the project to the latest AVPro in case that solves it.

Worryingly, we have also found that the Lenovo Mirage headsets still update their Daydream version even when automatic updates are disabled - only 3 out of our 11 headsets are still usable until we fix this bug! :O

I would be interested to know if anyone else can reproduce this on the current version of Daydream.

 

Yeah we're still on ExoPlayer 2.8.4....2.9.0 will come soonish.

Which version of AVPro Video are you using? We recently released 1.9.5 which has some related Android fixes.

Thanks,

 

The current live version is 1.8.9, so I'll see what happens when I update to 1.9.5. Thanks!

 

Okay, we now have the latest AVPro in our project and can confirm that the problem still exists.
Will continue testing and report back - any ideas in the meantime would be appreciated.

 

that's a shame....I don't suppose Lenovo / Google have a list of what changed in that version?

 

I'm sure that streaming 5 videos at once is beyond the intended design of most phones
Phone hardware only has limited GPU video decoding capabilities, and memory is very low....iOS devices in particular suffer from memory issues when playing videos - they tend to allocate a lot of memory for playing a single video.

Firstly I would highly recommend making sure that you're using the latest version of AVPro Video (version 1.9.5) as many improvements have been made. Also on iOS make sure that you're using the YCbCr option as this helps to save memory for video decoding. Then on Android, if you're still having issues, try switching the API to MediaPlayer from ExoPlayer. All of these options are in MediaPlayer > Platform Specific > Android/iOS.
In general you should also try to make your Unity app use less memory. There are various documents available to tips on how to do this. For one thing you should go into Graphics settings and remove all built-in shader types that you don't need. Also in auto-graphics API options, if there are two APIs listed, try only keeping one of them.

I see that you're using a very old version of the plugin (1.6.6). You should upgrade to the latest version (1.9.5) as we've made many improvements to memory usage and other things.

Thanks,

 

Yes that's true...having a single display is best. Hope you're winning there...

Thanks,

 

Kind of..On Windows desktop (only) we support playing back the audio via Unity (in MediaPlayer > Platform Specific > Windows > Audio Output). This uses a script called AudioOutput. You could modify this script to boost the volume.

Thanks,

 

Currently no...you have to remove the MediaPlayer component and recreate it if you want to change those options. Those options change a lot of fundamentals of how the player is created, so we can't switch them dynamically. In a future version when we improve the API we will try to make all of these options changable.

Thanks,

 

Documentation for the Daydream platform let alone the Lenovo Mirage itself is horrendously thin...we have tried to contact their support about it but get no response. We are going back through all of our code and project settings to try and fix this by stabbing in the dark - but thanks for your help!

 

First thing I would try: disable multi-threaded rendering

 

Hi,

I found following exception occuring on the android while opening the video clip. My devices are Galaxy S7(OS 8) and Pixel 2(OS 9) and Pixel 3(OS 9). Most of exeptions occurred on Pixel 2,3.
The version number of avpro is 1.9.5

I want to fix this issue as soon as possible because it makes crashes.
Let me know what should I do or if you need more detailed information.

Thanks in advance.

 

Thanks, I will try it.

 

How often does the crash happen?
Are you able to replicate it with one of the demo scenes?
Are you using the FB360 Audio feature?

Thanks,

 

Hi Andrew,

We recently updated our AVPro video to 1.9.5, and since the update we have been unable to build for mobile because of a shader error.

Code (CSharp):

1. Shader error in 'VR Composers': 'Convert420YpCbCr8ToRGB' : no matching overloaded function found; 'constructor' : not enough data provided for construction at line 168 (on gles)
2.  
3. Compiling Vertex program with MONOSCOPIC STEREO_DEBUG_OFF HIGH_QUALITY_OFF APPLY_GAMMA_OFF USE_YPCBCR
4. Platform defines: UNITY_NO_DXT5nm UNITY_NO_RGBM UNITY_ENABLE_REFLECTION_BUFFERS UNITY_FRAMEBUFFER_FETCH_AVAILABLE UNITY_NO_CUBEMAP_ARRAY UNITY_NO_SCREENSPACE_SHADOWS UNITY_PBS_USE_BRDF3 UNITY_NO_FULL_STANDARD_SHADER SHADER_API_MOBILE UNITY_HARDWARE_TIER1 UNITY_COLORSPACE_GAMMA

And the same on Gles3 and Vulkan

Shader error in 'VR Composers': undeclared identifier 'Convert420YpCbCr8ToRGB' at line 168 (on gles3)

The faulty line of code:

fixed4 col = fixed4(Convert420YpCbCr8ToRGB(ypcbcr), 1.0);

which is in the

Code (CSharp):

1. #if USE_YPCBCR
2.     #if SHADER_API_METAL || SHADER_API_GLES || SHADER_API_GLES3
3.                 float3 ypcbcr = float3(tex2D(_MainTex, uv).r, tex2D(_ChromaTex, uv).rg);
4.     #else
5.                 float3 ypcbcr = float3(tex2D(_MainTex, uv).r, tex2D(_ChromaTex, uv).ra);
6.     #endif
7.                 fixed4 col = fixed4(Convert420YpCbCr8ToRGB(ypcbcr), 1.0);
8. #else
9.                 fixed4 col = tex2D(_MainTex, uv);
10. #endif

block.

Playing in the Unity editor works fine without any errors.

Build device is a samsung galaxy S8 edge running android 8.0

Do you know how to fix this?

The VR Composers shader is basically the AVProVideo/Unlit/Opaque android OES ONLY shader, but with a small edit to it. But the same error occurs using the standard Unlit/opaque shader

 

Hello Again,

I am implementing the option 2 as suggested.
There is single video file playing in one MediaPlayer.

We try to display different parts of the video on different planes using offset and scale of ApplyToMesh component.
Please see the drawing below. (Main video with dimensions on the right and planes on the left)

I cannot get the offset and scale values for each plane correct.
Probably I don't have the correct understanding.
Can you help me understanding how those work?
For example what would be the offset and scale values of the ApplyToMesh components of the planes in the drawing?

Thanks in advanced.
Cheers,
Doruk

 

Hey, thanks for your help. I can confirm this does not make a difference I'm afraid.

 

Hi,

Yes in 1.9.5 we removed Convert420YpCbCr8ToRGB and replaced it with ConvertYpCbCrToRGB..which now has an extra matrix parameter for the transform. This was to improve the accuracy of the conversion. You will need to update any of your customised shaders to use this new method, or copy the old method into your shader. Here is the method for reference:

Code (CSharp):

1. INLINE FLOAT3 Convert420YpCbCr8ToRGB(FLOAT3 ypcbcr)
2. {
3. #if 1
4.     // Full range [0...255]
5.     FLOAT3X3 m = FLOAT3X3(
6.         1.0,  0.0,      1.402,
7.         1.0, -0.34414, -0.71414,
8.         1.0,  1.77200,  0.0
9.     );
10.     FLOAT3 o = FLOAT3(0.0, -0.5, -0.5);
11. #else
12.     // Video range [16...235]
13.     FLOAT3X3 m = FLOAT3X3(
14.         1.1643,  0.0,      1.5958,
15.         1.1643, -0.39173, -0.81290,
16.         1.1643,  2.017,    0.0
17.     );
18.     FLOAT3 o = FLOAT3(-0.0625, -0.5, -0.5);
19. #endif
20.  
21. #if defined(SHADERLAB_GLSL)
22.     return m * (ypcbcr + o);
23. #else
24.     return mul(m, ypcbcr + o);
25. #endif
26. }

Also in 1.9.5 we actually introduced a bug into the OES shader code..so please upgrade to 1.9.6 which will be released shortly.

Thanks,

 

Hi All,

AVPro Video version 1.9.6 has just been released!

You can find the updated version on the Asset Store and the free trial version on our website.

Changes include:

macOS

• Fixed a Metal rendering issue where the watermark didn’t display in the editor
• Fixed issue with Hap and Metal rendering in the editor
• Upgraded to latest Hap version (1.6.9)

iOS / tvOS

• Fixed a crash in iOS 9 and lower that could happen when closing a player
• Fixed packed transparent shaders to expand range correctly when using YpCbCr mode

Android

• Fixed bug where using the OES shader would cause video to display pink/black, this was a regression introduced in 1.9.5

General

• Shaders refactored

Thanks to everyone that reported bugs that were fixed in this release

Please report any issues here on the forum, or to our customer support email

Thanks,

 

I'm currently evaluating AVProVideo for a client project, but when using the latest trial version in Unity 2018.2.11f1 (on MacOS 10.12.6) every video in every sample scene just loops the first few frames. The audio plays fine.

Any idea what is wrong here?

>> Read the same problem on page 56. This a a known problem apparently. I switched to OpenGL, and everything is working ok now.

Still, would't it be a good idea to list known bugs or something. If I'm going to base a long-term project on this asset, I have to be sure about long-term support.

 

You're right - this is a known limitation of our plugin. We will add a section to the documentation that list these. This particular issue is only on macOS and only when using the Metal graphics API with a specific GPU (so far we've only heard about it on Macbook Pros). We haven't managed to fix it as we don't have the hardware to replicate it. For now the only option is to switch to OpenGL. Hopefully we'll be able to get this issue happening in our studio so we can work out the problem.

Out of interest, which version of AVPro Video are you using (it should print the version to the console window)? We just released 1.9.6 which we were hoping would fix this issue..

Thanks,

 

You should be able to set the Scale to 0.33333, 0.5 for all ApplyToMesh instances
Then you can adjust the offset value for each video, so:

TopLeft = 0.0 , 0.5
TopMiddle = 0.3333 , 0.5
TopRight = 0.6666, 0.5
BotLeft = 0.0 , 0.0
BotMiddle = 0.3333 , 0.0
BotRight = 0.6666, 0.0

Hope that helps,

 

Thanks for your answer. This test was with v1.9.6.

One more question:
The application I'm developing with AVPro Video will download video's from a backend and store them locally. The first version should support Windows and MacOS, later versions will also be released on Android and iOS. I'm searching for a way to store these video's while protecting them from usage in other players. I understand DRM is currently not supported, but the Android version supports a file offset feature. Is it possible to use this on Windows/MacOS/iOS as well, as a fast/easy protection? It would be better than nothing, and prevent playback for most users.

 

Thanks for the quick update with the OES shader fix, solved my issues

Just in case others run into this: After updating to 1.9.5(/6) from an earlier version the following dex error might occur (as it did in my case) when building for Android:

CommandInvokationFailure: Unable to convert classes into dex format. 

resulting in a build failure.

This is caused by the new "audio360-exo28.jar" being added in the Assets/Plugins/Android/ directory. You can remove the old "audio360-exo26.jar" to resolve this error.

 

@AndrewRH The Unity asset store doesn't appear to explicitly mention this, but how is the licensing handled for this asset? Is it one license per seat?
I'm working as lead developer on a project but will have to share the project with a colleague who may do some edits to the code base. Do we both need a license?

Great to have AVPro Video in the sale btw

 

Thanks for checking in with us.

On the Unity Asset store, I believe that all assets are site licensed (any number of developers at a single company site/location), except for editor-extensions, which are per-seat.

The terms are here:
https://unity3d.com/legal/as_terms

I'm no lawyer, so I don't know how it applies to you and your colleague in different locations.

Thanks,

 

Yes that's right, the old exo26 JAR should be deleted. Thanks for mentioning this. We will add it to the FAQ.

 

I'm afraid the file-offset feature is currently only on Android. We don't have support for this on other platforms and we don't have plans to add it in the near future.

 

Hello,
i got huge problems with tearing on an Intel Iris Plus 655 and Windows 10 while loading/switching Video. Resolution is 1920x1080.

It really looks terrible. I tried everything to force Vsync, but no success.

Have you got any tips for me?

Best regards

 

Could you give more details? Have a screenshot? Are you able to reproduce it using our demo scenes + media?

Thanks,

 

Hi, thank you for your quick response.

I attached an short video, only had to black out the corner In the video you can see the artefacts.

We tried with the demo scenes, same problem. We also have the problem with the demo media.

There seems to be a problem with the graphic card and unity/ AV Pro.

 

I've seen this sort of triangle artifact before when using Intel chipset for decoding. I always assumed it was some sort of optimisation in the Intel decoder...

 

I don't know if you're aware of it, but playing anything other than .NET 3.5 on Android has been broken since September.

https://issuetracker.unity3d.com/is...-when-executing-certain-sslstream-constructor

There are only 12 votes for this bug, but it's completely shut down the player on Android for me. Checking the log, you see:

Unable to lookup library path for 'libc', native render plugin support disabled.
Unable to find libc​

Is there another workaround I'm not aware of? Do you have a fix of your own? Going back to .NET 3.5 would be quite painful for me.

 

Is there a way to increase playbackrate to more than x4?

 

I wasn't able to replicate this.

I tried in Unity 2018.2.11f1, using AVPro Video 1.9.6, with IL2CPP enabled using .Net 4.x. It worked fine.

The only issue that I had was when enabling ARM64 architecture, I had to manually go to Plugins/Android/libs/arm64-v8a and set the platform of all of those files to Android, and their architecture to ARM64.

If you're using 1.9.6 and the above settings, and still having problems, please send us a copy of your project or a cut-down reproducible scene (www.renderheads.com/contact/) thanks,

 

It depends on the platform, video encoding, and video resolution, bitrate, frame-rate, etc.
In the UI we have limited it to 4x, but via the API you should be able to increase it more.

 

The crash occurs once in 5 times.
I can't reproduce it with demo scenes. (Some of demo scenes causes crashes but it caused by another issue)
I'm using 'facebook audience network' which require exoplayer. Could this be the cause? Is there anything I especially need to know to use these 'facebook audience network' and 'avpro video' together?

Thanks.

 

I made sure I am using the same settings you describe above, and I am getting the same error. Then I tried loading one of your demo scenes. That worked. What was the difference, then?

I narrowed it down to 3 things. You can take it from here...

1. Start with Demo 4
2. I use a stream source. Try https://videos3.earthcam.com/fecnetwork/tokyo1.flv/chunklist_w480902807.m3u8
3. Check the Persistent box
4. Change the Force File Format to Smooth Streaming
5. Build for Android
6. Run and watch nothing show up.

Expected: A live stream from Tokyo Harbor where the rabbit usually is

Actual: A blank video stream and 404 errors in logcat. Turn off Persistence and change the Force Type back to Unknown and you'll see Tokyo.

 

Hi Andrew

Thanks for the wonderful plugin. I am getting a serious hickups when opening HAP videos, even with relative small (~5mb) files. What is the best practise for reducing load duration?

All the best!

 

Please could you let me know what is the model of your machine? Ideally a screenshot of the About screen would really help. We're trying to narrow down which machines this happens on, as it isn't happening on any of our mac machines here. Thanks,

 

Do you mean when you load the file? Our DirectShow video player isn't asynchronous, so the main thread is blocked during loading. If you're using MOV, try swapping the container to AVI..or vice-verse, as the container type may affect loading time. Or if possible try using multiple MediaPlayer components, and pre-load all the Hap videos upfront.

 

I meant, how frequently can you get it to happen - every single time, every second time, once in a hundred?

If you can't reproduce it in the demo scenes, only in your own scenes, then yes it is likely that sometimes else is causing the problem. That facebook exoplayer extension does sound like a likely candidate. I have never used it before though. I wonder what version of ExoPlayer they're using. We use 2.8.4, so if they're using a different version then there could be a conflict. You could also try making AVPro Video use the MediaPlayer API instead of ExoPlayer. You can toggle this in MediaPlayer > Platform Specific > Android > Video API.

 

Ahh, I see - thanks for explaining. So the link you gave (m3u8) is a "HLS" stream, not "Smooth Streaming" which is a Microsoft stream and not very popular. When you force the stream type and it isn't correct, then it won't be able to load it. We will improve the Force File Format list so they are more descriptive in the next update. Thanks,

 

Does AVPRO android support using the video texture as a vertex shader (displacement) in ES3.1 (gearvr/oculus go)? We are having to abandon UE4 since they are commited now to a huge feature regression with their media textures.

 

Hi Tom,

Yes, the texture is just a normal texture...so you can use it for anything, including a displacement map, and access it in the vertex shader.

I didn't know UE4 was reducing media texture support - do you have a link?

Thanks,

 

@AndrewRH Function request: support external clock or external frame sequence number for synchronization.Thanks

 

Thanks Andrew! We decided to go for another direction ATM. But this will come handy in a different project for sure.
Cheers!