Correct video format for video playback ?

Hi CatherineP,
Please keep different issues in separate threads. Otherwise they become hard to manage.

 

Hi komodor,
You can find more information about the VideoPlayer and other features here:

https://forum.unity3d.com/threads/preview-of-some-5-6-features.445778/

 

Hi all,

Thanks for having a look at our new VideoPlayer and giving feedback!

As you are realizing, full documentation has yet to be done so in the mean time it'll be my pleasure to assist.

About codecs:

komodor, your question about supported formats is a frequent one and the answer is multifacteted. Generally speaking, the supported codecs are:

• H.264 for video with AAC for audio, typically found in .mp4, .m4v and .mov files.
• VP8 vor video and Vorbis for audio, typically found in .webm files.

These are the codecs that will work in most situations, both in the editor and in the target platform. H.264 is often the best choice as it is supported in hardware on most platforms - with Android also supporting VP8 in hardware. However, we're trying to have support for whatever the editor platform supports. For instance:

• On Windows, we're using Media Foundation for reading/transcoding movie files.
• On OSX, we're using AVFoundation for reading/transcoding movie files.
• On all editor platforms, including Linux, we have a software version of the VP8 video codec and Vorbis audio codec.

So in theory, all formats handled by these APIs are supported in the editor. I'm saying "in theory" because we do have some filtering based on filename extensions that may go away in the future (not sure yet). At this time, here's the list:

• All editor platforms: ogv, vp8, webm
• Additionally, on Windows and OSX: mov, dv, mp4, m4v, mpg, mpeg
• Additionally, on Windows: avi, asf, wmf

We will try to lift some restrictions and simply let the platform's native APIs tell us whether a given file format is supported, but this ties into how Unity's asset pipeline works so we cannot simply claim that every filename extension "might be a movie file", hence the current limitation.

All target platforms have their own different capabilities that we'd like to encode in the editor so that you don't have to keep track of them. Each one has different resolution, bitrate, codec, profile and audio restrictions and recommendations. Using Android as an example, the official recommendations are https://developer.android.com/guide/appendix/media-formats.html. But every Android device / model / OS version has its own specifics so it wouldn't be appropriate for us to impose limitations based on the general recommendations. Some Android devices like the Nvidia Shield TV have no problem playing 4K videos. We could emit warnings when a movie is found to be "out of spec", at best.

Then comes the discussion about whether you'll ask the movie to be transcoded during import. When the importer transcodes the movie, it will encode it in either H.264 or VP8 based on what the target platform is know to support best (this is the "Auto" mode). For instance on iOS, like on many platforms, this would be H.264 but on Linux this would be VP8. Again here, feedback is incomplete so you don't get to see what "Auto" translates into for the target platform; this will come. You can also override the codec choice if you have a good reason. A good reason could be: on Windows, H.264 works in hardware most of the time, but on certain Windows versions it won't work above HD. Forcing the codec to VP8 in this case means we'll revert to the software codec, which works identically on all Windows variants, but comes at a higher CPU usage cost. There's also the reality that transcoding takes time, and often a huge amount of it depending mostly on resolution. Furthermore, our current asset pipeline doesn't support aborting the transcode task, which is a (known) major flaw. We'll try to address this throughout the beta cycle.

So if you decide to not transcode your movie file, then you're essentially telling Unity that you know what you're doing and that you know the movie in its current format is supported on the target platform. This is the default choice when you import a movie, which one may find questionable since it may lead to surprises. But given the time it takes to transcode most videos, we still prefered to have this as the default than to send Unity into a uninterruptible transcode each time movie assets are added/reimported.

About error reporting:

As CatherineP points out: error reporting is most of the time absent; we are aware and are working on this. The console messages, when present, are vague at best but this will be improved. One thing that makes good error reporting harder is that many of the APIs we are using will only report errors asynchronously. The request to start decoding a given video will succeed, but a background event gets fired later on telling us about a failure. So one cannot rely on proper exception throwing to catch errors. This is why the scripting API has a "errorReceived" C# event: we hope to funnel as many error descriptions through there as possible, as well as through console logging.

About audio output:

Audio can be read from all supported movie files with the notable exception of .ogv. This has just been completed internally and will appear in a later beta version.

It's understandable that many will find the audio output hard to figure out initially as the implementation is not yet completed. Here's the general idea behind the audio output modes:

• Audio Source: This causes the movie's audio to be sent to the specified AudioSource(s): one audio source per multi-channel audio track in the movie. This means the movie's audio can be processed through Unity's audio engine alongside any other sound in the game. The AudioSource you assign to the track can then be configured normally for sending the samples to the wanted mixer group and all AudioSource controls will work as expected. When an AudioSource is used this way, its AudioClip field can still be used and both the AudioClip and the VideoPlayer's contribution will be mixed together.
• Direct: Some APIs - notably AVFoundation on OSX - make it possible to output audio directly from the movie decoding stage. It seemed convenient to expose this capability if all that is needed is to output the audio from the movie without further processing from Unity. This is what the "Direct" mode is about. It will not be possible to support this on all platforms, but we may end up simulating it for completeness sake, and also because this would still represent an economy over forcing the user to have an AudioSource object.

About HoloLens:

And to answer CatherineP's question about HoloLens specifically: no work has been done to support this platform yet. The current platform lineup is: Windows, OSX, Linux, Android, iOS, WebGL, Tizen and (in the next beta) XboxOne. Although one of the long term goals is to support better the VR workflows, this initial release is more about general hardware-assisted video playback than specifically about VR features/platforms. I've asked a developer on the HoloLens team to see if he can get videos to play in there. We'll add adequate error reporting if needed, or provide a fix if no major development is required. Thanks for trying it out and bringing this to our attention!


Again, thanks for experimenting with this and for your patience while we're working on the remaining bits and pieces!

Dominique

 

Thanks for your good words Studiomaurer!

We're also interested in hearing what you have to say when things work as expected! So if you can let us know what platform and types of videos you're using, other users will be able to refer to your experience to know what they can expect!

 

CatherineP : I just educated myself on the type of url that is needed to read the stream from the HoloLens, using instructions from this web page : https://blog.kloud.com.au/2016/09/01/streaming-hololens-video-to-your-web-browser/

The type of URL that has to be used (e.g. taken from the page: https://[USERNAME]:[PASSWORD]@192.168.1.6/api/holographic/stream/live_high.mp4?holo=true&pv=true&mic=true&loopback=true ) may very well trip our internal logic. As soon as we can spend time on the device, we'll see if this is the case.

Another thing that we didn't have time to try yet is "live" streams, i.e. streams for which there is no known duration. This also could be problematic. At this point, I feel we could continue chatting in another thread (if you don't mind opening one) as this doesn't seem codec-related and we'll be able to have HoloLens-specific discussions. It *is* hard to get a confirmation that the stream really is using H.264, but the fact that the filename uses the .mp4 extension is a strong hint. I went from the Microsoft doc for the HoloLens portal: https://developer.microsoft.com/en-us/windows/holographic/device_portal_api_reference

I'll let you know when we have time to experiment on our side!

 

Hi @hangemhigh , yes it does work for the devices we were able to test with. Please file a bug giving the details of what you are doing that leads to a crash as well as the full device description so we can reproduce and fix.

Sorry it's not working for you; hopefully we'll get you a fix quickly!

Dominique

 

Yes! Native vp8 alpha is supported as webm does have a standard way to encode alpha. For instance, Chrome supports this. When importing, depending on the target device, you may choose to not transcode at all (fine for desktop platforms) or transcode (and enable Keep Alpha in the importer) if you are targeting anything else. For instance, although Android supports VP8 in hardware, it doesn't support webm files that have vp8 video with alpha. In these cases, we store the alpha side-by-side with the color channels and re-assemble the RGBA result during decoding.

 

Yeah... Holiday period has slowed down our pace: we'll get to it asap!

 

Hey ileonidmas!

Sorry for the bumpy ride. I'm using a Nexus 6 locally without problems, and our test suites also run consistently on Nvidia Shield TV. Our Android QA team has also run it on other devices; I'll post more details here when people are back from vacation.

We are aiming to support Jelly Bean (API level 16, 4.1.x) and up, but I found a bug in the pre-Lollipop (API level < 21, pre-5.0) that causes intermittent trouble so stay on API level >= 21 for now if possible.

As for how to use videos, importing them as video clips (no need to transcode as long as your videos are H.264 or VP8), or streaming them via http works fine or even using file:// (if you have a video on the device's local storage). You can also put them in StreamingAssets; we have tests that verify this works.

For instance, you can try using this URL to see if you can get streaming to work: http://www.quirksmode.org/html5/videos/big_buck_bunny.mp4

Just create a VideoPlayer in your scene and set its Source to "URL" and paste this URL in the field.

It could be that the OS version you are using has specifics that we haven't encountered yet; please report back here (or preferably in a Android-specific video player thread so we keep things separated) and of course, feel free to file bugs if you have enough info for us to work from!

Hope to get this under control soon,

Dominique

 

Hey Cascho01!

Thanks a lot for reporting this. The VideoClipImporter does not offer color space conversion for now. But we do have a lot of image processing utilities ready that could take care of this (at the cost of one generation loss). I'll see if this can be added to the importer in time or 5.6.

A better approach however would be for this to be done in a shader. We already have a shader that performs the YUV->RGB conversion and it could very well perform colorspace conversion. We are looking at opening up this shader in order to offer other transformations such as 360 projections, stereoscopic management etc. But this is definitely not for 5.6 so in the mean time you'll have to either pre-process your videos into a linear colorspace, or manage the color conversion yourself by writing a BT.709-to-linear shader that receives the decoded video (presumably from a RenderTexture) if this is how you intend to use them.

I'll try and get back to you with better info/workarounds if we have time to fit this into 5.6. Feel free to open a thread about the VideoPlayer's (lack of) color management if you want to keep discussing this; input appreciated!

Dominique