Search Unity

  1. We've closed the job boards. If you're looking for work, or looking to hire check out Unity Connect. You can see more information here.
    Dismiss Notice
  2. We're running great holiday deals on subscriptions, swag and Asset Store packages! Take a peek at this blog for more information!
    Dismiss Notice
  3. Check out our Unite Austin 2017 YouTube playlist to catch up on what you missed. More videos coming soon.
    Dismiss Notice
  4. Unity 2017.2 is now released.
    Dismiss Notice
  5. The Unity Gear Store is here to help you look great at your next meetup, user group or conference. With all new Unity apparel, stickers and more!
    Dismiss Notice
  6. Introducing the Unity Essentials Packs! Find out more.
    Dismiss Notice
  7. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice
  8. Unity 2017.3 beta is now available for download.
    Dismiss Notice

Correct video format for video playback ?

Discussion in '5.6 Beta' started by komodor, Dec 13, 2016.

  1. komodor

    komodor

    Joined:
    Jan 2, 2013
    Posts:
    31
    Hi,
    thanks for that video feature, it's actually pretty cool.

    Anybody can tell me how to encode videos to have them with sound? What are supported formats and codecs for this new videoplayer?

    Thanks in advance
     
  2. CatherineP

    CatherineP

    Joined:
    Apr 17, 2014
    Posts:
    3
  3. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    432
    Hi CatherineP,
    Please keep different issues in separate threads. Otherwise they become hard to manage.
     
  4. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    432
    Hi komodor,
    You can find more information about the VideoPlayer and other features here:

    https://forum.unity3d.com/threads/preview-of-some-5-6-features.445778/
     
  5. CatherineP

    CatherineP

    Joined:
    Apr 17, 2014
    Posts:
    3
    This is actually the same question: which codec does the feature support? (although I should have phrased it more precisely)

    I'm not sure if the Hololens is using H.264/AAC for its live streaming, or if it's a proprietary codec. But given Unity's positioning on the VR market, I'm hoping someone in the video team has considered that use case and can give me the answer.
     
  6. komodor

    komodor

    Joined:
    Jan 2, 2013
    Posts:
    31
    ty!

    edit: actually there's not much information about it, just the usual stuff like - do it our way it'll work ... however when i use embeded video, there's no sound, when i use URL (what is exactly what i want), there's no sound ... if i play that video in system videoplayer, it has sound, the unity videoplayer component can see the track (shows 2 channels) ...

    so i am expanding the question to general:

    what has one to do to be able to play video with sound in unity 5.6 ?
     
    Last edited: Dec 14, 2016
  7. zyzyx

    zyzyx

    Joined:
    Jul 9, 2012
    Posts:
    186
  8. komodor

    komodor

    Joined:
    Jan 2, 2013
    Posts:
    31
    oh, TY!
     
  9. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    94
    Hi all,

    Thanks for having a look at our new VideoPlayer and giving feedback!

    As you are realizing, full documentation has yet to be done so in the mean time it'll be my pleasure to assist.

    About codecs:

    komodor, your question about supported formats is a frequent one and the answer is multifacteted. Generally speaking, the supported codecs are:
    • H.264 for video with AAC for audio, typically found in .mp4, .m4v and .mov files.
    • VP8 vor video and Vorbis for audio, typically found in .webm files.
    These are the codecs that will work in most situations, both in the editor and in the target platform. H.264 is often the best choice as it is supported in hardware on most platforms - with Android also supporting VP8 in hardware. However, we're trying to have support for whatever the editor platform supports. For instance:
    • On Windows, we're using Media Foundation for reading/transcoding movie files.
    • On OSX, we're using AVFoundation for reading/transcoding movie files.
    • On all editor platforms, including Linux, we have a software version of the VP8 video codec and Vorbis audio codec.
    So in theory, all formats handled by these APIs are supported in the editor. I'm saying "in theory" because we do have some filtering based on filename extensions that may go away in the future (not sure yet). At this time, here's the list:
    • All editor platforms: ogv, vp8, webm
    • Additionally, on Windows and OSX: mov, dv, mp4, m4v, mpg, mpeg
    • Additionally, on Windows: avi, asf, wmf
    We will try to lift some restrictions and simply let the platform's native APIs tell us whether a given file format is supported, but this ties into how Unity's asset pipeline works so we cannot simply claim that every filename extension "might be a movie file", hence the current limitation.

    All target platforms have their own different capabilities that we'd like to encode in the editor so that you don't have to keep track of them. Each one has different resolution, bitrate, codec, profile and audio restrictions and recommendations. Using Android as an example, the official recommendations are https://developer.android.com/guide/appendix/media-formats.html. But every Android device / model / OS version has its own specifics so it wouldn't be appropriate for us to impose limitations based on the general recommendations. Some Android devices like the Nvidia Shield TV have no problem playing 4K videos. We could emit warnings when a movie is found to be "out of spec", at best.

    Then comes the discussion about whether you'll ask the movie to be transcoded during import. When the importer transcodes the movie, it will encode it in either H.264 or VP8 based on what the target platform is know to support best (this is the "Auto" mode). For instance on iOS, like on many platforms, this would be H.264 but on Linux this would be VP8. Again here, feedback is incomplete so you don't get to see what "Auto" translates into for the target platform; this will come. You can also override the codec choice if you have a good reason. A good reason could be: on Windows, H.264 works in hardware most of the time, but on certain Windows versions it won't work above HD. Forcing the codec to VP8 in this case means we'll revert to the software codec, which works identically on all Windows variants, but comes at a higher CPU usage cost. There's also the reality that transcoding takes time, and often a huge amount of it depending mostly on resolution. Furthermore, our current asset pipeline doesn't support aborting the transcode task, which is a (known) major flaw. We'll try to address this throughout the beta cycle.

    So if you decide to not transcode your movie file, then you're essentially telling Unity that you know what you're doing and that you know the movie in its current format is supported on the target platform. This is the default choice when you import a movie, which one may find questionable since it may lead to surprises. But given the time it takes to transcode most videos, we still prefered to have this as the default than to send Unity into a uninterruptible transcode each time movie assets are added/reimported.

    About error reporting:

    As CatherineP points out: error reporting is most of the time absent; we are aware and are working on this. The console messages, when present, are vague at best but this will be improved. One thing that makes good error reporting harder is that many of the APIs we are using will only report errors asynchronously. The request to start decoding a given video will succeed, but a background event gets fired later on telling us about a failure. So one cannot rely on proper exception throwing to catch errors. This is why the scripting API has a "errorReceived" C# event: we hope to funnel as many error descriptions through there as possible, as well as through console logging.

    About audio output:

    Audio can be read from all supported movie files with the notable exception of .ogv. This has just been completed internally and will appear in a later beta version.

    It's understandable that many will find the audio output hard to figure out initially as the implementation is not yet completed. Here's the general idea behind the audio output modes:
    • Audio Source: This causes the movie's audio to be sent to the specified AudioSource(s): one audio source per multi-channel audio track in the movie. This means the movie's audio can be processed through Unity's audio engine alongside any other sound in the game. The AudioSource you assign to the track can then be configured normally for sending the samples to the wanted mixer group and all AudioSource controls will work as expected. When an AudioSource is used this way, its AudioClip field can still be used and both the AudioClip and the VideoPlayer's contribution will be mixed together.
    • Direct: Some APIs - notably AVFoundation on OSX - make it possible to output audio directly from the movie decoding stage. It seemed convenient to expose this capability if all that is needed is to output the audio from the movie without further processing from Unity. This is what the "Direct" mode is about. It will not be possible to support this on all platforms, but we may end up simulating it for completeness sake, and also because this would still represent an economy over forcing the user to have an AudioSource object.
    About HoloLens:

    And to answer CatherineP's question about HoloLens specifically: no work has been done to support this platform yet. The current platform lineup is: Windows, OSX, Linux, Android, iOS, WebGL, Tizen and (in the next beta) XboxOne. Although one of the long term goals is to support better the VR workflows, this initial release is more about general hardware-assisted video playback than specifically about VR features/platforms. I've asked a developer on the HoloLens team to see if he can get videos to play in there. We'll add adequate error reporting if needed, or provide a fix if no major development is required. Thanks for trying it out and bringing this to our attention!


    Again, thanks for experimenting with this and for your patience while we're working on the remaining bits and pieces!

    Dominique
     
    TSLoire and zyzyx like this.
  10. CatherineP

    CatherineP

    Joined:
    Apr 17, 2014
    Posts:
    3
    Thanks a lot, Dominique! (and I maintain that you are my favorite technical comment writer ever :) )

    (Just to be clear, I'm trying to play the output of the Hololens video feed on a Windows platform. But I understand that it might not be part of the initial scope... it would just be very nice if it came to be eventually. I'll be happy to help if you need a guinea pig)
     
  11. Studiomaurer

    Studiomaurer

    Joined:
    Sep 5, 2012
    Posts:
    17
    Thanks for that Video Player, I have been waiting for so many years.
    First impression is very good. Got 10 Videos running simultaneously on 3xUHD Displays.
    Performs beautifully...
     
  12. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    94
    Thanks for your good words Studiomaurer!

    We're also interested in hearing what you have to say when things work as expected! So if you can let us know what platform and types of videos you're using, other users will be able to refer to your experience to know what they can expect!
     
  13. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    94
    CatherineP : I just educated myself on the type of url that is needed to read the stream from the HoloLens, using instructions from this web page : https://blog.kloud.com.au/2016/09/01/streaming-hololens-video-to-your-web-browser/

    The type of URL that has to be used (e.g. taken from the page: https://[USERNAME]:[PASSWORD]@192.168.1.6/api/holographic/stream/live_high.mp4?holo=true&pv=true&mic=true&loopback=true ) may very well trip our internal logic. As soon as we can spend time on the device, we'll see if this is the case.

    Another thing that we didn't have time to try yet is "live" streams, i.e. streams for which there is no known duration. This also could be problematic. At this point, I feel we could continue chatting in another thread (if you don't mind opening one) as this doesn't seem codec-related and we'll be able to have HoloLens-specific discussions. It *is* hard to get a confirmation that the stream really is using H.264, but the fact that the filename uses the .mp4 extension is a strong hint. I went from the Microsoft doc for the HoloLens portal: https://developer.microsoft.com/en-us/windows/holographic/device_portal_api_reference

    I'll let you know when we have time to experiment on our side!
     
    harschell likes this.
  14. hangemhigh

    hangemhigh

    Joined:
    Aug 2, 2014
    Posts:
    55
    @DominiqueLrx, are you sure that the new VideoPlayer works on Android devices? It crashes on my Android device. You mentioned that Android is supported. If so, then I have to file for a bug report.
     
  15. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    94
    Hi @hangemhigh , yes it does work for the devices we were able to test with. Please file a bug giving the details of what you are doing that leads to a crash as well as the full device description so we can reproduce and fix.

    Sorry it's not working for you; hopefully we'll get you a fix quickly!

    Dominique
     
  16. hangemhigh

    hangemhigh

    Joined:
    Aug 2, 2014
    Posts:
    55
    I will file a bug report today. This is my 4th bug report for Unity 5.6 beta about the VideoPlayer. Hats off to you and other Unity engineers for bringing us the new VideoPlayer API. That's what I've been waiting for and it works on my Windows. Just few bugs that needs to be fixed.
     
  17. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    1,251
    Is the Alpha channel supported in vp8? or in any other? Thanks
     
  18. Egil-Sandfeld

    Egil-Sandfeld

    Joined:
    Oct 8, 2012
    Posts:
    18
    Do you need more bug reports on this? I experience it crashing on Android as well when starting to play the video, while it runs perfectly on Windows if you do the same things. Here's the logcat from when I press the button to start the video:

    Code (CSharp):
    1. 12-22 14:25:41.772 12441-12452/? V/VRManagerServer: (VRManagerService.java:443) yTqX [setThreadSchedFifo pkg=dk.vendeval.world pid=14975 tid=15017 prio=0]
    2. 12-22 14:25:41.772 3471-3513/? V/VRManagerService: ---- demoteProcessThread 'dk.vendeval.world' requestPriority(pid=14975, tid=15017, prio=0 ----
    3. 12-22 14:25:41.782 12441-12514/? V/VRManagerServer: (VRManagerService.java:443) yTqX [setThreadSchedFifo pkg=dk.vendeval.world pid=14975 tid=15062 prio=0]
    4. 12-22 14:25:41.782 3471-4493/? V/VRManagerService: ---- demoteProcessThread 'dk.vendeval.world' requestPriority(pid=14975, tid=15062, prio=0 ----
    5. 12-22 14:25:46.202 3153-3153/? A/DEBUG: pid: 14975, tid: 15017, name: UnityMain  >>> dk.vendeval.world <<<
    6. 12-22 14:25:46.242 3153-3153/? A/DEBUG:     #01 pc 005e9560  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
    7. 12-22 14:25:46.242 3153-3153/? A/DEBUG:     #02 pc 005eafc8  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
    8. 12-22 14:25:46.242 3153-3153/? A/DEBUG:     #03 pc 005ea098  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
    9. 12-22 14:25:46.252 3153-3153/? A/DEBUG:     #04 pc 005e7f78  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
    10. 12-22 14:25:46.252 3153-3153/? A/DEBUG:     #05 pc 009b8dfc  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
    11. 12-22 14:25:46.252 3153-3153/? A/DEBUG:     #06 pc 009b6204  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
    12. 12-22 14:25:46.252 3153-3153/? A/DEBUG:     #07 pc 009b6038  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
    13. 12-22 14:25:46.252 3153-3153/? A/DEBUG:     #08 pc 0081912c  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
    14. 12-22 14:25:47.212 3471-15652/? W/ActivityManager:   Force finishing activity dk.vendeval.world/com.unity3d.player.UnityPlayerActivity
    15. 12-22 14:25:47.332 3471-3585/? W/ResourcesManager: getTopLevelResources: /data/app/dk.vendeval.world-1/base.apk / 1.0 running in null rsrc of package dk.vendeval.world
    16. 12-22 14:25:47.382 3471-4009/? I/WindowState: WIN DEATH: Window{99561d u0 d0 dk.vendeval.world/com.unity3d.player.UnityPlayerActivity}
    17. 12-22 14:25:47.412 3471-3585/? D/ISSUE_DEBUG: InputChannelName : 50d8153 Application Error: dk.vendeval.world
    18. 12-22 14:25:47.422 3471-4805/? I/ActivityManager: Process dk.vendeval.world (pid 14975)(adj 0) has died(247,1310)
    19. 12-22 14:25:47.432 3471-4805/? D/ActivityManager: isAutoRunBlockedApp:: dk.vendeval.world, Auto Run ON
    20. 12-22 14:25:47.462 3471-3585/? D/StatusBarManagerService: manageDisableList userId=0 what=0x0 pkg=Window{50d8153 u0 d0 Application Error: dk.vendeval.world}
    21. 12-22 14:25:47.502 3471-3585/? V/WindowStateAnimator: Finishing drawing window Window{50d8153 u0 d0 Application Error: dk.vendeval.world}: mDrawState=DRAW_PENDING
    22. 12-22 14:25:47.652 3471-4805/? E/WindowState: The process of this window already died: Window{99561d u0 d0 dk.vendeval.world/com.unity3d.player.UnityPlayerActivity} : android.os.DeadObjectException
    23. 12-22 14:25:47.652 3471-4805/? W/WindowState: Failed to report 'resized' to the client of Window{99561d u0 d0 dk.vendeval.world/com.unity3d.player.UnityPlayerActivity}, removing this window.
    24. 12-22 14:25:47.752 3471-3585/? V/WindowStateAnimator: Finishing drawing window Window{50d8153 u0 d0 Application Error: dk.vendeval.world}: mDrawState=DRAW_PENDING
    25. 12-22 14:25:47.792 3471-3585/? V/WindowStateAnimator: Finishing drawing window Window{50d8153 u0 d0 Application Error: dk.vendeval.world}: mDrawState=HAS_DRAWN
    It crashes the app during this.
     
  19. hangemhigh

    hangemhigh

    Joined:
    Aug 2, 2014
    Posts:
    55
    That's not how to file for a bug report. From Unity Editor, go to Help->Report a Bug... Then report the bug.


    @DominiqueLrx

    Updated to b2 but this bug is still not fixed.
     
  20. ileonidmas

    ileonidmas

    Joined:
    Mar 16, 2015
    Posts:
    3
    Which android device are you using? I tried nexus 5 and samsung galaxy S6 but both didn't work. They are not crashing, but also don't run. I have a scene with empty plane or sphere (I tried to use different objects). I tried puting videos directly on VideoPlayer from assets folder, then loading from StreamingAssets and Resources, but nothing works. I tried changing codec to H264. I tried also to stream the video from the link I got in latest documentation, but also no luck. So basically my questions are:
    Which device did you use?
    Which folder you keep the video in and do you specify the codec?
    Is there something specific you need to do to run android because I it works perfectly on iphone and windows.

    Thank you very much for your time and I really apreciate your help.
    Leo
     
  21. Muckel

    Muckel

    Joined:
    Mar 26, 2009
    Posts:
    407
    hi,
    well checked the video feature...
    doesn't work here on all my Android devices....
    It works in Editor/PC/Mac but not on Mobile....
    looks like they are not supported @Moment
    Also got it not run on any of my iOS devices...

    so finally I think mobile devices are not yet supported... somewhere i have read that they will later supported...
    M.
     
  22. ileonidmas

    ileonidmas

    Joined:
    Mar 16, 2015
    Posts:
    3
    For me it worked on iOS though) I used iphone 7 for testing. This is why I am curious, why on iPhone it works and doesn't on android devices.
     
  23. Muckel

    Muckel

    Joined:
    Mar 26, 2009
    Posts:
    407
    interesting to hear!
    need a iPhone 7 to test now ;-)
    @ileonidmas what Xcode Version you are using?
     
  24. silverfire330

    silverfire330

    Joined:
    Jul 17, 2015
    Posts:
    3
    Same, also with Galaxy S6 and Google Pixel. Is there anyone who has gotten VideoPlayer to work on any Android phone? (Who doesn't work for Unity :p)
     
  25. ileonidmas

    ileonidmas

    Joined:
    Mar 16, 2015
    Posts:
    3
    I used 8.2
     
    Muckel likes this.
  26. Cascho01

    Cascho01

    Joined:
    Mar 19, 2010
    Posts:
    960
    I tried the new videoplayer in linear color space and it looks too bright:
    In gamma it´s fine.
    Is it possible that it only renders correct in gamma color space for now?
    Do we get a switch to prepare the movie for the colorspace that we use?
     
    Last edited: Jan 2, 2017
  27. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    94
    Yes! Native vp8 alpha is supported as webm does have a standard way to encode alpha. For instance, Chrome supports this. When importing, depending on the target device, you may choose to not transcode at all (fine for desktop platforms) or transcode (and enable Keep Alpha in the importer) if you are targeting anything else. For instance, although Android supports VP8 in hardware, it doesn't support webm files that have vp8 video with alpha. In these cases, we store the alpha side-by-side with the color channels and re-assemble the RGBA result during decoding.
     
    Lars-Steenhoff likes this.
  28. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    94
    Yeah... Holiday period has slowed down our pace: we'll get to it asap!
     
  29. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    94
    Hey ileonidmas!

    Sorry for the bumpy ride. I'm using a Nexus 6 locally without problems, and our test suites also run consistently on Nvidia Shield TV. Our Android QA team has also run it on other devices; I'll post more details here when people are back from vacation.

    We are aiming to support Jelly Bean (API level 16, 4.1.x) and up, but I found a bug in the pre-Lollipop (API level < 21, pre-5.0) that causes intermittent trouble so stay on API level >= 21 for now if possible.

    As for how to use videos, importing them as video clips (no need to transcode as long as your videos are H.264 or VP8), or streaming them via http works fine or even using file:// (if you have a video on the device's local storage). You can also put them in StreamingAssets; we have tests that verify this works.

    For instance, you can try using this URL to see if you can get streaming to work: http://www.quirksmode.org/html5/videos/big_buck_bunny.mp4

    Just create a VideoPlayer in your scene and set its Source to "URL" and paste this URL in the field.

    It could be that the OS version you are using has specifics that we haven't encountered yet; please report back here (or preferably in a Android-specific video player thread so we keep things separated) and of course, feel free to file bugs if you have enough info for us to work from!

    Hope to get this under control soon,

    Dominique
     
    gthiruna likes this.
  30. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    94
    Hey Cascho01!

    Thanks a lot for reporting this. The VideoClipImporter does not offer color space conversion for now. But we do have a lot of image processing utilities ready that could take care of this (at the cost of one generation loss). I'll see if this can be added to the importer in time or 5.6.

    A better approach however would be for this to be done in a shader. We already have a shader that performs the YUV->RGB conversion and it could very well perform colorspace conversion. We are looking at opening up this shader in order to offer other transformations such as 360 projections, stereoscopic management etc. But this is definitely not for 5.6 so in the mean time you'll have to either pre-process your videos into a linear colorspace, or manage the color conversion yourself by writing a BT.709-to-linear shader that receives the decoded video (presumably from a RenderTexture) if this is how you intend to use them.

    I'll try and get back to you with better info/workarounds if we have time to fit this into 5.6. Feel free to open a thread about the VideoPlayer's (lack of) color management if you want to keep discussing this; input appreciated!

    Dominique
     
  31. tomihr2

    tomihr2

    Joined:
    Oct 25, 2010
    Posts:
    24
    I am trying to make VideoPlayer work on Samsung Galaxy Tab A SM-T280, and just won't work, it works on Samsung Galaxy S7, Nexus 7, LG G3, but no mater what I do on Samsung Galaxy Tab A SM-T280 I got nothing.

    Tomislav
     
  32. wilmaster1

    wilmaster1

    Joined:
    Nov 16, 2012
    Posts:
    2
    How did you get it to work on the other devices? I tried 2 different S6 phones & a Huawei P9 Lite, neither of them worked with any combination of settings I tried (different transcoding options,video files, RenderImage, Direct mode etc etc)
     
  33. tomihr2

    tomihr2

    Joined:
    Oct 25, 2010
    Posts:
    24
    I have encoded video without sound.
    Sound extracted as mp3 file and played in separate AudioSource.
    Also used OGV video format.