Search Unity

Video Challenges playing back 8K HEVC video on Windows 11 Intel NUC device

Discussion in 'Audio & Video' started by kennyatinsight, Oct 2, 2023.

  1. kennyatinsight

    kennyatinsight

    Joined:
    Sep 11, 2023
    Posts:
    3
    I'm working on an app where one of the core features it the ability to play back high quality videos. We have it set up to download videos from a remote URL to the local file system and then play back due to the size of videos.

    Our current set up works perfectly with HD videos (1920x1080) but we are going to be dealing with 8K videos. We have been provided these in the following codecs: "Timecode, MPEG-4 AAC, HEVC"

    When we try to play back on the main device we're targeting - Intel NUC running Windows 11, the following error is showing up:

    WindowsVideoMedia error 0x80004004 while reading filename.mov

    Context: IMFSourceReader::WaitForSample in StepAllStreams
    Error details: Operation aborted

    Track types:
    Audio Track, type: %10%16
    Video Track [7680 x 4320], type: HEVC
    Resolution exceeds the Microsoft Media Foundation HEVC
    decoder limit of 4096 x 2304 on Windows 10 and above.
    Install HEVC Video Extensions to enable HEVC format support on compatible devices.
    https://www.microsoft.com/en-ca/p/hevc-video-extensions-from-device-manufacturer/9n4wgh0z6vhq
    https://www.microsoft.com/en-us/p/hevc-video-extensions/9nmzlz57r3t7

    I've tried installing the software at the link above after paying the 99 cents, and while the built-in Media Player app in Windows 11 seems to be able to playback the video better, I'm still experiencing the same error when trying to play the video back within our Unity app.

    I've tried upgrading to the latest Unity LTS version, restarting Unity, rebooting the PC, and even uninstalling and re-installing the HEVC codec and then rebooting the PC again.

    My suspicion is that while the new codec is installed, the Microsoft Media Foundation Library that does video playback in Unity is not using it.

    Specs for the Intel NUC:

    Processor  12th Gen Intel(R) Core(TM) i5-1250P 1.70 GHz
    Installed RAM  4.00 GB (3.57 GB usable)
    System type  64-bit operating system, x64-based processor
    Pen and touch  No pen or touch input is available for this display​

    The app is build with the latest Unity LTS which at the time I'm posting this is 2022.3.10f1

    I would appreciate any help in resolving this issue. Thanks!
     
  2. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    260
    Hi Kenny!

    There clearly is something that is upsetting Microsoft Media Foundation in the HEVC clip that is being played.

    The error code that we're reporting (0x80004004) is telling you that the codec has aborted (WINCODEC_ERR_ABORTED). Here's a very useful website to translate these obscure error codes: https://www.magnumdb.com/search?q=0x80004004

    First thing to try would be to play HEVC content that is not at the limit of what the device can do. So trying 4K or even 2K would be a good start. If this does play, then you can visit the Windows task manager and, while playback is ongoing, look at the Performance tab, in the GPU subtab, and there should be a graph showing activity for hardware-accelerated video decoding. If the graph is indeed active, it's because the Intel decoder is doing its job. If it's inactive, it's because for some reason the hardware-accelerated decoder is not kicking in and what you get is instead the software decoding (which is capped at 4K).

    Another thing I'd explore is removing the Timecode track from the movie you're playing in case it's interfering with our decoding logic. See this link for example for an explanation of how this can be done with the ffmpeg command-line tool: https://www.tal.org/tutorials/remove-timecode-track-ffmpeg

    It would also be good to have more detailed information about the movie file you're trying to play. I understand these are big files so sharing one of them here may not be feasible (though we can take a look if you can share a URL). Another way of getting information is using the mediainfo command-line tool. If you have it installed, then mediainfo -f the-file-to-inspect.mp4 will spit out a lot of info that you can paste here so we can take a look and perhaps find ideas of what next to try.

    If you don't have it installed and would prefer not to, then you can always use the online version (it apparently doesn't upload your content to a server): https://mediaarea.net/MediaInfoOnline.

    Let us know what you find and maybe we can get to the bottom of this.

    Dominique Leroux
    A/V developer at Unity
     
  3. kennyatinsight

    kennyatinsight

    Joined:
    Sep 11, 2023
    Posts:
    3
    Thanks for all the info! First off - 4K video is playing perfectly fine with our set up. I've tried playing back the video in Media Player with both 4k and 8k with the HEVC codec installed on the Intel NUC. Both are showing the GPU is in use, as well as using the video decoding.

    I never knew about the Timecode track and that article calls out Davinci Resolve Studio as the main culprit as far as adding that to the files. According to the metadata on the videos we have, that is the software used to create the videos we're working with.

    I tried the steps to remove the Timecode track and now our Unity app is crashing with out of memory errors, so we're looking into switching up our target hardware to something more powerful and with more RAM. I don't remember seeing that happen before removing the Timecode tracks.

    We do have access to Adobe Media Encoder, so if there's a format we can encode to that would work better, that's an option. Ideally once we know what format works best with Unity, we could just have them export any future videos in that format.

    Options we're looking at: adding RAM to the Intel NUC we're using (4GB for a system with shared video memory seems a bit low), switching to an upgrade M2 Mac Mini since Apple devices have built-in hardware for HEVC decoding.

    Also a little more context on our app setup: we've got a config file to define which videos should play back in which parts of the app and then it streams the video to a render texture after it downloads the video.

    This is for a client project, so I don't feel comfortable sharing the file itself, but here's what we get from running mediainfo on the original file.


    Format : MPEG-4
    Format profile : QuickTime
    Codec ID : qt 0000.02 (qt )
    File size : 1.71 GiB
    Duration : 1 min 20 s
    Overall bit rate : 183 Mb/s
    Frame rate : 60.000 FPS
    Encoded date : 2023-05-10 01:57:07 UTC
    Tagged date : 2023-05-10 01:57:07 UTC
    Writing application : Blackmagic Design DaVinci Resolve Studio
    Video
    ID : 1
    Format : HEVC
    Format/Info : High Efficiency Video Coding
    Format profile : Main 10@L6.2@Main
    Codec ID : hvc1
    Codec ID/Info : High Efficiency Video Coding
    Duration : 1 min 20 s
    Bit rate : 182 Mb/s
    Width : 7 680 pixels
    Height : 4 320 pixels
    Display aspect ratio : 16:9
    Frame rate mode : Constant
    Frame rate : 60.000 FPS
    Color space : YUV
    Chroma subsampling : 4:2:0
    Bit depth : 10 bits
    Scan type : Progressive
    Bits/(Pixel*Frame) : 0.092
    Stream size : 1.71 GiB (100%)
    Encoded date : 2023-05-10 01:57:07 UTC
    Tagged date : 2023-05-10 01:57:07 UTC
    Color range : Limited
    Color primaries : BT.709
    Transfer characteristics : BT.709
    Matrix coefficients : BT.709
    Codec configuration box : hvcC
    Audio
    ID : 2
    Format : AAC LC
    Format/Info : Advanced Audio Codec Low Complexity
    Codec ID : mp4a-40-2
    Duration : 1 min 20 s
    Bit rate mode : Constant
    Bit rate : 320 kb/s
    Channel(s) : 2 channels
    Channel layout : L R
    Sampling rate : 48.0 kHz
    Frame rate : 46.875 FPS (1024 SPF)
    Compression mode : Lossy
    Stream size : 3.08 MiB (0%)
    Default : Yes
    Alternate group : 1
    Encoded date : 2023-05-10 01:57:07 UTC
    Tagged date : 2023-05-10 01:57:07 UTC
    Other
    ID : 3
    Type : Time code
    Format : QuickTime TC
    Duration : 1 min 20 s
    Frame rate : 60.000 FPS
    Time code of first frame : 00:00:05:02
    Time code of last frame : 00:01:25:36
    Time code, stripped : Yes
    Language : English
    Default : No
    Encoded date : 2023-05-10 01:57:07 UTC
    Tagged date : 2023-05-10 01:57:07 UTC
     
  4. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    260
    Hi, thanks for the reply!

    Are you saying that 4K is also playing fine in Unity? If so, does it appear to be using hardware acceleration?

    Maybe this is a good sign (i.e. the playback is able to get a bit further than before...). But still not great.

    First thing I would try, given the mediainfo output you provided, is to use the Main HEVC profile instead of Main 10. The Main profile is 8-bits, which is definitely something that's more widely working in Unity than 10 bits. 10 bits is fine in some scenarios, but since I'm not familiar with this specific graphics device it's hard for me to know in advance.

    There are chances HEVC will work better on a Mac, since support is builtin (and doesn't depend on vendor-specific plugins). But again, I don't remember trying 8K/10bits on a Mac so I'm somewhat extrapolating.

    Understandable. Thanks for sharing the media info output, this is indeed useful. Like I said, 10 bits is something I'd avoid initially and try to with 4K, 30FPS, 8 bits. You can scale up from there (resolution, bit depth, frame rate) to see how far you can get.

    Sorry that this process is a bit painful, but hopefully we can find a good compromise.

    Dominique
     
  5. kennyatinsight

    kennyatinsight

    Joined:
    Sep 11, 2023
    Posts:
    3
    Thanks for the great feedback. I wasn't aware of the 8bit vs. 10 bit color profiles.

    If I do 4k video, the playback in Unity works fantastic and looks like it's using the video decoder on the Macs I have access to.

    When I play back the 4k or 8k video Windows Media Player on the Intel NUC, it shows the Video Decode and shared GPU usage going up like crazy. If I attempt to play back on the Intel NUC in Unity, I am not seeing the Video Decode in the GPU performance being used at all. The shared GPU memory usage does go up.

    Does Unity require the video to be in any specific format to use the hardware video decoding? What I'm hearing is try the following:

    • Remove the timecode track (tried, it seemed to help a little)
    • Switch from 10 bit color to 8 bit (working on this one)
    • Downgrade from 60fps to 30fps (I can try this, but hope quality doesn't suffer too much)
    • Try 4k video instead of 8k (hoping this would be a last resort as quality and wow factor are an important part of the experience we're trying to build)
    Any ideas or recommendations if H264 or H265 would be better encodings for this, or is there another one that may work out better?

    I've got access to Adobe Media Encoder and am pretty comfortable working with Handbrake or FFMPEG.
     
  6. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    260
    Hi again!

    OK, are you saying that video playback of HEVC content on Windows with Unity is working in any way (I assume not yet, but just to make sure I understand where you're at)? Also, did you have a chance to try 8k on Mac? Just curious...

    It's not Unity specifically that is requiring a specific encoding, it's the vendor's plugin (the one you installed from the Windows Store for 1US$). Depending on the gpu manufacturer, the constraints and capabilities will differ. The "vanilla" decoder from Microsoft (the one presumably using software decoding) is limited to 4K and supports both 8 and 10 bits. Reference: https://learn.microsoft.com/en-us/windows/win32/medfound/h-265---hevc-video-decoder

    I don't know what are the capabilities built into the Intel version of this plugin, which is why I'm suggesting to start with low demands and increase complexity/weight to find where things stop working. 8K is typically "out of spec" for Microsoft's own HEVC decoder, but I know it can work with some vendors (e.g.: I've seen it work on Nvidia).

    H.264 will probably not get you to 8K: it's possible in theory (the spec allows it) but in practice a lot of encoders or software will prevent going beyond 4K because encoding levels are not defined for these higher resolutions. So H.265, as far as hardware-accelerated codecs go, is your only choice.

    You can also try vp8, in a webm file, which is "limited" to 16k x 16k. I would recommend making your own encoding outside of Unity however. Our vp8 encoder, until very recently - 2023.3.0a3/2023.2.0b3/2023.1.7f1/2022.3.7f1- produced low-quality results because of parameter errors on our side. And it's also quite slow, so large resolutions will be painful. Because vp8 is using only the CPU for decoding however, on desktops at least, the decoding performance may suffer if your machine isn't fast enough. Multiple cores won't help in this case: the decoding happens in background but is currently constrained to a single core. When exploring this area, I often turn to https://trac.ffmpeg.org/wiki/Encode/VP8

    Hope this helps you get a bit further! Have a great week-end,

    Dominique