Search Unity

Bug Videos recorded on iPhone 12 onwards play washed out/desaturated

Discussion in 'Audio & Video' started by tomkail_betterup, Oct 3, 2022.

  1. tomkail_betterup

    tomkail_betterup

    Joined:
    Nov 17, 2021
    Posts:
    106
    We've noticed that any video recorded on iPhone 12 onwards appears washed out and tinted in a weird way when played via VideoPlayer using url pointing at the local file on device.
    We're using Unity 2021.3.5f1.
    We believe that the problem is that iPhone 12 on record video in HDR, which uses the HEVC (H.265) format.
    Can anyone from Unity speak to this?
     
    tomkail_betterup, Oct 3, 2022
    #1

  2. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    Can you output the file metadata with MediaInfo if you can or share the file in DM? We officially support HEVC on iPhone 12 and most of our devices used to test are iPhone 12, so it should work. Do you get any errors other than the artifacts? Can you share a screenshot of what you mean?
    https://docs.unity3d.com/Manual/VideoSources-FileCompatibility.html
     
    The_Island, Oct 4, 2022
    #2

  3. tomkail_betterup

    tomkail_betterup

    Joined:
    Nov 17, 2021
    Posts:
    106
    Here's our video - left is how it should look, right is the same video loaded in Unity, using a RawImage. We don't have any filters, post effects or anything like that.
    I'll DM you the video now!
     

    Attached Files:

    tomkail_betterup, Oct 4, 2022
    #3
    The_Island likes this.

  4. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    Here is the output from MediaInfo. There are three potential issues.
    Code (CSharp):
    1. Video
    2. ID                                       : 1
    3. Format                                   : HEVC
    4. Format/Info                              : High Efficiency Video Coding
    5. Format profile                           : Main 10@L4.1@Main
    6. HDR format                               : Dolby Vision, Version 1.0, dvhe.08.04, BL+RPU, HLG compatible
    7. Codec ID                                 : hvc1
    8. Codec ID/Info                            : High Efficiency Video Coding
    9. Duration                                 : 59 s 470 ms
    10. Bit rate                                 : 15.2 Mb/s
    11. Width                                    : 1 920 pixels
    12. Clean aperture width                     : 1 920 pixels
    13. Height                                   : 1 080 pixels
    14. Clean aperture height                    : 1 080 pixels
    15. Display aspect ratio                     : 16:9
    16. Rotation                                 : 90°
    17. Frame rate mode                          : Variable
    18. Frame rate                               : 30.000 FPS
    19. Minimum frame rate                       : 28.571 FPS
    20. Maximum frame rate                       : 30.000 FPS
    21. Color space                              : YUV
    22. Chroma subsampling                       : 4:2:0
    23. Bit depth                                : 10 bits
    24. Bits/(Pixel*Frame)                       : 0.244
    25. Stream size                              : 108 MiB (99%)
    26. Title                                    : Core Media Video
    27. Encoded date                             : UTC 2022-09-08 14:28:48
    28. Tagged date                              : UTC 2022-09-08 14:29:46
    29. Color range                              : Limited
    30. Color primaries                          : BT.2020
    31. Transfer characteristics                 : HLG
    32. Matrix coefficients                      : BT.2020 non-constant
    33. Codec configuration box                  : hvcC+dvvC
    34.  
    1. Your HEVC clip has a bit depth of 10 bits, which is not well supported on every platform. For example, Windows doesn't support it.
    2. I don't know if you saw it, but the color in your video is shifted. The reason is your video color space is BT.2020, and we only support BT.709 and BT.601.
    3. And as you mention, the video has HDR, and the blurriness is probably for that. We would need to investigate more on the subject, but the HDR aspect of it possibly means there are multiple images per frame for multiple exposure levels.
    I don't think it is something we can easily fix or find a workaround for. I would appreciate it if you could write your use case on our roadmap https://unity.com/roadmap/unity-platform/audio-video. It helps us plan for these kinds of needs and allows us to prioritize what to work on next. I hope it helps!
     
    Last edited: Oct 5, 2022
    The_Island, Oct 4, 2022
    #4

  5. tomkail_betterup

    tomkail_betterup

    Joined:
    Nov 17, 2021
    Posts:
    106
    Thanks for this!
    The blurriness is me - wanted to hide my coworker's face :)

    I'll triple check this with my team, but I'm certain the clip was recorded using the standard iPhone 12 (or in other cases, 12 and above) camera app. We've had some reports from our users too.
    We think it's the case that the bit depth and color space is something that Apple is doing by default - but it's very strange that your team hasn't been able to reproduce it!
    Perhaps our issue is caused by the use of this plugin to access the phone's photo library - although I can't quite see why that would change anything, since it's ultimately just returning a file path.
    I just tried loading the file itself into Unity and it's imported with the wrong colors when both in the preview and when loaded via a VideoPlayer.
    upload_2022-10-5_11-18-28.png
     

    Attached Files:

    tomkail_betterup, Oct 5, 2022
    #5

  6. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    Oh, I see. I thought the "washed out and tinted in a weird way" was the blurriness and was not sure :confused:
    To be clear, the bit depth will probably work on some platforms. But it is in the danger zone where it will break on some platforms because the platform doesn't support it like Windows. HEVC is another thing some platforms just don't have or don't support a bit depth of 10. So if you plan to support multiple platforms, you will maybe have some issues with these clips.

    If the tint and washed is the color then you can test this theory by converting the clip to bt709.
    https://stackoverflow.com/questions...scode-uhd-h265-to-sdr-h264-without-color-loss

    And finally no, Apple doesn't do the YUV conversion. We are doing it. It is why we have to support BT.2020 for it to work.
     
    The_Island, Oct 5, 2022
    #6

  7. tomkail_betterup

    tomkail_betterup

    Joined:
    Nov 17, 2021
    Posts:
    106
    Right! Is it then weird that we're seeing this on iOS? Our app is iOS only, and so aside from testing in editor we're recording and viewing videos on iOS exclusively.
     
    tomkail_betterup, Oct 7, 2022
    #7

  8. tomkail_betterup

    tomkail_betterup

    Joined:
    Nov 17, 2021
    Posts:
    106
    Hi! Is there any update on this? I've tried reading up on formats but this isn't something I really understand (and appears to be quite complex!).

    It'd be useful if you could help us establish if this is an error in Unity or something in our setup that we need to correct for.

    We're both creating video and testing entirely with Apple devices, so it seems unlikely that the issue is due to the platforms we're using.
     
    tomkail_betterup, Oct 11, 2022
    #8

  9. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    We don't support BT.2020 so not a bug. You can convert your file using this command
    ffmpeg -i "IMG_5127 2.mov" -vf zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=tonemap=hable:desat=0,zscale=t=bt709:m=bt709:r=tv,format=yuv420p -c:v h264 -crf 19 -preset ultrafast color_output.mp4

    I tested it with your file and it does show the right color now. I will send it to you.
     
    The_Island, Oct 11, 2022
    #9

  10. tomkail_betterup

    tomkail_betterup

    Joined:
    Nov 17, 2021
    Posts:
    106
    I see! Thanks very much, this has been incredibly helpful. We're looking into some asset store FFMPEG converters now, so that snippet should be handy.
    I'll update this thread with a solution if we find one.

    The converted video you sent works great, although I should note for other devs that the video file doesn't import into Unity as a Video Asset, but rather as a DefaultAsset - but it loads fine via local or remote URL.

    The one thing that still confuses me is that video recorded on iPhone 12+ is supported and are some of the devices tested by your team - we've seen this pretty reliably from users using the default camera app on those devices, so I'm a bit unsure what we've done differently! Is it that you've been explicitly testing non-BT.2020 recordings from those devices because it's not supported?
     
    tomkail_betterup, Oct 12, 2022
    #10

  11. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    Weird I will have a look.
    We are making sure the VideoPlayer is supported on the iPhone 12+. We don't test that video recorded on iPhone 12+ is supported, and we don't support every possible file configuration. What we share with the recorder is the codec but everything else is different. Contrary to AVFoundation, when we need to add a feature, we need to implement it on all platforms Unity support. So it is always more challenging to add a feature that is not supported on all platforms. While saying that the fact the video recorder on iPhone is using BT.2020, shows that BT.2020 is now mature enough that companies start using it. It is probably something we need to look into but I can't give you an ETA for this.
     
    The_Island, Oct 13, 2022
    #11

  12. tomkail_betterup

    tomkail_betterup

    Joined:
    Nov 17, 2021
    Posts:
    106
    tomkail_betterup, Oct 14, 2022
    #12