[Released] Demolition Media Hap Pro Sync — Network Synced GPU video playback

Hap is a video codec which allows to decode high-resolution videos using mainly GPU power and gives ability to play videos with alpha channel transparency.

It proved to be the only one solution which managed to preserve glitch-free high-framerate video playback in our mixed reality playscape with Unity running with 8 projectors. We tried other solutions, but they just didn't work. Finally, we made this plugin.

This plugin version has an ability to sync the video playback across multiple machines in a LAN network:

Get it on the Asset Store
Try it before buying: Demo app

Main features:
— Hardware accelerated Hap video playback without any external codecs installation needed. All fravours of Hap are supported.
— Sync playback across multiple machines using UDP networking
— Use Unity Timeline to control the playback
— Control video playback with your own clock source
— Ability to manually specify current video frame index
— Texture updates are completely in background threads - main and render threads are unaffected
— Multichannel 5.1/7.1 audio support using Unity AudioSource
— High quality video with the new Hap R codec
— Low CPU usage. Frames are mostly decompressed on the GPU.
— Play 4k @ 120 fps, 8k @ 60 fps or 10k @ 60fps videos (on supported hardware), play tons of smaller videos at once. Extremely fast frame-precise seeking.
— Playback speed control.
— Transparent videos with Hap Alpha, Hap Q Alpha and Hap R codecs. First plugin to support Hap Q Alpha/Hap R in Unity.
— Chunked Hap support for even faster multi-threaded decoding.
— Works with DX11 rendering on Windows.
— Audio output through the Unity Native Audio plugin (without network syncing) and AudioSource (with netowrk syncing, multichannel audio supported).
— Preload video to memory if your SSD isn't fast enough.
— Configurable RAM/GPU memory usage.
— Suits for both programmers and artists: C# API along with IMGUI/uGUI/Render to Material wrappers provided.
— Example scenes with the typical usage scenarios.

Requirements for this asset version:
— Unity 2019 - Unity 2020
— Windows 8.1/10
— DirectX 11

 

Demo app just got updated - fixed bug where it wasn't possible to change the video size using G hotkey

 

Hi,
is the Demolition Media Hap standard player included in the Demolition Media Hap Pro Sync?
(and are Unity 2021 and 2022 supported?)
Thanks

 

Hi,
we bought the asset,
unfortunately after half a day of testing we got erratic behaviours
and both server and client did not play smootly
a resync with mouse click made it work better again till now.
We will keep on testing but could you please give some feedback/tips? maybe a resync after each loop ?

videos are 1920x1920 30 fps normal Hap (8 chunk from after effects)
the setup for each pc is two 4k screen with two cameras/canvas panel playing the same video (scaled twice),
hardware is gpu Nvidia Geforce rtx 3080 , cpu i7 10700k 3.80Ghz, nvme drive
for the test the 2 pc are cabled directly (no switch or router)
unity version is 2021.2.19

here some short video of the stuttering

Thank you

 

(the password to watch videos is "test")

 

Hi, the standard player is included. Unity 2021 and 2022 should be ok

 

Hi!
1. Did you try first launching clients and then server?
2. Did you also try with a router? Shouldn't be a problem actually
3. Did you try doing the same setup with the demo app available out there (with the sample videos/with your videos)?
4. Can you send your videos / sample project please? (you can send the link to demolition.studios.rocks@gmail.com)

 

@cameranebbia If nothing of the above helps, somewhat informative thing will be to record a video (preferrably a slowmotion using iphone) of the demo app playback with all the information visible. The best is to have both computers screens on one vertical video, where one screen is located above another like below:

 

@cameranebbia Another thing to watch out for: make sure VSync is enabled on server/client.
Either use QualitySettings.vSyncCount = 1 (done in scripts included with the package by default) for build, or enable it in the Editor window.

 

HI,
thanks for the reply,
1. Did you try first launching clients and then server? Yes
2. Did you also try with a router? Shouldn't be a problem actually, tried with switch and also direct cable (->latency is not the issue)
3. Did you try doing the same setup with the demo app available out there (with the sample videos/with your videos)? Not yet, I will
4. Can you send your videos / sample project please? (you can send the link to demolition.studios.rocks@gmail.com) OK, i will

This weekend I launched another test, same set up and same problem after around 10 hours, the issue is clear on the client machine, here is a video with the info resyncing and glitching (password is "test")

the server was not smoothed but also not as bad as the client.
Again after a mouse click (but not with a space bar pause/play) the playing became smoothed
The issue appears after at least half a day, so testing will take time.
Pretty sure everything was vsynched,
I will update you and in case send project files.
Best

 

just read the demolition_media_hap_pro_sync_demo_app manual:
do you recommend using nvidia sorround for two display (1 video spanned across the screens)?
I'm using two display with two cameras and two canvas (cause it's easier for frame blending)

 

Hi,
new tests seem not encouraging really,
for the details I've sent you a mail at demolition.studios.rocks@gmail.com

 

There currently seem to be a problem while running playback for long periods of time (~10hours) indeed.
Working on a fix. I've sent you a mail

 

New version 2.0.1 just published on the Asset Store:
— Fixed sync stability while running for long time (several hours and more). Thanks @cameranebbia for reporting the issue in the posts above
— HAP HDR - play 16-bit per color component floating point HDR videos. Ideal for playing things like 10-bit color depth footage. Or playing back encoded ambient occlusion/arbitrary data. Unsigned format recommended. Encoding possible with TouchDesigner
— 10-15% faster decoding compared to previous release - play even more videos.
— DX12/Vulkan/OpenGL rendering supported in addition to DX11 (DX11 recommended).
— Added Unity 2022/2023 support.
— Added Linear color workflow support.
— Ability to disable all logging or INFO-level messages using by setting DEMOLITION_MEDIA_DISABLE_LOGS / DEMOLITION_MEDIA_DISABLE_INFO_LOGS define in the Project Settings

 

Demo app got updated with the latest plugin version. Download here

 

Hi. Just wanted to mention that VSync should be enabled in the Project Settings, for the plugin to work correctly! There were some cases recently where some users didn't do so, and the playback sync was broken because of that. Thanks everyone!

 

I've just implemented a playlist functionality for SYNC Server/Client to be able to switch videos during playback. Also added an example of sending custom commands from the server to the clients + couple of other improvements. The asset store package will get updated in some time, meanwhile if you want to try it, I can send it by e-mail

 

• • I need this plugin, but I want to use MP4

 

Hi. Usually .mov container is used, but .avi is likely working too. I think .mp4 won't work for Hap codec

 

I don't need hap code. I want to use the sync feature.

 

• • • Only you .sync

 

Sorry, you have to figure it out by yourself in that case. This forum thread is only for Hap codec related discussions :]

 

I tested the program once, I changed the synchronized GUI drawing screen to UGUI drawing screen, and then I tested the self-contained video as two copies, but the result is a little unsatisfactory, a gap will appear in the middle when the speed is fast, and it is obvious

 

Hi. Could you please upload your full Unity project to some file hosting and send me the link?
You can either post it here or send to demolition.studios.rocks@gmail.com
I will take a look

 

Sorry for the late reply, I will send the file to your email address later today.

 

Thanks, I have the project now.
I've sent you an additional question in response.
I think I can help you to fix that behavior, but need some additional info first

 

I didn't get your reply (yet), but I think the problem might be that you're running both server and client instances on a single machine. It's not meant to work like that. Please test over different machines in the LAN for expected results!

 

Hi everybody!
A new major plugin version 3.0.0 is out!
New features:
— 100% GPU decoding with new GDeflate compression format (now no CPU overhead at all)
— 20-90% smaller file sizes (!) with GDeflate compression format which is really nice
— Added playlist & send data support for sync client/server
— Update to FFmpeg 6.0
— Fix flickering in some cases
— Added more example scenes

Starting from now, a new encoder app is available for free download: https://github.com/DemolitionStudios/shutter-encoder. I took an open source Shutter Encoder and improved support for Hap:

• Added fast Hap R encoding path (both RGB and RGBA modes supported - also use for high quality videos with alpha channel)
• Multi-threaded Hap encoding, which is very fast on multi-core CPU (15% faster compared to the original app)
• Support for fully GPU based format based on GDeflate compression algorithm

 

Hey,

I tested this - and it seems to work really nicely, good job!

Looking at the implementation, it seems like you wrapped a custom FFMpeg build for this - would it be possible to add support for HAP HDR? Not sure what pixel formats FFMpeg supports - this would probably need at least 16-bit floating point?

We currently use HAP HDR for high dynamic range content, and the only thing that can encode it is Touch Designer, which is essentially impossible to integrate into our automated production pipeline.

I started writing my own implementation using this

https://github.com/knarkowicz/GPURealTimeBC6H

The encoding part seems to work really nice and fast (currently it takes around 0.7 secs to load & encode a 12k EXR frame on mobile 2080 gpu) but I have serious problems in wrapping the raw data into playable format, so any help would be appreciated.

br,
Sami

 

Hi Sami,
Really nice you've tested it! Feel free to share any kind of comparison of old decoding VS new in your real-life scenario (in case you have - that'd be good to know some numbers)

Regarding the BC6 encoding - actually I was trying to do exactly the same lately in order to provide more feature complete HAP encoding support in our custom ffmpeg / Shutter Encoder. I think it was at least almost successful, but there are still some problems to solve (some GPUs give errors). Would be interesting to know how far did you get (I assume you wrapped GPURealTimeBC6H into a library?)
We can discuss it personally via support e-mail so it gets finished sooner

 

I actually got so far that I'm able to read an EXR file and encode it + see the results on the screen (it seems to indeed work), but I did all of this inside Unity (2023). I'm happy to share the project if it's of any help to you.

br,
Sami

 

Now it's done.
Please try the latest release: https://github.com/DemolitionStudios/shutter-encoder/releases/tag/v9.0
You can select encoding profile (default is Quality) and input data normalization factor (default is 1.0) as well through Shutter Encoder's Advanced features, or via packaged ffmpeg's command line arguments (see ffmpeg -h encoder=hap). In the future release I will replace Hap H with Hap HDR everywhere, for now it's Hap H

Encdoing EXR image sequences is also possible through Shutter Encoder, you just need to specify framerate in the "Image sequence" section

 

You're my hero! I'll give it a try tomorrow!

br,
Sami

 

Oh btw did you use the GPU code I linked above, or some other lib?

br,
Sami

 

Close, but no cigar, seems like the EXR sequence encoding (at least) flips some RGB channels or something. It also seems to somehow mess with the color space (interprets the linear EXR with Rec709 curve or something).

Top = your result, bottom = raw source linear color space EXR image.

 

Another test, this time from some random mp4 video - it didn't encode it at all (shows just white on Touch Designer). As a test, I encoded the same video to Hap-R using Shutter and it seems to work correctly.

I'd suggest that for compatibility testing, download a demo version of Touch Designer, it works but just scales the output to 1280 x 800 or so.

 

Here the problem is most likely in normalization. It's 1.0 by default, and for video you probably want to use 255.

Try with that option set, should be better. It's also possible to set it to 255 for RGBA input data by default (I guess that's debatable)

 

Right. The only good solution on the internet. I wrapped it into a library (took most of the time) and integrated into ffmpeg.

I can reproduce the problem above, so I will work on it (more efforts need to be put into that development)

 

Hi @Ookoohko
Following up our Hap HDR encoding discussion, a new release is out, which should solve the issues above (normalization factor for RGBA32 is 255 by default now): https://github.com/DemolitionStudios/shutter-encoder/releases/tag/v9.1

 

Good news - the encoding part now works, the channels are no longer flipped!

Bad news - the EXR parser (or whatever) still seems to assume some things, here's a comparison of the raw EXR file (top) and the resulting HAP file (bottom) dropped into touch designer.

The EXR is in linear color space, but the output video seems to be auto-converted to Rec709 gamma space, which should not happen (the encoder should *not* touch the color space unless specifically wanted). This happens either in your EXR loader or the encoding side, I assume.

I can drop you a frame sequence if you want to try it out, just drop the raw EXR frame and the HapHDR file into Touch Designer, they should look identical.

 

Hi @Ookoohko
I tried to compare inside TouchDesigner, and came to the following: it might be that TouchDesigner is displaying EXR files differently compared to other software. Take a look:
Left is exr file inside GIMP, middle is exr file inside TD, and right is Hap inside TD.
Hap colors look identical to what GIMP shows. Also it looks identical to what's on the official preview of that exr sequence. Do you think we still should try to match the TD's middle result, even though it may be non-standard?

Another interesting thing to try: what if we encode the exr data to Hap HDR inside TD and compare to what we currently get? Not sure if we can take it as a reference, but it might worth checking.

Btw I think you can apply an additional color transformation (like Linear -> sRGB) as a shader pass

 

Here is the result of gamma correction applied to the GIMP imag - mathematical operation is pow(color, 1/2.2).
Now is looks same as inside exr inside TD.


I'm pretty sure we shouldn't encode such transformed colors inside HAP - as we want to keep the linear nature of HDR data.

You can do like this in TD:

 

Yeah, this is exactly what I mean - whatever is in the EXR should go in exactly 1:1 as it is (no color space encoding, and no gamma correction applied).

Anyway I'll investigate more, it may indeed be an issue with how TD displays EXR images.

br,
Sami

 

So, I tested a bit more.

if I drop an exr seq into touch designer and encode it to HAP HDR, the output result is "dark" (as it should be when the source file is linear). Touch designer treats the data as raw until you manually apply an OCIO config or gamma correction to the pipeline, so the way TD treats the EXR data is indeed correct - it does not mangle the data in *any* way unless specifically told so.

If I view the same EXR sequence using a professional EXR viewer (such as DJV) without deliberately applying a display curve, the result is also "dark", as it should be.

https://darbyjohnston.github.io/DJV/

However, as a comparison, I tried opening the EXR in Affinity Photo and in Photoshop, and it shows up as "bright" in both of them, which means that they have already applied a gamma correction curve to the image. This is probably because image editing software doesn't deal with raw data - they convert stuff to the monitor color space, so I assume GIMP does the same thing.

So: *something* in your pipeline also applies the gamma curve to the EXR data before encoding the HapHDR, which ends up with a technically wrong result. The output data is no longer in linear color space, but in sRGB 2.2 gamma space, even though it now looks "correct" on the monitor.

Good question is which part does that? My guess is that it's some default option in FFMPEG that you have to force off for EXR images.

 

Here's a comparison, a screenshot from DJV (which displays the "raw" data of an EXR correctly), file encoded by TD (also dark, raw data) and file encoded by your encoder (the bright one, with gamma curve applied).

 

in DJV you can choose the display color space (for example "none" for raw data or "srgb" for displaying in srgb monitors). If you choose "none" the result is 100% raw and the image looks dark.

 

You are right. Shutter Encoder had a hidden parameter to convert to sRGB indeed.
FFmpeg part was good if using from command line.
Please try the new release: https://github.com/DemolitionStudios/shutter-encoder/releases/tag/v9.2

 

Cool, I'll try it tomorrow, thanks!

br,
Sami

 

Hi @Ookoohko . How is it working now?
Also, please check PMs (Inbox button on the top of the forum page). I've just sent you one

 

Hi everyone.
Attached is the hot fix for playing multiple files with GDeflate GPU decompression. The compute shader wasn't re-instantiated in each Media script instance. Just replace the attached script