[Released] Demolition Media Hap — multi-platform 8k@60fps GPU video playback

Hap is a video codec which allows to decode high-resolution videos using mainly GPU power and gives ability to play videos with alpha channel transparency.

It proved to be the only one solution which managed to preserve glitch-free high-framerate video playback in our mixed reality playscape with Unity running on 8 FullHD projectors. We tried other solutions, but they just didn't work. Finally, we made this plugin.

Main features:
— Native hardware Hap video playback support without any codec installation needed.
— Low CPU/memory usage. Frames are decompressed on the GPU
— Works with all the render interfaces (DX9, DX11, OpenGL, Metal) on the desktop platforms.
— Both 32 bit and 64 bit supported! Even on OS X!
— Best playback performance achievement: 8k@60fps Hap, 4k@60fps Hap Q
— Play multiple videos at once with minimal CPU overhead
— Transparent videos with Hap Alpha codec!
— Audio output through the Unity Native Audio API
— Can preload video to memory if your SSD isn't fast enough
— Suits for both programmers and artists: C# API, IMGUI/uGUI wrappers provided
— Demonstration scenes for typical use scenarios
— Demo version provided (see the link on the asset store)

If you work with video in Unity, which includes, but not limited to

• VR projects
• multi-display/projector walls and playscapes
• playing alpha channel-enabled content
• VJing
• other cool stuff

get it on the Asset Store: https://www.assetstore.unity3d.com/en/#!/content/78908

 

Thanks for Great Assets!

Can I use this plugin with playback speed rate change?
I want to change playback speed in scene.

 

Hey! Thanks for your request. I will add playback speed control possibility to the upcoming new version of the plugin. Do you need the audio to be played as well with the varying speed, or what you need is only the video (audio could be a bit more tricky to add) ?

 

thanks for your quick reply!

I want only Video playback speed change (currently).

 

Hi, thanks for the asset. My video is now FLYING! I wonder if you have the API documented somewhere. I can snoop around in the C# files but I'd rather look at some documentation.
I fully concur with izm1252 about manipulating playback speed. Yes please! If it helps you crank out an update sooner to focus on just the video playback speed and not the audio playback speed too then please do that, at the moment my primary application would be just video.

 

does it do any benefit for mobile VR devices like GearVR? Playing 8k on there would be awesome

 

@adriaanwormgoor, @izm1252

I have a workable version with playback speed controls here. Please directly contact me via lev.panov@gmail.com, I will send you the needed asset version, and publish on asset store if it works for you!

And I will definitely think of how it's possible to make some nice-looking docs out of the C# files! Thanks for the idea!

 

I think it will work in VR environment, since you can render the video onto any material you want! There is an example scene which shows how to do it in the package. You can check it without buying the asset, using the demo version here: https://github.com/DemolitionStudios/DemolitionMediaHap. Important: for 8k you will need a really fast SSD!

Please let me know how it goes! I'm highly interested whether it works, unfortunately have no VR equipment nearby

 

I've just submitted a new version 0.9.3 with playback speed controls and some other changes.
Let's wait couple of days, so it gets approved by the Asset Store admin!

 

Updated demo version available for download here!

 

@leavittx : is it possible to use on mobile (high end Samsung phones, for example) ?
Real 4K or 8k video would be of a great help when doing 360 mono/stereo videos !

 

@Pode Hi there! No plans to support mobile targets right now! Our main aim is to play videos of killer resolutions, but only on desktop platforms. However things may change in the future, since there is a technical possibility, but as far as I know plenty of other nice-working video assets exist for mobiles

 

@leavittx yes they are other assets, but nothing for HAP.

 

How might I got about recording or capturing video with transparency? in particular lets say from a unity camera or rendertexture

 

@Carpet_Head Hi there! You might use one of these for your needs:
https://www.assetstore.unity3d.com/en/#!/content/2670 (costs money but should work out of the box)
https://www.assetstore.unity3d.com/en/#!/content/509 (not sure how difficult would be to adapt it to capture video)
https://github.com/keijiro/FFmpegOut (in an early stage, but looks very promising)

 

@leavittx If the purchase of resources can provide all the source code? I see there are a lot of dll files.

 

Hey! I've replied via e-mail!

 

New version 0.9.3 is on the asset store!

Changelog:
— Playback speed can be controlled
— Framedrop can be enabled or disabled using the C# API
— Overall stability improvements
— Fix resource leaks (GPU and CPU)
— Updated the IMGUI example scene: set active segment, playback speed, etc
— New Hap Alpha example video

Get it here

 

@izm1252 @adriaanwormgoor Check it out!

 

Thanks very much!I have already responded to your mail.

 

I'd actually like to know this well. We're working on a project where we need to play back as many high-res (2K+) videos as possible, and ran into a wall with other HAP solutions because we couldn't see what they were doing outside of DLLs.

 

Hi I would need to use your plugin to stream RTMP, RTSP and potentially ASX, I tried RTMP in the demo and got an error for FindCodedError, where do i need to install the codecs for the the plugin to find them?

 

@leavittx
Hi! Thanks for the plug-in, we bought the full version and our 4k 360 videos play on the Oculus rift now at 90+ FPS.
We did have one issue, however, the texture quality appears to be downgraded.
Is there any easy fix to this? Are we facing a known problem?
We're using the render to material script and our video appears to be pixelated. We've checked the texture after pausing the scene and the size appears to be correct - 3840 x 2160.
We've also tried using a higher resolution sphere to render the video on, but that's not helped either. It'd be amazing if you could help us out here.

 

Are you wondering about the source code? If yes, the native dll's C++ source code isn't included in the asset store package! And I don't think you really need it for anything! Shouldn't be a problem to run a lot 2k+ videos! Some people managed to run 120 HD videos at once. And if you have questions about the plugin architecture, please ask.

 

This version of my plugin doesn't have anything to do with network streams by any means! Although it has the possibility to open them internally (for other video codecs like h264), for Hap codec it doesn't really makes sense due to high data throughput! Why would you want to do that with Hap?

 

Hey! Thanks for your message! Now I can finally confirm that the plugin works in VR.
Regarding your problem: the video could appear slightly downgraded due to DXT1/5 compression artifacts, but not that much really (gradient areas could look not as smooth as in the source video). Pixelization shouldn't be caused by the Hap codec!

Could you please check on the test scenes included into the asset (render a 1/4 part of the texture if you don't have a 4K display) if the pixelization takes place there as well? If that doesn't helps, you can also play the video in an external player like QtAV. Let me know!

 

The windows version is on sale! https://www.assetstore.unity3d.com/en/#!/content/78907

 

@leavittx I can confirm that the pixelation is not happening inside QtAV for our footage.
We also have a bunch of Media scripts running at the same time, but I made a test scene with just two media scripts running (one to render the video and one for an alpha mask because we need transparency and also shadows to work) and the footage looks pixelated there as well. Any advice?

 

@sunilnayak first of all, please check whether the pixelization takes place in the 02_Hap_ControlsIMGUI example scene included into the Asset Store package! I can do it for you if you send me the videos.

I also wonder whether you're using Hap or Hap Q? Try the latter if you haven't yet.

And how do you encode your videos? I suggest to do it using FFmpeg. AE-like programs could produce broken videos. See the documentation on that included into asset for the instructions!

You can also write me directly via e-mail lev.panov@gmail.com

 

Just noticed that it's 30% off for the full version of my plugin on the Asset Store these days! Be fast before 30.04!

 

@leavittx Sorry about the late reply, we were testing out a lot of different things and found out that it was the source footage itself that was causing the issue. Unfortunately, 4K in 360 (inside a headset) doesn't look as good as we thought it would. But your plugin works perfectly! Thanks again!

Details: We were using HapQ and ffmpeg to encode the footage to HapQ but we needed AE in the pipeline as we had some keying to do. However, we tested the footage we obtained directly from the camera (after converting it to HapQ using ffmpeg) to see if AE was breaking it and the footage looked just as pixelated. We used KodakSP360 cameras to record said footage.

Thanks for your prompt replies!

 

Hi! Testing this asset right now, looks good. It would be great to also have audio following playback speed... Would that be possible?

 

Hi, I’m using the trial version and it’s great, but I have a somewhat unusual problem – playback is too fast! Here’s what I mean…

I’m using AV Movie Capture to record the game output in non-realtime. Frame rates can drop down to around 15 fps during recording. Now the HAP video clip I’m using in the scene has a 60 fps rate and it plays back at that speed even when the rest of the scene is down to 15 fps. As a result, AV Movie Capture records only 1 out of every 4 frames of the HAP video clip. So, upon playback, the HAP clip is sped up by 4 times.

My request is, can you make playback speed of the HAP movie actually match the playback speed of the scene, when the scene is going slower than 60 fps ? Or another way of saying it is, can there be a 1 to 1 correspondence between the frames of the video clip and the frames of the scene?

Do this and you’ve got another customer!

 

@jeroenvdv I will try to do it in the next version of the plugin! Would be a nice feature indeed! Thank you

 

@cerrec I think it can be easily achieved by manually calling Media.StepForward() method every time you need a new video frame to be displayed! Not sure how smooth the audio playback will be in this case, but probably you don't need it in 15 fps case anyway, do you? Let me know if it helps!

 

I like the phrase 'easily achieved' but I have absolutely no idea what Media.StepForward() is. Could you please elaborate?

 

Here's another problem - The Color Space of my project is set to Linear. But the Hap video clip is washed out on playback. If I set the Color Space to Gamma, the video looks correct, but the rest of the scene is too dark. Since I want to continue working in Linear Color Space, is there any way for your plug-in to set the video clip to Gamma so it will blend in correctly with the scene?

 

@cerrec are you familiar with C#? For what you want you need to write some C# code!
I would do something like that in your situation:

1. Attach a new script to your Movie object (it should already has Media script attached!) with a name similar to MovieFrameUpdate
2. In the Update() method of your script, you need to keep track of whether the current frame has changed. I assume you can use Time.frameCount for that.
3. When the new video frame is needed, get the Media component of the current object and execute StepForward() on it.

The code is roughly the following:

Code (CSharp):

1. using UnityEngine;
2. using DemolitionStudios.DemolitionMedia;
3.  
4. RequireComponent(typeof (Media))]
5. public class MovieFrameUpdate : MonoBehaviour
6. {
7.     private int lastFrame = -1;
8.  
9.     void Update()
10.     {
11.         int currentFrame = Time.frameCount;
12.         if (currentFrame == lastFrame)
13.             return;
14.  
15.         lastFrame = currentFrame;
16.  
17.         var media = GetComponent<Media>();
18.         media.StepForward();
19.     }
20. }
21.  

Don't forget to set the Play On Open flag to false!

Regarding your second question, I need more time to get an idea of how to do that.

 

@cerrec okay, I've just tried setting Color Space to Linear in the Player Settings and building one of the example scenes. And it works correctly, I think! I can see both the video and the reset of the scene (i.e. the IMGUI elements).
Here is what exactly I get. Color space set to Gamma:

Color space set to Linear:


The Unity tutorial on the Gamma/Linear workflows approves that both linear and gamma textures can be used in linear modes, so I see no problem here. Maybe you can give me some more details of the problem you're facing?

 

Thanks for the responses.

I tried your script – it entered in with no errors (after I added ‘ [ ’ in front of RequiredComponent).
Unfortunately it had no effect. I know a little about C#, but not enough to do any real programming.
BTW, I’m using the excellent FFmpegOut to record the play, if you want to try it yourself.

As for the gamma issue, here’s a more detailed explanation of my setup. I have an ordinary 3D scene with a ground, 3D objects, skybox, lights, etc. The project setting for Player is set to Linear. The scene is lit so everything looks good with this setting. The Hap video clip is mapped onto a plane which is an object in the scene. The shader for the plane is set to Unlit. With this setup, the video appears washed out and over exposed. However, if I set the Player to Gamma, the video is rendered correctly, but the rest of the scene is now under exposed and very contrasty. So the Linear and Gamma settings appear to affect everything in the scene globally. What would be great is if you could isolate the video from that and give the user the ability to set Linear and Gamma for it independently.

Hope that helps.

 

Does this support stereoscopic playback for VR HMDs?

 

Hi,

Purshased Demolition Media HAP for OSX and PC, tried all scene example on unity 5.6.2p1 on MAC OSX 10.12.5, HAP codec installed.
In play editor mode all works fine, video are playing, but if I build for OSX, video don't play, no error message !

Editor:
https://screencast.com/t/1KlFiTiIr2

build:
https://screencast.com/t/oh2O33HrL4

Thanks for help

 

Hi, any news ? sended mail and post on forum, no response... Tested project with 5.4.2p3 too, same problem

 

@marcusestes Hey! It's totally possible to play top-bottom or left-right stereoscopic videos with my asset! But with some hand work at the moment. You need to keep track of which eye is currently being rendered by the VR system and correspondingly change the texture coordinates of the video material. I think I can include the needed scripts to the package, but I'd like someone to help me with testing them first (since I don't own any VR gear). If you (or anybody else) are interested with giving it a try, please write me an e-mail to lev.panov@gmail.com. Thanks!

 

@stephane-cottin Hey! I think I've found what the problem is. Please check your mail!

 

@cerrec Hey! At that point, this is slightly out of the forum thread scope. If you could send me a simple version of a Unity project which shows your problem, then I may be able to help you. You can find my e-mail two posts above! Cheers

 

Thanks for help, works fine now ! my setting : https://screencast.com/t/uCofZOLUWF6

 

Hi,
I keep getting this error when I try to run a 4K HEVC clip.

[DemolitionMedia] OpenFailed: FindCodecError
UnityEngine.Debug:LogError(Object)
DemolitionStudios.DemolitionMedia.MediaopulateErrors() (at Assets/DemolitionMedia/Scripts/Components/Media.cs:434)
DemolitionStudios.DemolitionMedia.Media:Update() (at Assets/DemolitionMedia/Scripts/Components/Media_Unity.cs:134)​

Please can you tell me what I might be doing wrong? Should I be downloading the Codec seperately?
This is using the demo version.
thanks,
V

 

@vishal_changrani Hey! No codec installation is needed! But you need to re-encode your source HEVC videos to the Hap format. It allows to play videos with very low CPU usage. Please see the .pdf file included in the unity package for details on how to do it. I think ffmpeg command-line tool will work nicely for you!

 

@leavittx I've recently purchased and I'm running into an issue while using your plugin, after a while of usage the unity player sporadically crashes. The application has two media components and is swapping a RenderToUGUI sourceMedia as well as swapping the Media sources. I've tried turning off Enable Audio, Use Native Audio Plugin, Open On Start, Play On Open and all combinations of which and still hasn't helped and in some instances made it worse.

error.log
Unity Player [version: Unity 5.5.2f1_3829d7f588f3]
nvwgf2umx.dll caused an Access Violation (0xc0000005)
in module nvwgf2umx.dll at 0033:7eda0e8c.
Error occurred at 2017-08-16_151814.

Read from location 000000cc caused an access violation.

It seems likes it's crashing the gpu though i'm not sure, I've updated to the latest GPU drivers.

I've made sure that I've downloaded the latest Asset from the Asset Store, Calling Open and then Play doesn't seem to always fire if Open On Start, Play On Open are not both set to true.

Any advice or direction would be greatly appreciated.

EDIT: Only crashing the final outputted build of the project and the crash doesn't get caught by the debugger.