NatReader - Free Video Decoding API

Get NatReader on GitHub.

Hi all! It's been a while since I've announced a new API, but I'm really excited for this one. I'd like to introduce NatReader API. NatReader is a lightweight video decoding API. I began working on NatReader because I had use for a transcoder. My other API, NatCorder, already handles the encoding aspect of transcoding. NatReader completes the other half.

Another goal of mine with NatReader, one which I need help from the community to achieve, is to create a video player plugin that works and works well for everyone. Unity's VideoPlayer has bugs here and there; and AVPro Video is prohibitively expensive. Hopefully, this is a start to a complete featureset.

NatReader Features Include:
- Extremely easy thread-safe API. Use a `foreach` loop to access frames in a media file.
- Support for decoding on iOS, Android, and macOS (editor and standalone). WebGL and Windows coming soon.
- Support for transcoding when used with NatCorder.
- Open source. Yes, you read that right. Submit pull requests with additions and improvements!

Features I intend to see in the future include:
- Support for streaming from remote URL's
- Support for decoding audio for full video playback

___

I would love to hear any and all the feedback you have. What features should we look into? What should we prioritize? Let's create the best solution for video decoding in Unity Engine together.

 

Hello @Lanre . Downloaded the project from GitHub. Didn't change a thing (besides opening it with Unity 2018.3.14f1 instead of 2018.3.13f1). The example scene when played in Editor just shows white rectangle and gives this error:

EntryPointNotFoundException: NRCreateMP4FrameReader
NatReader.MP4FrameReader..ctor (System.String uri, System.Single startTime, System.Single duration) (at Assets/NatReader/Plugins/Managed/MP4FrameReader.cs:64)
NatReader.Examples.Playback+<Start>d__2.MoveNext () (at Assets/NatReader/Examples/Playback/Playback.cs:19)
UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) (at C:/buildslave/unity/build/Runtime/Export/Coroutines.cs:17)

Trid to build to Android, it only shows Unity skybox. What could be the problem?
Thank you in advance.

 

The Windows backend isn't complete. On Android, you have to pay special attention to the path of the MP4 file being decoded. Check the logs for info on why it isn't working.

 

Thanks for the answer. I attach the complete log of what I'm getting on Android from the demo scene. The project is exactly as it is on GitHub, didn't change a thing, except for slight Unity version difference, but I don't really think there should be much difference between 2018.3.13f1 and 2018.3.14f1. Correct me if I'm wrong.

 

On Android, you have to place the video in the app's persistent data path. In your case, it isn't able to find the video.

 

Hi,

I get an error with the FrameReader, after i have found the video on Android:

2020-02-04 11:48:40.420 32767-350/com.jabii.jabiiapp I/Unity: Duration: 16.08867 Size: (1280, 720) Framerate: 9 URI: file:///storage/emulated/0/Android/data/com.jabii.jabiiapp/files/recording_2020_02_04_11_48_18_153.mp4
UnityEngine.DebugLogHandler:Internal_Log()
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
Jabii.Controllers.<SetThumbnail>d__30:MoveNext() (at /Users/aawezz/Jabii/Unity Jabii/Assets/Scripts/Controllers/EvaluationController.cs:254)
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
UnityEngine.MonoBehaviour:StartCoroutineManaged2()
UnityEngine.MonoBehaviour:StartCoroutine(IEnumerator)
Jabii.Controllers.EvaluationController:OnReady() (at /Users/aawezz/Jabii/Unity Jabii/Assets/Scripts/Controllers/EvaluationController.cs:181)
Jabii.Controllers.<DoChangeScene>d__6:MoveNext() (at /Users/aawezz/Jabii/Unity Jabii/Assets/Scripts/Controllers/Navigation/NavigationController.cs:97)
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

(Filename: /U
2020-02-04 11:48:40.512 32767-1017/com.jabii.jabiiapp E/AndroidRuntime: FATAL EXCEPTION: FrameReader
Process: com.jabii.jabiiapp, PID: 32767
java.lang.Error: FATAL EXCEPTION [FrameReader]
Unity version : 2018.3.6f1
Device model : HUAWEI SLA-L22
Device fingerprint: HUAWEI/SLA-L22/HWSLA-Q:7.0/HUAWEISLA-L22/C432B130:user/release-keys

Caused by: java.lang.NoSuchMethodError: No static method copyFrame(Ljava/nio/ByteBuffer;IIILjava/nio/ByteBufferV in class Lapi/natsuite/natrender/GLBlitEncoder; or its super classes (declaration of 'api.natsuite.natrender.GLBlitEncoder' appears in /data/app/com.jabii.jabiiapp-1/base.apk:classes2.dex)
at api.natsuite.natreader.MP4FrameReader$3.onImageAvailable(MP4FrameReader.java:138)
at android.media.ImageReader$ListenerHandler.handleMessage(ImageReader.java:658)
at android.os.Handler.dispatchMessage(Handler.java:105)
at android.os.Looper.loop(Looper.java:156)
at android.os.HandlerThread.run(HandlerThread.java:61)

 

I responded to your email. Download the latest code from GitHub; this issue has been fixed.

 

@Lanre I'm using NatReader to generate video thumbnail.
My code:

Code (CSharp):

1.                 using (var reader = new MP4FrameReader(galleryPath + child.GetComponent<VideoItem>().name))
2.                 {
3.                         Texture2D frameTexture = new Texture2D(640, 1386, TextureFormat.RGBA32, false);
4.                         frameTexture.LoadRawTextureData(pixelBuffer);
5.                         frameTexture.Apply();
6.                         ShowToast(frameTexture.width.ToString());
7.                         int width = frameTexture.width;
8.                         int height = frameTexture.height;
9.                         var size = width < height ? width : height;
10.                         Rect rect = new Rect(0, 0, size, size);
11.                         Texture2D t2d = TextureTools.CropWithRect(frameTexture, rect, TextureTools.RectOptions.Center, 0, 0);
12.                         child.GetComponent<RawImage>().texture = t2d;
13.                         yield return eof;
14.                         Destroy(frameTexture);
15.                         break;
16.                     }
17.                 }

I'm reading only the first frame from the reader. I have multiple videos with different resolutions. My doubt is, it possible to get video height and width from the returnded framebuffer?

 

The frame reader exposes the frame size (reader.frameSize.width and reader.frameSize.height).

 

Thank you for your response.

 

I'm getting this error in iOS


2020-06-07 12:23:47.361487+0530 ios[2281:1008812] NSURLConnection finished with error - code -1002
2020-06-07 12:23:47.365437+0530 ios[2281:1008069] NatReader Error: MP4FrameReader failed to find video track for asset at '/var/mobile/Containers/Data/Application/0752FA99-154B-4FD9-8E9D-939B6887AAED/Documents/files/tmp/recording-1591512349.mp4'

note: The video is present in the path and it is being played by a video player. This error is not occurring in android.

Please help me to fix this.
Thanks in advance!

 

The download finishes with an error code, so the downloaded file is most likely corrupt. This is most likely the cause of the issue. Resolve that and try again.

 

I'm not downloading the video. I'm recording the video using natcorder and I'm using Natreader to get thumbnail.


more Logs:

-> applicationWillResignActive()
2020-06-07 17:39:56.105978+0530 ios[2332:1055535] FCM: Connected to FCM
-> applicationDidBecomeActive()
2020-06-07 17:39:59.116250+0530 ios[2332:1055535] NatCorder: Prepared MP4Recorder video encoder at resolution 840x1120@25.000000Hz with average bitrate 5909760 and keyframe interval 3s
2020-06-07 17:40:03.910160+0530 ios[2332:1055535] NatCorder: MP4Recorder finishing
2020-06-07 17:40:14.433993+0530 ios[2332:1055535] [Snapshotting] Snapshotting a view (0x12c8237f0, UIInputSetContainerView) that has not been rendered at least once requires afterScreenUpdates:YES.
2020-06-07 17:40:17.692982+0530 ios[2332:1056335] NSURLConnection finished with error - code -1002
2020-06-07 17:40:17.697945+0530 ios[2332:1055535] NatReader Error: MP4FrameReader failed to find video track for asset at '/var/mobile/Containers/Data/Application/BE18848E-CC26-4ED6-AB8C-F6543F1F0C7E/Documents/files/tmp/recording-tmp.mp4'

 

Then why is the video in `tmp`? NatCorder never saves to a temp folder. Are you modifying NatCorder to save to a different location? Also, you'll need to prepend the "file://" protocol to your path when creating the frame reader.

 

Yes, I'm moving the file after recording is finished.
pretending "file://" helped me. Thanks a lot.

 

Sounds good. I'm going to remove this requirement soon.

 

Been using NatCorder for our project and have really enjoyed it! Currently trying to figure out a way to merge more than one .mp4 file together with it's audio. Would love to see audio decoding in the future. I assume ffmpeg is my best option for now.

 

Hm NatReader doesn't support audio, so your best option is FFmpeg.