Loading image files in native code

Hey all,

We are using a "trick" of using images as .bytes (TextAsset assets). At runtime we load these into a Texture2D object, and then create a sprite from that texture (or use directly as a texture). We do this since it dramatically decreases the game package size (the .bytes files for PNG images result in a smaller game package).

The problem is that for larger files, this process is very slow at runtime and becoming noticeable by the players.

BTW - we are targetting mobile platforms.

Specifically, the method that takes the longest is Texture.LoadImage (leaving the generated GC from reading the file aside).

I thought about one way of optimizing this by performing the texture creation in native code (http://docs.unity3d.com/ScriptRefer...ptReference/Texture2D.LoadRawTextureData.html)

Has anyone tried this method before? I know it probably has its own quirks and limitations, but i wonder if that could give any performance improvements over loading the image using LoadImage ?

 

So, we moved all of our images into asset bundles for this reason. The downside of using png/jpg is:

- All loading is done on the main thread, causing aa sizable glitch (up to 100ms on our low end devices)
- Memory is allocated on the mono heap for the .bytes
- Texture is not compressed on the GPU. Texture compression is a 4:1 to 16:1 savings depending on platform/format.

I haven't tried using LoadRawTextureData before, but we talked about it quite a bit and are considering moving some of our pipeline to it if Unity cannot fix the cache server issues soon (they've fixed several, but often Unity won't generate the same hash or upload a correct hash- this causes our project to re-import every time we build it on a new machine, which can easily take 20 hours).

From my understanding:

- You'll need to compress the textures into each format you support (PVR, ETC, ETC2, DXT, etc). For uncompressed, you can likely processes it into the raw bytes of a color[], but I'm not sure what it expects there.
- Loading will still allocate on the mono heap
- Loading will still happen on the main thread, causing a glitch, but likely many times faster than the png/jpg technique as it won't have to decompress the data into a large buffer, just upload the bytes.

If you give this a try, I'd love to know your findings on upload times.

 

That is just an idea, i was actually looking for people who implemented something similar, to see if this really has any benefits.

When you say "we moved all of our images into asset bundles for this reason.", do you mean that all of your assets are now in asset bundles, but what improvement that does that give you exactly ?

Also, are you images stored as .bytes or .png inside the asset bundles ?

 

Neither, they are compressed into platform specific versions for each compression type we support, then loaded dynamically using the asset bundle system, which allows them to be uploaded to the GPU without blocking the main thread.

 

How does the asset bundle system allow loading without blocking the main thread ? i believe it just uses the standard loading APIs (even when doing that async)

 

Sorry, I wasn't explicit enough. Here's what happens when you download a PNG:

- WWW or WebRequest gets data from server
- Bytes are copied to the mono heap when you call .bytes, and you pass that to the loading API
- bytes are sent to C++ side and decompressed from PNG to uncompressed texture data
- Texture is uploaded to the GPU
- We measured times of 100+ ms for large textures under this scenario

Once you grab the byes and send it to the loading function (usually in the same call), it's all blocking the main thread.

When you load a compressed texture from an asset bundle using WWW:

- WWW gets data from the server
- Bytes are copied to the mono heap and sent to the asset bundle loading API (back to C++)
- Texture is uploaded to the GPU in ready compressed form, asynchronously (as of 5.2?)
- Texture is between 1/4 and 1/16th the size of the PNG depending on compression format/options

The main advantage here is that you avoid the PNG decompression, which can be slow, and work with textures that are many times smaller since they are compressed.

When you load a compressed texture from an asset bundle using WebRequest (5.3+):

- WebRequest gets data form the server, decompresses the asset bundle, and lets you know it's ready to load (not on main thread)
- Texture data is uploaded to the GPU in ready compressed form, asynchronously.

Here, we avoid the mono heap completely, as well as most of the work on the main thread. Only the Texture2D object exists in C# land, which is just a wrapper around the actual texture.

 

How exactly are you loading the texture from the asset bundle?

The only API i know using WWW is something like:

www.assetBundle (and then calling the normal asset bundle loading APIs).

 

http://docs.unity3d.com/Manual/UnityWebRequest.html

Note that calling .assetBundle doesn't load the bytes of the asset bundle into memory- it's just a wrapper around the data, which remains on the C++ side until you load data into the C# side- and Texture2D is just a wrapper around a raw texture, which can remain on the C++ side unless you use ReadPixels on it..

 

Thanks. i am still kind of missing something:

1. How do you load textures (or any assets) from asset bundles using the WWW class? from what i know, you can load an asset bundle using a few different ways, but after that, you load assets from it only by using the API from the AssetBundle class.

2. Why does loading things from an asset bundle any different than loading assets in any other way? how is that more efficient ?

3. Could you perhaps post a few lines of code that demonstrate how you're using things? perhaps that could really help us out in tweaking our system to be more efficient.

 

1. Yes, you use the asset bundle API to load the texture from the asset bundle (WWW is just used to download the bundle). Unity has a project on the asset store with an asset bundle loading system you can download to learn more. You can load a texture2D in the exact way you load any other asset from an asset bundle.

2. Because the data doesn't have to be marshalled between the c# and c++ side of Unity; only the wrapper class (Texture2D) has to exist on the C# side, the texture itself can exist in C++ and be uploaded to the GPU without going through the mono heap. If you load a PNG via .bytes, you load that byte array into the mono heap because unity doesn't know to bypass all of this; it's just generic array of data which you can do stuff with; Unity doesn't know it's a texture. When you use the LoadTexture API, you pass that byte array back to the C++ side, where it decodes the texture into an uncompressed texture and uploads it to the GPU.

Also, texture compression is pretty useful in saving memory/bandwidth, so having your textures compressed in hardware native formats is a huge win.

3. I suggest you download the asset bundle example project from Unity. You can then assign your textures to bundle(s), load the bundle, then load the texture(s) out of the bundle. If you rewrite the example project to use UnityWebRequest instead of WWW, then the textures can be loaded without allocating the texture data in the mono heap (it will stay on the C++ side, only the texture2D object will be allocated on the mono heap side).

Note that asset bundles have a lot of undocumented issues on Unity versions before 5.3.2p2; such as not being able to load more than ~200 of them on iOS, etc. We've been working closely with Unity to fix some of these issues, as our game has over 5000 asset bundles and we've been a major stressor of the system.

 

Sort of related to the rest of this discussion, but I've implemented a caching system using LoadRawTextureData. For the vast majority of projects using asset bundles is likely better but I've not tried them yet. (Specific case, wanted end users to be able to edit textures without using unity)

My files are stored .png, and I do some processing and combine channels / remove alpha / compress if needed. Once I have the texture2D as I want it, I store the GetRawTextureData bytes to disc (compressed with LZ in my case).

Ignoring loading from disc and decompressing (can be threaded / preloaded), the time it takes to load a 2048x2048 RGB is ~8 ms using LoadRawTextureData. For a 2048x2048 DXT1 it takes only about 1 ms. Both have mipmaps included, so it's 16 MiB in 8 ms, and 2.7 MiB in 1 ms. My game currently only has 2048x2048 texture atlases using this cache, so I can't measure the speeds of other sizes but it seems to scale quite nicely.

For comparison, my initial cache creation takes about 50-400 ms per texture depending on configured complexity.

This is measured on an i5-4670k & gtx 770, but even if a pc is say, 3 times slower, the ms penalty of loading ~1k or 2K DXT textures is still in usable ranges for realtime usage. Unless you happen to trigger a GC run.

P.S: I'm also using LoadRawTextureData in a graph UI texture, and that works quite well too =).

 

Are there any limitations when using LoadRawTextureData ? what should be the source image formats, etc?
BTW we are targetting mobile, i assume the numbers should be different to what you're seeing on a desktop platform.

 

It's mostly what you'd expect it to be. It takes the raw pixel data, without any overhead / decoding/ encoding. So a RGB format needs RGBRGBRGB-like bytes, while a DXT1 format will want the full 4x4 block encoding. It also needs mipmaps if the texture2d has them enabled; they're simply behind eachother in the array, mip 0 first.

See http://docs.unity3d.com/ScriptReference/Texture2D.LoadRawTextureData.html for obvious reasons. The only thing I missed was a (byte[], offset, length) overload, but they seem to have sneakily added an (IntPtr, length) overload I didn't know about which can be used as one I suppose.

For mobile, if you're going this way, you probably want to have an editor script that converts the compressed assets to (compressed?) byte[] and saves it somewhere (streaming asset folder iirc?).

There's also this http://docs.unity3d.com/Manual/AsyncTextureUpload.html, but I've got no clue about it.

 

Thanks for the info !

I've done a little experiment with LoadRawTextureData / GetRawTextureData. The raw data is (as expected) a lot larger than the original texture (png or jpg). You say that you store it compressed? and then when you load it from disk you decompress it before loading the data into a texture?

The big issue here is memory usage as i see it (since it's probably faster, if you dont consider the extra complexity that is needed).

 

Yeah, I'm compressing them with CLZF iirc. Some examples, all mipmapped;
1) 2048x2048 RGB specularity/emissive/smoothness; Raw: 16.8 MB, LZ: 635.2 kB, PNG: 167.3 kB.
2) 2048x2048 RGB normalmap; Raw: 16.8 MB, LZ: 4.9 MB, PNG: 2.3 MB
3) 2048x2048 DXT albedomap; Raw: 2.8 MB, LZ: 130.9 kB, PNG: 99.5 kB

Personally I have a lot of 2048x2048 mipmapped atlas textures, so I can decompress it into a buffer that's reused. So I only need 1 raw RGB buffer and 1 raw DXT buffer. The other garbage created is the LZ compressed byte[] read from disc. Not a lot that can be done about that. If the textures vary in size/format/mipmap a lot, it'll create a lot of decompressed garbage.

But given that there is now an overload that takes an IntPtr into LoadRawTextureData, it may be possible to alter the decompressor to use the marshall alloc, and then manually free the data. I'm not sure about .net file IO giving a pointer option, but if so it may be an option as well. Though I'm not 100% sure if you can use unsafe c# code on mobile.

 

@jbooth i still don't understand the advantage of packing textures into an asset bundle (even if they are compressed) vs. having them available in the resources folder or even directly in the scene.

For example - what makes the way you suggest better than, say, including compressed textures in the Resources folder and then loading them (async) using Resources.LoadAsync<T> ? i still get a Texture2D object (that unity creates for me, without having to go through the mono heap). The texture is also compressed (e.g: uploaded to the GPU that way) so i am saving the bandwidth and all the advantages you mentioned.

What is the extra benefit of having asset bundles in this case ? (strictly for the loading part, i agree that the main game download size would be smaller if i can download bundles that are not part of the original game package).

 

There is no advantage to asset bundles if:

1. You only ship on one texture compression format
2. You don't care about app size

For our game, we support 6 compression formats across multiple platforms; and as far as I know there isn't a way to include a resource only on certain platforms; so we'd have all 6 textures on every platform, even if that platform doesn't support 5 of them. Additionally, because webGL is one of our platforms, and webGL requires that all resources be loaded before the first scene starts, it would cause a massive delay in loading on webGL..

 

Some interesting information here.
Is there any way to save a Texture from unity to the disk as a dxt compressed file (ie dds file)?
I mean during run time.

 

https://docs.unity3d.com/ScriptReference/Texture2D.GetRawTextureData.html

This will return the raw byte[]; if you want to load it outside of unity, there's likely some header information required to make it a .dds file, but that will get the raw data..

 

Isn't a raw byte[] an uncompressed huge file texture?
I wanted to get from the server images and then cache them locally with the quickest possible loading.
Maybe assetBundles is the only way - that will have to have the server to support that.

 

bytes are just bytes - in this case, the raw compressed dxt data (with no additional compression applied, or whatever texture format is native to your platform). An asset bundle is definitely a better way to go though, in that it will be additionally compressed beyond the dxt data, be platform independent, cache automatically for you, and only be re-downloaded on a client when it's actually changed..

 

Since when asset bundles are platform independent? From my experience, ios and android are using different compression. When I use an ios asset bundle on android and vice versa, I get magenta textures