Compressing Asset Bundles in WebGL

I have setup my webserver and game to deploy the .unityweb files using gzip compression and this is working very well. However, I am trying to do the same with the asset bundles and having less success.

I gave the asset bundles the extension 'unity3d' and tried two approaches.

1. Gzip them after the build and include an .htaccess file which does "AddEncoding gzip .unity3d". This fails with an error "Error while downloading Asset Bundle: Failed to decompress data for the AssetBundle".

2. Leave the asset bundles LZMA compressed but not Gzipped, and instead put "SetOutputFilter DEFLATE" into .htaccess applied to unity3d files. This appears to work - I see the compressed file sizes in the Chrome Network tab, and the game runs. The downside here is the browser is having to compress the files at runtime for each request.

It looks to me like the .unityweb files are actually gzipped while the asset bundles are not, and consequently Unity expects to gunzip the .unityweb files but not the asset bundles and therefore gives the error listed above.

What is the best way to approach this?

Thanks!

 

Build the assetbundles as uncompressed in Unity and then use an external program to gzip them. Let me know how it goes, been considering doing this for Brotli compression.

It would be interesting if someone could benchmark the difference in load time (unity engine handling decompression vs the browser). I suspect the browser approach is better, but how much (is it worth it)?

 

It would also be nice if (just for the webgl platform) @Marco-Trivellato provided us an option to compress asset bundles using gzip/brotli within Unity. Using an external program to do this is a production killer and the Unity installation already includes a gzip/brotli compressor!

 

I'll only get to try this on Monday, but in the meantime - are you suggesting not using LZMA compression and then manually using Gzip? If I serve these with appropriate encoding, will Unity then happily use them? It won't moan about the 'Failed to decompress data' again?

Regarding the difference in load time, I think it's a clear win for serving gzipped and letting the browser handle the decompression. Firstly, it isn't hard to setup server-side (barring this forum post of course!), and secondly the browser can handle the decompression in compiled native code in a separate thread while Unity would have to do so in Javascript on the main thread.

 

I agree that in many cases this is not a production killer. But in our situation, we've got like 40 or so people making asset bundles across a spectrum of environments. Ensuring that those external tools work correctly on all those platforms is challenging. In addition, some of those 40 people are not what I'd call "technical" (e.g. writers). Thus if something goes wrong with those tools, it may not be noticed until later when the root of the problem is not so obvious. Huge workflow ramifications for us.

These of course are not impossible problems to solve. But as you aptly said, it always comes down to priorities. At the moment, much easier from a workflow standpoint to compress with LZ4.

On a somewhat related note, when using the "CachedXMLHttpRequest plug-in", it caches the LZ4 file itself right? So we still have to decompress when loading from cache correct? I've been debating (when priorities allow...) on seeing if LZ4 is even a net win. I suspect so (especially for those with poor connections), but not entirely sure with the decompression overhead and our use of caching.

 

I've tried the suggestion above but still get the error. Here is what I'm doing:

1. In Unity Editor, set the asset bundle name to "main.unity3d"
2. Execute the following in a build script: "BuildPipeline.BuildAssetBundles( "Assets/AssetBundles/webgl", BuildAssetBundleOptions.UncompressedAssetBundle, BuildTarget.WebGL );"
3. In the same script, compress each unity3d file with gzip, then remove the extension 'gz'
4. Include an .htaccess file that says "AddEnconding gzip .unity3d"
5. Run the game and observe an error when main.unity3d is downloaded: "Error while downloading Asset Bundle: Failed to decompress data for the AssetBundle 'AssetBundles/webgl/main.unity3d'."

What have I done wrong?

 

And in the network tab the headers are correct? In other words, when you download the asset bundle file through a browser (not Unity), is it the raw/uncompressed (non-gzip) asset bundle?

 

I believe they are.

HTTP/1.1 200 OK

Date: Tue, 15 May 2018 19:36:08 GMT

Server: Apache/2.4.7 (Ubuntu)

Last-Modified: Tue, 15 May 2018 14:22:45

GMT ETag: "1aa14-56c3f55fdf740"

Accept-Ranges: bytes

Content-Length: 109076

Keep-Alive: timeout=5, max=100

Connection: Keep-Alive

Content-Encoding: gzip

That 100kb content length is the gzipped size - the file that appeared on my client computer is 500kb (uncompressed) and is called main.unity3d. This is what Unity appears to be seeing as well. This is what I don't get - Unity is being given an uncompressed asset bundle from the browser and then moans about being unable to decompress it... it seems like it's expecting it to be compressed but it isn't.

 

And the checksum of the original uncompressed asset bundle (before you gzipped it) matches the file you downloaded from the browser? Can you also confirm that the original asset bundle "works" (better even, does the asset bundle you download from the browser work if you rehost it on your sever with gzip encoding off)?

 

Is there an available API to use Unity's compressor to do that? Or do we still have to use a third party solution for that sort of script?

 

I can just say we ended up doing more or less what Marco suggested. We use the built-in FileStream/GZipStream in .net to compress our (uncompressed) bundles that have been built by Unity. It actually works pretty well and not a "production killer" that I first thought it would be. (Sorry Marco!)

 

Hm, that's interesting. How did you load them? Did you keep the .gz extension and use that as the url and it just works? Or did you have to remove the .gz extension and configure the host to set the encoding to gzip?

 

Ah figured it out, just overwrite the file with the compressed version (no .gz) and set the content-encoding on the host, then it just works!