Feedback Reducing catalog.json's large file size

Hi there! We're using Addressables for our WebGL game so we can reduce file sizes. However, we've noticed our catalog.json file is getting really large; we're nearly up to 1mb and it keeps growing as we continue to develop.

We know it's cached, but first time loads are ultimately the thing we're trying to reduce.

Something we were hoping would be possible would be for Unity to compress then catalog.json on build, and decompress it once downloaded - it appears it can be compressed to about 1/5th of the size using brotli!

We suspect that another approach is to break up our project and use LoadContentCatalogAsync to only load catalogs (that then load assets) when required and have a "primary" file thats super small; but we don't know if this'll work or not.

Is anyone from Unity able to speak to this?

 

Oh brilliant, thank you! We've switched to using GUID, which has saved a few bytes - but the compress catalog setting doesn't appear to do anything? The catalog is still built as the same json file, and the file size doesn't appear to be further reduced.
We're on addressables 1.19.17

 

Ah I see! It WAS creating catalog.bundle in Library/com.unity.addressables/aa/[Platform], but it also creates catalog.json in the remote build path. Is this to be expected?

 

I think what would really be useful here is to have the remote catalog compressed as well. We have other JSON/text assets sitting on a custom server that are simply ZIP compressed. As a part of the Addressables workflow it would be nice if the remote catalogs could also be a compressed file. I notice the Addressable code is often hardcoded to expect a *.json (and *.hash) file however.

 

I think what would really be useful here is to have the remote catalog compressed as well.As the project grows, the resources will also grow, and the remote catalog.json will become larger and larger. If it is not compressed, it will take a very long time to download from the CDN, and there is no download progress prompt. Wait until after executing UpdateCatalogs, it would be best if the remote catalog.json can be compressed.

 

As the project grows, the resources will also grow, and the remote catalog.json will become larger and larger. If it is not compressed, it will take a very long time to download from the CDN, and there is no download progress prompt. Wait until after executing UpdateCatalogs, it would be best if the remote catalog.json can be compressed.

 

I think what would really be useful here is to have the remote catalog compressed as well.As the project grows, the resources will also grow, and the remote catalog.json will become larger and larger. If it is not compressed, it will take a very long time to download from the CDN, and there is no download progress prompt. Wait until after executing UpdateCatalogs, it would be best if the remote catalog.json can be compressed.

 

Our remote catalog json file is over 10mb and will continue to grow as we add new content to our game. We have already optimized the catalog as far as we can by using only GUID identifiers and optimizing catalog load paths.

If I zip the catalog file it goes down to about 1.5mb which is much more acceptable.

Has anyone got a custom provider they can share to load the catalog from a zip instead of json file?

 

I encountered same issue and found a simple way to solve it: just enable gzip compression for .json on your http server. The UnityWebRequest used for downloading catalog.json will handle it correctly. It works on Android and iOS, but I'm not sure if it works on other platforms.

 

Hi thanks for your reply. We are using amazon S3 and Cloudfront CDN. Do you know how I can set that up to work with these?

 

Sorry, I'm not familar with them.

 

I am also looking at reducing the download size of our bundle. It doesn't work yet with S3/Cloudfront, but I think it is due to the fact that we have not uploaded it as the correct content type. We marked them as 'application/octet-stream' because this is what we do to our bundles when we are uploading them via our build tools. The json catalog is just another file so we give it the same treatment..



I noticed though I can edit the type to application/json and duplicate the file. The duplicated file will be gzip'ed correctly if you try to download it sending the correct request header (accept-encoding). It works for both S3 and Cloudfront. I did noticed that the original object still did not respond to accept-encoding header though. It always returned uncompressed. So it seems your object has to be created with application/json to start with. You can't change it later and have it work.

I have not yet verified if this works with Unity web requests. I am not sure if they are sending accept-encoding headers... from the sounds of it from this thread it looks like they are.. So it should 'just work'.

You can use a program like Insomnia to download your files and test for yourself. Send a GET request to the file and make sure you add the appropriate header to the request. The response size should change accordingly.






TLDR

In 'theory' all you have to do to make it work with S3/CF is to make sure your content type for your json file is `application/json`. The Unity side should 'just work'.. Though I have not tested this yet. But that's all you need to do on the S3/CF side.