7zip, lzma, LZ4, fastLZ, zip/gzip & brotli native multiplatform plugins.

Hi all.

A small fix for the decompress asset bundle function (lzma plugin):

Code (CSharp):

1.  
2.  
3.   public static  int decompressAssetBundle(byte[] inBuffer,  ref byte[] outbuffer){
4.  
5.      int offset = 0;
6.    
7.      for(int i=0; i<inBuffer.Length; i++) {
8.        if(i>160) break;
9.        if(inBuffer[i] == 0x5d) {
10.          if(inBuffer[i+1] == 0x00) {
11.            if(inBuffer[i+2] == 0x00) {
12.              if(inBuffer[i+3] == 0x08) {
13.                offset = i;  break;
14.              }
15.            }
16.          }
17.        }
18.      }
19.    
20.      if(offset==0 || offset>160) return 4;
21.  
22.      GCHandle cbuf = GCHandle.Alloc(inBuffer, GCHandleType.Pinned);
23.      IntPtr ptrBundle = new IntPtr(cbuf.AddrOfPinnedObject().ToInt64() + offset);
24.      int uncompressedSize = (int)BitConverter.ToUInt64(inBuffer,offset+5);
25.      Array.Resize(ref outbuffer, uncompressedSize);
26.      GCHandle obuf = GCHandle.Alloc(outbuffer, GCHandleType.Pinned);
27.    
28.      int res = Lzma_Uncompress(ptrBundle, inBuffer.Length-offset, uncompressedSize, obuf.AddrOfPinnedObject(), true);
29.  
30.      cbuf.Free();
31.      obuf.Free();
32.  
33.      //if(res!=0){/*Debug.Log("ERROR: "+res.ToString());*/ return res; }
34.  
35.      return res;  
36.    }
37.  
38.  

Edit: the update is up on the AssetStore.

 

just after importing the plugin i am pretty annoyed. it has folders all over the asset directory, first thing i have to do is clean up the mess it created.

 

Hi.

The asset has 2 parts.Folders that demonstrate the features of the plugins AND the _plugin_packages folder from where you can install individual plugins in a plugins only directory.

Thank you.

 

update version 2.08

- Compatibility fixes for Windows Phone 8.1 and WSA8.1. Sdk8.1 should work out of the box now without the need to provide your own IO functions.

- Added a benchmark scene.

 

Can I use it locally to reduce build size on android? i.e., Compress in editor and then on android phone decompress textures and materials and assign to respective models?

 

Yes you can. But it depends on the use case.

For textures, if you compress some raw files you will not have the internal etc compression.

Some people use this to compress asset bundles, extract and then load them.

 

Oh its you,I dint noticed....Multitasking lot of assets..nice

My use case is as follow,
Currently my apk size is going way to above 100mb , So I am planning to compress all prefabs and its references (tex,materials,fbx,scripts) in one 7z file then on run time decompress it on device...Will it work that way? WIll my references to prefab will be intact ? The prefab are basically race tracks , UI Canvas etc.

 

Hi (Yes, I work on various projects simultaneously, including contract jobs. Makes me less bored.)

On the topic now.
I am afraid that this would not work. Unity makes internal conversions on these files. The only thing you could do about them is to create asset bundles and compress them with the highest compression you can. Then you would decompress the assetbundles and load from there.

One other use case is when you have data that can be loaded on runtime, like wavefront obj models, databases etc.

 

mm thanks for info, seems like need to go over asset bundles architecture to understand and implement it....Thanks for clarifying...

Keep "A dog's life software" alive with insta classic ocean ...All the best

 

Thanks. I want to use the ocean in a project of mine, and I am a speed freak, so expect good things in the future.

 

update version 2.09

- Added bitcode supported plugins for iOS.

 

Is 7ZIP support the extracting volume?

 

Hi. Sorry, it does not support multiple volumes for now.

 

Hi,
I just bought the 7zip library and am deeply impressed! Cool stuff!
Is there also a Apple TV platform version planned?

Thanks!

 

Hi.

If you look in the Plugins/iOS folder there are various iOS variations zipped.
tvOS and watchOS are included.
Unzip the plugin that you want and you should be good.

Thank you.

 

Thanks for the fast response (and just-in-time implementation)!

 

version 2.1 (Major update)

- Added Brotli support for all platforms except WebGL & watchOS.

- zip: added gzip support. Can operate on buffers with or without headers/footers.

- zip: added progress for individual file decompression.

- Android, Linux, iOS, MacOSX got chmod support to set file permissions.

- iOS: made the plugins much smaller in size.

- Reorganized the demo folders. You are adviced to delete the demo folders before updating the sample scenes.

 

update version 2.11

zip/iOS: hotfix to fix a GcHandle crash.

LZ4: added compress buffer to a specific location of a fixed size buffer.

LZ4: added decompress an lz4 buffer from a specific location of a fixed size buffer

 

Hi, any source code available?
I never buy any asset without source, you never know when it may stop be supported.

 

Sorry, no.

But as I state in my asset publisher homepage:

 

Hi,

Only with LZMA?.

Edit :
Can we do this in another thread?

 

Hi. There are various buffer compress/decompress functions for all the plugins. Not Only lzma.
Yes you can run this on separate threads.

Here is the documentation of the plugins:

 

Thanks for the reply.

Edit :
Has more features than i thought. Seem very good.

 

Hi.

I have just got this asset and I have been trying to replace sharpzip lib I was using.

So far so good.. except, I can't find the way to remove/delete entry from the zip file.

How do I do this one? There isn't any example of this, or any obvious function from lzip

 

Just to add, I don't want to extract all files and then manually recreate zip file...

I just want a simple way of removing entry from a zip file without creating mess.

 

Hi. It is not possible to do this for various reasons.

Most of the end user compression/decompression packages recreate the zip after extracting and deleting the file in question.

 

Oh.. I am not sure why it can't be done as I am doing it using sharpzip.. I am trying to use your plugin because of Android platform.

Anyway.

If I use :

lzip.buffer2File(9, fileName, zipEntryName, data, true))

I want to know if it overwrites zip entry if it already exists.

any idea?

 

Hi. I am not aware how sharpzip does it. But in any case some rewriting of the archive must be done.
Perhaps they have a higher level function which does the reziping of the extracted files minus the file you want to delete.

The buffer2File function will not overwritte. It will add a new archive in the file.

 

Sharpzip has API that you can search for a particular entry in the zip, and then just call remove to remove the entry.

Anyway, so if I have the entry already in the zip file, and want to overwrite it, then do I have to extract everything , delete and add one , and then zip up everything again?

Wow.. I am not sure what to say.. but it seems like it will take time to do decompress and then compress again. It won't be too much of an issue with small sized zip file, but for the big size zip file, this kind of operation can take some time. This will result in a lot slower time than sharpzip. (despite that you mentioned that it is faster.. it may only be faster for doing a particular operations.)

In normal , usual usage case, you would want to have more direct control over the zip file content, such as delete entry , overwrite entry, check for existence of the entry etc.. without having to decompress or compress whole zip file over and over again. Sharpzip may be doing this on background. It could be hiding this operation from the user, but it does have the simple function to call upon.

 

To make a replacement in a zip file surely has some rewriting. They modify the file. So they rewrite something for sure.

And if something goes wrong during that operation you lose your file.

You can check for the existence of an entry by checking the file size of an entry. If it returns a negative number the entry does not exist.

Anyway if you are not happy with the lib I would be happy to provide a refund.

 

Hi Elias.

I am enjoying the plugin but I would really like to use this on other console platform such as XboxOne.

Do you think this is going to be possible? I only need zip compress/decompress buffer support. I would be glad to help/work with you to implement this.

Please let me know. saehoon@pixellore.com

Thanks!

 

Hi castor76.

The plugins that are in the 'Plugins\WSA\UWP' directory should be XboxOne compatible since they are compiled as universal.

So they should work.

If you are having issues contact me to solve them.

 

Thanks! I will try them!

 

Hi Elias, by any chance have you tried this for 5.4.0?

 

I have just run a test. All plugins seem to work fine except the zip plugin on Windows.
I will investigate more to see if it is something on my end.

edit2: There are no issues with unity5.4. I found some issues though with the zip plugin for windows and already uploaded the update to the asset store.

 

update version 2.13

zip/Windows: better unicode support. Unity5.4 compatibility fix.

 

thanks!

 

update version 2.14

gzip: Fixed Android issues and added a 2nd ungzip function.

 

update version 2.15

Linux: fixed preprocessor directives for Unity5.4.x and up.

 

update version 2.16

ZIP All Platforms-gzip: Fixed a bug that was introduced in a previous update that had to with the outbuffer length. Now the gzip functions behave as intended.

ZIP Windows: Now you can treat a zip file that is loaded in a byte buffer as a file.

ZIP ios/gzip: updated the gzip functions.

 

I've got a project that requires me to download a zip file via www.bytes in WebGL, and then decompress the zip, and use the assets that are in the file (images and text). From what I understand, there's no way to save the zip to a file in WebGL, so it all has to be done in memory via byte arrays.

Unfortunately, when I run this code:

IEnumerator DownloadZip(string url)
{

WWW www = new WWW(url);
yield return www;

byte[] bytes = www.bytes;

int files = lzip.getTotalFiles(null, bytes);
Debug.Log("Contains " + files + " files");

}

My log shows

Contains -1 files

From what I've seen, I should be able to pass in a byte array rather than a filename, and get the individual files, right? Is it possible to do what I'm trying to do?

Thanks!

 

Hi. The WebGl plugins do not support 'FileBuffers as Files'.

While the code to support this did compile with emscipten, it didn't work. So I left WebGL out of this.
But I will try again with a newer version.

If you run the demo in WebGl, it downloads a small zip and performs various operations on it.
Doesn't this work for you?

 

Also try to use Application.temporaryCachePath to store the zip file there and use entry2Buffer to extract from there.

I guess you already have read this: https://blogs.unity3d.com/2016/09/20/understanding-memory-in-unity-webgl/

 

Thanks for your response! I had seen that article before but I didn't realize that Unity had implemented a pseudo-file system using memory. I searched all around and everyone said file operations weren't possible - use playerprefs instead. However, this looks promising. I'll give it a try.

Thanks!

 

version 2.17

zip/Windows: A small fix to the getTotalFiles function to allow to work correctly with FileBuffers.

 

Hi, I'm trying to decompress a zip on Android from the StreamingAssets folder into PersistentData but my app keeps crashing when it gets to line 28 below:

Code (CSharp):

1.  
2. string from = Application.streamingAssetsPath + "/myZip.zip";
3. string to = Application.persistentDataPath;
4.  
5. Debug.Log( "[AndroidStartUp] LOADING ZIP WITH WWW" );
6.  
7. WWW buffer = new WWW( from );
8.  
9. while ( buffer.isDone == false )
10. {
11.     yield return null;
12. }
13.  
14. Debug.Log( "[AndroidStartUp] BUFFER ERROR: "+ buffer.error );
15.  
16. if ( buffer.error == null )
17. {
18.  
19.     Debug.Log( "[AndroidStartUp] COPYING FILES FROM: " + from );
20.     Debug.Log( "[AndroidStartUp] TO PERSISTENT DATA: " + to );
21.     Debug.Log( "[AndroidStartUp] STARTING DECOMPRESSION" );
22.  
23.     mZipDecompression = lzip.decompress_File( null, to, mCompressionProgress, buffer.bytes );
24.  
25.  
26.     Debug.Log ( "[AndroidStartUp] DECOMPRESSION: "+ mZipDecompression );
27.  
28.     while ( mZipDecompression == 0 )
29.     {
30.         yield return null;
31.     }
32.  
33.     Debug.Log ( "[AndroidStartUp] DECOMPRESSION: "+ mZipDecompression );
34.  
35.     if ( mZipDecompression < 0 )
36.         DecompressionFailed();
37.     else if ( mZipDecompression == 1 )
38.         DecompressComplete();
39. }
40. else
41. {
42.     Debug.Log("[AndroidStartUp] ERROR READING FROM STREAMING ASSETS");
43. }

Is there a better way of doing this? or is there something i am doing wrong?
I have also tried moving the decompress into its own thread but i still get a crash.

Thanks

 

Hi. The problem comes from the way you are yielding the function.

If you use it like this it will work fine:

Code (CSharp):

1.     private int mZipDecompression;
2.  
3.     IEnumerator androidStreaming() {
4.         string from = Application.streamingAssetsPath + "/testZip.zip";
5.         string to = Application.persistentDataPath;
6.         Debug.Log( "[AndroidStartUp] LOADING ZIP WITH WWW" );
7.         WWW buffer = new WWW( from );
8.         yield return buffer;
9.         Debug.Log( "[AndroidStartUp] BUFFER ERROR: "+ buffer.error );
10.         if ( buffer.error == null )
11.         {
12.             Debug.Log( "[AndroidStartUp] COPYING FILES FROM: " + from );
13.             Debug.Log( "[AndroidStartUp] TO PERSISTENT DATA: " + to );
14.             Debug.Log( "[AndroidStartUp] STARTING DECOMPRESSION" );
15.             int[] mCompressionProgress = new int[1];
16.  
17.             mZipDecompression = lzip.decompress_File( null, to, mCompressionProgress, buffer.bytes );
18.             Debug.Log ( "[AndroidStartUp] DECOMPRESSION: "+ mZipDecompression );
19.  
20.             if ( mZipDecompression < 0 )
21.                 Debug.Log("DecompressionFailed");
22.             else if ( mZipDecompression == 1 )
23.                 Debug.Log("DecompressComplete");
24.         }
25.         else
26.         {
27.             Debug.Log("[AndroidStartUp] ERROR READING FROM STREAMING ASSETS");
28.         }
29.     }

 

Amazing thank you. Worked like a charm

 

Am i able to pull out a list of file names from a directory within the StreamingAssets folder on Android using this plugin?

 

Hi.

Yes you can.

There are 2 ways to do this.

1. Using www call and then call the appropriate functions that treat the www.bytes as a file.

2. Use a www call to get the www.bytes of the file -> save to ApplicationPersistantDataPath -> decompress from there.