7zip, lzma, LZ4, fastLZ, zip/gzip & brotli native multiplatform plugins.

Hi there!


We purchased your zipping plugin for unity, and so far it works pretty well!


But we have a huge issue that breaks our entire app when trying to unzip files on android, if the zip files contain folders or files that contain characters like ö / Ä / ü

The resulting unzipped folders will have strange symbols in them, and they can’t be deleted from the device and cause all sorts of troubles in file manager applications. We can't even delete them via the windows explorer on the devices SD card.


The code we used ist he following:


lzip.decompress_File(_zipPath, _localDataPath, _unzipProgress, null, progress2);



On windows, the folders are extracted properly, as well as zips that don’t have ö/ä/ü/é and so on.


Thanks in advance,

Christian

 

Hello.
It is a known issue of using unicode characters for zips.
For example the 7zip application is more compatible then winrar.
With which app were the zips created?

 

update version 2.51

- ZIP: iOS/tvOS: fixed duplicate symbol issue with Unity2018.


update version 2.50

- ZIP: iOS/tvOS: fixed some conflict issues with the Facebook SDK.

 

update version 2.52 - ZIP: fixed some C# issues with the compress_dir and compress_file_list functions.

update version 2.51 - ZIP: iOS/tvOS: fixed duplicate symbol issue with Unity2018.

 

Hi. I've been using a free compression/decompression tool for the past couple years and it just seems to bug out sometimes. Mine is a big open-world game so there can end up being a lot of data over time and compression makes a big difference, plus I like that it makes it a little harder to mess with the files directly. I'm going to purchase and implement your package today or tomorrow. I wanted to ask if you could recommend which type of compression would be best for my case. Pretty thick JSON files around 200kb max compressed right now, usually more like 1-20kb depending on how much a player interacts/builds with/on any given terrain.
Thank you.

 

Hello.

It seems that in terms of speed and compression brotli would be your first choice.

Here is a screenshot of a benchmark I ran:

 

In version 2.49, putting a streamed byte array (about 100mb) in lzip.decompress_File() causes a frequent crash.
ex. lzip.decompress_File(null, filePath, null, www.bytes, null);

Can you solve the problem?
spec
asset 2.49
unity 5.6.6f2
android ver 5.1.1~9.0

here is crash report

Code (CSharp):

1. # UnityMain(14771)
2.  
3. SIGSEGV(SEGV_MAPERR)
4.  
5. 1 #00 pc 01a951e8 libil2cpp.so il2cpp::vm::Object::AllocatePtrFree(unsigned int, Il2CppClass*) (C:\Program Files\Unity\Editor\Data\il2cpp\libil2cpp\vm/Object.cpp:63) [armeabi-v7a]
6. 2 #01 pc 01a9a280 libil2cpp.so il2cpp::vm::Array::NewSpecific(Il2CppClass*, int) (C:\Program Files\Unity\Editor\Data\il2cpp\libil2cpp\vm/Array.cpp:93) [armeabi-v7a]
7. 3 #02 pc 00a6b97c /data/app/com.app.game-GKvkiHMsyO7A-KPKmAAK1g==/lib/arm/libunity.so [armeabi-v7a]
8. //4th line is when decompress_File api is called
9. 4 #03 pc 015b11b0 libil2cpp.so U3CdecompressU3Ec__Iterator0_MoveNext_m510608782 (C:\Users\buildpc\Desktop\gameCN_Release\Temp\StagingArea\Il2Cpp\il2cppOutput/Bulk_Assembly-CSharp_0.cpp:24375) [armeabi-v7a]
10. 5 #04 pc 017a5b54 libil2cpp.so SetupCoroutine_InvokeMoveNext_m3444040730 (C:\Users\buildpc\Desktop\gameCN_Release\Temp\StagingArea\Il2Cpp\il2cppOutput/GeneratedInterfaceInvokers.h:36) [armeabi-v7a]
11. 6 #05 pc 00d6e9a0 libil2cpp.so RuntimeInvoker_Void_t1185182177_Il2CppObject_IntPtr_t(MethodInfo const*, void*, void**) (C:\Users\buildpc\Desktop\gameCN_Release\Temp\StagingArea\Il2Cpp\il2cppOutput/Il2CppInvokerTable.cpp:4708) [armeabi-v7a]
12. 7 #06 pc 01a631ac libil2cpp.so il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2CppException**) (C:\Program Files\Unity\Editor\Data\il2cpp\libil2cpp\vm/Runtime.cpp:497) [armeabi-v7a]
13. 8 #07 pc 004d6150 /data/app/com.app.game-GKvkiHMsyO7A-KPKmAAK1g==/lib/arm/libunity.so [armeabi-v7a]
14. 9 #08 pc 004e66c0 /data/app/com.app.game-GKvkiHMsyO7A-KPKmAAK1g==/lib/arm/libunity.so [armeabi-v7a]
15. 10 #09 pc 004eafa8 /data/app/com.app.game-GKvkiHMsyO7A-KPKmAAK1g==/lib/arm/libunity.so [armeabi-v7a]
16. 11 #10 pc 004eaba8 /data/app/com.app.game-GKvkiHMsyO7A-KPKmAAK1g==/lib/arm/libunity.so [armeabi-v7a]
17. 12 #11 pc 001dac60 /data/app/com.app.game-GKvkiHMsyO7A-KPKmAAK1g==/lib/arm/libunity.so [armeabi-v7a]
18. 13 #12 pc 0031c394 /data/app/com.app.game-GKvkiHMsyO7A-KPKmAAK1g==/lib/arm/libunity.so [armeabi-v7a]
19. 14 #13 pc 0031d268 /data/app/com.app.game-GKvkiHMsyO7A-KPKmAAK1g==/lib/arm/libunity.so [armeabi-v7a]
20. 15 #14 pc 005f5654 /data/app/com.app.game-GKvkiHMsyO7A-KPKmAAK1g==/lib/arm/libunity.so [armeabi-v7a]
21. 16 #15 pc 005f8468 /data/app/com.app.game-GKvkiHMsyO7A-KPKmAAK1g==/lib/arm/libunity.so [armeabi-v7a]
22. 17 #16 pc 0001f1e3 /data/app/com.app.game-GKvkiHMsyO7A-KPKmAAK1g==/oat/arm/base.odex [armeabi]
23. 18 java:
24. 19 com.unity3d.player.UnityPlayer.nativeRender(Native Method)
25. 20 com.unity3d.player.UnityPlayer.c(Unknown Source:0)
26. 21 com.unity3d.player.UnityPlayer$c$1.handleMessage(Unknown Source:157)
27. 22 android.os.Handler.dispatchMessage(Handler.java:103)
28. 23 android.os.Looper.loop(Looper.java:214)
29. 24 com.unity3d.player.UnityPlayer$c.run(Unknown Source:20)

 

Hello.

Can you please send me in a private message, a more detailed code example that you are using so I can reproduce the issue.

Thank you.

 

Hi Elias,

Do you have, or could you add UWP support? I purchased the plugin, was surprised it didn't exist. Their ecosystem has been super confusing, I assumed that by Windows and "Added WSA" support, that carried over, but I can't find any documentation about WSA support in what I just purchased from the store.

 

Hi shawww. WSA support has been removed due to many certification issues. If you don't care about these issues I could provide you the plugins for uwp.

Otherwise please send me your invoice so I can arrange a refund with Unity Support.

 

update version 2.53 [Major update]

- ZIP: WebGL has now full zip support. Obviously not on the file system but on memory buffers. (bz2 method not supported yet.)
- ZIP: Webgl got zlib buffer support again. (Was misplaced in a previous update.)
- ZIP: Added the ability to find merged/hidden zip/gzip archives in files or buffers and extract/get info from them.
- ZIP: iOS/tvOS: fixed potential issues with admob/facebook/google sdk.
- ZIP: Windows/Android: fixed bz2 issue.
- ZIP: bz2 method support for inmemory compression. (not for macos/ios/tvos/webgl.)
- All Plugins-tvOS: fixed preprocessor issues on newer versions of Unity.
- plugin binary sizes got smaller.
- updated the benchmark script.

 

update version 2.54

- ZIP: Critical fix for the inMemory zip creation functions. All users that use these functions should update to this version.
- ZIP: Added a public function to the inMemory struct for getting the byte[] buffer of the inMemory created zip.

 

update version 2.55

- ZIP: Using compress_File/compress_File_List/compressDir to append multiple files to a zip were very slow. Now compress_File_List and compressDir work very fast.
- ZIP: For the same reason the inMemory class got some low level functions to allow much faster appending of multiple entries. See examples and docs.



If you are publishing for MacOS on the AppStore please do the following to codesign the bundle:

1. Remove all the meta files from inside the .bundle.
2. Remove (if any) folder named CodeSignature.
3. Find the Info.plist file and change the bundleID to one of your own.

 

update version 2.56

- ZIP: Hotfix for the compressDir/compress_File_List functions. The plugin could crash when thousands of files were going to get compressed.

 

Hi,

During compiling for iOS I get this error and a few more:
Assets/ADL-Plugins/zip/Plugins/lzip.cs(1068,58): error CS1503: Argument 3: cannot convert from 'System.IntPtr' to 'string[]'

using 2019.2.17

 

Sorry for that.

It slipped with the previous hotfix update. I will send you shortly the lzip.cs that will fix that and update the Asset store package.

 

update version 2.57

- ZIP: Process of decompression in bytes now returned as ulong.
- ZIP: Fixed an issue with ios/tvOS caused by the 1.64 update.

 

Hey there!
I have a problem with decompressing 7z archives.

With some archives, it works, but with others, only a few files are extracted, with all others missing.

I don't get any error, just the status code -2.
The archives were compressed with 7zip, lzma2 on "Fast".
Again, it works with other archives that were compressed in the exact same way.

This is the code I use:
lzma.doDecompress7zip(path, outpath, false, true);

Paths are fine, it works with some files.

For example, this file will only extract 3 out of ~60 files.

 

Hello. Let me test this and I get back to you.

 

Edit:

For some reason in a previous update the bcj filter was ruled out.
I have fixed this issue now and update asap.

Please tell me the platform you are using so I can send you the updated plugins.

 

Thanks man!

I only use it on Windows (x86_64).

 

thx. Sending now.

The issue was only for the win platform.

 

your link on the front page is outdated fyi.

Is there any way to decompress a file into a buffer?
edit: I guess you just buffer prior to output or after input, nevermind.

 

Thx. Updated.

 

update version 2.58

- ZIP: fixed WebGL issues on Unity 2019.x
- fixed ios/tvos issue with bitcode plugins on xcode 11.x
- Lzma/7zip: fixed issue with the windows plugins. Now bcj and ppmd are back.

The ios/tvos plugins include xcode 10.3 and 11.4 bitcode compiled versions.

 

@elias_t i have two questions that prevent me from directly using assetbundle.loadfromfile from zip file
1、is there an api to get entry offset so i can pass it to assetbundle.loadfromfile?
2、how can i compress files using compression method store, currently even i set level of compression to 0 its compression method is deflate?
thanks

 

Hello.

1. Please use the lzip.getZipInfo(string fileName) function.
It will fill the public static struct list lzip.zinfo.

Code (CSharp):

1.     // a struct to store zip entry information
2.     public struct zipInfo{
3.         public short VersionMadeBy;
4.         public short MinimumVersionToExtract;
5.         public short BitFlag;
6.         public short CompressionMethod;
7.         public short FileLastModificationTime;
8.         public short FileLastModificationDate;
9.         public int CRC;
10.         public int CompressedSize;
11.         public int UncompressedSize;
12.         public short DiskNumberWhereFileStarts;
13.         public short InternalFileAttributes;
14.         public int ExternalFileAttributes;
15.         public int RelativeOffsetOfLocalFileHeader;
16.         public string filename;
17.         public string extraField;
18.         public string fileComment;
19.     };

2. I will look into it. There might be an update needed to add the store method automatically when cpmpression level is set to 0.

Edit: I believe that with cpmpression level 0 you should be able using the zinfo struct list to use assetbundle.loadfromfile.

 

@elias_t you mean find zipinfo by zipinfo.filename and pass zipinfo.RelativeOffsetOfLocalFileHeader directly to assetbundle.loadfromfile?

 

Hello.

I thought this was the original question.

I will make some tests here and report.

 

It seems that right now it is not doable.
I will try to find a solution for your request.

Edit: But in case you use the 7zip app to create zip files with the store option enabled, to get the correct offset use this:

Code (CSharp):

1. offsetInFile = lzip.zinfo[x].RelativeOffsetOfLocalFileHeader + 30 + lzip.zinfo[x].filename.Length;

 

@elias_t waiting for you solution, currently using sharpzipcodelib as alternative

 

Ok. I think I solved this.
I must run tests now for all platforms before a release.

If you need a specific platform right now please send a pm.

 

@elias_t thanks

 

@elias_t Just purchased the asset. Is there any way to use file queries like for example ' name.* ' rather than having to explicitly write the file extension? I'm trying to select a file with any file format in a byte buffer using lzip.entry2Buffer, but it doesn't seem to be working correctly when I use a file query.

My code (changed second parameter to "file.*" to illustrate the issue):

Code (CSharp):

1. public static byte[] BytesFromZip(string zipPath)
2.         {
3.             if (lzip.validateFile(zipPath, string fileSelector))
4.             {
5.                 byte[] bytes = lzip.entry2Buffer(zipPath, "file.*");
6.                 return bytes;    
7.             } else
8.             {
9.                 Debug.LogError("File to unzip was not valid!");
10.                 return null;
11.             }

 

Hello.

As of now, not directly.

But you can use any of the get zip info functions (getFileInfo, getZipInfo) that fill the public static info lists (lzip.ninfo ,lzip.zinfo) and then find your file there.

 

Hey there,

I've been trying to use this plugin to unzip/decompress a .zip file that we download from an AWS S3 server on an Oculus Quest. I've been using the lzip.cs script and calling the decompress_File() method but I get -1 as a response and False when trying to falidate the zip file after downloading. This only occurs on the Oculus Quest headset, and doesn't occur when I move the downloaded zip file over to my Windows PC and try the same method, which works perfectly.

The zip file that we download is very large (2.97 GB because it includes three 1 GB video files) so passing in the file as a byte array isn't really possible as far as I've seen.

I'm pretty positive we are getting the correct permissions such as:
WRITE_EXTERNAL_STORAGE
READ_EXTERNAL_STORAGE

This is how I've written the extract method:

Code (CSharp):

1. /// <summary>
2.     /// A coroutine that extracts files from a source file to the VideoLocation folder.
3.     /// </summary>
4.     /// <param name="sourceFile">The full path to the zip file to be extracted.</param>
5.     IEnumerator ExtractFiles(string sourceFile)
6.     {
7.         int[] prog = new int[1];
8.         ulong[] proc = new ulong[1];
9.         int result = 0;
10.         bool isValid = lzip.validateFile(sourceFile);
11.  
12.         debugBox.text = "Validate: " + isValid;
13.  
14.         if (isValid)
15.         {
16.             try
17.             {
18.                 new System.Threading.Thread(delegate ()
19.                 {
20.                     result = lzip.decompress_File(sourceFile, VideoLocation, prog, null, proc);
21.                 }).Start();
22.             }
23.             catch (Exception e)
24.             {
25.                 debugBox.text = "ERROR WHEN DECOMPRESSING FILE: " + e.Message;
26.             }
27.  
28.             while (prog[0] < 3)
29.             {
30.                 try
31.                 {
32.                     debugBox.text += "\n1. prog: " + prog[0] + " | proc: " + proc[0] + " | res: " + result;
33.                 }
34.                 catch (Exception e)
35.                 {
36.                     debugBox.text += "\n1. ERROR USING 0: " + e.Message;
37.                 }
38.  
39.                 yield return new WaitForSeconds(1f);
40.             }
41.         }
42.  
43.         yield break;
44.     }

Are there other permissions I have to ask for? Or is there a different function that I need to use in order to decompress this zip file on the Oculus Quest? Let me know if I need to clarify anything else.

 

Hello.
Oculus Quest is on Android right?

In that case try to use a split zip if possible. (1.zip. 1.z01, ...)

 

Yes Oculus Quest is Android. The reason why we are using a singular large zip file is so that we only have to make a request to the AWS S3 servers one time, so we don't want to download multiple split zip files. Unless you mean to split up the zip after downloading.

 

No, no. I asked if you had the possibility to create the split zip on the server.

I will send you a newly compiled plugin and see if that helps you.

 

I don't believe we can since we are directly uploading the file to the S3 bucket and not using an inbetween website uploader or splitter.

 

OK. I have successfully compiled a new arm64-v8a plugin which decompressed a 2.55 GB file.
This should work for you.

I am sending this in a pm and later I will update the package.

 

Update version 2.6 (ZIP: Important feature and maintenance update)

- ZIP: Setting level of compression to zero will now use the STORE method.

- ZIP: Added the setCancel() function to cancel the compress_File, compressDir, compress_File_List, decompress_File, extract_entry, entry2Buffer functions.

- ZIP: Added the ability to get byte level progress when compressing files.

- ZIP: Android arm64-v8a plugin should decompress files larger then 2GB now.

- ZIP: Windows: compression with unicode characters in path should work correct now.

- ZIP: The getEntrySize function returns a ulong value now.

- ZIP: The getFileInfo function filled Lists for uncompressed, compressed and localOffset are filled with UInt64 values now.

- ZIP: The getFileInfo function fills a new list localOffset. Useful for archives with the STORE method to find the position in the zip. (you should add + 30 bytes + name.length to get the final offset)

- ZIP: Demo script updated to reflect the new changes.

- 7zip: When extracting a 7z archive the output folder should be created when not existing now.

 

update version 2.6.1

- ZIP: Hot fix for zips that contain the __MACOSX folder.
- ZIP: Added simulator plugin for iOS.

 

I'm trying to use brotli (per suggestion) but I keep getting decompression errors.
my compressed bytes before File.WriteAllBytes() have a length of 54298
and the file when opened with File.ReadAllBytes() has a length of 54298
but I'm getting the a false return with the bool decompressBuffer(), and creating a 3 million byte test array for the int decompressBuffer() is returning a -1 (failed to write output) ?
This is 2018.4.23, Win10.

Code (CSharp):

1.         byte[] compressedBytes = new byte[0];
2.         int[] progress = new int[1];
3.         bool compSuccess = brotli.compressBuffer(jsonBytes, ref compressedBytes, progress);
4.         File.WriteAllBytes(path, compressedBytes);

Code (CSharp):

1.             byte[] compressedBytes = File.ReadAllBytes(path);
2.             byte[] decompressedBytes = new byte[0];
3.             bool decompSuccess = brotli.decompressBuffer(compressedBytes, ref decompressedBytes);

 

I will check this and get back to you.
It would help if you could send the jsonbuffer as an example to reproduce it here.

Edit1: Alternatively try to use the useFooter flag on both functions.

Edit2: I just ran a test with a 58 kb Json file and it all worked as expected.

 

Just waking up, will try as suggested and post a JSON file for ya. Fyi, it's around 2.6 Mb, 54kb compressed.

Edit: I don't see a use footer for compress, setting useFooter to true for decompress, did seem to have fixed it for the first 3-4 tries. Then it started giving me:

Code (CSharp):

1. ArgumentOutOfRangeException: Non-negative number required.
2. Parameter name: newSize
3. System.Array.Resize[T] (T[]& array, System.Int32 newSize) (at <e1319b7195c343e79b385cd3aa43f5dc>:0)
4. brotli.decompressBuffer (System.Byte[] inBuffer, System.Byte[]& outBuffer, System.Boolean useFooter) (at Assets/3rd Party/ADL-Plugins/brotli/Plugins/brotli.cs:342)

and now it's giving me decompression failed again. It's also adding 600-800Mb memory usage to the editor with each attempt?

Edit2: I've attached a rar with the uncompressed and the compressed (that I am getting errors trying to uncompress) files.

Edit3: lzip seems to be working fine. Maybe it's just a brotli bug for larger files? Seems to be cause memory issues. I'll stick with lzip for now, I don't have a preferences either way to be honest.

 

Thank you for the detailed input.
I will make my tests today and try for a fix.

 

Update version 2.6.2

- 7zip: Windows: Unicode issues resolved.
- 7zip: Added a cancel function when decompressing 7zip archives with multiple entries.
- 7zip: Linux: FileBuffers disabled for now on Linux since they are not stable.
- ZIP: Hotfix: the previous update broke the decompression of zip file. It is now fixed!
- ZIP: Added 2 new functions to gzip and ungzip archives on the File System.

 

Ok. I found the issue.

For compressed buffers that add more then one metaBlock however the usage of the useFooter flag is needed.
I will upload a new update today.

 

update version 2.6.3

- Brotli: Fixed the buffer functions.