[RELEASED] zip / gzip native multiplatfrom plugin

No I do not mean this.

I mean that if your entry resides in a directory you must give the full path to extract it.

Like: lzip.entry2Buffer(ppath + "/testZip.zip","dir1/dir2/test2.bmp",ref reusableBuffer2);

Do the other functions work except entry2Buffer? Does it work ok in the editor and not on the device?

If you still face issues please send me the zip file in question.

 

No, there isn't any directory in the zip file. I think they do work in the editor.

Here is link to the file in question.

https://www.dropbox.com/s/j0jqeo18egkb14s/SaveGame_Slot0.dat?dl=0

 

I just double checked and the the same code checking the zip file does work under editor. But not on build. The only difference I can see is the Application.PersistentPath is different when run in editor, for obvious reason. The app obviously has the right to write files on the disk , it's how the zip file is copied to the location in the first place.

If the tool is using some kind of temp location to process the zip file, do you think it could be the problem?

 

I did some debugging and it seems like :

if (siz <= 0)
{
Debug.Log("lzip : siz is 0");
return null;
}

siz gets to be zero...and that is why it is returning null.
The reason for siz to be zero is beyond me. I was trying to extract the GameStats.dat file from the zip

BTW, I am not running on any mobile device, I am just testing it on local machine.

 

Ok, this is driving me nuts, because when i kept trying to get entry from zip file, it works in a random manner. Sometimes works, sometimes it doesn't. when it doesn't work res becomes null. It almost feels like a timing issue too. Very strange.
It works like 20% of time.

My previous analysis about siz getting 0 is not happening anymore for some reason. but res becoming null is still happening in random manner.

I keep pressing button to extract entry from zip file and that is the result. Random.

 

I also get this error when i use lzip:getFileInfo

Error 2:21:38 Could not allocate memory: System out of memory!
Trying to allocate: 2508541143B with 16 alignment. MemoryLabel: TempOverflow
Allocation happend at: Line:177 in C:\buildslave\unity\build\Runtime/Allocator/ThreadsafeLinearAllocator.cpp
Memory overview

[ ALLOC_DEFAULT ] used: 252126123B | peak: 0B | reserved: 278093635B
[ ALLOC_TEMP_JOB ] used: 0B | peak: 0B | reserved: 2097152B
[ ALLOC_GFX ] used: 9420360B | peak: 0B | reserved: 12531332B
[ ALLOC_CACHEOBJECTS ] used: 2382888B | peak: 0B | reserved: 10485760B
[ ALLOC_TYPETREE ] used: 1151516B | peak: 0B | reserved: 4194304B
[ ALLOC_PROFILER ] used: 7451076B | peak: 0B | reserved: 16777216B
[ ALLOC_TEMP_THREAD ] used: 32768B | peak: 0B | reserved: 4128768B
UnityEngine.Windows.Directory:Exists(String)
lzip:getFileInfo(String, String, Byte[])
ZipManager_Windows10eleteBufferedData(String)
<DeleteBufferedDataNew>d__34:MoveNext()
UnityEngine.MonoBehaviour:StartCoroutine(IEnumerator)
<StartGame>d__10:MoveNext()
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

When this happens it leaves uziplog.txt file with entry names and their compressed and uncompressed file sizes..

 

Hi again.

I have made tests with your zip file on a windows 10 phone and it runs fine.

But now I see some issues with il2cpp on windows10 local machine. I will have to investigate more
if it is a unity issue and report it as a bug or not.

 

As I stated above it must be an issue with il2cpp on windows 10 local machine.
Try to compile and run with .net and see if the same happens.

 

Hi.

I have just tried 64 bit build, and it seems to work much better. I have not seen any issue yet. So perhaps I was on the edge of problem with memory on my previous attempts when I was using 32bit build.

It's either there is problem with 32bit version (either tool or unity or whatever) or it's the memory limitation that caused this random failure behaviour. I am more inclined to the memory issue we could have faced on 32bit build since there is particular error on that when using getfileinfo.

 

I can't run .net version because it won't work with my game. It's complicated, but it is because of other plugins that isn't playing nicely with netcore difference in UWP.

 

Hm... This is strange also. You shouldn't have any memory problems with getfileinfo. In the worst case it uses some kilobytes.

Are your uwp plugins correctly assigned in the editor inspector as 32 bit and 64 bit?

 

Also something else.

Please run the demo that came with the package and report to me if all the functions run correctly with your compilation setup.

 

Now I can't even run an empty scene on Unity5.4.2 uwp il2cpp.
There must be some issues with uwp and il2cpp that I am not aware of.

Trying with unity5.5 now.

 

Yeah , strange. 2508541143B sounds a lot.

And yeah I think the plugins are configured correctly at least according to what the folder name is suggesting. WSAPlayer and to 32 and 64 bit according to the folder name. SDK = UWP , scripting backend = any.

Unity 5.5.0p1

I will try with demo scene too.

 

Hi.

I am still having issue with 32bit build on UWP 10 Windows.

It's strange that it works fine in 64bit version. But in order to submit to the store , I need to support 32bit version as well.

It will be really great if you can help to solve this issue.

res keeps returning null

 

Hi.

Yes. I will try to test tonight on 32bit uwp win 10 and report back.

 

Thanks.

I also found strange issue with zipArchive string variable. After getting the siz (which seems to return correctly) it seems to be corrupted.. somehow. becoming null. Perhaps that is why res is null.

I have no idea , and it may well be il2cpp bug, but it's driving me nuts. I even tried this:

string zipArchiveCopy = zipArchive

... get siz (works)

then check zipArchiveCopy and zipArchive, they both seems to be null at this point just before actually using zipEntry2Buffer.

if there is bug with string operations with il2cpp 32bit, then this kind of error should be happening everywhere, but it isn't so it could be some very specific case issue. I am not sure. I have tried to log entry string variable and it even won't let me. It just throws an exception after getting siz.

 

Thanks for the new input.
I will replace some functions with a different approach and compile new dlls.
When I have them ready and tested, I will send them to you in a private message.

 

update version 1.39

- fix for a WSA/x86/il2cpp crash.

 

the progress1 of function decompress_File does not count folder?
I can not compare the progress1 to getTotalFiles() to know whether the deompress is over.
I have no idea. thanks

 

Hi. Yes, the progress of the files does not take into account the folders.
The getTotalFiles should return the total number of files and not the folders.

 

The return of getTotalFiles at version 1.39 does count the folders.
what should I do to confirm whether the decompress is over ?

 

edit: yes the getTotalFiles returns files + folders.

Anyway, you can overcome this by using these global vars after calling a getFileInfo on a zip file:

Code (CSharp):

1. public static int zipFiles, zipFolders;// global integers that store the number of files and folders in a zip file.

lzip.zipFiles will give you the true total files.

edit: (the zipFolders is not working for now)

 

windows 7, unity 5.4.4.f1

 

Scratch my previous answer. getTotalFiles returns files+folders.

You can get the true number of files by using my previous answer: lzip.zipFiles after calling getFileInfo on a zip file.

 

A new question

WWW www = new WWW(url);
yield return www;
int totalFiles = lzip.getTotalFiles(null, www.bytes);
lzip.decompress_File(null, targetDir, progress1, www.bytes, progress2);

the decompress is OK, but getTotalFiles returns -1

lzip.getFileInfo & lzip.zipFiles work rightly

-----------------------------------------------------------------

like the above code, while using the FileBuffer param, the progress1 takes in account the folders!!!!!

 

Hi the getTotalFiles does not work with file buffers. use the lzip.zipFiles var.

As for the second comment yes, you discovered something here. The filebuffer extraction takes into account
the directories. This happens only on windows. Will fix this for the next version.

 

Hello, just purchased your asset.
What I need it for is: compress some images (2048x2048 RBGA32 - 16MB), store the data in a binary file, load it and decompress it at runtime.

At the moment I'm using LZ4 compress/decompress buffer methods and it's not working.

Here is the code I'm using:

Code (CSharp):

1. byte [] compress (byte [] _data) {
2.         byte [] result = new byte [1];
3.  
4.         LZ4.compressBuffer (_data, ref result, 1);
5.  
6.         return result;
7.     }
8.  
9.     byte [] decompress (byte [] _data) {
10.  
11.         byte [] result = new byte [1];
12.  
13.         if (LZ4.decompressBuffer (_data, ref result))
14.             return result;
15.         else
16.             return null;
17.     }

To get the image data I'm using Texture2D.GetRawTextureData() method and to re-create the image I'm using Texture2D.LoadRawTextureData().

I checked and the size of the original byte array is the same with the same of the compressed and then uncompressed byte array, but the image is not loading correctly (see the attached screenshot).

Can you point me in the right direction? Also, if you want I can share the test project with you.

Thanks,
Daniel

 

Hi.

Thank you for your purchase.

(This thread belongs to the zip only plugin. The correct support thread is this: https://forum.unity3d.com/threads/7...p-brotli-native-multiplatform-plugins.211273/ )

Yes send to me via private messaging the project so I can look at it.

 

I'm trying to build an UWP app with your library. I can compile it for Universal 10 SDK but not for Universal 8.1 SDK.

Assets\Plugins\lzip.cs(695,9): error CS0117: 'UnityEngine.Windows.File' does not contain a definition for 'Move'

Could you look into this?

 

Hi.

That slipped through for 8.1. If you don't need the delete_entry/replace_entry functions in 8.1 you can safely comment them out until I upload the fix for this.

 

update version 1.41

- Added zip64 support.

- Added low level functions (except for WSA) - see demo and lzip.cs.

- WSA8.1 fixes.

- Added a setEncoding function for Windows and WSA10 to set text encoding on file names.

- Added a validate function to check sanity of zip files.

- Removed WebGL support since it is almost impossible to work on Unity5.5.

- The getTotalFiles will now return only the true files number (no folders and no zero byte files).

The next update will aim to support encryption.

 

update version 1.42

- hotfix for zip files which use backslashes instead of slashes for directories
- Linux/MacOS fixes for the low level functions on standalone builds.
- ios: watchOS & tvOS. Added simulator plugins.

Since some users requested it, WebGL support will be reintroduced with support for zip and gzip buffers only.

 

Because some people wanted to use raw inflate/deflate on streams/buffers I want to point out that using the gzip/unGzip functions without headers & footers, the functions will use raw inflate/deflate operations.

 

update version 1.43

- Reintroduced WebGL support for compression/decompression of zlib/gzip buffers.

 

Interested in buying, but I'm just a little confused about your wording about extracting a single file.

You say extracting a single file out of a zip file, but also mention getting the size info of many files in a zip file. I'm assuming that you can extract multiple files out of a zip file, but just one at a time is this correct?

 

Hi.

You can extract all of them with a single command. Or you can extract a single file by name or by index with the new low level commands.

Here is the instructions file that explains the functions:

 

Hello. I am trying to work with plugin on UWP platform and having next error:

DllNotFoundException: Unable to load DLL 'libzipw': Не найден указанный модуль. (Exception from HRESULT: 0x8007007E)
at lzip.zipCompressBuffer(IntPtr source, Int32 sourceLen, Int32 levelOfCompression, Int32& v)
at lzip.compressBuffer(Byte[] source, Byte[]& outBuffer, Int32 levelOfCompression)
at GZipHelper.<CompressData>d__7.MoveNext()
at Job.<doWork>d__21.MoveNext()
at UnityEngine.SetupCoroutine.InvokeMoveNext(IEnumerator enumerator, IntPtr returnValueAddress)
at UnityEngine.SetupCoroutine.$Invoke1(Int64 instance, Int64* args)
at UnityEngine.Internal.$MethodUtility.InvokeMethod(Int64 instance, Int64* args, IntPtr method)
(Filename: <Unknown> Line: 0)

What should I do?

 

Hi.

It seems that the plugin has not been copied over to your wsa project?

Make sure in the unity3d inspector to have your plugins assigned for the appropriate platforms. (x86/ARM/x86_64).

 

Some heads up.

I have managed to add AES256 encryption and AES256 and/or ZipCrypto decryption.
I also added bz2 compression/decompression method besides the deflate/inflate one.

Deflate64 could come later.

Right now it has been tested on Windows + Android.
In the next days when the other platforms pass the tests it will be pushed to the AssetStore.

 

Hi Elias,

Thanks for all your hard work on this great asset! I'm wondering if you can send me a copy of the software license? Cheers!

 

Hi. Up until now the lib is based on public domain sources. So the only license that applies is the unity asset store license.

The next version will use zlib.

 

Thanks!

 

Hi.. wiht this pluggin i can zin/unzip large files in split parts... eg: file.zip,file.zip.0001,file.zip.002 i need use this in android platform... thanks

 

Hi. With the current version no. But there is a new version coming that will allow this. The only thing that is holding it back are iOS compatibility fixes being done right now.

 

Hi, Elias.
Could you please provide x64 Windows debug version of your library with pdb?
It looks like it have some memory leaks which leads to random crashes while working with many (hundreds) zip files.


I wrote some code to test zip operations using some of your code from testZip.cs (from package). Here is it:

Code (CSharp):

1. using System;
2. using System.Diagnostics;
3. using System.IO;
4. using UnityEngine;
5. using UnityEditor;
6. using System.Collections.Generic;
7.  
8. public class TestZipPack : MonoBehaviour
9. {
10.     private static int _loopn;
11.  
12.     [MenuItem( "ZipPack/Start test with many zips" )]
13.     static void StartTest()
14.     {
15.         var zips = Directory.GetFiles( @"D:\ziptest\", "*.zip", SearchOption.AllDirectories );
16.         var totalCount = 100;
17.         float progressStep = 1f / zips.Length;
18.         var sw = new Stopwatch();
19.         sw.Start();
20.         bool needBreak = false;
21.         for ( _loopn = 0; _loopn < totalCount; _loopn++ )
22.         {
23.             if ( needBreak ) break;
24.             float progress = 0;
25.             for ( var index = 0; index < zips.Length; index++ )
26.             {
27.                 var zip = zips[index];
28.                 if ( EditorUtility.DisplayCancelableProgressBar( "Testing zips in a loop",
29.                     string.Format( "loop: {0}/{1} file: {2}/{3} Mem: {4}", _loopn + 1, totalCount, index + 1, zips.Length, GC.GetTotalMemory( false ) ),
30.                     progress += progressStep ) )
31.                 {
32.                     needBreak = true;
33.                     break;
34.                 }
35.                 ZipPackTest( zip );
36.             }
37.         }
38.         sw.Stop();
39.         UnityEngine.Debug.LogFormat( "Time: {0}", sw.ElapsedMilliseconds );
40.         EditorUtility.ClearProgressBar();
41.     }
42.  
43.     static void ZipPackTest( string zipPath )
44.     {
45.         // Create a pointer to a zip archive and initialize it.
46.         IntPtr za = lzip.zipArchive( zipPath ); //Can be used also with file buffers
47.  
48.         if ( za == IntPtr.Zero )
49.         {
50.             UnityEngine.Debug.LogError( " ###### Pointer returned ZERO ######" );
51.         }
52.         else
53.         {
54.             //Display total files in a zip (without the folders)
55. //            UnityEngine.Debug.LogError( "total files: " + lzip.zipTotalFiles( za ) );
56.  
57.             //Create a file stat structure pointer
58.             IntPtr fs = lzip.zipFileStat();
59.  
60.  
61.             //Create a List of the file stat struct to store the info of the entries
62.             List<lzip.fileStat> fsl = new List<lzip.fileStat>();
63.  
64.             // iterate through all the entries
65.             for ( int i = 0; i < lzip.zipTotalEntries( za ); i++ )
66.             {
67.                 //initialize the previous created file stat structure
68.                 lzip.initZipFileStat( za, fs, i );
69.  
70.                 lzip.fileStat fstat = new lzip.fileStat();
71.  
72.                 lzip.fstatFillSizes( fs, ref fstat );
73.  
74.  
75.                 // ! change the text encoding to your own needs !
76.                 fstat.name = lzip.fillNameString( fstat.nameSize, fs, System.Text.Encoding.UTF8 );
77.                 if ( fstat.commentSize > 0 ) fstat.comment = lzip.fillCommentString( fstat.commentSize, fs, System.Text.Encoding.UTF8 );
78.  
79. //                UnityEngine.Debug.LogError( fstat.index + " : " + fstat.name /*+" : "+fstat.comment*/ );
80.  
81.                 //add the fstat struct to the list for further use.
82.                 fsl.Add( fstat );
83.             }
84.  
85.             //free the file stat struct
86.             lzip.freeZipFileStat( fs );
87.  
88.             //close the low level zip archive and release its pointer
89.             lzip.closeZipArchive( za );
90.  
91.             for ( int i = 0; i < fsl.Count; i++ )
92.             {
93.                 if ( fsl[i].isDirectory ) continue;
94.                 byte[] bytes = new byte[1];
95.                 if ( !lzip.entry2Buffer( zipPath, fsl[i].name, ref bytes, null ) )
96.                     UnityEngine.Debug.LogError( "Can't extract entry " + fsl[i].name );
97.             }
98.         }
99.     }
100. }

Test folder should contain about 300-400 zips.

Please investigate..

 

Hi. The issue seems to be the low level get info functions. They are removed in the next version.

All the platforms except iOS are ready. If you want I can send that version for you to try.

 

Yep, you can send me this version in PM to test it.
And could you tell me when new version will be finished and available in Unity Store?

PS. Thanks for quick reply!

 

PM sent.

 

Two points:

1.
I upgraded from "Zip / gzip Multiplatform Native Plugin" to
"7ZIP, lzma,..." and paid 21,27€ instead of upgrade 10,63€

2.
I purchased the upgrade because I wanted to get rid of following issue:
My Zipfile contains a texture named "fläche.jpg" and the unzipped jpg is named "fl├ñche.jpg".
Could you help here?

Thanks very much!