Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

7zip, lzma, LZ4, fastLZ, zip/gzip & brotli native multiplatform plugins.

Discussion in 'Assets and Asset Store' started by elias_t, Nov 16, 2013.

  1. codes

    codes

    Joined:
    Nov 26, 2012
    Posts:
    10
    Hi, i'm trying to update an entry in an exist zipfile.
    lzip.buffer2File(3, persistZipPath, entryName, www.bytes, true);
    byte[] data = lzip.entry2Buffer(persistZipPath, entryName);
    But the data is always null after update.
    The plugin version is 2.16. Can you give me some advises? Thank you.
     
  2. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    Hi. You cannot update an entry right now.

    You must extract your zip and recreate it without the file you want to update and then add the updated file.
     
  3. codes

    codes

    Joined:
    Nov 26, 2012
    Posts:
    10
    OMG, that is completely unacceptable....
    Are there any chances to add this feature in the future?
     
  4. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    Hi.

    1. In the features of the package there was no mention of the ability to replace a file in the zip file.
    2. As I stated before, the procedure to replace a file is almost the same as to recreate it.
    3. Yes there will be an update to allow such a functionality.
     
  5. codes

    codes

    Joined:
    Nov 26, 2012
    Posts:
    10
    ok,thanks any way.
    Think this situation. One zip file with 10000 or more entries and I just want to update one entry of them, the procedure would be a effect killer on mobile platforms.
    Maybe I should try some other methods.
     
  6. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    I will try in the next days to have a first version of the replace function.
    But it will still create a new file by copying the rest and update the new file.
    The in-place replace is more risky to loose your original file if something goes wrong.
     
  7. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    Hi.

    I have added 2 new functions that work without recompression the content of the zip file:

    delete_entry and replace_entry.

    I am making tests in all supported platforms and will push an update to the Asset Store in a day or two.
     
  8. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    update version 2.19

    zip/All platforms: added the delete_entry function to delete an entry in a zip file.

    zip/All platforms: added the replace_entry function to replace an entry in a zip file(from a filepath or a buffer).

    (These 2 functions will not recompress the original archive. A tmp file is created where the compressed data of the other archives is copied and after that the deletion or replacement occurs.)
     
    Last edited: Dec 8, 2016
  9. hayashiyui

    hayashiyui

    Joined:
    Nov 8, 2012
    Posts:
    6
    I got this build error when building to WebGL in Unity 5.5.0f3.

    error: Linking globals named 'XXH32': symbol multiply defined!
    Traceback (most recent call last):
    File "/Applications/Unity/PlaybackEngines/WebGLSupport/BuildTools/Emscripten/emcc", line 13, in <module>
    emcc.run()
    File "/Applications/Unity/PlaybackEngines/WebGLSupport/BuildTools/Emscripten/emcc.py", line 1482, in run
    final = shared.Building.llvm_opt(final, link_opts, DEFAULT_FINAL)
    File "/Applications/Unity/PlaybackEngines/WebGLSupport/BuildTools/Emscripten/tools/shared.py", line 1592, in llvm_opt
     
  10. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    Hi. I haven't tested lately webgl with unity5.5. Let me run some tests and I'll get back to you.
     
  11. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    Hi. It seems that there are some conflicting functions with the lz4 webgl plugin.
    If you are not using it remove it and you should be able to compile fine.

    I will rename some functions in the lz4 webgl plugin to make it work on unity5.5 in the next days.
     
  12. hayashiyui

    hayashiyui

    Joined:
    Nov 8, 2012
    Posts:
    6
    Thanks for your reply. I will wait for update since I'd like to use lz4.
     
  13. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    Hi again.

    After many tests it seems that the lz4 WebGL plugin will not work for unity5.5 versions and up.
    So I will remove it in the next update.

    I made a lot of modifications to make it work, but for now it seems a lost cause.

    I would advice to use the flz plugin, since in terms of speed, it is comparable to the lz4 one.
     
  14. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    update version 2.2

    Removed WebGL support for the LZ4 plugin since there are too many conflicts with Unity5.5.
    If you have the LZ4 WebGL plugin in your project and you are on Unity5.5 you are advised to removed it.
     
  15. hayashiyui

    hayashiyui

    Joined:
    Nov 8, 2012
    Posts:
    6
    I will try fastlz or anything else. Thank you.
     
  16. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    update version 2.21

    - fix for a WSA/x86/il2cpp crash.
     
  17. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    update version 2.24

    - Zip: Added zip64 support.

    - Zip: Added low level functions (except for WSA) - see demo and lzip.cs.

    - Zip: WSA8.1 fixes.

    - Zip: Added a setEncoding function for Windows and WSA10 to set text encoding on file names.

    - Zip: Added a validate function to check sanity of zip files.

    - Zip: The getTotalFiles will now return only the true files number (no folders and no zero byte files).

    - All plugins: Removed WebGL support since it is almost impossible to work on Unity5.5.

    The next update will aim to support encryption for the zip plugin.


     
  18. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    update version 2.25

    - Zip: hotfix for zip files which use backslashes instead of slashes for directories

    - Zip: Linux/MacOS fixes for the low level functions on standalone builds.

    - Zip/ios: watchOS & tvOS. Added simulator plugins.


    In a next update WebGL will be reintroduced for zip/gzip, flz, lzma supporting buffers only.
     
  19. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    update version 2.26

    - Reintroduced webGL support for zip/gzip, flz and lzma compression/decompression of buffers.



    * Because some people wanted to use raw inflate/deflate on streams/buffers I want to point out that using the gzip/unGzip functions without headers & footers, the functions will use raw inflate/deflate operations.
     
  20. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    update version 2.27

    - Added webGL LZ4 support for compression / decompression of buffers.
     
  21. Brogan89

    Brogan89

    Joined:
    Jul 10, 2014
    Posts:
    244
    Hi. I am trying to figure out how to zip files within a memory stream and return base64. I used this method previously
    Code (CSharp):
    1. public string ZipStream(Dictionary<string, byte[]> files)
    2. {
    3.     using (MemoryStream ms = new MemoryStream())
    4.     {
    5.  
    6.         using (ZipFile zip = new ZipFile())
    7.         {
    8.             foreach (var fileName in files.Keys)
    9.             {
    10.                 zip.AddEntry(fileName, files[fileName]);
    11.             }
    12.  
    13.             zip.Save(ms);
    14.         }
    15.  
    16.         ms.Close();
    17.         return System.Convert.ToBase64String(ms.ToArray());
    18.     }
    19. }
    It worked on all platforms other than WebGL, so I'm trying to replace this same function based on the testGzipWebGL.cs file. Trying to use the same logic my function looks like this
    Code (CSharp):
    1.     public string ZipStream(Dictionary<string, byte[]> files)
    2.     {      
    3.         foreach (var fileName in files.Keys)
    4.         {
    5.             try
    6.             {
    7.                 byte[] outBytes = new byte[lzip.gzipUncompressedSize(files[fileName])];
    8.                 var x = lzip.gzip(files[fileName], outBytes, 1);
    9.                 Debug.Log(x);
    10.  
    11.                 if (outBytes != null)
    12.                     return System.Convert.ToBase64String(outBytes);
    13.             }
    14.             catch(System.Exception e)
    15.             {
    16.                 Debug.LogError(e);
    17.             }          
    18.         }      
    19.  
    20.         return null;
    21.     }
    but it throughs System.OverflowException: Number overflow. on the line
    Code (CSharp):
    1. byte[] outBytes = new byte[lzip.gzipUncompressedSize(files[fileName])];
    in the foreach loop.

    Please help :D
     
  22. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    Hi.

    You are trying to get the uncompressed size of a zlib buffer that probably has not gzip compatible headers to supply the uncompressed size.

    Try to gzip compress your buffers (including headers) or try to use the decompressBuffer function that works on zlib buffers.
     
  23. Brogan89

    Brogan89

    Joined:
    Jul 10, 2014
    Posts:
    244
    Ok, thanks, ill try it out.

    Just a bit more insight into what im trying to do.
    I am get the bytes array from Texture2D.EncodeToPNG, but nothing is a file, i dont want to make files, its all happening in runtime. nothing is stored locally. the base64 is to send to server.

    Ill play around with it a bit more. I just don't fully understand it is all.
     
  24. Brogan89

    Brogan89

    Joined:
    Jul 10, 2014
    Posts:
    244
    Hey man I have spent quite a bit of time trying to get this work and am starting to give up.
    I am trying to zip a file of 2 PNG images, in a stream and return a base64 string of the zipped file. In another library I am using this code

    Code (CSharp):
    1.        
    2. using (MemoryStream ms = new MemoryStream())
    3.         {
    4.             // create zip file
    5.             using (ZipFile zip = new ZipFile()) // from Iconic.Zip
    6.             {        
    7.                 // forach file
    8.                 foreach (var fileName in files.Keys)
    9.                 {
    10.                     // add to zip
    11.                     zip.AddEntry(fileName, files[fileName]);
    12.                 }
    13.                 // save zip file in stream
    14.                 zip.Save(ms);
    15.             }
    16.             // close stream
    17.             ms.Close();
    18.  
    19.             // get base64 string
    20.             base64 = System.Convert.ToBase64String(ms.ToArray());
    21.         }
    However, it is not working for WebGL which we need support for.
    In your library I have gotten this far (try to use the same logic as the above code)

    Code (CSharp):
    1.      
    2.  using (MemoryStream ms = new MemoryStream())
    3.         {
    4.             byte[] fileBuffer = null;
    5.             var zip = lzip.zipArchive("", fileBuffer);
    6.  
    7.             foreach (var fileName in files.Keys)
    8.             {
    9.                 byte[] outBuffer = null;
    10.                 lzip.compressBuffer(files[fileName], ref outBuffer, 1);
    11.                 lzip.entry2Buffer("", fileName, ref outBuffer, fileBuffer);
    12.             }
    13.  
    14.             if (fileBuffer != null)
    15.             {
    16.                 ms.Write(fileBuffer, 0, fileBuffer.Length);
    17.             }
    18.             else
    19.             {
    20.                 Debug.LogError("FileBuffer null");
    21.             }
    22.  
    23.             ms.Close();
    24.             base64 = System.Convert.ToBase64String(ms.ToArray());
    25.         }
    Bu it is returning null.

    Is there a way using your library to produce the same result as the first block of code. That library was very easy to figure out but this is quite complicated and the only example i found for WebGL was gzip and was only unzipping.

    To be clear I want to be able to
    1. create a zip file in memory.
    2. add pngs to that zip.
    3. return a base64 string of the zip file.

    Any help would be appreciated thank you.
     
  25. Brogan89

    Brogan89

    Joined:
    Jul 10, 2014
    Posts:
    244
    @elias_t , don't worry anymore, we are going to change the way we call our API using gzip instead, since it seems easier.
     
  26. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    Ok. Let me know how things turn out.


    On WebGL you will not be able to do this since it works only on zlib and gzip buffers.
     
    Brogan89 likes this.
  27. Brogan89

    Brogan89

    Joined:
    Jul 10, 2014
    Posts:
    244
    Hey yea, not getting on very well. It's crashing Unity. I'm sure it's because of something I am doing though.
    Could you please provide an example of how to gzip in a memory stream? Or does it have to be from a file on disk?
     
  28. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367

    Hi.

    Various reasons could lead to this.

    1. You are compressing uncompressible data.
    2. The gzip buffer does not contain headers.


    The files from the example are on disk just for the demo. They could be just downloaded buffers.

    Another user had such issues and the problem came from the code that created the gzip buffers.

    Send me your code where you compress your data over a private message and I will provide you a working example.
     
  29. sumpfkraut

    sumpfkraut

    Joined:
    Jan 18, 2013
    Posts:
    242
    Is it possible to add a list of textures to a archive?
    I want to save multiple textures in one file, and import them back in one file. (WebGL and Standalone)

    The example compress just one file.
    lzip.compress_File(9, path);
     
  30. witcher101

    witcher101

    Joined:
    Sep 9, 2015
    Posts:
    516
    can you tell me how is this better than the current default compression for webgl.
    Does this reduce file size considerably??
     
  31. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    Hi.

    In the example script are all possible variations.

    You can use lzip.compressDir to make a zip out of file in a directory.

    Or you can use lzip.compress_File having the append flag set to true, so you can append to a zip file.

    On WebGL you can compress/decompress buffers only.
     
  32. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    The plugin gives you the ability on WebGL to compress/decompress buffers.

    One scenario is to use www.bytes to get a buffer and decompress it on the fly.
     
  33. Aledebarande

    Aledebarande

    Joined:
    Oct 6, 2015
    Posts:
    15
    Hi Elias !

    Your plugin is great, and does for me the job for several app.
    Now I'm venturing in something new with WebGL, and I need to know what is the equivalent method of

    lzip.decompress_File(FullPath, LocalDir, progress);

    that works in webGL ?

    ... Or any sample code that will take an www.bytes as a .zip file and uncompress its content to the persistant data path :) (...zip file containing multiple files)
     
    Last edited: Sep 20, 2017
  34. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    Hi.

    Most of the functions used to work on WebGL.

    But later Unity changed the security restrictions and it was not possible to download and decompress a file.

    To treat a www.bytes buffer as a zip archive was never possible since the beginning due to some restrictions with the emscripten compiler.

    So the best option is to use saved gzip/zlib buffers and uncompress them.
     
    Aledebarande likes this.
  35. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    update version 2.28 [Major zip plugin update]

    - Zip: Added Encryption/Decryption support.

    - Zip: Added bz2 as an alternative compression/decompression method of zip archives.

    - Zip: WSA supports now FileBuffers.

    - Zip: On WSA encryption is supported only on UWP10 x64.

    - Zip: Better Unicode support on Windows platforms.

    - Zip: Faster and more reliable getInfo function. (intermediate file is not used anymore.)

    - Zip: Low level functions have been removed due to inconsistency between platforms.

    - Zip: Added the ability to compress a list of files into a single archive.
     
  36. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    HOTFIX notification for the zip plugin iOS builds (2.28 version and up):

    In your project's "Build Settings->Linking->Other Linker flags" add -lz

    Screen Shot 2017-09-25 at  22.41.37.png
     
  37. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    Hi all.

    There is an issue with the Zip plugin regarding the bz2 method in the last update on MacOS/iOS platforms. Avoid using this method for now on these platforms until a fix.
     
  38. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    update version 2.28.2

    - ZIP: iOS/MacOS Hotfix.

    - ZIP: bz2 method removed for MacOS/iOS/watchOS/tvOS because it is not working correctly.
     
  39. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    update version 2.29

    - ZIP: Removed encryption/decryption for WSA due to certification reasons.
     
  40. Aledebarande

    Aledebarande

    Joined:
    Oct 6, 2015
    Posts:
    15
    Hi ! I have a problem building app for iOS ; Xcode throw me a :


    Showing All Messages
    ld: warning: object file (/Users/jean-christopheromain/Desktop/iOS/Libraries/Plugins/iOS/libzipw.a(dll.o)) was built for newer iOS version (8.0) than being linked (7.0)
    [several more]

    [...]

    Showing All Messages
    "_zipExtractToFile", referenced from:
    _lzip_zipExtractToFile_m3248955922 in Bulk_Assembly-CSharp-firstpass_0.o
    (maybe you meant: _lzip_zipExtractToFile_m3248955922)

    etc... (several more)

    and also this warning :

    Showing All Messages
    ld: warning: object file (/Users/jean-christopheromain/Desktop/iOS/Libraries/Plugins/iOS/libzipw.a(dll.o)) was built for newer iOS version (8.0) than being linked (7.0)


    Any information or tips about this link problem ?
    The previous builds were working fine....
     
  41. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    Hi. Do you have this using the bitcode enabled plugin?
    In that case I can compile the bitcode plugins with a lower version of xcode.
     
  42. Aledebarande

    Aledebarande

    Joined:
    Oct 6, 2015
    Posts:
    15
    Well I don't use bitcode in Xcode (for now)
    How can I manage using bitcode enabled plugins or not ?
     
  43. Aledebarande

    Aledebarande

    Joined:
    Oct 6, 2015
    Posts:
    15
    Tryed to remove bitcode ios folder from pluggin and redo a ios build but same link error ::

    Undefined symbols for architecture armv7:
    "_zipEntryExists", referenced from:
    _lzip_zipEntryExists_m1479011174 in Bulk_Assembly-CSharp_0.o
    (maybe you meant: _lzip_zipEntryExists_m1479011174)

    Looks like the linker could'nt find the referenced bundles
     
  44. Aledebarande

    Aledebarande

    Joined:
    Oct 6, 2015
    Posts:
    15
    Ok works now....
    This absolutely wasn't related to 7zip plugin but to how Unity handles pluggins for multiplatforms..
    Ended up to remove all ADT files and re-import from asset store now it works fine !

    Thanks !
     
  45. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    Oh I see. Good to know.

    Was about to send you xcode 7.3 compiled plugins.
     
  46. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    If you want to use the bitcode compiled plugins, unzip the one you want from the bitcode folder and replace the one that is currently in use.
     
    Aledebarande likes this.
  47. Aledebarande

    Aledebarande

    Joined:
    Oct 6, 2015
    Posts:
    15
    Ah ah thanks a lot Elias but right now I don't have this need, this is really nice from you !
    (I'll bother you later if we have tu use bitcode and/or xcode 7.3 but things works like a charm now !)
    Thanks a lot !
     
  48. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    update version 2.30

    - ZIP: Added the getEntryDateTime function, to get DateTime for a specific entry

    - Updated the file sample links of the demos, due to a server switch.
     
  49. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,367
    update version 2.31

    - ZIP: Added the entry2FixedBuffer function to decompress an entry to a fixed size buffer.

    - ZIP: The getFileInfo is now much faster on zip archives with thousands of files.
     
  50. ldlan

    ldlan

    Joined:
    Jan 12, 2016
    Posts:
    1
    I need compress in linux and decompress in android, but is required glibc-2.14, But i don't want upgrade system.How can I help it.