Search Unity

7zip, lzma, LZ4, fastLZ, zip/gzip & brotli native multiplatform plugins.

Discussion in 'Assets and Asset Store' started by elias_t, Nov 16, 2013.

  1. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Hi all.

    A small fix for the decompress asset bundle function (lzma plugin):

    Code (CSharp):
    1.  
    2.  
    3.   public static  int decompressAssetBundle(byte[] inBuffer,  ref byte[] outbuffer){
    4.  
    5.      int offset = 0;
    6.    
    7.      for(int i=0; i<inBuffer.Length; i++) {
    8.        if(i>160) break;
    9.        if(inBuffer[i] == 0x5d) {
    10.          if(inBuffer[i+1] == 0x00) {
    11.            if(inBuffer[i+2] == 0x00) {
    12.              if(inBuffer[i+3] == 0x08) {
    13.                offset = i;  break;
    14.              }
    15.            }
    16.          }
    17.        }
    18.      }
    19.    
    20.      if(offset==0 || offset>160) return 4;
    21.  
    22.      GCHandle cbuf = GCHandle.Alloc(inBuffer, GCHandleType.Pinned);
    23.      IntPtr ptrBundle = new IntPtr(cbuf.AddrOfPinnedObject().ToInt64() + offset);
    24.      int uncompressedSize = (int)BitConverter.ToUInt64(inBuffer,offset+5);
    25.      Array.Resize(ref outbuffer, uncompressedSize);
    26.      GCHandle obuf = GCHandle.Alloc(outbuffer, GCHandleType.Pinned);
    27.    
    28.      int res = Lzma_Uncompress(ptrBundle, inBuffer.Length-offset, uncompressedSize, obuf.AddrOfPinnedObject(), true);
    29.  
    30.      cbuf.Free();
    31.      obuf.Free();
    32.  
    33.      //if(res!=0){/*Debug.Log("ERROR: "+res.ToString());*/ return res; }
    34.  
    35.      return res;  
    36.    }
    37.  
    38.  

    Edit: the update is up on the AssetStore.
     
    Last edited: Dec 8, 2015
  2. ludos

    ludos

    Joined:
    Nov 14, 2011
    Posts:
    53
    just after importing the plugin i am pretty annoyed. it has folders all over the asset directory, first thing i have to do is clean up the mess it created.
     
  3. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Hi.

    The asset has 2 parts.Folders that demonstrate the features of the plugins AND the _plugin_packages folder from where you can install individual plugins in a plugins only directory.

    Thank you.

    Untitled.png
     
  4. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    update version 2.08

    - Compatibility fixes for Windows Phone 8.1 and WSA8.1. Sdk8.1 should work out of the box now without the need to provide your own IO functions.

    - Added a benchmark scene.

    benchmark.jpg
     
    fantastisch_ likes this.
  5. idurvesh

    idurvesh

    Joined:
    Jun 9, 2014
    Posts:
    493
    Can I use it locally to reduce build size on android? i.e., Compress in editor and then on android phone decompress textures and materials and assign to respective models?
     
  6. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Yes you can. But it depends on the use case.

    For textures, if you compress some raw files you will not have the internal etc compression.

    Some people use this to compress asset bundles, extract and then load them.
     
    idurvesh likes this.
  7. idurvesh

    idurvesh

    Joined:
    Jun 9, 2014
    Posts:
    493
    Oh its you,I dint noticed....Multitasking lot of assets..nice

    My use case is as follow,
    Currently my apk size is going way to above 100mb , So I am planning to compress all prefabs and its references (tex,materials,fbx,scripts) in one 7z file then on run time decompress it on device...Will it work that way? WIll my references to prefab will be intact ? The prefab are basically race tracks , UI Canvas etc.
     
  8. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Hi :) (Yes, I work on various projects simultaneously, including contract jobs. Makes me less bored.)

    On the topic now.
    I am afraid that this would not work. Unity makes internal conversions on these files. The only thing you could do about them is to create asset bundles and compress them with the highest compression you can. Then you would decompress the assetbundles and load from there.

    One other use case is when you have data that can be loaded on runtime, like wavefront obj models, databases etc.
     
    idurvesh likes this.
  9. idurvesh

    idurvesh

    Joined:
    Jun 9, 2014
    Posts:
    493
    mm thanks for info, seems like need to go over asset bundles architecture to understand and implement it....Thanks for clarifying...

    Keep "A dog's life software" alive with insta classic ocean ...All the best
     
  10. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Thanks. I want to use the ocean in a project of mine, and I am a speed freak, so expect good things in the future.
     
    idurvesh likes this.
  11. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    update version 2.09

    - Added bitcode supported plugins for iOS.
     
    fantastisch_ likes this.
  12. wuhuan

    wuhuan

    Joined:
    Mar 12, 2014
    Posts:
    7
    Is 7ZIP support the extracting volume?
     
  13. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Hi. Sorry, it does not support multiple volumes for now.
     
  14. CADS

    CADS

    Joined:
    Jun 20, 2013
    Posts:
    13
    Hi,
    I just bought the 7zip library and am deeply impressed! Cool stuff!
    Is there also a Apple TV platform version planned?

    Thanks!
     
  15. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Hi.

    If you look in the Plugins/iOS folder there are various iOS variations zipped.
    tvOS and watchOS are included.
    Unzip the plugin that you want and you should be good.

    Thank you.
     
  16. CADS

    CADS

    Joined:
    Jun 20, 2013
    Posts:
    13
    Thanks for the fast response (and just-in-time implementation)! :)
     
  17. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    version 2.1 (Major update)

    - Added Brotli support for all platforms except WebGL & watchOS.

    - zip: added gzip support. Can operate on buffers with or without headers/footers.

    - zip: added progress for individual file decompression.

    - Android, Linux, iOS, MacOSX got chmod support to set file permissions.

    - iOS: made the plugins much smaller in size.

    - Reorganized the demo folders. You are adviced to delete the demo folders before updating the sample scenes.
     
    fantastisch_ likes this.
  18. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    update version 2.11

    zip/iOS: hotfix to fix a GcHandle crash.

    LZ4: added compress buffer to a specific location of a fixed size buffer.

    LZ4: added decompress an lz4 buffer from a specific location of a fixed size buffer
     
    fantastisch_ and zyzyx like this.
  19. 00christian00

    00christian00

    Joined:
    Jul 22, 2012
    Posts:
    808
    Hi, any source code available?
    I never buy any asset without source, you never know when it may stop be supported.
     
  20. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Sorry, no.

    But as I state in my asset publisher homepage:

     
  21. ZJP

    ZJP

    Joined:
    Jan 22, 2010
    Posts:
    2,646
    Hi,

    Only with LZMA?. :(

    Edit :
    Can we do this in another thread?
     
    Last edited: Mar 10, 2016
  22. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Hi. There are various buffer compress/decompress functions for all the plugins. Not Only lzma.
    Yes you can run this on separate threads.

    Here is the documentation of the plugins:
     

    Attached Files:

    • doc.zip
      File size:
      17.9 KB
      Views:
      119
  23. ZJP

    ZJP

    Joined:
    Jan 22, 2010
    Posts:
    2,646
    Thanks for the reply. :cool:

    Edit :
    Has more features than i thought. Seem very good.
     
    Last edited: Mar 10, 2016
  24. castor76

    castor76

    Joined:
    Dec 5, 2011
    Posts:
    1,506
    Hi.

    I have just got this asset and I have been trying to replace sharpzip lib I was using.

    So far so good.. except, I can't find the way to remove/delete entry from the zip file.

    How do I do this one? There isn't any example of this, or any obvious function from lzip
     
  25. castor76

    castor76

    Joined:
    Dec 5, 2011
    Posts:
    1,506
    Just to add, I don't want to extract all files and then manually recreate zip file...

    I just want a simple way of removing entry from a zip file without creating mess.
     
  26. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Hi. It is not possible to do this for various reasons.

    Most of the end user compression/decompression packages recreate the zip after extracting and deleting the file in question.
     
  27. castor76

    castor76

    Joined:
    Dec 5, 2011
    Posts:
    1,506
    Oh.. I am not sure why it can't be done as I am doing it using sharpzip.. I am trying to use your plugin because of Android platform.

    Anyway.

    If I use :

    lzip.buffer2File(9, fileName, zipEntryName, data, true))

    I want to know if it overwrites zip entry if it already exists.

    any idea?
     
  28. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Hi. I am not aware how sharpzip does it. But in any case some rewriting of the archive must be done.
    Perhaps they have a higher level function which does the reziping of the extracted files minus the file you want to delete.

    The buffer2File function will not overwritte. It will add a new archive in the file.
     
  29. castor76

    castor76

    Joined:
    Dec 5, 2011
    Posts:
    1,506
    Sharpzip has API that you can search for a particular entry in the zip, and then just call remove to remove the entry.

    Anyway, so if I have the entry already in the zip file, and want to overwrite it, then do I have to extract everything , delete and add one , and then zip up everything again?

    Wow.. I am not sure what to say.. but it seems like it will take time to do decompress and then compress again. It won't be too much of an issue with small sized zip file, but for the big size zip file, this kind of operation can take some time. This will result in a lot slower time than sharpzip. (despite that you mentioned that it is faster.. it may only be faster for doing a particular operations.)

    In normal , usual usage case, you would want to have more direct control over the zip file content, such as delete entry , overwrite entry, check for existence of the entry etc.. without having to decompress or compress whole zip file over and over again. Sharpzip may be doing this on background. It could be hiding this operation from the user, but it does have the simple function to call upon.
     
  30. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    To make a replacement in a zip file surely has some rewriting. They modify the file. So they rewrite something for sure.

    And if something goes wrong during that operation you lose your file.

    You can check for the existence of an entry by checking the file size of an entry. If it returns a negative number the entry does not exist.

    Anyway if you are not happy with the lib I would be happy to provide a refund.
     
  31. castor76

    castor76

    Joined:
    Dec 5, 2011
    Posts:
    1,506
    Hi Elias.

    I am enjoying the plugin but I would really like to use this on other console platform such as XboxOne.

    Do you think this is going to be possible? I only need zip compress/decompress buffer support. I would be glad to help/work with you to implement this.

    Please let me know. saehoon@pixellore.com

    Thanks!
     
  32. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Hi castor76.

    The plugins that are in the 'Plugins\WSA\UWP' directory should be XboxOne compatible since they are compiled as universal.

    So they should work.

    If you are having issues contact me to solve them.
     
  33. castor76

    castor76

    Joined:
    Dec 5, 2011
    Posts:
    1,506
    Thanks! I will try them!
     
  34. jchowdownKabam

    jchowdownKabam

    Joined:
    Aug 12, 2016
    Posts:
    4
    Hi Elias, by any chance have you tried this for 5.4.0?
     
  35. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    I have just run a test. All plugins seem to work fine except the zip plugin on Windows.
    I will investigate more to see if it is something on my end.

    edit2: There are no issues with unity5.4. I found some issues though with the zip plugin for windows and already uploaded the update to the asset store.
     
    Last edited: Aug 13, 2016
    idurvesh likes this.
  36. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    update version 2.13

    zip/Windows: better unicode support. Unity5.4 compatibility fix.
     
    fantastisch_ likes this.
  37. jchowdownKabam

    jchowdownKabam

    Joined:
    Aug 12, 2016
    Posts:
    4
  38. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    update version 2.14

    gzip: Fixed Android issues and added a 2nd ungzip function.
     
  39. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    update version 2.15

    Linux: fixed preprocessor directives for Unity5.4.x and up.
     
  40. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    update version 2.16

    ZIP All Platforms-gzip: Fixed a bug that was introduced in a previous update that had to with the outbuffer length. Now the gzip functions behave as intended.

    ZIP Windows: Now you can treat a zip file that is loaded in a byte buffer as a file.

    ZIP ios/gzip: updated the gzip functions.
     
  41. glenrhodes

    glenrhodes

    Joined:
    Oct 25, 2013
    Posts:
    55
    I've got a project that requires me to download a zip file via www.bytes in WebGL, and then decompress the zip, and use the assets that are in the file (images and text). From what I understand, there's no way to save the zip to a file in WebGL, so it all has to be done in memory via byte arrays.

    Unfortunately, when I run this code:

    IEnumerator DownloadZip(string url)
    {

    WWW www = new WWW(url);
    yield return www;

    byte[] bytes = www.bytes;

    int files = lzip.getTotalFiles(null, bytes);
    Debug.Log("Contains " + files + " files");

    }

    My log shows

    Contains -1 files

    From what I've seen, I should be able to pass in a byte array rather than a filename, and get the individual files, right? Is it possible to do what I'm trying to do?

    Thanks!
     
  42. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Hi. The WebGl plugins do not support 'FileBuffers as Files'.

    While the code to support this did compile with emscipten, it didn't work. So I left WebGL out of this.
    But I will try again with a newer version.

    If you run the demo in WebGl, it downloads a small zip and performs various operations on it.
    Doesn't this work for you?
     
  43. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Also try to use Application.temporaryCachePath to store the zip file there and use entry2Buffer to extract from there.

    I guess you already have read this: https://blogs.unity3d.com/2016/09/20/understanding-memory-in-unity-webgl/
     
  44. glenrhodes

    glenrhodes

    Joined:
    Oct 25, 2013
    Posts:
    55
    Thanks for your response! I had seen that article before but I didn't realize that Unity had implemented a pseudo-file system using memory. I searched all around and everyone said file operations weren't possible - use playerprefs instead. However, this looks promising. I'll give it a try.

    Thanks!
     
  45. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    version 2.17

    zip/Windows: A small fix to the getTotalFiles function to allow to work correctly with FileBuffers.
     
  46. c_andrews

    c_andrews

    Joined:
    Jan 26, 2015
    Posts:
    42
    Hi, I'm trying to decompress a zip on Android from the StreamingAssets folder into PersistentData but my app keeps crashing when it gets to line 28 below:

    Code (CSharp):
    1.  
    2. string from = Application.streamingAssetsPath + "/myZip.zip";
    3. string to = Application.persistentDataPath;
    4.  
    5. Debug.Log( "[AndroidStartUp] LOADING ZIP WITH WWW" );
    6.  
    7. WWW buffer = new WWW( from );
    8.  
    9. while ( buffer.isDone == false )
    10. {
    11.     yield return null;
    12. }
    13.  
    14. Debug.Log( "[AndroidStartUp] BUFFER ERROR: "+ buffer.error );
    15.  
    16. if ( buffer.error == null )
    17. {
    18.  
    19.     Debug.Log( "[AndroidStartUp] COPYING FILES FROM: " + from );
    20.     Debug.Log( "[AndroidStartUp] TO PERSISTENT DATA: " + to );
    21.     Debug.Log( "[AndroidStartUp] STARTING DECOMPRESSION" );
    22.  
    23.     mZipDecompression = lzip.decompress_File( null, to, mCompressionProgress, buffer.bytes );
    24.  
    25.  
    26.     Debug.Log ( "[AndroidStartUp] DECOMPRESSION: "+ mZipDecompression );
    27.  
    28.     while ( mZipDecompression == 0 )
    29.     {
    30.         yield return null;
    31.     }
    32.  
    33.     Debug.Log ( "[AndroidStartUp] DECOMPRESSION: "+ mZipDecompression );
    34.  
    35.     if ( mZipDecompression < 0 )
    36.         DecompressionFailed();
    37.     else if ( mZipDecompression == 1 )
    38.         DecompressComplete();
    39. }
    40. else
    41. {
    42.     Debug.Log("[AndroidStartUp] ERROR READING FROM STREAMING ASSETS");
    43. }
    Is there a better way of doing this? or is there something i am doing wrong?
    I have also tried moving the decompress into its own thread but i still get a crash.

    Thanks
     
    Last edited: Oct 26, 2016
  47. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Hi. The problem comes from the way you are yielding the function.

    If you use it like this it will work fine:

    Code (CSharp):
    1.     private int mZipDecompression;
    2.  
    3.     IEnumerator androidStreaming() {
    4.         string from = Application.streamingAssetsPath + "/testZip.zip";
    5.         string to = Application.persistentDataPath;
    6.         Debug.Log( "[AndroidStartUp] LOADING ZIP WITH WWW" );
    7.         WWW buffer = new WWW( from );
    8.         yield return buffer;
    9.         Debug.Log( "[AndroidStartUp] BUFFER ERROR: "+ buffer.error );
    10.         if ( buffer.error == null )
    11.         {
    12.             Debug.Log( "[AndroidStartUp] COPYING FILES FROM: " + from );
    13.             Debug.Log( "[AndroidStartUp] TO PERSISTENT DATA: " + to );
    14.             Debug.Log( "[AndroidStartUp] STARTING DECOMPRESSION" );
    15.             int[] mCompressionProgress = new int[1];
    16.  
    17.             mZipDecompression = lzip.decompress_File( null, to, mCompressionProgress, buffer.bytes );
    18.             Debug.Log ( "[AndroidStartUp] DECOMPRESSION: "+ mZipDecompression );
    19.  
    20.             if ( mZipDecompression < 0 )
    21.                 Debug.Log("DecompressionFailed");
    22.             else if ( mZipDecompression == 1 )
    23.                 Debug.Log("DecompressComplete");
    24.         }
    25.         else
    26.         {
    27.             Debug.Log("[AndroidStartUp] ERROR READING FROM STREAMING ASSETS");
    28.         }
    29.     }
     
  48. c_andrews

    c_andrews

    Joined:
    Jan 26, 2015
    Posts:
    42
    Amazing thank you. Worked like a charm
     
  49. c_andrews

    c_andrews

    Joined:
    Jan 26, 2015
    Posts:
    42
    Am i able to pull out a list of file names from a directory within the StreamingAssets folder on Android using this plugin?
     
  50. elias_t

    elias_t

    Joined:
    Sep 17, 2010
    Posts:
    1,187
    Hi.

    Yes you can.

    There are 2 ways to do this.

    1. Using www call and then call the appropriate functions that treat the www.bytes as a file.

    2. Use a www call to get the www.bytes of the file -> save to ApplicationPersistantDataPath -> decompress from there.