Search Unity

Data caching on Unity 5.6 is not working

Discussion in 'WebGL' started by XSUnity, Apr 27, 2017.

  1. bebo77

    bebo77

    Joined:
    Aug 3, 2012
    Posts:
    10
  2. vigoAtWork

    vigoAtWork

    Joined:
    Apr 25, 2016
    Posts:
    6
    The regression is that it's simply been removed from UnityLoader.js ;) For at least 5.6.2p1, the uploaded "patch" adds back IndexedDB-caching for the data-file (everything else usually fits in the browser cache). For brevity, it uses
    https://github.com/localForage/localForage, and simply adds a "useCache" code path to UnityLoader.downloadJob(), which is otherwise unchanged. Obviously, localForage and the original UnityLoader have to be included before "unityloader-fixup", e.g. change

    Code (Html):
    1.  
    2. ...
    3. <script src="/path/to/UnityLoader.js"></script>
    4.  
    5. <script>
    6. UnityLoader.instantiate(...);
    7. </script>
    8. ...
    9.  
    to

    Code (Html):
    1.  
    2. ...
    3. <script src="/path/to/UnityLoader.js"></script>
    4. <script src="/path/to/localforage.js"></script> <!-- https://github.com/localForage -->
    5. <script src="/path/to/unityloader-fixup.js"></script> <!-- the script below -->
    6.  
    7. <script>
    8. UnityLoader.instantiate(...);
    9. </script>
    10.  
    Without caching, my company had traffic in excess of 1 TB per day, with only a few thousand WebGL users.

    Edit: inline code seems easier; variable names are courtesy of Chrome's "pretty print" feature ;)
    Code (JavaScript):
    1. localforage.config({ driver: [ localforage.INDEXEDDB ] });
    2.  
    3. // patch UnityLoader's downloadJob
    4. UnityLoader.downloadJob = function(e, t) {
    5.     var url = t.parameters.url;
    6.     var useCache = (url.indexOf(e.dataUrl) == url.length - e.dataUrl.length);
    7.  
    8.     t.parameters.onprogress && t.parameters.onprogress({ type: "progress" });
    9.  
    10.     if (useCache) {
    11.       localforage.getItem(url).then(
    12.         function(item) {
    13.           if (!item) {
    14.             console.log("getItem(" + url + ") successful - not stored yet");
    15.             download(url, useCache, e, t);
    16.             return;
    17.           }
    18.  
    19.           console.log("getItem(" + url + ") successful - retrieved " + item.byteLength + " bytes");
    20.           decompress(item, t);
    21.         },
    22.         function(err) {
    23.           console.error("getItem(" + url + ") failed - err = " + err);
    24.           download(url, false, e, t); // most likely not recoverable
    25.         }
    26.       );
    27.     } else {
    28.       download(url, useCache, e, t);
    29.     }
    30.  
    31.     function decompress(buffer, t) {
    32.       UnityLoader.Compression.decompress(buffer, function(e) {
    33.         t.parameters.onload && t.parameters.onload({ type: "load" });
    34.         t.complete(e);
    35.       });
    36.     }
    37.  
    38.     function download(url, useCache, e, t) {
    39.       var r = new XMLHttpRequest;
    40.       r.open("GET", url);
    41.       r.responseType = "arraybuffer";
    42.       r.onload = function() {
    43.         var buffer = new Uint8Array(r.response);
    44.  
    45.         if (useCache) {
    46.           localforage.setItem(url, buffer).then(
    47.             function() {
    48.               console.log("setItem(" + url + ") successful - stored " + buffer.byteLength + " bytes");
    49.               decompress(buffer, t);
    50.             },
    51.             function(err) {
    52.               console.error("setItem(" + url + ") failed - err = " + err);
    53.               decompress(buffer, t);
    54.             }
    55.           );
    56.         } else {
    57.           decompress(buffer, t);
    58.         }
    59.       }
    60.       t.parameters.onprogress && r.addEventListener("progress", t.parameters.onprogress);
    61.       r.send();
    62.     }
    63.   }
     

    Attached Files:

    Last edited: Nov 8, 2017
  3. nfmelendez

    nfmelendez

    Joined:
    Feb 26, 2015
    Posts:
    20
    Having the same problem here, please fix it is blocker for us.
     
  4. amateurd

    amateurd

    Joined:
    Nov 1, 2016
    Posts:
    23
    This sounds wonderful - if you have time can you list the steps required? I wasn't sure whether you needed to edit an existing file and what to include.

    Thanks
     
  5. bebo77

    bebo77

    Joined:
    Aug 3, 2012
    Posts:
    10
    ...I do not understand how to use it! In the wiki talk about installation...
    But is Unity's team still doing nothing?
    How Much Should We Stay With This Problem?
     
  6. XSUnity

    XSUnity

    Joined:
    Nov 18, 2012
    Posts:
    17
    Marco, the problem is serious for all of us. It's been 2 months since your last response and more than 4 months since you switched to the new caching system.

    The last version that have working cache is 5.5.4.p3. We have tried everything till the last stable ( 2017.1.0 p5 ).

    What have to be done, in order to take us more seriously ? The new system is NOT working. We are using 2 different unity versions to build (one for web gl and one for mobile since 5.5.4 have issues on mobile ), this is costing us time and money.

    How much more do we have to wait for a working cache system ?

    P.S.
    Yes, we've split our game into asset bundles, and the main build size is now 50 mb, and still NO CACHE
     
  7. Marco-Trivellato

    Marco-Trivellato

    Unity Technologies

    Joined:
    Jul 9, 2013
    Posts:
    1,653
    We know this is important and we have been working on a solution. We will update you very soon.
     
  8. daniel-granat

    daniel-granat

    Joined:
    Jun 2, 2013
    Posts:
    1
  9. ortin

    ortin

    Joined:
    Jan 13, 2013
    Posts:
    221
    Any new info?
    "very soon" seems to have expired
     
  10. bebo77

    bebo77

    Joined:
    Aug 3, 2012
    Posts:
    10
    I hope for the release Unity 2050.9
     
  11. roka

    roka

    Joined:
    Sep 12, 2010
    Posts:
    457
    I dreamed a dream ........ ^^
     
  12. Marco-Trivellato

    Marco-Trivellato

    Unity Technologies

    Joined:
    Jul 9, 2013
    Posts:
    1,653
    As you noticed, this has been taking more time than expected. However, a new caching solution is currently being tested and reviewed. You should be able to try it in a few weeks.
     
    blacKode likes this.
  13. kognito1

    kognito1

    Joined:
    Apr 7, 2015
    Posts:
    321
    By any chance will this work with WebGLCachedXMLHttpRequest (or will that be updated)? That was the superior method of caching the binary imo.
     
  14. mmortall

    mmortall

    Joined:
    Dec 28, 2010
    Posts:
    81
    PlayerPrefs also not working in latest Unity 5.6 in WebGL builds? Is it related issue?

    Reproduced in Google Chrome. Works ok in Firefox
     
  15. ciorbyn

    ciorbyn

    Joined:
    Oct 17, 2013
    Posts:
    28
    PlayerPrefs have problems both in 5.5 and 5.6.

    I'm relying on a database for many games, because sometimes with chrome (the most used browser) are deleted on their own.
     
  16. blacKode

    blacKode

    Joined:
    May 21, 2015
    Posts:
    38
    Thanks Marco !

    * fisrt post issue XSUnity, Apr 27, 2017 ... (half-year and the issue is very very important for webgl users) must recognize that it is not serious.
     
  17. paxtonmason

    paxtonmason

    Joined:
    Sep 17, 2012
    Posts:
    8
    Sorry to pester you, but our team is also blocked on this issue. Any update on the release of the caching fix?
     
  18. Marco-Trivellato

    Marco-Trivellato

    Unity Technologies

    Joined:
    Jul 9, 2013
    Posts:
    1,653
    A fix will be available in 2018.1 and we are in the process of back-porting to 2017.3
     
    blacKode likes this.
  19. roka

    roka

    Joined:
    Sep 12, 2010
    Posts:
    457
    It's an april fool?

    You create a new file format without checking if the caching system work with it and now you need more than 6 months for fixing that?

    Like @cvnbcvnb said, it's really not serious. Also, 5.6 users will not have the choice to upgrade their version. It's call "How to force everybody to move on the new subscription system".

    A lot of reports has been made by us and most of them are important and not fixed after a lot of months.

    So yeah, today, i'm not happy. This thread has been started in April ...... seriously .....
     
    ortin and blacKode like this.
  20. Marco-Trivellato

    Marco-Trivellato

    Unity Technologies

    Joined:
    Jul 9, 2013
    Posts:
    1,653
    We are not creating a new format. No changes will be required on your side, at least in the 2017.3 version.

    The first step is to back port to to 2017.3, then we will assess the risk of back-porting to earlier versions.
     
    blacKode likes this.
  21. roka

    roka

    Joined:
    Sep 12, 2010
    Posts:
    457
    Sorry, but according that more than 6 months is needed for fixing a simple file extension, i'm sure that it will not be back-ported.

    I was referring to the .unityweb extension.

    My post of May :

    Your answer :
    The .unityweb have started this thread.
     
    blacKode likes this.
  22. Marco-Trivellato

    Marco-Trivellato

    Unity Technologies

    Joined:
    Jul 9, 2013
    Posts:
    1,653
    @roka The problem has been introduced by the new embedding/loading mechanism. The unityweb extension is just one part of it.

    Again, there won't be a new extension/format.
     
    ilmario and blacKode like this.
  23. ilmario

    ilmario

    Joined:
    Feb 16, 2015
    Posts:
    69
    @Marco-Trivellato
    I was not aware of this thread while wondering why .unityweb data caching was working less great than expected. Didnt even know that the data-files used to be cached in the indexedDB. So I created this issue a few days ago:
    https://feedback.unity3d.com/sugges...-supporting-browser-caching-solution-proposal

    I already got an answer back stating that the issue is being worked on and a solution is near (must be talking about the same thing as you are). I also asked if the solution will contain splitting the .unityweb files into multiple smaller chunks (ensuring that each is well below the 50Mb default cache entry size of Firefox, Chrome being more dynamic regards to the limit, but usually caches only even smaller files).

    I also wanted to know if the files would get named so that each build/version would be guaranteed to be fetched as soon as its update (not resulting in accidental stale caches). I mean, this could likely happen if the unityweb-filenames are static, and caching them only happens via direct browser HTTP caching (Cache-Control - header).
    But maybe thats not an issue with indexedDb, which I know less about?

    Anyway, Unity 2018.1 sounds like a far far date, especially considering that I'm using Unity 2017.1.2p2, and 2017.2 (and the beta .3) all seem to unpolished for me to use them, yet.
     
    blacKode likes this.
  24. Marco-Trivellato

    Marco-Trivellato

    Unity Technologies

    Joined:
    Jul 9, 2013
    Posts:
    1,653
    Hi @ilmario,

    Yes, the solution we are working is what I was referring to, though it does not include any splitting of the data file. Having said that, I think it makes sense to consider it.

    The new data caching that we implemented for 2018.1 (which we are considering back-porting), relies on E-Tags so if the file was updated on the server, it will cause the data file to be re-downloaded otherwise it will be fetched from indexed-db.

    On this topic, I should also mention that in the Publishing Settings you can enable "Name Files as Hashes" to have unique file names for each build.
     
    Last edited: Nov 12, 2017
    blacKode and ilmario like this.
  25. melefans

    melefans

    Joined:
    Aug 1, 2016
    Posts:
    36
    this issue has been solved ?
    5.6.4.p3 data cache not work.:(
     
    Last edited: Dec 26, 2017
  26. Marco-Trivellato

    Marco-Trivellato

    Unity Technologies

    Joined:
    Jul 9, 2013
    Posts:
    1,653
    Data caching has been fixed in 2017.3 we just released.

    Has anyone had a chance to try it yet? It would be good to get feedback on it before back-porting to earlier versions.
     
    Last edited: Dec 23, 2017
    amateurd and blacKode like this.
  27. melefans

    melefans

    Joined:
    Aug 1, 2016
    Posts:
    36
    thank you for your reply.
    I will upgrade version as soon as possible, the feedback results for you。:)

    update : unity 2017.3 work nice !!:)
     
    Last edited: Dec 26, 2017
  28. blacKode

    blacKode

    Joined:
    May 21, 2015
    Posts:
    38

    Works nice for me with Apache and IIS... in Chrome, Edge and Firefox.


    But no works with nodejs express.

    Save to indexedDB works, but UnityLoader.js set request header Cache-Control to no cache and express with this adjustment no send status code 304 (always sends 200) and never load from indexedDB.


    I fix editing UnityLoader.js, and changing:
    ... and works.

    Thank you.

    Regards...
     
    Last edited: Dec 25, 2017
  29. amateurd

    amateurd

    Joined:
    Nov 1, 2016
    Posts:
    23
    It is working well for most browsers (2017.3). Thank you. Has a problem with Microsoft Edge. Every couple of attempts, it returns an error "indexedDB database could not be opened." Next time you try it works fine. Seems to be completely random. Delaying the request until Edge is "ready" might help.
     
    blacKode likes this.
  30. blacKode

    blacKode

    Joined:
    May 21, 2015
    Posts:
    38
    Same for me in Chrome...
     
    amateurd likes this.
  31. ciorbyn

    ciorbyn

    Joined:
    Oct 17, 2013
    Posts:
    28
    The cache now works, but I do not know why Multiplayer games using photon now have serious problems.

    The browser page it comes often frozen and I have to kill it from the task manager.
    maybe some problems with websockets...
    Does anyone have the same problem?

    Every time I update my games, there is a new WEBGL problem... pfff!!! :mad:
     
    Last edited: Mar 29, 2018