Search Unity

  1. Looking for a job or to hire someone for a project? Check out the re-opened job forums.
    Dismiss Notice
  2. Unity 2020 LTS & Unity 2021.1 have been released.
    Dismiss Notice
  3. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Mangled POST requests with UnityWebRequest on 2018.4.x [SOLVED?]

Discussion in 'WebGL' started by Kumo-Kairo, Apr 2, 2020.

  1. Kumo-Kairo

    Kumo-Kairo

    Joined:
    Sep 2, 2013
    Posts:
    343
    UPDATE
    We have managed to fix this by manually patching the framework file after build.
    See this post for more info.

    Reference patching tool can be found at https://github.com/SWAG-MASHA/UnityWebRequestWebPatcher
    ---

    We're experiencing a strange issue sending POST request in certain browsers on certain machines. The request body seems to be completely mangled (looks like random bytes of memory)
    upload_2020-4-2_13-52-50.png
    upload_2020-4-2_13-52-58.png

    We haven't been able to find the culprit of the problem - it's relatively rare (it only happens on most browsers of 4 of the machines of our QA department and 1 browser of our developer). But even 3% reproduce-ability among our players means a lot of negative feedback, so we can't leave it as it is.

    We tried updating to the latest 2018.4.19, but it doesn't solve the problem

    I can't really find any info on these Unity forums or on the internet and can't file a bug because we don't have a 100% working way of reproducing it.

    Does anyone have any info regarding this? Or anyone experiencing something similar?
     
    Last edited: Apr 9, 2020
  2. roka

    roka

    Joined:
    Sep 12, 2010
    Posts:
    485
    I have the same on my 2 games and it seem that the UnityWebRequest POST function is affected.
    Around 3/5 % of my players has been affected , so it has been really hard to find the problem.

    I have moved everything to GET in order to solve the situation quickly but my code look horrible.
     
  3. roka

    roka

    Joined:
    Sep 12, 2010
    Posts:
    485


    Affected players get this instead of "GetInfo:ffisjs87564fddd54qs658"
    I'm using Unity 2019.2.11f1
     
    Last edited: Oct 7, 2020
  4. Kumo-Kairo

    Kumo-Kairo

    Joined:
    Sep 2, 2013
    Posts:
    343
    We have managed to find the culprit of the problem.
    Taking a look at the .wasm.framework.unityweb code and seeing the _JS_WebRequest_Send method, it tries to http.send the raw bytes of the body before converting them to a string:
    Code (JavaScript):
    1. http.send(HEAPU8.subarray(ptr, ptr + length));
    It seems like most browsers on most systems can correctly detect raw bytes and convert them to a string automatically. But some can't (those 3% of cases).

    What we did - we wrote a patching tool that takes compressed framework.unityweb file, decompresses it, changes the http.send code so it takes a pre-converted string (to a UTF-8 using a TextDecoder), and compresses it back to gzip again.
    TextDecoder is a relatively unsupported tool, but it's more widely supported than wasm itself, so it shouldn't be a problem to use. There are also a lot of third-party js implementations that convert a sequence of UTF-8 bytes to a string.

    It took some time to manage the cryptic nature of those compressed files, but POST requests work correctly now (including the internal ones like unity analytics)
    Unity requires certain comments to be present in the gzip archive and we couldn't manage to find where exactly does Unity adds that comment to a gzip archive (7za instance which is embedded in Unity doesn't support adding comments).

    It would be great to get some comment of the Unity Tech guys on this problem.

    I will post a patching tool here a bit later.
     
    Last edited: Apr 3, 2020
  5. roka

    roka

    Joined:
    Sep 12, 2010
    Posts:
    485
    Maybe you can patch the WebRequest.js file present in :
    - Program Files\Unity\Hub\Editor\"YourUnityVersion"\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\lib instead of using your tool???

    Also, posting your solution can be usefull. Thank you
     
  6. Kumo-Kairo

    Kumo-Kairo

    Joined:
    Sep 2, 2013
    Posts:
    343
    I have just checked - files UnityNative.asm.js and UnityNative.js both contain this code, and it can be patched right on the Unity install.

    It doesn't really suit our case though as we use a separate CI tool that automatically builds new versions. And patching it on a Unity instance instead of on a post-build process will increase the maintenance pressure on our dev-ops team. It's also less transparent on what's going on and why certain builds on certain machines work (which have Unity patched) and some don't. Explicit is better than implicit.

    But again, maybe someone's running a one-dev-machine setup and rarely updates Unity, so engine patching can be a viable option
     
    Last edited: Apr 3, 2020
  7. roka

    roka

    Joined:
    Sep 12, 2010
    Posts:
    485
  8. Kumo-Kairo

    Kumo-Kairo

    Joined:
    Sep 2, 2013
    Posts:
    343
    roka likes this.
  9. roka

    roka

    Joined:
    Sep 12, 2010
    Posts:
    485
  10. Tarrag

    Tarrag

    Joined:
    Nov 7, 2016
    Posts:
    215
    Hi i'm suffering from this too , @roka could you please share some insight how you post with get as you mention at the top of this thread pls?

    thanks a bunch for your help
     
  11. Kumo-Kairo

    Kumo-Kairo

    Joined:
    Sep 2, 2013
    Posts:
    343
    @Tarrag I have updated the first post with our solution which we have essentially shipped with
     
  12. Tarrag

    Tarrag

    Joined:
    Nov 7, 2016
    Posts:
    215
    Hi @Kumo-Kairo thanks a bunch, yes I tested decompressed and gzip and neither worked for me. I built and then applied the patch. Probably didn't work as you mention in your post cos i'm using 2019 (tested on 2019.3.0f5, 2019.3.0f3, 2019.3.9)

    FYI I also raised this issue as a bug with Unity, I'll post when I hear back
     
  13. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    446
    Got here from another thread. Using C# to replace JS framework code is too complicated.
    You can try this instead:

    Under Assets/Plugins/WebGL create a file .jslib file (e.g. webrequest_patch.jslib) with this content:
    Code (JavaScript):
    1. var LibraryWebrequestSend = {
    2.   JS_WebRequest_Send: function (request, ptr, length) {
    3.     var td = new TextDecoder('utf-8');
    4.     var sp = td.decode(HEAPU8.subarray(ptr, ptr + length));
    5.     var http = wr.requestInstances[request];
    6.     try {
    7.         if (length > 0)
    8.             http.send(sp);
    9.         else
    10.             http.send();
    11.     } catch (e) {
    12.         console.error(e.name + ': ' + e.message);
    13.     }
    14.   }
    15. };
    16. mergeInto(LibraryManager.library, LibraryWebrequestSend);
    That should replace the existing JS function.
     
    Kumo-Kairo likes this.
  14. roka

    roka

    Joined:
    Sep 12, 2010
    Posts:
    485
    I made a development build and the jslib replace it perfectly on the wasm.framework.unityweb.
    Well done. I do not have tested if it work but it's in the good way.
     
    De-Panther likes this.
  15. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    633
    Just now found out about this issue. Added this to our work list.

    Btw, whether one sends a Uint8Array() or a string should only affect the Content-Type: MIME header that is sent along the request. Converting the Uint8Array to a string will cause the browser to just undo the conversion, wasting performance. I believe the proper fix here will be to be explicit about the Content-Type field that is being set, and not rely on browsers automatically specifying it. Though will need to test in practice to confirm.
     
  16. Kumo-Kairo

    Kumo-Kairo

    Joined:
    Sep 2, 2013
    Posts:
    343
    Yeah, maybe.
    We have just dropped the web version altogether, all problems solved (including performance drop of doing-undoing the conversion) :)
     
unityunity