Search Unity

Strange webrequest answer !p

Discussion in 'Windows' started by starmindfr, Jul 19, 2019.

  1. starmindfr

    starmindfr

    Joined:
    Nov 7, 2016
    Posts:
    34
    Hello

    First i am using Steam relase with steamworks and Unity 2019.2.0b7 (64-bit)
    moni and net standard 2.0 with Standalone x86_64

    Issue is only on few clients computers, all the tests system we are here are working fine and lots of customers too

    But on few when we use the following code :

    Code (CSharp):
    1. WWWForm form = new WWWForm();
    2.         form.AddField("valeur", "versions");
    3.  
    4. #if (UNITY_STANDALONE)
    5.         form.AddField("platform", "STANDALONE");
    6.         Debug.Log("Release STANDALONE");
    7. #endif
    8.  
    9.         UnityWebRequest www = UnityWebRequest.Post(url, form);
    10.         www.downloadHandler = new DownloadHandlerBuffer();
    11.  
    12.     IEnumerator WaitForRequestVersion(UnityWebRequest www)
    13.     {
    14.         yield return www.SendWebRequest();
    15.  
    16.  
    17.         // check for errors
    18.         if ((www.error == null) && (www.responseCode < 400))
    19.         {
    20.             //Debug.Log("WWW Ok!: " + www.downloadHandler.data.ToString());
    21. #if (UNITY_STANDALONE_WIN)
    22.             Debug.Log("Version init debug: " + www.downloadHandler.text);
    23. #endif
    the downloadhandler.txt value read is "!p" instead of json "{"texte":"6.64","result":"OK"}"

    Computer where issue appear could open a web browser tool and type the url + post data then the json show fine.

    Only from the unity code we get the same answer "!p" for all the url we are calling. Also it's not shown as a www error code as the IF check show the debug log value "!p"

    The target server use PHP PHP/5.5.38 and Smarty


    I could not find any reference to a web answer "!p" so i have no idea how server could send that instead the json (still all works fine on others computers and is same come for android devices without issues)

    thanks for your help
     
  2. mgear

    mgear

    Joined:
    Aug 3, 2010
    Posts:
    5,345
    if you test with Postman or similar tools, is there any difference?
    since putting the url in browser is GET request, but your code makes POST request.
     
  3. starmindfr

    starmindfr

    Joined:
    Nov 7, 2016
    Posts:
    34
    yes postman reply is perfect (using POST with the arguments the same way unity should do) on the customer computer, only unity webrequest provide garbage. And a wrong POST request inside Postman provide another message (i included this case in the php code) and never a "!p" i have no idea what could be "!p"
     
  4. starmindfr

    starmindfr

    Joined:
    Nov 7, 2016
    Posts:
    34
    maybe its a BOM issue i have done lots of check and obviously the downloadHandler.text is not valid so i try to compare the BYTES betweens 2 computers, the result received is allways the same so it should be sames bytes but :

    On Brasil - Port computer with issue i received :

    Code (CSharp):
    1.  local default encoding:System.Text.UTF8Encoding
    2. STEAM LANGUAGE english
    3. Version init debug: !p
    4. Version init debug www size: 34
    5. Version init debug www txt try2: !p
    6. Version init debug www txt try3: IXAABHsidGV4dGUiOiI2LjEiLCJyZXN1bHQiOiJPSyJ9Aw==
    On EN_en computer i get :
    Code (CSharp):
    1.  
    2. local default encoding:System.Text.UTF8Encoding
    3. STEAM LANGUAGE english
    4. Version init debug: {"texte":"6.1","result":"OK"}
    5. Version init debug www size: 29
    6. Version init debug www txt try2: {"texte":"6.1","result":"OK"}
    7. Version init debug www txt try3: eyJ0ZXh0ZSI6IjYuMSIsInJlc3VsdCI6Ik9LIn0=
    source code
    Code (CSharp):
    1.            
    2. Debug.Log("Version init debug: " + www.downloadHandler.text);
    3.             byte[] results = www.downloadHandler.data;
    4.  
    5.             Debug.Log("Version init debug www size: " + results.Length);
    6.             //Debug.Log("Version init debug www txt try: " + results.ToString());
    7.  
    8.             string str = System.Text.Encoding.UTF8.GetString(results);
    9.             Debug.Log("Version init debug www txt try2: " + str);
    10.             string strNew = System.Convert.ToBase64String(results);
    11.             Debug.Log("Version init debug www txt try3: " + strNew);
    12.  
    13.             string str3 = System.Text.Encoding.UTF8.GetString(results, 3, results.Length - 3);
    14.             Debug.Log("Version init debug www txt try4: " + str3);
    so the bugged computer receive a 34 bits size raw data instead 29 the try3 text line is raw data

    So now i need help in order to find a way to have downloadHandler.text processing the raw data to text , but for all my game code (maybe +100 webrequests)

    Edit : Base64 decoder seems to show the expected answer with few extra codes... i have no idea why on this system then its in base64
     
    Last edited: Jul 20, 2019
  5. Aurimas-Cernius

    Aurimas-Cernius

    Unity Technologies

    Joined:
    Jul 31, 2013
    Posts:
    2,247
    This looks like an encoding mishandling.
    Could you print all response headers and check the raw received bytes against what your server has sent?
    If encoding is provided by HTTP headers, UWR should pick it, and default to UTF8 if no encoding provided.
     
  6. starmindfr

    starmindfr

    Joined:
    Nov 7, 2016
    Posts:
    34
    nice idea i will try that to know what is encoded also what i try to figure out is why between 2 Windows computers 64 bits the webrequest handling is not same, is this related to .net sdk / fixes or asp on the cliet ?
     
  7. Aurimas-Cernius

    Aurimas-Cernius

    Unity Technologies

    Joined:
    Jul 31, 2013
    Posts:
    2,247
    In general there shouldn't be any difference. Since you mention, that the issue is observed on computers with different locale, that's the place to start.
    First question now is: is the server sending different bytes or is it the client failing to convert those bytes to text?
     
  8. starmindfr

    starmindfr

    Joined:
    Nov 7, 2016
    Posts:
    34
    thanks a lot aurimas cernius i am waiting the new data from customers but the server php code have no update / change it should the same result for all clients

    at least here is the data from test system where all is ok :

    Code (CSharp):
    1. header key:o2switch PowerBoost=Server
    2. header key:Tue, 23 Jul 2019 15:59:44 GMT=Date
    3. header key:text/html; charset=utf-8=Content-Type
    4. header key:29=Content-Length
    5. header key:keep-alive=Connection
    6. header key:Accept-Encoding=Vary
    7. header key:PHP/5.5.38=X-Powered-By
    8. header key:Thu, 19 Nov 1981 08:52:00 GMT=Expires
    9. header key:no-store, no-cache, must-revalidate, post-check=0, pre-check=0=Cache-Control
    10. header key:no-cache=Pragma
    11. header key:PHPSESSID=589c95808e5eebd4d3ab981cd22feb0c; path=/=Set-Cookie
    12. Version internet return :ReachableViaLocalAreaNetwork
    also note that if using a web brower with an html online POST tool the result received is OK inside the browser of customer getting issue insid steam (not sure how it's parsed in javascript)
     
    Last edited: Jul 23, 2019
  9. Aurimas-Cernius

    Aurimas-Cernius

    Unity Technologies

    Joined:
    Jul 31, 2013
    Posts:
    2,247
    UnityWebRequest should find this header and turn bytes into string using UTF-8 encoding.
    However, if a different encoding is specified here, it will be used, unless unsupported (fallback to UTF-8 then).

    So, what you need now is exact headers and bytes (downloadHandler.data) from failing machine. Even better is to use some debugging proxy like Fidder to get the raw request and response for analysis.
     
  10. starmindfr

    starmindfr

    Joined:
    Nov 7, 2016
    Posts:
    34
    hello so here is the customer result :
    Code (CSharp):
    1.  
    2. raw data of downloadhandler: IXAABHsidGV4dGUiOiI2LjEiLCJyZXN1bHQiOiJPSyJ9Aw==
    3. text received using txt.64deconding: EOT{"texte":"6.1","result":"OK"}ETX
    4. header key:Server=o2switch PowerBoost
    5. header key:Date=Tue, 23 Jul 2019 20:26:12 GMT
    6. header key:Content-Type=text/html; charset=utf-8
    7. header key:Transfer-Encoding=chunked
    8. header key:Connection=keep-alive
    9. header key:Vary=Accept-Encoding
    10. header key:X-Powered-By=PHP/5.5.38
    11. header key:Expires=Thu, 19 Nov 1981 08:52:00 GMT
    12. header key:Cache-Control=no-store, no-cache, must-revalidate, post-check=0, pre-check=0
    13. header key:Pragma=no-cache
    14. header key:Set-Cookie=PHPSESSID=acb15bca061a53d775b4ec25414f95ce; path=/
    15. header key:Content-Encoding=br
    so EOT and ETX are the systems chars before and after the text part braking the .data to downloadhander.txt process giving instead only !p
    and this only on few computers

    on computer woriking fine same .data bytes :
    bytes couts 29 instead 32
    downloader.data : eyJ0ZXh0ZSI6IjYuMSIsInJlc3VsdCI6Ik9LIn0=
    downloader.txt {"texte":"6.1","result":"OK"}
    no ETX or EOT

    edit : i just notice the chunked transfert on the bugged client and i have

    Code (CSharp):
    1.        UnityWebRequest www = UnityWebRequest.Post(url, form);
    2.         www.downloadHandler = new DownloadHandlerBuffer();
    3.         www.certificateHandler = new AcceptAllCertificatesSignedWithASpecificKeyPublicKey();
    4.         www.chunkedTransfer = false;
    so i disabled all compress methods i could see on httpaccess and instead ETX EOT i get whites stanges signs []

    also last header line is content-encoding=br i dont have this on valids test systems
     
    Last edited: Jul 23, 2019
  11. Aurimas-Cernius

    Aurimas-Cernius

    Unity Technologies

    Joined:
    Jul 31, 2013
    Posts:
    2,247
    This looks a very interesting issue indeed. Content-Encoding=br means brotli compression. Chunked transfer is unlikely to have anything to do with this.
    I don't think you can avoid debugging this issue with proxy (Fiddler, Charles or similar). We need to examine the exact request and response to pinpoint the source of the problem.
     
  12. starmindfr

    starmindfr

    Joined:
    Nov 7, 2016
    Posts:
    34
    Hello Aurimas sadly the end user dont have needed knowledge for such process so i dont know if it will possible.

    From google i see Brotli could lead to a double compression issue. I will try to rewrites rules on httpaccess but i still dont understand why the headers are not process / sames with same unity code without headers in the webrequest
     
  13. Aurimas-Cernius

    Aurimas-Cernius

    Unity Technologies

    Joined:
    Jul 31, 2013
    Posts:
    2,247
    Do you have control of the server? If yes, then you should be able to do logging on server and extract all the data from there (and compare with users data to check if some proxy didn't modify anything).
     
  14. starmindfr

    starmindfr

    Joined:
    Nov 7, 2016
    Posts:
    34
    yes i have an access to server and could do few things but no sessions or telnet and log is quite huge i will see what i could get.

    also i started to check the main httaccess and figured out that the compressions methods are not sames between browsers so maybe on the customer side the browser use another compression / default :

    Code (CSharp):
    1. <IfModule mod_deflate.c>
    2.     SetOutputFilter DEFLATE
    3.     <IfModule mod_setenvif.c>
    4.         # Netscape 4.x has some problems...
    5.         BrowserMatch ^Mozilla/4 gzip-only-text/html
    6.        
    7.         # Netscape 4.06-4.08 have some more problems
    8.         BrowserMatch ^Mozilla/4\.0[678] no-gzip
    9.        
    10.         # MSIE masquerades as Netscape, but it is fine
    11.         # BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
    12.        
    13.         # NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
    14.         # the above regex won't work. You can use the following
    15.         # workaround to get the desired effect:
    16.         BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
    17.        
    18.         # Don't compress images
    19.         SetEnvIfNoCase Request_URI .(?:gif|jpe?g|png)$ no-gzip dont-vary
    20.     </IfModule>
    21.    
    22.     <IfModule mod_headers.c>
    23.         # Make sure proxies don't deliver the wrong content
    24.         Header append Vary User-Agent env=!dont-vary
    25.     </IfModule>
    26.  
    27. </IfModule>
    28. # AddHandler application/x-httpd-php53 .php
    29. suPHP_ConfigPath /home/xxxx/public_html/
     
  15. starmindfr

    starmindfr

    Joined:
    Nov 7, 2016
    Posts:
    34
    after a real time server log check i found something

    this is the line of the computer with issue :

    177.157.xxx.xxx - - [24/Jul/2019:22:01:30 +0200] "POST /android_version_check.php?cprotect=1 HTTP/1.1" 200 29 "-" "Mozilla/5.0"

    mozilla/5.0 is just user agent i forced today in header but the critical thing is that this is added on my url : cprotect=1

    this is the result for my test system where all is ok :

    "POST /android_version_check.php HTTP/1.1" 200 29 "-" "Mozilla/5.0"

    i see only one other http line with cprotect from some sort of browser outside of unity and not expected maybe a box for referencing or other.and it's not from the customer IP

    cprotect make me think about crawl protect (i dont use) or cpanel protect but i dont go far with this
     
    Last edited: Jul 24, 2019
  16. starmindfr

    starmindfr

    Joined:
    Nov 7, 2016
    Posts:
    34
    edit : After review with the server hosting services ?cprotect=x is from a front web security tool. After update of rules cprotect flage disapear.

    So now the main thing i see is brotli compatible header added where it should not be as unity user agent is same for all the clients and i dont have brotli enable.

    Also from server log 29 bytes are sent, and in unity 34 are shown.
     
    Last edited: Jul 27, 2019
  17. Aurimas-Cernius

    Aurimas-Cernius

    Unity Technologies

    Joined:
    Jul 31, 2013
    Posts:
    2,247
    Looks like there is some proxy in between which breaks things.
     
  18. starmindfr

    starmindfr

    Joined:
    Nov 7, 2016
    Posts:
    34
    yes i try to figure out what it could be but end user have very low computer level and so it's difficult to get a list of possibles proxy .

    As he seems to have them i tried both bytefence and avast but all is fine with them on test system.
     
  19. starmindfr

    starmindfr

    Joined:
    Nov 7, 2016
    Posts:
    34
    seems i could reproduce with adding

    www.SetRequestHeader("Accept-Encoding", "br");
    but seems i could not force on customer system a header to avoid this i tried :

    www.SetRequestHeader("Accept-Encoding", "identity");


    but customer system still use same encoding :

    header key:Content-Encoding=br
    linked to
    header key:Transfer-Encoding=chunked
     
    Last edited: Aug 2, 2019
  20. starmindfr

    starmindfr

    Joined:
    Nov 7, 2016
    Posts:
    34
    i solved the issue last thing i tried was 2 things updating a little the httaccess to disable gzip and brotli for the unity agent but i beleive the more effective was adding an header flag in a php script i use in all my php process :

    Code (CSharp):
    1. <?php
    2. header('Content-Encoding: identity', true);
    and in the htaccess

    Code (CSharp):
    1. <IfModule mod_deflate.c>
    2.     SetOutputFilter DEFLATE
    3.     <IfModule mod_setenvif.c>
    4.         # Unity test
    5.         BrowserMatch ^UnityPlayer no-gzip no-br
    6.         BrowserMatch ^UnityWebRequest no-gzip no-br
    7.  
    8.         Header append Vary User-Agent env=!dont-vary
    9.     </IfModule>
    10. </IfModule>