Strange webrequest answer !p

Hello

First i am using Steam relase with steamworks and Unity 2019.2.0b7 (64-bit)
moni and net standard 2.0 with Standalone x86_64

Issue is only on few clients computers, all the tests system we are here are working fine and lots of customers too

But on few when we use the following code :

Code (CSharp):

1. WWWForm form = new WWWForm();
2.         form.AddField("valeur", "versions");
3.  
4. #if (UNITY_STANDALONE)
5.         form.AddField("platform", "STANDALONE");
6.         Debug.Log("Release STANDALONE");
7. #endif
8.  
9.         UnityWebRequest www = UnityWebRequest.Post(url, form);
10.         www.downloadHandler = new DownloadHandlerBuffer();
11.  
12.     IEnumerator WaitForRequestVersion(UnityWebRequest www)
13.     {
14.         yield return www.SendWebRequest();
15.  
16.  
17.         // check for errors
18.         if ((www.error == null) && (www.responseCode < 400))
19.         {
20.             //Debug.Log("WWW Ok!: " + www.downloadHandler.data.ToString());
21. #if (UNITY_STANDALONE_WIN)
22.             Debug.Log("Version init debug: " + www.downloadHandler.text);
23. #endif

the downloadhandler.txt value read is "!p" instead of json "{"texte":"6.64","result":"OK"}"

Computer where issue appear could open a web browser tool and type the url + post data then the json show fine.

Only from the unity code we get the same answer "!p" for all the url we are calling. Also it's not shown as a www error code as the IF check show the debug log value "!p"

The target server use PHP PHP/5.5.38 and Smarty


I could not find any reference to a web answer "!p" so i have no idea how server could send that instead the json (still all works fine on others computers and is same come for android devices without issues)

thanks for your help

 

if you test with Postman or similar tools, is there any difference?
since putting the url in browser is GET request, but your code makes POST request.

 

yes postman reply is perfect (using POST with the arguments the same way unity should do) on the customer computer, only unity webrequest provide garbage. And a wrong POST request inside Postman provide another message (i included this case in the php code) and never a "!p" i have no idea what could be "!p"

 

maybe its a BOM issue i have done lots of check and obviously the downloadHandler.text is not valid so i try to compare the BYTES betweens 2 computers, the result received is allways the same so it should be sames bytes but :

On Brasil - Port computer with issue i received :

Code (CSharp):

1.  local default encoding:System.Text.UTF8Encoding
2. STEAM LANGUAGE english
3. Version init debug: !p
4. Version init debug www size: 34
5. Version init debug www txt try2: !p
6. Version init debug www txt try3: IXAABHsidGV4dGUiOiI2LjEiLCJyZXN1bHQiOiJPSyJ9Aw==

On EN_en computer i get :

Code (CSharp):

1.  
2. local default encoding:System.Text.UTF8Encoding
3. STEAM LANGUAGE english
4. Version init debug: {"texte":"6.1","result":"OK"}
5. Version init debug www size: 29
6. Version init debug www txt try2: {"texte":"6.1","result":"OK"}
7. Version init debug www txt try3: eyJ0ZXh0ZSI6IjYuMSIsInJlc3VsdCI6Ik9LIn0=

source code

Code (CSharp):

1.            
2. Debug.Log("Version init debug: " + www.downloadHandler.text);
3.             byte[] results = www.downloadHandler.data;
4.  
5.             Debug.Log("Version init debug www size: " + results.Length);
6.             //Debug.Log("Version init debug www txt try: " + results.ToString());
7.  
8.             string str = System.Text.Encoding.UTF8.GetString(results);
9.             Debug.Log("Version init debug www txt try2: " + str);
10.             string strNew = System.Convert.ToBase64String(results);
11.             Debug.Log("Version init debug www txt try3: " + strNew);
12.  
13.             string str3 = System.Text.Encoding.UTF8.GetString(results, 3, results.Length - 3);
14.             Debug.Log("Version init debug www txt try4: " + str3);

so the bugged computer receive a 34 bits size raw data instead 29 the try3 text line is raw data

So now i need help in order to find a way to have downloadHandler.text processing the raw data to text , but for all my game code (maybe +100 webrequests)

Edit : Base64 decoder seems to show the expected answer with few extra codes... i have no idea why on this system then its in base64

 

nice idea i will try that to know what is encoded also what i try to figure out is why between 2 Windows computers 64 bits the webrequest handling is not same, is this related to .net sdk / fixes or asp on the cliet ?

 

thanks a lot aurimas cernius i am waiting the new data from customers but the server php code have no update / change it should the same result for all clients

at least here is the data from test system where all is ok :

Code (CSharp):

1. header key:o2switch PowerBoost=Server
2. header key:Tue, 23 Jul 2019 15:59:44 GMT=Date
3. header key:text/html; charset=utf-8=Content-Type
4. header key:29=Content-Length
5. header key:keep-alive=Connection
6. header key:Accept-Encoding=Vary
7. header key:PHP/5.5.38=X-Powered-By
8. header key:Thu, 19 Nov 1981 08:52:00 GMT=Expires
9. header key:no-store, no-cache, must-revalidate, post-check=0, pre-check=0=Cache-Control
10. header key:no-cache=Pragma
11. header key:PHPSESSID=589c95808e5eebd4d3ab981cd22feb0c; path=/=Set-Cookie
12. Version internet return :ReachableViaLocalAreaNetwork

also note that if using a web brower with an html online POST tool the result received is OK inside the browser of customer getting issue insid steam (not sure how it's parsed in javascript)

 

hello so here is the customer result :

Code (CSharp):

1.  
2. raw data of downloadhandler: IXAABHsidGV4dGUiOiI2LjEiLCJyZXN1bHQiOiJPSyJ9Aw==
3. text received using txt.64deconding: EOT{"texte":"6.1","result":"OK"}ETX
4. header key:Server=o2switch PowerBoost
5. header key:Date=Tue, 23 Jul 2019 20:26:12 GMT
6. header key:Content-Type=text/html; charset=utf-8
7. header key:Transfer-Encoding=chunked
8. header key:Connection=keep-alive
9. header key:Vary=Accept-Encoding
10. header key:X-Powered-By=PHP/5.5.38
11. header key:Expires=Thu, 19 Nov 1981 08:52:00 GMT
12. header key:Cache-Control=no-store, no-cache, must-revalidate, post-check=0, pre-check=0
13. header key:Pragma=no-cache
14. header key:Set-Cookie=PHPSESSID=acb15bca061a53d775b4ec25414f95ce; path=/
15. header key:Content-Encoding=br

so EOT and ETX are the systems chars before and after the text part braking the .data to downloadhander.txt process giving instead only !p
and this only on few computers

on computer woriking fine same .data bytes :
bytes couts 29 instead 32
downloader.data : eyJ0ZXh0ZSI6IjYuMSIsInJlc3VsdCI6Ik9LIn0=
downloader.txt {"texte":"6.1","result":"OK"}
no ETX or EOT

edit : i just notice the chunked transfert on the bugged client and i have

Code (CSharp):

1.        UnityWebRequest www = UnityWebRequest.Post(url, form);
2.         www.downloadHandler = new DownloadHandlerBuffer();
3.         www.certificateHandler = new AcceptAllCertificatesSignedWithASpecificKeyPublicKey();
4.         www.chunkedTransfer = false;

so i disabled all compress methods i could see on httpaccess and instead ETX EOT i get whites stanges signs []

also last header line is content-encoding=br i dont have this on valids test systems

 

Hello Aurimas sadly the end user dont have needed knowledge for such process so i dont know if it will possible.

From google i see Brotli could lead to a double compression issue. I will try to rewrites rules on httpaccess but i still dont understand why the headers are not process / sames with same unity code without headers in the webrequest

 

yes i have an access to server and could do few things but no sessions or telnet and log is quite huge i will see what i could get.

also i started to check the main httaccess and figured out that the compressions methods are not sames between browsers so maybe on the customer side the browser use another compression / default :

Code (CSharp):

1. <IfModule mod_deflate.c>
2.     SetOutputFilter DEFLATE
3.     <IfModule mod_setenvif.c>
4.         # Netscape 4.x has some problems...
5.         BrowserMatch ^Mozilla/4 gzip-only-text/html
6.        
7.         # Netscape 4.06-4.08 have some more problems
8.         BrowserMatch ^Mozilla/4\.0[678] no-gzip
9.        
10.         # MSIE masquerades as Netscape, but it is fine
11.         # BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
12.        
13.         # NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
14.         # the above regex won't work. You can use the following
15.         # workaround to get the desired effect:
16.         BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
17.        
18.         # Don't compress images
19.         SetEnvIfNoCase Request_URI .(?:gif|jpe?g|png)$ no-gzip dont-vary
20.     </IfModule>
21.    
22.     <IfModule mod_headers.c>
23.         # Make sure proxies don't deliver the wrong content
24.         Header append Vary User-Agent env=!dont-vary
25.     </IfModule>
26.  
27. </IfModule>
28. # AddHandler application/x-httpd-php53 .php
29. suPHP_ConfigPath /home/xxxx/public_html/

 

after a real time server log check i found something

this is the line of the computer with issue :

177.157.xxx.xxx - - [24/Jul/2019:22:01:30 +0200] "POST /android_version_check.php?cprotect=1 HTTP/1.1" 200 29 "-" "Mozilla/5.0"

mozilla/5.0 is just user agent i forced today in header but the critical thing is that this is added on my url : cprotect=1

this is the result for my test system where all is ok :

"POST /android_version_check.php HTTP/1.1" 200 29 "-" "Mozilla/5.0"

i see only one other http line with cprotect from some sort of browser outside of unity and not expected maybe a box for referencing or other.and it's not from the customer IP

cprotect make me think about crawl protect (i dont use) or cpanel protect but i dont go far with this

 

edit : After review with the server hosting services ?cprotect=x is from a front web security tool. After update of rules cprotect flage disapear.

So now the main thing i see is brotli compatible header added where it should not be as unity user agent is same for all the clients and i dont have brotli enable.

Also from server log 29 bytes are sent, and in unity 34 are shown.

 

yes i try to figure out what it could be but end user have very low computer level and so it's difficult to get a list of possibles proxy .

As he seems to have them i tried both bytefence and avast but all is fine with them on test system.

 

seems i could reproduce with adding

www.SetRequestHeader("Accept-Encoding", "br");
but seems i could not force on customer system a header to avoid this i tried :

www.SetRequestHeader("Accept-Encoding", "identity");


but customer system still use same encoding :

header key:Content-Encoding=br
linked to
header key:Transfer-Encoding=chunked

 

i solved the issue last thing i tried was 2 things updating a little the httaccess to disable gzip and brotli for the unity agent but i beleive the more effective was adding an header flag in a php script i use in all my php process :

Code (CSharp):

1. <?php
2. header('Content-Encoding: identity', true);

and in the htaccess

Code (CSharp):

1. <IfModule mod_deflate.c>
2.     SetOutputFilter DEFLATE
3.     <IfModule mod_setenvif.c>
4.         # Unity test
5.         BrowserMatch ^UnityPlayer no-gzip no-br
6.         BrowserMatch ^UnityWebRequest no-gzip no-br
7.  
8.         Header append Vary User-Agent env=!dont-vary
9.     </IfModule>
10. </IfModule>