Search Unity

  1. All Pro and Enterprise subscribers: find helpful & inspiring creative, tech, and business know-how in the new Unity Success Hub. Sign in to stay up to date.
    Dismiss Notice
  2. Dismiss Notice

Best HTTP Released

Discussion in 'Assets and Asset Store' started by BestHTTP, Sep 11, 2013.

  1. lvvova

    lvvova

    Joined:
    Dec 13, 2018
    Posts:
    1
    @BestHTTP

    Hello.
    I got problem with freeing memory:

    When i download file i can't unload it from my memory.

    Already tryed disable cookies, cache.

    Define Symbols:
    BESTHTTP_DISABLE_ALTERNATE_SSL;BESTHTTP_DISABLE_COOKIES;BESTHTTP_DISABLE_CACHING

    Here code:
    https://pastebin.com/sVTrqyTs



    I think I'm making smth wrong but I tried almost everything.
     
  2. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
    @yathish9158

    With what protocol are you using it? Websocket itself has no reconnect logic.

    You can set the plugin's log level to All and send the produced log to so i can see what's happening. However, if there's poor network conditions it's not guaranteed that the plugin can reconnect to the server if it could previously.
     
  3. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
    @Ivvova

    1.) There's no need to dispose the request and response manually, the plugin going to do it automatically after the callback.
    2.) The plugin doesn't use Resources, so Resources.UnloadUnusedAssets(); has no effect.
    3.) Cache and cookies doesn't use much memory, so you wouldn't gain too much disabling them either.
    4.) You created a non-streaming request, the response Data is a byte[] containing the whole response. In this case, it contains 100 Mb of data.
    5.) You are releasing the Data back to the plugin's memory pool. So even if you call GC.Collect a few lines down, you just added one another reference to that 100Mb of data making it impossible to the GC to reclaim.

    So a few suggestions:

    1.) If you don't want to download large contents, you can use streaming.
    2.) You can disable MemoryPooling, but in the end you might end up with more garbage as the plugin can't reuse already allocated large chunks of memory. So instead of disabling it, i would suggest to lower the RemoveOlderThan's value.
     
  4. ArseniyMaryin

    ArseniyMaryin

    Joined:
    Nov 2, 2018
    Posts:
    9
    Hi,
    I have performed a speed test - downloaded file with size 248182482 bytes via BestHttp.
    On Huawei pad this file was downloaded in 66 sec.

    UseStreaming = true, streamFragmentSize = 64 * 1024

    UnityWebRequest can download the same file on the same device in 28 sec.

    So I tried to understand, why BestHttp is 2 times slower, than UWR.

    In HTTPResponse class, in function ReadRaw, I measured
    1. "Pure" time it takes for a socket to send bytes (Span1)
    2. The time WaitWhileHasFragments function waits (Span2)

    Stopwatch stopwatch1 = new Stopwatch();
    Stopwatch stopwatch2 = new Stopwatch();
    while (contentLength > 0)
    {
    readBytes = 0;
    do
    {
    int readbuffer = (int)Math.Min(2147483646, (uint)contentLength);
    stopwatch1.Start();
    int bytes = stream.Read(buffer, readBytes, Math.Min(readbuffer, buffer.Length - readBytes));
    stopwatch1.Stop();
    if (bytes <= 0)
    throw ExceptionHelper.ServerClosedTCPStream();
    readBytes += bytes;
    contentLength -= bytes;
    // Progress report:
    baseRequest.Downloaded += bytes;
    baseRequest.DownloadProgressChanged = this.IsSuccess
    #if !BESTHTTP_DISABLE_CACHING && (!UNITY_WEBGL || UNITY_EDITOR)
    || this.IsFromCache
    #endif
    ;
    } while (readBytes < buffer.Length && contentLength > 0);
    if (baseRequest.UseStreaming)
    {
    // If reading from cache, we don't want to read too much data to memory. So we will wait until the loaded fragment processed.
    stopwatch2.Start();
    WaitWhileHasFragments();
    stopwatch2.Stop();
    if (gzipped)
    {
    var decompressed = Decompress(buffer, 0, readBytes);
    FeedStreamFragment(decompressed, 0, decompressed.Length);
    }
    else
    FeedStreamFragment(buffer, 0, readBytes);
    }
    else
    output.Write(buffer, 0, readBytes);
    };
    Debug.Log("Span1 = " + stopwatch1.Elapsed.TotalSeconds);
    Debug.Log("Span2 = " + stopwatch2.Elapsed.TotalSeconds);



    Results:

    2021-05-11 17:54:00.417 29568-29824/com.my.AddressablesTest I/Unity: Span1 = 42.2962665
    2021-05-11 17:54:00.417 29568-29824/com.my.AddressablesTest I/Unity: Span2 = 22.8999408

    It seemed suspiciously for me, that WaitWhileHasFragments is wasting 23 sec. But there is a comment:
    // If reading from cache, we don't want to read too much data to memory. So we will wait until the loaded fragment processed.

    WaitWhileHasFragments fuctions looks like this:

        void WaitWhileHasFragments()
    {
    #if !UNITY_WEBGL || UNITY_EDITOR
    while (baseRequest.UseStreaming &&
    #if !BESTHTTP_DISABLE_CACHING
    //this.IsFromCache &&
    #endif
    HasStreamedFragments())
    {
    #if NETFX_CORE
    await System.Threading.Tasks.Task.Delay(16);
    #else
    System.Threading.Thread.Sleep(16);
    #endif
    }
    #endif
    }


    Why //this.IsFromCache is commented out? If we are not reading from cache, we don't have to wait.
    So I changed this function like so:

        void WaitWhileHasFragments()
    {
    #if !BESTHTTP_DISABLE_CACHING && (!UNITY_WEBGL || UNITY_EDITOR)
    while (baseRequest.UseStreaming && this.IsFromCache && HasStreamedFragments())
    {
    #if NETFX_CORE
    await System.Threading.Tasks.Task.Delay(16);
    #else
    System.Threading.Thread.Sleep(16);
    #endif
    }
    #endif
    }


    And test file is downloading now in 42 sec.
    BestHttp version is 1.10.3

    What can you say about this?
     
  5. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
    @ArseniyMaryin

    That you should try out Best HTTP/2. I made a lot of improvements to speed up upload and download speed.
     
  6. deiva

    deiva

    Joined:
    Jul 12, 2016
    Posts:
    15
    I have using Best http 2.4.0 Latest plugin for Socket IO3 code

    While using socket connection most of time I'm getting this error and also after socket IO3 joining the room unable to receive a event from server to unity.

    {"tid":1,"div":"WebSocketTransport","msg":"OnBinary Packet parsing","ex": [{"msg": "Unexpected token readed 'BeginArray' while 'BeginObject' is expected.", "stack": " at GameDevWare.Serialization.Serializers.ObjectSerializer.Deserialize (GameDevWare.Serialization.IJsonReader reader) [0x00026] in D:\\Pictures\\mopub testing\\Assets\\Plugins\\GameDevWare.Serialization\\Serializers\\ObjectSerializer.cs:69 \r\n at GameDevWare.Serialization.JsonReaderExtentions.ReadValue (GameDevWare.Serialization.IJsonReader reader, System.Type valueType, System.Boolean nextToken) [0x00096] in D:\\Pictures\\mopub testing\\Assets\\Plugins\\GameDevWare.Serialization\\JsonReaderExtentions.cs:727 \r\n at BestHTTP.SocketIO3.Parsers.MsgPackParser.ReadParameters (BestHTTP.SocketIO3.Socket socket, BestHTTP.SocketIO3.Events.Subscription subscription, GameDevWare.Serialization.IJsonReader reader) [0x00098] in D:\\Pictures\\mopub testing\\Assets\\Best HTTP\\Examples\\SocketIO3\\Parsers\\MsgPackParser.cs:240 \r\n at BestHTTP.SocketIO3.Parsers.MsgPackParser.ReadData (BestHTTP.SocketIO3.SocketManager manager, BestHTTP.SocketIO3.IncomingPacket packet, GameDevWare.Serialization.IJsonReader reader) [0x000ee] in D:\\Pictures\\mopub testing\\Assets\\Best HTTP\\Examples\\SocketIO3\\Parsers\\MsgPackParser.cs:304 \r\n at BestHTTP.SocketIO3.Parsers.MsgPackParser.Parse (BestHTTP.SocketIO3.SocketManager manager, BestHTTP.PlatformSupport.Memory.BufferSegment data, BestHTTP.SocketIO3.TransportEventTypes transportEvent) [0x0010e] in D:\\Pictures\\mopub testing\\Assets\\Best HTTP\\Examples\\SocketIO3\\Parsers\\MsgPackParser.cs:146 \r\n at BestHTTP.SocketIO3.Transports.WebSocketTransport.OnBinary (BestHTTP.WebSocket.WebSocket ws, System.Byte[] data) [0x00053] in D:\\Pictures\\mopub testing\\Assets\\Best HTTP\\Source\\SocketIO.3\\Transports\\WebSocketTransport.cs:172 "}],"stack":" at SocketIO3.Transports.WebSocketTransport.OnBinary (WebSocket.WebSocket ws, System.Byte[] data) [0x0007a] in D:\\Pictures\\mopub testing\\Assets\\Best HTTP\\Source\\SocketIO.3\\Transports\\WebSocketTransport.cs:176 \r at WebSocket.WebSocket.<OnInternalRequestUpgraded>b__47_1 (WebSocket.WebSocketResponse ws, System.Byte[] bin) [0x0000e] in D:\\Pictures\\mopub testing\\Assets\\Best HTTP\\Source\\WebSocket\\WebSocket.cs:439 \r at WebSocket.WebSocketResponse.Core.IProtocol.HandleEvents () [0x000f7] in D:\\Pictures\\mopub testing\\Assets\\Best HTTP\\Source\\WebSocket\\WebSocketResponse.cs:540 \r at Core.ProtocolEventHelper.ProcessQueue () [0x00087] in D:\\Pictures\\mopub testing\\Assets\\Best HTTP\\Source\\Core\\ProtocolEvents.cs:69 \r at HTTPManager.OnUpdate () [0x0000d] in D:\\Pictures\\mopub testing\\Assets\\Best HTTP\\Source\\HTTPManager.cs:416 \r at HTTPUpdateDelegator.Update () [0x00020] in D:\\Pictures\\mopub testing\\Assets\\Best HTTP\\Source\\HTTPUpdateDelegator.cs:171 ","ctxs":[{"TypeName": "SocketManager", "Hash": -392288000}],"t":637568698150178555,"ll":"Exception","bh":1}
    UnityEngine.Debug:LogError(Object)
    BestHTTP.Logger.UnityOutput:Write(Loglevels, String) (at Assets/Best HTTP/Source/Logger/UnityOutput.cs:22)
    BestHTTP.Logger.ThreadedLogger:ThreadFunc() (at Assets/Best HTTP/Source/Logger/ThreadedLogger.cs:130)
    BestHTTP.PlatformSupport.Threading.<>c__DisplayClass5_0:<RunLongLiving>b__0() (at Assets/Best HTTP/Source/PlatformSupport/Threading/ThreadedRunner.cs:94)
    System.Threading.ThreadHelper:ThreadStart(Object)
     
  7. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
  8. deiva

    deiva

    Joined:
    Jul 12, 2016
    Posts:
    15
    Yes. We using message MsgPackParser
     
    Last edited: May 18, 2021
  9. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
    @deiva

    Could you send a repro project (client + server)? Or at least related code fragments from the client and server.
     
  10. deiva

    deiva

    Joined:
    Jul 12, 2016
    Posts:
    15
    @BestHTTP

    I have attached unity and server code for Socket.
     

    Attached Files:

    Last edited: May 18, 2021
  11. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
    @deiva

    Could you try out with v2.5.0 that released yesterday?
     
  12. deiva

    deiva

    Joined:
    Jul 12, 2016
    Posts:
    15
    Currently we using V2.5.0. Still we facing the issue.
    public static string UserAgent = "BestHTTP/2 v2.5.0";
     
  13. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
    @deiva

    In this case could you narrow down what's message causing problems? I just tried out and it's working as expected.

    This was my server:
    Code (JavaScript):
    1. var app = require('http').createServer();
    2.  
    3. const customParser = require('socket.io-msgpack-parser');
    4.  
    5. var io = require('socket.io')(app, {
    6.     parser: customParser
    7. });
    8.  
    9. io.listen(3000);
    10.  
    11. io.of("bt_multiplayer").on("connection", (socket) => {
    12.     socket.on("get_lobbies", () => {
    13.         const lobbies = {
    14.             lobbies: [
    15.                 {
    16.                     id: "001",
    17.                     is_enabled: true,
    18.                     maintenance: false,
    19.                     room_type: "custom",
    20.                     overs: 123,
    21.                     balls_per_over: 3,
    22.                     max_players: 100,
    23.                     min_players: 20,
    24.                     waiting_time: 4,
    25.                     bot_players: false
    26.                 },
    27.  
    28.                 {
    29.                     id: "002",
    30.                     is_enabled: false,
    31.                     maintenance: true,
    32.                     room_type: "roomie",
    33.                     overs: 321,
    34.                     balls_per_over: 6,
    35.                     max_players: 64,
    36.                     min_players: 8,
    37.                     waiting_time: 2,
    38.                     bot_players: true
    39.                 }
    40.             ]
    41.         };
    42.  
    43.         socket.emit("lobbies_list", lobbies);
    44.     });
    45.  
    46.     let _countdown = 5;
    47.     let wait_time = setInterval(() => {
    48.         if (_countdown <= 0) {
    49.             clearInterval(wait_time);
    50.         } else {
    51.             socket.emit("wait_time_countdown", { countdown: _countdown });
    52.             _countdown--;
    53.             console.log("Countdown: ", _countdown);
    54.         }
    55.     }, 1000);
    56.  
    57.     socket.on("disconnect", () => {
    58.         clearInterval(wait_time);
    59.     });
    60. });
    And client:
    Code (CSharp):
    1. Socket nsp;
    2.  
    3. public override void Do_Test()
    4. {
    5.     SocketOptions options = new SocketOptions();
    6.     options.AutoConnect = false;
    7.     options.Reconnection = true;
    8.     options.Timeout = TimeSpan.FromSeconds(10.0f);
    9.     options.QueryParamsOnlyForHandshake = false;
    10.     ObservableDictionary<string, string> query = new ObservableDictionary<string, string>();
    11.  
    12.     query["token"] = "QWER-REWQ-ASDF-FDSA";
    13.     options.AdditionalQueryParams = query;
    14.  
    15.     var Manager = new SocketManager(new Uri("http://localhost:3000"), options);
    16.     Manager.Parser = new MsgPackParser();
    17.     Manager.Open();
    18.  
    19.     nsp = Manager.GetSocket("/bt_multiplayer");
    20.     nsp.On<ConnectResponse>("connect", OnConnected);
    21.     nsp.On("disconnect", OnDisconnected);
    22.     nsp.On<Lobbies>("lobbies_list", OnLobbiesList);
    23.     nsp.On<WaitTimeCountdown>("wait_time_countdown", OnWaitTimeCountdown);
    24. }
    25.  
    26. private void OnWaitTimeCountdown(WaitTimeCountdown obj)
    27. {
    28.     Debug.Log($"OnWaitTimeCountdown: {obj.countdown}");
    29. }
    30.  
    31. private void OnLobbiesList(Lobbies l)
    32. {
    33.     Debug.Log($"LobbiesList: {l?.lobbies?.Length}");
    34.  
    35.     foreach (Lobby lob in l.lobbies)
    36.     {
    37.         Debug.LogError("Lobbies " + lob.id);
    38.  
    39.         nsp.Emit("find_room", l.lobbies[1].id);
    40.     }
    41. }
    42.  
    43. private void OnDisconnected()
    44. {
    45.     Debug.Log("Disconnected!");
    46. }
    47.  
    48. private void OnConnected(ConnectResponse resp)
    49. {
    50.     Debug.Log($"Connected with sid: {resp.sid}");
    51.  
    52.     nsp.Emit("get_lobbies");
    53. }
    upload_2021-5-18_14-4-47.png
     
  14. deiva

    deiva

    Joined:
    Jul 12, 2016
    Posts:
    15
    Screenshot from 2021-05-18 18-09-41.png error.JPG The only issue is we are unable to get the room events. Kindly check server code.


    Client Code:

    Code (CSharp):
    1.  
    2.  
    3. void Start(){
    4. nsp = Manager.GetSocket("/bt_multiplayer");
    5. nsp.On<ConnectResponse>("connect", OnConnected);
    6.  
    7. nsp.On("direct_event", () => {
    8. Debug.LogError("Event triggered Directly");
    9. });
    10.  
    11. nsp.On("room_event", () => {                    
    12. Debug.LogError("Event triggered from room");
    13. });
    14. }
    15.  
    16. private void OnConnected(ConnectResponse resp)
    17. {
    18. Debug.LogError("Connected");
    19. nsp.Emit("join_room");
    20. }
    Server:

    Code (JavaScript):
    1. var app = require("http").createServer();
    2.  
    3. const customParser = require("socket.io-msgpack-parser");
    4.  
    5. var io = require("socket.io")(app, {
    6.   parser: customParser,
    7. });
    8.  
    9. io.listen(3000);
    10.  
    11. io.of("bt_multiplayer").on("connection", (socket) => {
    12.   console.log("Connected");
    13.   socket.on("join_room", () => {
    14.     console.log("Join Room");
    15.     socket.emit("direct_event");
    16.  
    17.     socket.join("ROOM_12345");
    18.     io.to("ROOM_12345").emit("room_event");
    19.   });
    20. });
     
    Last edited: May 18, 2021
  15. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
    @deiva

    The client connects, sends the "join_room" event, receives "direct_event", but the server doesn't sent the "room_event":
    upload_2021-5-18_15-22-52.png
     
  16. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
    Used DEBUG=* environment variable for the server log
     
  17. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
    @deiva What's strange is though that the server (socket.io v4.1.2 on my end) sends binary and text websocket messages, however data sent msgpack encoded should sent binary only.
    Your errors are orignate from binary messages to make it more confusing...
     
  18. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
    @deiva I might found one or two issues in socket.io, going to continue investigate and if i'm right report them.
     
  19. deiva

    deiva

    Joined:
    Jul 12, 2016
    Posts:
    15
  20. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
    @deiva Added some details to the issue.
     
  21. wicea

    wicea

    Joined:
    Jan 23, 2016
    Posts:
    16
    @BestHTTP
    Hello. I have a question related to the issue that I created - https://github.com/Benedicht/BestHTTP-Issues/issues/60

    I'm using HTTPRequest.ProcessingStarted to determine request execution time and correct my local time to the server time.

    Code (CSharp):
    1. var req = new HTTPRequest(uri, HTTPMethods.Get);
    2. req.Callback += (request, response) =>
    3. {
    4.     TimeSpan requestExecutionTime = DateTime.UtcNow - request.ProcessingStarted;
    5. };
    6. req.Send();
    I read about Timing API and now a bit confused about my implementation. Is it more accurate and correct to retrieve request execution time, like this way? Assuming that I'm not using caching.

    Code (CSharp):
    1. var req = new HTTPRequest(uri, HTTPMethods.Get);
    2. req.Callback += (request, response) =>
    3. {
    4.     var firstTiming = request.Timing.FindFirst(TimingEventNames.Request_Sent);
    5.     var lastTiming = request.Timing.FindLast(TimingEventNames.Response_Received);
    6.  
    7.     TimeSpan requestExecutionTime = lastTiming.When - firstTiming.When;
    8. };
    9. req.Send();
     
  22. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
    @wicea I would recommend to use the Timing API, it can give a more realistic view about the timing as it going to include redirects too.
     
  23. umityayla

    umityayla

    Joined:
    May 21, 2019
    Posts:
    24
    HTTPUpdateDelegator.Update takes quite some time, any idea how can we get around this?
     

    Attached Files:

  24. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
    @umityayla

    Those high times are most probably from the callbacks added for HTTPRequests. HTTPUpdateDelegator.Update is where the callbacks are called from.

    Mono.JIT: Contains samples that relate to just-in-time compilation of a scripting method. When a function is executed for the first time, Mono compiles it and Mono.JIT represents this compilation overhead.
    (https://docs.unity3d.com/Manual/profiler-markers.html)
     
  25. ArseniyMaryin

    ArseniyMaryin

    Joined:
    Nov 2, 2018
    Posts:
    9
    Hi, some remarks about the latest BestHttp 2.5.0:

    1. Debug stack trace collection in BufferPool.cs needs to be switched off. BestHttp is extremely slow in Editor otherwise.
    2. It seems that SendBufferSize and ReceiveBufferSize are not optimal at least for one device - Huawei MediaPad M3.

    On my Internet, BestHttp 2.5.0 can download a 250 MB file in 45 seconds, but if I comment these strings:

    Client.SendBufferSize = HTTPManager.SendBufferSize;
    Client.ReceiveBufferSize = HTTPManager.ReceiveBufferSize;

    the same file can be downloaded in 28 seconds.
     
  26. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
    @ArseniyMaryin

    1.) It's fixed in the next release i plan to do this week.
    2.) I'm going to try to investigate it, thanks.
     
  27. IOU_RAY

    IOU_RAY

    Joined:
    Jul 25, 2017
    Posts:
    76
    Just wanted to chime in and say that I've been using BestHTTP for years now for websockets to custom built servers hosted through mono on linux via AWS EC2s.

    My infrastructure was complicated migrating away from back-end services to custom and the headaches behind it were nasty, but being able to facilitate quick integration with websockets (instead of regular tcp sockets through flash back in the day) was a godsend.

    At any given moment there's thousands of players live on the game with stable connections and I have no instabilities caused by using BestHTTP that I'm aware of.

    Unity has been a nightmare with its engine bugs, and every other service/library has been underwhelming except this one from my experience.

    So in any case, I just wanted to say...thanks! Best asset on the store by far, and I only use the bare minimum basics of it. I can't believe how awesome and far the features for it stretch.
     
    BestHTTP likes this.
  28. BestHTTP

    BestHTTP

    Joined:
    Sep 11, 2013
    Posts:
    1,590
    Thanks @IOU_RAY, i'm really happy that you have a great experience with the plugin! Can ask you o post this as a review in the asset store too? That would be really helpful, i'm sure the plugin's review page is a better place for those who want to decide to by the plugin. Anyhow, i'm glad you wrote it down and last but not least, what's the game? I'm really enjoying trying out games using the plugin. :)
     
  29. IOU_RAY

    IOU_RAY

    Joined:
    Jul 25, 2017
    Posts:
    76
    Hehe, I did 4 years ago already! Just wanted to bombard further with compliments this much time after :)

    upload_2021-6-18_17-9-33.png

    Cheers.
     
    BestHTTP likes this.
unityunity