Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

Question Are there any standard/reliable low overhead database read/write solutions?

Discussion in 'Multiplayer' started by BagoDev, Sep 25, 2022.

  1. BagoDev

    BagoDev

    Joined:
    Jan 2, 2017
    Posts:
    21
    Currently in performance optimization iterations for my game and one particular issue I've been struggling with solving is GC memory allocations on external database calls.

    Are there any standard/reliable low overhead database read/write solutions?

    Our current implementation utilizes Amazon.DynamoDBv2 library on our Unity authoritative servers to make UpdateItemAsync() calls with serialized player data to our DynamoDB tables in our cloud environment. In terms of functionality, everything works fine for saving and retrieval, with dynamo having very fast write/read times <2ms internally, but the overhead for the calls from the server->cloud is a bit brutal.



    I've highlighted in red one particular call that has a very small payload allocating 87kb~ (a complete character save is roughly 10937 characters, and averages 330kbs~ GC alloc), even with an empty payload the overhead for the save seems to be around 60kb~. As you can imagine this problem becomes an issue when we start adding more players to the mix. (I'm aware there are other allocation issues on the profiler list, just trying to hit the largest ones first)

    I've tried:
    • pool the async/awaits,
    • pool the memory streams that house the byte[] data in AttributeValueUpdates of the AWS request,
    • pool/pre-allocate anything and everything I can related to the call itself,
    • consolidate the small saves into a single call to update entire character
    • breakup the big saves to have more frequent smaller payloads calls
    nothing seems to make a significant impact on the GC allocation and CPU impact. I can't help but think this is a naive implementation, and these issue persists for one of these reasons:
    • the library was not meant to handle this kind of environment and does not provide a 0/low allocation implementation (https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/DynamoDBv2/TDynamoDBClient.html)
    • I'm missing something upstream in the request chain that is causing the library to allocate more than it needs to
    • this stack while functional, is not performant enough for games or lacks some standard used in games. Another type of implementation is required
    • I don't understand something fundamental about why how the GC is being allocated in this particular instance even with pooling and pre-allocation solutions
    Been hammering away at this for about a week now and I'm kinda at my wits end. Any help or advice would be greatly appreciated.
     
  2. BagoDev

    BagoDev

    Joined:
    Jan 2, 2017
    Posts:
    21
    An interesting change made today based on reading -> https://stackoverflow.com/questions...ts-async-await-start-literally-another-thread

    led me to try wrapping my update call with a Task.Run:

    await Task.Run(() => Client.UpdateItemAsync(request));


    This resulted in a huge improvement in performance, time went from 40-50ms~ to 10-15ms~. Though the memory GC allocation remains roughly the same. If I am to understand this right it is pushing this into another thread (because its synchronous part implementation is large?), and because I don't need or use the return (FireAndForget) I don't run into issues of having to update something on the original thread.

    Even if I can't somehow reduce the GC overhead here in this case, the performance gain processing time wise would in theory allow more time for the incremental GC to process garbage? Maybe utilizing this on things I can't reduce overhead on combined with pooling and preallocating where I can will be enough to hit our CCU target.

    Continuing to investigate...
     
  3. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,275
    Note that anything async is extremely slow in Unity, compared to non-Unity C# projects.
    Probably best to just avoid async in Unity. I don't think they are even aware that this is a problem.
    You could always try to find a native implementation and call that from Unity.
    Or pick a different database with a more simple API that doesn't require async.
     
  4. BagoDev

    BagoDev

    Joined:
    Jan 2, 2017
    Posts:
    21
    So something like `UnityWebRequest`? I mean wrapping these calls with an API is not a huge deal, just more work. I was warned however before starting this project that UnityWebRequests had very poor performance. I guess now would be as good a time as ever to try. Other than doing web requests, I'm not all sure what other native solutions I'd have.
     
  5. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,275
    I think so.
    I remember debugging Unity's super slow asset store upload tool.
    It used one of the built in HTTP APIs, probably UnityWebRequest.
    Which in turn used async sockets internally.
    Which is why it was so slow.

    This is also why our TCP transport needs 2 threads per connection, which is not great.
    We tried all async methods, including SocketAsyncEventArgs, Begin/EndSend, async/await etc.
    All of them are super slow in Unity.

    Hoping that the transition from mono to netcore would get rid of those issue in the future :)
     
    BagoDev likes this.
  6. Punfish

    Punfish

    Joined:
    Dec 7, 2014
    Posts:
    280
    Async or not does not affect the GC outcome, you're getting that from something else.

    Worth noting, that the async keyword is not threaded, but threads are async. To run async, simply put, means to execute code non-blocking / potential out of order.

    If you are retrieving database content I would thread not async. This will increase performance by taking the load off the main thread. As mentioned though, your GC is coming from something else so you may want to sort that as well.
     
  7. JamesFrowenDev

    JamesFrowenDev

    Joined:
    Oct 10, 2015
    Posts:
    18