Question Are there any standard/reliable low overhead database read/write solutions?

Currently in performance optimization iterations for my game and one particular issue I've been struggling with solving is GC memory allocations on external database calls.

Are there any standard/reliable low overhead database read/write solutions?

Our current implementation utilizes Amazon.DynamoDBv2 library on our Unity authoritative servers to make UpdateItemAsync() calls with serialized player data to our DynamoDB tables in our cloud environment. In terms of functionality, everything works fine for saving and retrieval, with dynamo having very fast write/read times <2ms internally, but the overhead for the calls from the server->cloud is a bit brutal.



I've highlighted in red one particular call that has a very small payload allocating 87kb~ (a complete character save is roughly 10937 characters, and averages 330kbs~ GC alloc), even with an empty payload the overhead for the save seems to be around 60kb~. As you can imagine this problem becomes an issue when we start adding more players to the mix. (I'm aware there are other allocation issues on the profiler list, just trying to hit the largest ones first)

I've tried:

• pool the async/awaits,
• pool the memory streams that house the byte[] data in AttributeValueUpdates of the AWS request,
  
• pool/pre-allocate anything and everything I can related to the call itself,
  
• consolidate the small saves into a single call to update entire character
• breakup the big saves to have more frequent smaller payloads calls

nothing seems to make a significant impact on the GC allocation and CPU impact. I can't help but think this is a naive implementation, and these issue persists for one of these reasons:

• the library was not meant to handle this kind of environment and does not provide a 0/low allocation implementation (https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/DynamoDBv2/TDynamoDBClient.html)
  
• I'm missing something upstream in the request chain that is causing the library to allocate more than it needs to
• this stack while functional, is not performant enough for games or lacks some standard used in games. Another type of implementation is required
• I don't understand something fundamental about why how the GC is being allocated in this particular instance even with pooling and pre-allocation solutions

Been hammering away at this for about a week now and I'm kinda at my wits end. Any help or advice would be greatly appreciated.

 

An interesting change made today based on reading -> https://stackoverflow.com/questions...ts-async-await-start-literally-another-thread

led me to try wrapping my update call with a Task.Run:

await Task.Run(() => Client.UpdateItemAsync(request));

This resulted in a huge improvement in performance, time went from 40-50ms~ to 10-15ms~. Though the memory GC allocation remains roughly the same. If I am to understand this right it is pushing this into another thread (because its synchronous part implementation is large?), and because I don't need or use the return (FireAndForget) I don't run into issues of having to update something on the original thread.

Even if I can't somehow reduce the GC overhead here in this case, the performance gain processing time wise would in theory allow more time for the incremental GC to process garbage? Maybe utilizing this on things I can't reduce overhead on combined with pooling and preallocating where I can will be enough to hit our CCU target.

Continuing to investigate...

 

Note that anything async is extremely slow in Unity, compared to non-Unity C# projects.
Probably best to just avoid async in Unity. I don't think they are even aware that this is a problem.
You could always try to find a native implementation and call that from Unity.
Or pick a different database with a more simple API that doesn't require async.

 

So something like `UnityWebRequest`? I mean wrapping these calls with an API is not a huge deal, just more work. I was warned however before starting this project that UnityWebRequests had very poor performance. I guess now would be as good a time as ever to try. Other than doing web requests, I'm not all sure what other native solutions I'd have.

 

I think so.
I remember debugging Unity's super slow asset store upload tool.
It used one of the built in HTTP APIs, probably UnityWebRequest.
Which in turn used async sockets internally.
Which is why it was so slow.

This is also why our TCP transport needs 2 threads per connection, which is not great.
We tried all async methods, including SocketAsyncEventArgs, Begin/EndSend, async/await etc.
All of them are super slow in Unity.

Hoping that the transition from mono to netcore would get rid of those issue in the future

 

Async or not does not affect the GC outcome, you're getting that from something else.

Worth noting, that the async keyword is not threaded, but threads are async. To run async, simply put, means to execute code non-blocking / potential out of order.

If you are retrieving database content I would thread not async. This will increase performance by taking the load off the main thread. As mentioned though, your GC is coming from something else so you may want to sort that as well.

 

According to the c# docs,

Task.Run

will run the code on a side thread

https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.task.run?view=netstandard-2.0