Question Should I use Protobuf with Unity? If so, how?

Hello, everyone.

I'm developing an app targeting Android and iOS, and I'm looking for alternatives on data serialization.

I know how to use basic binary serialization, but through my searches I've found that it's considerably less optimal than other libraries, such as Google Protobuf.

Is this true? Is it also true for using with Unity?

If so, how can I use Protobuf with Unity? The advice I seek is mainly "how to begin using protobuf". I've downloaded the NuGet packages Google.Protobuf and Google.Protobuf.Tools, but I have no idea where to go from there. I've checked the tutorials (https://developers.google.com/protocol-buffers/docs/csharptutorial, https://developers.google.com/protocol-buffers/docs/encoding, https://developers.google.com/protocol-buffers/docs/reference/csharp, https://developers.google.com/protocol-buffers/docs/reference/csharp-generated, https://developers.google.com/protocol-buffers/docs/proto3 ) but none seem to indicate how exactly to actually begin using the buffer.

I have been developing in Unity for some time, but I've never developed outside it, neither had formal education on programming, so I'm not very comfortable using the scripting tools or raw programming languages outside Unity.

 

What exactly does JSON serialization not get you? And do you know that this "protobuf" thing will get it?

If you have identified that Protobuf will get you what you need, now you need to identify if it CAN get you what you need in Unity3D. Since I've never heard of it or heard of it posted here, I'm going to hazard a guess that most Unity engineers have never touched it.

That kinda means it's up to you to do the legwork and engineering to qualify this third-party library.

The requirements for integration with Unity are quite well documented, but if you have specific issues, definitely hit google because if that library is even remotely usable with Unity, you will NEVER be the first guy to have specific errors.

 

Protobufs is a set of tools made for the purpose of serializing and communicating data between applications, usually in a client/server configuration. The steps to use protobufs are pretty much:

• Define your service and data models in a .proto file
• Use the protobufs compiler to generate code for your client and server separately. They will both use the same data model, but the server side will have hooks for a web application framework to integrate with, and the client side will have hooks for a
• Your server needs a web application framework with a protobufs integration to properly use the compiled results to serve requests. The most common one with the best integration is gRPC, but you can make your own for your favorite web application framework over HTTP for example. Here's an article about people using it with Spring MVC: https://golb.hplar.ch/2017/01/Protocol-Buffers-with-Spring-Integration.html#:~:text=Protocol Buffers is not limited to Java.&text=Before applications can send and,from the binary wire format.
• You will likewise need a client framework on the client side that can take the serialized bytes from protobufs and pass them to your server. If you use gRPC for your server, you would use the gRPC client for your client. Otherwise, use whatever is compatible with your server.

The benefit of this over JSON for example is that it provides a centralized data model for both your client and your server, and it has a binary Over-The-Wire protocol which is much smaller than JSON.

The disadvantage is that it's a lot more complicated to set up, and you also lose the ability to easily look at the over-the-wire data and manually inspect it.

If none of the above makes any sense to you, protobufs is probably not for you.

Let's step back for a minute. What's your use case for serializing data?

 

Thanks for the insights, guys.

Those things make "half sense" to me. I know somewhat what they mean, but not exactly.

My use for data serialization is that the app should be able to write and read data across platforms - the initial app will be Android and iOS, but I plan to release a Windows and MAC version later on.

The user save files should be synchronized with a server, and this server should keep this synchronization across different platforms, so if the user makes a change in something using their iOS, the change should be reflected on their Windows too, for example.

Another point is that the app will have a continued development, with multiple versions coming in after release, so the data files must be very compatible with modifications on the client version.

The data itself is not large, mainly a few dozen Lists of integers and strings, but since the target is Mobile I'm concerned with performance. The saving process should be seamless.

So, in short, what I need is:

• Fast serialization/deserialization for Android and iOS;
• Small-medium data files;
• Readability accross Android, iOS, MAC and Windows;
• Compatibility with new versions;

Does JSON (JsonUtility package?) provide me this, or should I indeed be looking for alternatives?

 

It certainly can do all this, but how you implement future upgrades will determine whether the last point is true.

JSON is great because it comes out as text and that is CRUCIAL to development and finding your bugs.

And if want it smaller, then compress it with something before writing it to disk.

 

I'd use a text based approach, such as JSON as already discussed, during development. It will be easier and faster to implement and debug. Then after you're in a stable state with all the related systems which use this, I'd look at the actual results, including data size and performance. Compare those results to your criteria. If acceptable, I'd probably just leave it be. Don't fix what isn't broken.

If you're not happy with the results, I'd probably switch to converting everything to binary using BitConverter for everything except strings, and System.Text.Encoding.UTF8.GetBytes for the strings. You'll likely find it to be the most size and performance efficient approach, though more difficult to maintain and debug than any other alternative. I believe all your target platforms default to little endian, so I wouldn't expect any compatibility issues across those platforms, nor would they likely change between Unity versions (since Unity didn't even write them).

 

Thanks for all the great tips, guys!

I'm going with this approach, sounds good and reasonable.
Thanks a lot