What is the most efficient way of managing large amounts of string data?

Hi all,

I would like to do some sort of a "notes" system. So the player finds notes scattered across the map and can collect them and read them and so on.

This is going to be a lot of text/string data and I'm wondering what's the best way to deal with this in regards to performance?

I could just create a script and store all the strings in arrays or maybe store it all on an xml and parse it all in Awake/Start. It would be easier to just be able to store it in an array or list of some sort of course but just wondering what the best way to approach something like this is.

Any advice or links would be awesome!

Thanks,

 

I'd say look into ScriptableObjects, so that way:

a) you can edit them in Unity

b) you can slot them into other prefabs and scenes in your game as you build your content

c) you can write editor scripts to organize and list them en-masse, such as for localization.

ScriptableObjects are basically custom-shaped bags of predefined data that live as files in your project, and you can edit them just like any other Unity item in the Unity inspector window. Check Youtubes for some tutorials... SOs sound perfect for what you're doing.

 

How much text are we talking about? A short story? A novel? The entirety of Wikipedia?

If you have so much text that it doesn't all fit in RAM at the same time, then it's time to discuss tricks for dynamically loading just the portion you currently need and release it when you're done.

If you aren't pushing that threshold, then any performance considerations of how you store it are almost certainly insignificant compared to how convenient it is for your development team to reference and edit.

 

such as this - https://docs.unity3d.com/Manual/class-ScriptableObject.html

that seems like it's just using variables from another class on another script.

I'll look more into it. thank you.

 

Good question.

Let's say the "notes" are a paragraph or two (maybe 5-20 lines) each and I have 150 -200 of these "notes". I'm not sure what this will be like for RAM.

 

C# strings use 2 bytes per character (UTF-16). Let's guess you have perhaps 100 characters per line, times 20 lines per note, times 200 notes = 400,000 characters ~= 0.8 MB.

That's insignificant on modern hardware.

 

I would ask who is writing these? Like what kind of interface does the writer need? For example,if it was a C# developer, I might be lazy and just write them directly inside the script that needs them, and throw them in a region to collapse it out of the way. If I was someone non-technical, I might set it up so it can be edited in maybe a spreadsheet then save as CSV with a non-standard deliminator (I usually use the "|" character).

Code (csharp):

1. private List<string> notes = new List<string>();
2.  
3. #region All them notes
4. private void initializeNotes()
5. {
6.     notes.Add("Which taco sauce is better, the red or the green?");
7.     notes.Add("Don't forget to do the laundry");
8.     notes.Add("Is Pluto a planet?  Of course not silly.  Pluto is a dog, and dogs cannot be planets.");
9.     notes.Add("So all I need for faster than light travel is negative energy?  I have that every morning already.");
10.     notes.Add("Which runs hotter, Brad Pitt or the RTX 3090?");
11.     notes.Add("Are the people of York embarrassed that New York is more famous?  What happened York?");
12. }
13. #endregion
14.  

 

What Uncle Joe asks above is a good point. You might find it better to work in something like google sheets, typing it in as a spreadsheet, then export as CSV and import into your game. All of that is easy to do, but I would definitely start with the easiest way first, as Joe outlined above, and build up from there.

 

So it seems like it's not as big of an issue as i thought.

Im thinking what ill do is store it all in a class in a separate script to keep it simple and tidy then call from it when i need to.

im surprised that what i consider a large amount of string data is actually not that large after all

thank you everyone

 

As a rule of thumb, text assets are generally very small compared to graphics or sound. Very unusual for their size to be any kind of issue.

 

Fair enough. I've been reading that where possible one should avoid the use of strings as it is performance heavy and so i thought I would do a bit of research before starting anything. I'm assuming that this is more to do with concatenating strings and so on rather than just storing them in a variable or array and calling it when required?

 

One notable thing about strings in C# is that they are immutable: once you have created a string object, its text can never be changed. Anything that looks like it is modifying a string is actually creating and returning a new string object--which involves a memory allocation. If you create a lot of strings you don't need and quickly throw them away, you're creating work for the garbage collector.

This is why e.g. Unity advises you not to create strings every frame; if you're displaying some variable on the screen as text, then instead of just blindly converting it every frame, it's usually a good idea to check whether the value has actually changed since the last frame, and reuse the same string object if not.

Also, a string variable could be considered expensive in comparison to, say, a float or an int. You wouldn't want to just replace numeric variables with string variables for no reason. Especially not in some class that you're going to instantiate a million copies of, because any size becomes big when you multiply it by a sufficiently big number.