Search Unity

  1. Unity 2020.1 has been released.
    Dismiss Notice
  2. We are looking for feedback on the experimental Unity Safe Mode which is aiming to help you resolve compilation errors faster during project startup.
    Dismiss Notice
  3. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Alternative to using string

Discussion in 'Data Oriented Technology Stack' started by Soaryn, Mar 23, 2018.

  1. Soaryn

    Soaryn

    Joined:
    Apr 17, 2015
    Posts:
    265
    If I have an entity that I'd want to hold a string or two, what would be the best way to go about doing that in the Entity Component System?

    I am aware strings and char are not blittable, but was curious if there were any soon to be replacements of these types for entities to hold as data.
     
    SubPixelPerfect likes this.
  2. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,044
    You should be able to do this if you want to hold a string that's a fixed size:

    Code (CSharp):
    1.   public unsafe struct EnemyShootState : IComponentData {
    2.     public float Cooldown;
    3.     public fixed char StringValue[20];
    4.   }
    That requires you to convert the String to a Char[] array and copy it over by hand. It also requires an unsafe context to access it, and can at most hold X characters (20 in this case, as defined after the StringValue name.

    Char is blittable in the runtime itself, I dont know if there's any special restrictions around the Unity ECS though.
     
  3. Soaryn

    Soaryn

    Joined:
    Apr 17, 2015
    Posts:
    265
  4. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,044
  5. RootCauseProductions

    RootCauseProductions

    Joined:
    Feb 1, 2018
    Posts:
    31
    According to MSDN, System.Char is non-blittable. Unicode characters are 1-4 bytes long so this causes problems. Also, arrays are not blittable so byte[] won't work. There is a code sample in this post that uses NativeArray<byte> to hold encoded strings. Not a pretty thing, but it works. Ideally, you wouldn't do string processing in your job due to the overhead that string garbage collection will create but it works if you need it.
     
  6. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,049
    Right now best answer is to keep it on monobehaviour. We are working on enabling this but right now it’s the best answer we got
     
  7. andywatts

    andywatts

    Joined:
    Sep 19, 2015
    Posts:
    89
    Can a ComponentSystem access a monobehaviour?

    The entityManager sees the mono thanks to GameObjectEntity.
    GetComponentData fails because it's not an IComponentData.


    In both hybrid and pure, I'm struggling to find a home for non-blittable data like strings and >14k arrays.
    It's instance data, so icomponentshared feels wrong.
    A mono is BAU, but I want access from systems.

    Am I missing something?
     
  8. Kender

    Kender

    Joined:
    Nov 16, 2012
    Posts:
    275
    I believe, it is possible to access strings out of ECS data flow. There's no need to pass strings around. It is possible to store them somewhere and systems may cache that storage on their initialization. For instance, you may have an array of strings ad you may refer them by id, passed around by int
     
    illinar, vanxining and andywatts like this.
  9. JohnHudeski

    JohnHudeski

    Joined:
    Nov 18, 2017
    Posts:
    126
    I was going to say something like this. Use a static Dictionary<string> and store the hash/key.

    Unless you are trying to modify the string
     
    vestigial likes this.
  10. Soaryn

    Soaryn

    Joined:
    Apr 17, 2015
    Posts:
    265
    The best solution I have rationalized thus far: Store a dictionary with the Entity as the key; however, GENERALLY in the past, statics have been a little on the iffy side when attempting to do running recompiles, meaning still having the simulation running, edit some code, then recompile the code to see live changes.

    However, in the current state of the ECS, live compiling is not something that has been addressed just yet, so I am sort of unsure where statics lie in terms of approaching. Singletons are typically *A* use case of a static, but I have refactored all of my code to use scriptable objects in place of static fields to allow for more modular systems.

    An edge case scenario that would require more work doing this dictionary route:
    EntityManager.GetComponentData<StringDataComponent>(entity) would be relatively useless in this regard; and as mentioned before I'm a little wary with the static approach. StringSystemDictionary.GetData(entity) would work as what I would see as more of a temp solution rather than the end solution.
     
  11. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    7,483
    Really you want to keep strings away from ECS as they have a massive memory footprint, the goal being to cram as much data into the processors caches that the whole processing streams without any snags and max through put.

    What about using a hashing function to 'index' your strings for access?

    I'm assuming the strings are just user information or debug data and not the core data you are processing.
     
  12. Soaryn

    Soaryn

    Joined:
    Apr 17, 2015
    Posts:
    265
    I have a broadcast notifier system that is composed of strings so it is a little unavoidable. I am the receiver of string information from a external API.
     
  13. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    585
    We definitely need something like NativeString that can work with IComponentData nicely.
     
    illinar and FROS7 like this.
  14. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    2,929
    Just map the strings to integers and stick them in a Dictionary<int,string> when you receive them. Your classes/structs that would have had a reference to the string, now reference it via id.

    Code (csharp):
    1.  
    2.  
    3. // Somwhere
    4.         public static Dictionary<int, string> Strings = new Dictionary<int, string>();
    5.  
    6.         // In classes/structs that need a reference
    7.         private int MyStringId;
    8.         public string MyString
    9.         {
    10.             get
    11.             {
    12.                 string value;
    13.                 if (Strings.TryGetValue(MyStringId, out value))
    14.                 {
    15.                     return value;
    16.                 } else
    17.                 {
    18.                     return null;
    19.                 }
    20.             }
    21.         }
    22.  
     
  15. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    7,483
    You should map the strings to tokens then you are sending less data, I'm presuming both systems have a common set of strings they work with and your not writing a dynamic system like those AI's that started developing their own language -> http://www.iflscience.com/technology/facebook-ai-develop-language-human-understand/
     
    KwahuNashoba likes this.
  16. Soaryn

    Soaryn

    Joined:
    Apr 17, 2015
    Posts:
    265
    I translate a json string that I receive into a scriptable object (before ECS); my hope was to use this data to create entities with the names and stringed values so that it would render based on the data. In a NON-ecs system, this works easily.

    The only known information is the topic that the data is being sent over, everything else is dynamic. This is real time data that I, again, have no control over the structure. For now, I will wait until Unity has a better solution to this as Joachim mentioned before. I already mentioned my standpoint and current solution using static dictionaries.

    Also @snacktime, while that does clean a little bit up, I wouldn't need to store an extra int as I have the entity ID

    Thanks though :)
     
  17. allenmathewmd

    allenmathewmd

    Joined:
    Apr 21, 2018
    Posts:
    2
    It's not ideal but you could use ISharedComponentData to store string values too. There is no requirement for the data to be blittable.
     
  18. Soaryn

    Soaryn

    Joined:
    Apr 17, 2015
    Posts:
    265
    IShared is to really be used for Non-changing and grouped data. Whilst the data, once cataloged is relatively immutable, in this case I have to assume all data will be unique which is not really ideal for the use case from what I understand. I also have to work on the assumption I don't have an end of input data.
     
  19. Soaryn

    Soaryn

    Joined:
    Apr 17, 2015
    Posts:
    265
    Hey Joachim, I noticed in the Berlin Q&A that 'arrays and strings' are being addressed, but that wouldn't happen to be sooner rather than later would it? I'd very much rather prefer to use a look up system already inplace rather than having an extra layer dictionary lookup on my side :)
     
    Kender and optimise like this.
  20. Necromantic

    Necromantic

    Joined:
    Feb 11, 2013
    Posts:
    116
    I think it's actually not that far off. He spoke about arrays in details but didn't really say anything about strings besides the title. Though I guess a string is just a character array internally anyway so it shouldn't really be that different.

    I wonder what the restrictions on arrays and strings will be. Since changing the length of them would mean rearranging the whole data layout of those strings and arrays for all the Entity components of that type in memory.
     
    Last edited: Jun 22, 2018
  21. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,507
    It might be like you are allowed to store NativeArray in a component data so it is just a fixed length address, then the address will actually link to memory somewhere else which allows variable length? (Could cause complexity with serialization etc.)
     
  22. Necromantic

    Necromantic

    Joined:
    Feb 11, 2013
    Posts:
    116
    Well, in the talk he said they would layout the data for all the arrays of the components in linear memory as well. So it can't be just pointers that point somewhere else or you'll lose all the cache optimization and basically have cache misses everywhere.
     
  23. GDevTeam

    GDevTeam

    Joined:
    Feb 14, 2014
    Posts:
    70
    New to DOTS. Joachim, can you update us on what the solution is for this technical hurdle is now? Got an example?

    Use case:
    Will have thousands of Prefabs at application launch in Scene. We need to have MouseOver and or Click on way to populate UI elements with each Prefab>Entity's related data (e.g. color, geo location, name, in-operation time, etc.). Each Entity has their own data. This was working fine in OOP, but of course we are dealing with a different beast here. Any help would be appreciated. I've been banging my head on a wall for days now.
     
  24. Enzi

    Enzi

    Joined:
    Jan 28, 2013
    Posts:
    340
    Pretty old thread. :)

    There are several ways to work around this. One is to use ISharedComponentData which can hold managed objects like strings.
    Another is to build a Singleton accessible or static Dictionary<Entity, string> as a lookup.

    In systems, strings should be avoided at all costs. There's also really no reason to have them or need them there.
    ECS for UI elements is also not recommended. A normal MonoBehaviour will be enough with IPointerEnterHandler, IPointerExitHandler to handle the update of texts.
     
    GDevTeam likes this.
  25. wg-siggig

    wg-siggig

    Joined:
    Mar 17, 2020
    Posts:
    36
    Or simply use the NativeString types? Burst even got some basic support now in v1.3 to do debug string writing and interpolation.
     
    sl1nk3_ubi, Kender and GDevTeam like this.
  26. GDevTeam

    GDevTeam

    Joined:
    Feb 14, 2014
    Posts:
    70
    Thanks Enzo, I'll look into these avenues.
     
  27. GDevTeam

    GDevTeam

    Joined:
    Feb 14, 2014
    Posts:
    70
    I saw this option late in the evening yesterday after a lot of searching. Wondering how NativeString64 would look in an example for this use-case. I can get NS64 to Debug.Log() after testing .isBlittable() showed true. What's the best way now to associate it with the instantiation process of Prefabs (Entities)?
     
  28. Kender

    Kender

    Joined:
    Nov 16, 2012
    Posts:
    275
    We also have FixedString. How is it different from NativeString?
     
    YurySedyakin likes this.
  29. Enrico-Monese

    Enrico-Monese

    Joined:
    Dec 18, 2015
    Posts:
    64
    I'd guess it lets you specify a fixed size, may help with performance/optimization
     
  30. Curlyone

    Curlyone

    Joined:
    Mar 15, 2018
    Posts:
    41
    You can put FixedString into a struct IComponentData, you cant with NativeString since its not blittable
     
    Kender likes this.
  31. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    1,562
    I have been using NativeString in ICDs before FixedString was a thing.
     
    GDevTeam likes this.
  32. Kender

    Kender

    Joined:
    Nov 16, 2012
    Posts:
    275
    Why do we need NativeString then?
     
  33. Brendon_Smuts

    Brendon_Smuts

    Joined:
    Jun 12, 2017
    Posts:
    75
    My understanding is that FixedString is essentially identical to NativeString and will serve as it’s replacement in order to be consistent with the other Fixed*** types that have been recently added.
     
    Kender likes this.
  34. sl1nk3_ubi

    sl1nk3_ubi

    Joined:
    Aug 21, 2019
    Posts:
    4
    Note that ECS also supports *class* component types if you must have variable length strings, so you could do something like this:

    Code (CSharp):
    1. class ManagedString : IComponentData
    2. {
    3.     public string Value;
    4. }
    And you would access that like a normal IComponentData via the regular API.
    If you need to store strings with an upper limit on their length, such as the player name, then using FixedString32 or FixedString64 would probably be more cache friendly.
     
  35. WAYN_Group

    WAYN_Group

    Joined:
    Mar 16, 2019
    Posts:
    426
    Note that these class component are not compatible with burst (them behing reference type), not sure they work with jobs either.
     
    GDevTeam likes this.
  36. GDevTeam

    GDevTeam

    Joined:
    Feb 14, 2014
    Posts:
    70
    This is what I'm looking at now. But I'm wondering how I tie this together. I've not seen any good Unity or third-party demonstrations of this technical hurdle fixed. Surely this is needed in just about every Unity application?
     
  37. GDevTeam

    GDevTeam

    Joined:
    Feb 14, 2014
    Posts:
    70
    Okay, here's where I'm at now. Anyone know what goes in where the question mark (?) goes? NativeArray<NativeString64> related.

    I'm once again pulling out my hair. I'm not seeing it. Or am I way off?

    I'm trying to add an array to a NativeMultiHashMap. (I want my Entities to have associated attributes or properties if you will, like color, scale/size, string label).

    Here's what I have so far. I feel I'm close now.

    Code (CSharp):
    1. NativeArray<NativeString64> sArray = new NativeArray<NativeString64>(1, Allocator.Persistent);
    2. public NativeMultiHashMap<int, NativeArray<NativeString64>> nmhm_pointTypes = new NativeMultiHashMap<int, NativeArray<NativeString64>>(100000, Allocator.Persistent);
    3.  
    4.                 sArray[i] = "One";
    5.                 sArray[i] = "Two";
    6.                 sArray[i] = "Three";
    7.  
    8.                 nmhm_pointTypes.Add(i, ?);  <---------- how do I code the sArray (or if you have another way?) into the position where the question mark is located?
    9.  
    10.                 NativeArray<NativeString64> value;
    11.                 NativeMultiHashMapIterator<int> iterator;
    12.                 if (nmhm_pointTypes.TryGetFirstValue(i, out value, out iterator))
    13.                 {
    14.                     do
    15.                     {
    16.                         //Debug.Log("KEY: " + 0 + " VALUE: " + value);
    17.                         Debug.Log("KEY: " + i + " VALUE: " + value);
    18.                     } while (nmhm_pointTypes.TryGetNextValue(out value, ref iterator));
    19.                 }
     
  38. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,029
    You can't. It's not allowed to have NativeArray in native hash map, native array of native arrays etc..
     
    GDevTeam likes this.
  39. GDevTeam

    GDevTeam

    Joined:
    Feb 14, 2014
    Posts:
    70
    No wonder. Thanks.
     
  40. Kender

    Kender

    Joined:
    Nov 16, 2012
    Posts:
    275
    Consider using FixedString instead
     
  41. Soaryn

    Soaryn

    Joined:
    Apr 17, 2015
    Posts:
    265
    What? How would that change anything in this scenario? Any blittable type is fine to use in a NativeContainer. The problem here was the fact that a NativeArray was being used. Most (if not all) NativeContainers are non-blittable. NativeStrings are not NativeContainers.
     
    GDevTeam likes this.
  42. GDevTeam

    GDevTeam

    Joined:
    Feb 14, 2014
    Posts:
    70
    I know you replied to Kender here. But thanks for the clarification nontheless Soaryn. :)
     
unityunity