Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Creating StringTable at Runtime

Discussion in 'Localization Tools' started by Erethan, Nov 3, 2020.

  1. Erethan

    Erethan

    Joined:
    Apr 27, 2015
    Posts:
    28
    Hi!
    I wish to be able to import files (e.g., csv) into new StringTables and insert them into the StringDatabase during runtime.

    I understand how it's done through the editor but I can't seem to figure out how to do this using only the `UnityEngine.Localization` namespace.

    What I have managed so far:
    Import the csvs files
    Create and add new locale to LocalizationSettings
    Create and add entries read from csv into a new StringTable

    What I am still missing:
    Get the SharedData table from the Database.
    Add the new StringTable to the Database.

    Thanks!
     
  2. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,142
    Hmm. Ok you would need to register the new tables with the LocalizedStringDatabase during play. The API for this is not public at the moment. Ill make a task to support user created StringTables in playmode.
    One thing you could do now and that I would recommend is to create a custom addressables provider to provide the StringTable. This is the approach we will go with in the future when we support loading additional formats in player.

    Look at ResourceProviderBase
     
    Last edited: Nov 3, 2020
  3. Erethan

    Erethan

    Joined:
    Apr 27, 2015
    Posts:
    28
    This is great! I'll look into it!


    Thanks, @karl_jones!
     
  4. Erethan

    Erethan

    Joined:
    Apr 27, 2015
    Posts:
    28
    I have created a IResourceProvider called
    CsvTableProvider
    and added as the first IResourceProvider in the ResourceManger providers by calling:

    AddressableAssets.Addressables.ResourceManager.ResourceProviders.Insert(0, new CsvTableProvider(_sharedDataTable));


    However, when I change to the newly added Locale, the LocalizedDatabase class calls
    Addressables.LoadAssetAsync<TTable>(tableAddress);
    and fails with four errors logged in the following order:

    1. Exception encountered in operation CompletedOperation, status=Failed, result= : Exception of type 'UnityEngine.AddressableAssets.InvalidKeyException' was thrown., Key=UI Text_en-GB, Type=UnityEngine.Localization.Tables.StringTable
    2. Failed to load table: CompletedOperation
    3. Exception: Exception of type 'UnityEngine.AddressableAssets.InvalidKeyException' was thrown., Key=UI Text_en-GB, Type=UnityEngine.Localization.Tables.StringTable
    4. Exception encountered in operation CompletedOperation, status=Failed, result=UnityEngine.Localization.Settings.LocalizedDatabase`2+TableEntryResult[[UnityEngine.Localization.Tables.StringTable, Unity.Localization, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null],[UnityEngine.Localization.Tables.StringTableEntry, Unity.Localization, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null]] : Failed to load table: CompletedOperation


    Here is the
    CsvTableProvider
    implementation:


    Code (CSharp):
    1. public class CsvTableProvider : ResourceProviderBase
    2. {
    3.     private SharedTableData _sharedDataTable;
    4.  
    5.     public override string ProviderId => "UnityEngine.ResourceManagement.ResourceProviders.AssetDatabaseProvider";
    6.  
    7.  
    8.     public CsvTableProvider(SharedTableData sharedDataTable)
    9.     {
    10.         _sharedDataTable = sharedDataTable;
    11.     }
    12.  
    13.     public override void Provide(ProvideHandle provideHandle)
    14.     {
    15.  
    16.         string filePath = ResourceLocationCsvPath(provideHandle.Location);
    17.  
    18.         StringTable st = ImportTable(filePath);
    19.  
    20.         object result;
    21.         if (provideHandle.Type.IsArray)
    22.         {
    23.             result = ResourceManagerConfig.CreateArrayResult(provideHandle.Type, new UnityEngine.Object[]{ st});
    24.         }
    25.         else if (provideHandle.Type.IsGenericType && typeof(IList<>) == provideHandle.Type.GetGenericTypeDefinition())
    26.         {
    27.             result = ResourceManagerConfig.CreateListResult(provideHandle.Type, new UnityEngine.Object[] { st });
    28.         }
    29.         else
    30.         {
    31.             result = st;
    32.         }
    33.  
    34.  
    35.         provideHandle.Complete(result, result != null, result == null ? new Exception($"Unable to load asset of type {provideHandle.Type} from location {provideHandle.Location}.") : null);
    36.     }
    37.  
    38.  
    39.     public override bool CanProvide(Type t, IResourceLocation location)
    40.     {
    41.         string path = ResourceLocationCsvPath(location);
    42.  
    43.         return
    44.             t.Equals(typeof(StringTable))
    45.             && (path != null);
    46.     }
    47.  
    48.  
    49.     private StringTable ImportTable(string filePath)
    50.     {
    51.         Locale newLocale = Locale.CreateLocale(Path.GetFileNameWithoutExtension(filePath));
    52.         List<(string key, string value)> entries = ReadCsvEntries(filePath);
    53.  
    54.         StringTable stringTable = ScriptableObject.CreateInstance<StringTable>();
    55.         stringTable.LocaleIdentifier = newLocale.Identifier;
    56.         stringTable.SharedData = _sharedDataTable;
    57.  
    58.  
    59.         for (int i = 0; i < entries.Count; i++)
    60.         {
    61.             stringTable.AddEntry(
    62.                 entries[i].key,
    63.                 entries[i].value);
    64.         }
    65.  
    66.  
    67.         return stringTable;
    68.     }
    69.  
    70.     private List<(string, string)> ReadCsvEntries(string filePath)
    71.     {
    72.  
    73.         List<(string, string)> entries = new List<(string, string)>();
    74.         DataTable csvTable = new DataTable();
    75.         using (var stream = new StreamReader(filePath))
    76.         {
    77.             using (var csvReader = new CsvReader(stream, CultureInfo.InvariantCulture))
    78.             {
    79.                 using (var dr = new CsvDataReader(csvReader))
    80.                 {
    81.                     csvTable.Load(dr);
    82.  
    83.  
    84.                     for (int i = 0; i < csvTable.Rows.Count; i++)
    85.                     {
    86.                         entries.Add(
    87.                             (csvTable.Rows[i][0] as string,
    88.                             csvTable.Rows[i][1] as string)
    89.                         );
    90.                     }
    91.                 }
    92.             }
    93.  
    94.         }
    95.         return entries;
    96.     }
    97.  
    98.     private string ResourceLocationCsvPath(IResourceLocation location)
    99.     {
    100.         if(_sharedDataTable.TableCollectionName.Length >= location.PrimaryKey.Length)
    101.             return null;
    102.  
    103.         string fileName = location.PrimaryKey.Substring(_sharedDataTable.TableCollectionName.Length + 1); //Expected StringTable PrimaryKey format: "{Collection.Name}_{CultureName}"
    104.         string filePath = Path.Combine(Application.streamingAssetsPath, fileName + ".csv");
    105.  
    106.         return File.Exists(filePath) ? filePath : null;
    107.  
    108.     }
    109.  
    110. }
     
  5. Erethan

    Erethan

    Joined:
    Apr 27, 2015
    Posts:
    28
    I have manage to do it but there is still some concepts (related to Addressables) that are not clear to me.
    Before, I've completely missed the IResourceLocator interface concept. That is why I did the ProviderId atrocity above.

    From creating my own IResourceLocator, I managed to set the ProviderId in CsvTableProvider as well as guarantee that my Location will have a Locator. (This is what caused the four exceptions from above).

    In case anyone wonders, below is a simple implementation of an IResourceLocator

    Code (CSharp):
    1.  
    2. public class CsvResourceLocator : IResourceLocator
    3. {
    4.     public string LocatorId => nameof(CsvResourceLocator);
    5.  
    6.     public IEnumerable<object> Keys => new object[0];
    7.  
    8.     public bool Locate(object key, Type type, out IList<IResourceLocation> locations)
    9.     {
    10.         if (!typeof(StringTable).IsAssignableFrom(type))
    11.         {
    12.             locations = null;
    13.             return false;
    14.         }
    15.  
    16.         locations = new List<IResourceLocation>();
    17.        
    18.         IResourceLocation[] noDependencies = new ResourceLocationBase[0];
    19.         locations.Add(new ResourceLocationBase(key as string, key as string, typeof(CsvTableProvider).FullName, type, noDependencies));
    20.        
    21.  
    22.         return true;
    23.     }
    24.  

    if anyone could point out any misconceptions from this snippet, I would be grateful. I have no idea what some parameters in ResourceLocationBase are for :)
     
  6. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,142
    I'm not too familiar with this area at the moment. I'll ask the Addressables teram if they have any advice.

    One thing I think you will need to do is add the locator and provider to Addressables:

    Code (CSharp):
    1. IEnumerator Start()
    2.     {
    3.         yield return Addressables.InitializeAsync();
    4.  
    5.         Addressables.ResourceManager.ResourceProviders.Add(new CsvTableProvider(null));
    6.         Addressables.AddResourceLocator(new CsvResourceLocator());
    7.     }
    Once I did this it started to work for me.

    Edit:

    It looks like what you have works pretty well now. Im going to pin this thread as it seems like something others will want to do and it's a good place to start :)
    We do plans to have support various custom providers and locators in the future so people can add modding support, pull google sheets in the player etc.
     
    Last edited: Nov 6, 2020
    Erethan likes this.
  7. miron83

    miron83

    Joined:
    Oct 28, 2015
    Posts:
    6
    Hi,
    I am currently trying to create a runtime translation updater in my application based on a CSV file downloaded from the server.
    The CSV file is generated on the server and has identical structure to that exported in the CSV Extension available from the String Table Collections object.

    I tried to use the script suggested by you but I totally don't know how to initialize it and upload CSV file - I suspect I need to run Provide (ProvideHandle provideHandle) but I don't know how to create ProvideHandle object with information about CSV file. In addition, it seems to me that the above script is used to add a new Locale with values.
    I will definitely need it in the future, but at the moment I am looking for a solution to update and save translations in the runtime.

    I used and tweaked a little bit the above class for CSV interpretation and creation of temp string table and wrote something like this:


    Code (CSharp):
    1. public IEnumerator UpdateTranslations(string _translationsCSVFilePath){
    2.        
    3.         // save starting locale
    4.         Locale _baseLocale = LocalizationSettings.SelectedLocale;
    5.  
    6.         // loop throu all available locales
    7.         foreach(Locale _locale in LocalizationSettings.AvailableLocales.Locales){
    8.    
    9.          
    10. LocalizationSettings.SelectedLocale = _locale;
    11.  
    12.             // get localized string table database based on active locale
    13.             AsyncOperationHandle<StringTable> _asyncStringTableDatabase =                                  LocalizationSettings.StringDatabase.GetTableAsync(LocalizedStringTables.TableReference);
    14.             yield return new WaitUntil(()=> _asyncStringTableDatabase.IsDone);
    15.             StringTable _stDatabase = _asyncStringTableDatabase.Result;
    16.          
    17.             // create string table corresponding to active locale (CSV contains cols from all locales).
    18.             // The safest way to get propper CSV structure is to use export CSV extension from your primary String Table Collection object in Unity inspector
    19.             StringTable _importedST = ImportTable(_translationsCSVFilePath, _locale);
    20.  
    21.            
    22. // this part finds and adds fields that are empty in selected language string database but exist in shared values and in imported StringTable.
    23.             // If fields in database are empty they can't be updated - _stDatabase.Values.Count = number of filled fields
    24.             // further more it will add new row in Localization Tables if the key is unique - it desn't exist
    25.             foreach(StringTableEntry _value in _importedST.Values){
    26.                 if(_value.Value != "" && _value.Value != null){
    27.                     if(_stDatabase.SharedData.Contains(_value.KeyId)){
    28.                         if(!_stDatabase.ContainsKey(_value.KeyId)){                                                
    29.                             _stDatabase.AddEntry(_value.Key, _value.Value);                        
    30.                         }
    31.                     }
    32.                 }          
    33.             }
    34.  
    35.             // this part updates the editor runtime string database values - will be visable in runtime. Also the Window > Asset Management > Localization Talbes, will also be updated but after you stop and run the app again from the editor
    36.             foreach(StringTableEntry _value in _stDatabase.Values){
    37.              
    38.                 if(_importedST[_value.Key] == null){ continue;} // for safety - skip that translation part if key from database doesnt exist in imported csv
    39.                 string _val = _importedST[_value.Key].Value;                          
    40.              
    41.                 if(_value.Value != _val){
    42.                     _value.Value = _val;
    43.                 }              
    44.             }          
    45.  
    46.             // get addressable string tables based on active locale. Just pulling the values will update displayed values on the device
    47.             AsyncOperationHandle<StringTable> _asyncAddressablesTable =  Addressables.LoadAssetAsync<StringTable>("Locale-" + _locale.Identifier.Code);
    48.             yield return new WaitUntil(()=> _asyncAddressablesTable.IsDone);          
    49.             StringTable _addressablesTable = _asyncAddressablesTable.Result;
    50.  
    51.             // This script will work only if in "Window > Asset Managmenet > Addressables > Groups" window, the "Simulate Groups (advanced)" option located in the "Play Mode Script" dropdown will be selected.
    52.    
    53.         }
    54.  
    55.         // switch to saved starting locale
    56.         LocalizationSettings.SelectedLocale = _baseLocale;
    57.  
    58.         AppManager.SetBaseMessages();
    59.     }
    60.  
    61.  
    62.  
    63.     private StringTable ImportTable(string _filePath, Locale _locale)
    64.     {
    65.         // Locale newLocale = Locale.CreateLocale(Path.GetFileNameWithoutExtension(_filePath));
    66.         List<(string _key, string _value)> _entries = ReadCsvEntries(_filePath, _locale.LocaleName);
    67.         StringTable _stringTable = ScriptableObject.CreateInstance<StringTable>();
    68.         _stringTable.LocaleIdentifier = _locale.Identifier;
    69.         _stringTable.SharedData = SharedTable;
    70.  
    71.  
    72.         for (int i = 0; i < _entries.Count; i++)
    73.         {
    74.             _stringTable.AddEntry(
    75.                 _entries[i]._key,
    76.                 _entries[i]._value);
    77.         }
    78.         return _stringTable;
    79.     }
    80.  
    81.     private List<(string, string)> ReadCsvEntries(string _filePath, string _localeName)
    82.     {
    83.         List<(string, string)> _entries = new List<(string, string)>();
    84.         DataTable _csvTable = new DataTable();
    85.         using (var _stream = new StreamReader(_filePath))
    86.         {
    87.             using (var _csv = new CsvReader(_stream, CultureInfo.InvariantCulture))
    88.             {
    89.        
    90.                 _csv.Read();
    91.                 _csv.ReadHeader();
    92.                 while (_csv.Read())
    93.                 {
    94.                     _entries.Add((
    95.                         _csv.GetField("Key"),
    96.                         _csv.GetField(_localeName.Replace(") (", ")("))));
    97. // it mightr be a bug but the  Locale identifier and he header in exported CSV file corresponding to the language differs - theres extra space between brackets
    98. // standard CSV Extension export -> English (United Kingdom)(en-GB), Locale English (United Kingdom) (en-GB)
    99. // the CSV names can be changed in CSV Extension, but keep in mind that the standard extension removes this space (I didn't notice it at first)
    100.                 }
    101.             }
    102.         }
    103.         return _entries;
    104.     }
    as you will notice the CSV file is used to create temporary StringTable instances with the Key, and value columns coresponding to the currently selected Locale. Then a few parts that update the values according to the created table - the values displayed in the editor player, the stringtables database (which is useful so that you do not have to do it manually each time) and AddressableTables, which I care about the most, i.e. changing the values on users' devices.
    However, I have a problem - updating values on users' devices only works if we run the above script. After restarting the application, the values return to their original values.
    I suspect it has something to do with updating the Catalogs, but when I run a script that I found on another post:

    Code (CSharp):
    1. private IEnumerator UpdateCatalogs()
    2.     {
    3.         List<string> catalogsToUpdate = new List<string>();
    4.         AsyncOperationHandle<List<string>> checkForUpdateHandle = Addressables.CheckForCatalogUpdates();
    5.  
    6.         checkForUpdateHandle.Completed += op =>
    7.         {
    8.             Debug.Log(op.Result.Count);
    9.             catalogsToUpdate.AddRange(op.Result);
    10.         };
    11.         yield return checkForUpdateHandle;
    12.         if (catalogsToUpdate.Count > 0)
    13.         {  
    14.             AsyncOperationHandle<List<IResourceLocator>> updateHandle = Addressables.UpdateCatalogs(catalogsToUpdate);
    15.             yield return updateHandle;
    16.         }
    17.     }

    ...it does not find any values to update.
    What am I doing wrong? How can I save modified values and StringTables in runtime on the device (not only in editor player runtime) so the next time i call Localized string i'll get the updated values?
     
    Last edited: Sep 15, 2021
  8. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,142
    Changes you make to a table at runtime wont persist after the application restarts, they are just in memory and will be lost when the application closes. Addressables does have a system for content updates: https://docs.unity3d.com/Packages/com.unity.addressables@1.19/manual/ContentUpdateWorkflow.html

    For what you are doing your script will always need to run in order to patch the String Table, this is fine to do. If the problem is that you need to pull a csv file from your server each time then it may be worth trying to cache the csv file locally, such as un PlayerPrefs and then only pull the table if you dont have one cached or every so often.
     
  9. miron83

    miron83

    Joined:
    Oct 28, 2015
    Posts:
    6
    Thanks for quick reply

    Well, I can always save a file to disk (actually, now I do this before using a CSV file) and download it again only when I require it.
    However, I was hoping to somehow skip the entire process of creating tables and updating the values each time the application starts - it is only a second or so, but each second is important...

    From what I read in the link mentioned, it is also possible to update by building remote directories, using external RemoteLoadPath.

    aaa.jpg
    I will also have to try it, but at first glance it seems to me that this method is not very optimal (i might be wrong) ... safe because the groups are generated by the unity editor, but time and resource intensive if there are many languages and even more phrases to update...
    Correct me if I'm wrong but from what I understand is that every time, apart from updating the values in StringTablesDatabase, I will also have to build AddressableGroups and push them to the server, these groups then will be retrieved from the server by the app each time the aplication starts and updated if I use Addressables.CheckForCatalogUpdates() ?
     
  10. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,142
  11. miron83

    miron83

    Joined:
    Oct 28, 2015
    Posts:
    6
    Ok, thanks for the advice,
    However, I think I'll go with serializing and deserializing tables generated from CSV to speed up the process.
    cheers
     
    karl_jones likes this.
  12. Mashimaro7

    Mashimaro7

    Joined:
    Apr 10, 2020
    Posts:
    727
    Hi, you referred me to here, I just had a look, would you be able to explain how addressables work? What is the CsvTableProvider? And the CsvResourceLocator? And how is this creating a new string table? Is that what this is doing? lol, sorry I'm so confused. I feel dumb today lol
     
  13. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,142
    Addressables is the system Localization uses to fetch data such as String Tables. By default it comes from Asset Bundles but you can create custom providers to get data from other sources such as Csv, Json etc.
    https://docs.unity3d.com/Packages/com.unity.addressables@1.19/manual/index.html
     
  14. AlfSgio

    AlfSgio

    Joined:
    Oct 11, 2020
    Posts:
    7
    I'm looking for a solution that does the same thing as what you're looking for.
    Can you find something relevant to make your StringTable update persistent in your app ?
     
  15. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,142
  16. AlfSgio

    AlfSgio

    Joined:
    Oct 11, 2020
    Posts:
    7
    Skibitsky likes this.
  17. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,142
    We have now released 1.4.2 which includes ITableProvider and ITablePostProcessor. These let you provide tables from custom locations and apply changes to a table when it first loads. These are a much simpler way than creating a custom addressables resource provider.
     
    Last edited: Oct 17, 2022
  18. Taro_FFG

    Taro_FFG

    Joined:
    Jun 24, 2022
    Posts:
    57
    Hi,
    regarding the docs of ITableProvider, it says it is useful for example for mods, however the example references
    LocalizationEditorSettings from UnityEditor so that won't work for loading a new table for a mod.
    In our case mods typically don't override base game content so having a separate table makes much more sense.
    It would be great if there was a snippet there for that use case as well.
     
  19. Taro_FFG

    Taro_FFG

    Joined:
    Jun 24, 2022
    Posts:
    57
    I see now why I am having issues with it.
    In our case we have two sources of data, the addressable and a dynamic source for additional tables during runtime.
    I guess the system is not designed for directly so I'll have to figure out a workaround.
     
  20. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,142
    Ah, the example should not be using LocalizationEditorSettings. That's a mistake. You can change that part to
    Code (csharp):
    1. var settings = LocalizationSettings.Instance
    Ill get the example fixed.

    Can you provide some more details? The system should be able to handle this.
     
  21. Taro_FFG

    Taro_FFG

    Joined:
    Jun 24, 2022
    Posts:
    57
    Thank you, sure. I've been toying with it a bit yesterday after the post.
    My intention would be to leave the addressable loading as is and have additional ITableProvider(s) to load dynamic content from text files that is not known during compile time.

    Our modded content and some of our official content is created using an editor that is available inside the runtime, not in editor. This loading happens once during initial loading of the runtime and after the user uses a package selection feature in the menu.
    The data is serliazed as json files and loaded from the client based on what packages users select in the runtime.
    This content currently is not containing addressables, just text/scripting data.

    My first impression of ITableProvider was that the localization system would allow me to maintain a collection of ITableProvider during runtime, one for each table I intend to have.
    It seems now that the system only expects a single provider to create all the tables for it.

    I currently see three potential routes of getting my desirend behavior, provided I'm not missing some feature of the system.

    A) I let the normal provider run in the beginning creating all the constant data from Addressables and swap it out after to load dynamic content when I need it. Would require the already loaded data to not get lost by a refresh later, for example locale selection change.
    B) Fork the package and change providers to a collection of providers and call them in a loop.
    C) Find the default provider, write a custom provider and call the default one to load base content from the custom provider and after that ran import my custom content.

    I haven't read through all the relevant code in the package yet so it is possible I'm missing something.
     
  22. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,142
    Do you expect that these modded tables to be requested before the user has selected the package they belong to? We only load tables as they are requested, so if the table is unknown at that point then we wont request it until its needed by your custom package, so in theory this should be fine. We dont want to load all tables and languages as its an overhead we dont need when the table and language is not being used.

    What about creating a custom table provider that contains a collection? This is how we designed it.

    Something like this:

    Code (csharp):
    1. [Serializable]
    2. public class TableProviderCollection : ITableProvider
    3. {
    4.     [SerializeReference]
    5.     public List<ITableProvider> myProviders;
    6.  
    7.     public AsyncOperationHandle<TTable> ProvideTableAsync<TTable>(string tableCollectionName, Locale locale) where TTable : LocalizationTable
    8.     {
    9.         foreach (var item in myProviders)
    10.         {
    11.             var operation = ProvideTableAsync<TTable>(tableCollectionName, locale);
    12.             if (operation.isValid())
    13.                 return operation;
    14.         }
    15.  
    16.         return default;
    17.     }
    18. }
    Now you can prioritize the order they are executed, we do the same for the locale selectors.
     
  23. Taro_FFG

    Taro_FFG

    Joined:
    Jun 24, 2022
    Posts:
    57
    Yes, I'm at a similar place now. I use a single ITableProvider to load from my list of selected packages.

    What threw me off was that the assignment of the ITableProvider to the database and the wording in the cods felt like I am replacing the default addressable loading behavior, meaning I either load from Addressables or not at all.

    After reading the code it became clear that the LoadTableOperation is checking for the "collectionName" and if the ITableProvider is not loading the addressable collection names, it is still loading the addressables.

    Thank you.
     
    karl_jones likes this.
  24. mfbaer

    mfbaer

    Joined:
    Oct 22, 2017
    Posts:
    3
    Thanks for the info in this thread. I am fairly new to Unity and am trying to figure out how to download data when needed. Specifically, I have created a new CustomTableProvider : ITableProvider and know how to load cached tables on demand (at the moment I download all translation data from the server on game start which defeats the point). However, I cannot figure out how to wait for a server request to finish before returning the AsyncOperationHandle<TTable>. Using the code in the docs, how would I await for my table entries to be downloaded from an online source (I am using firestore).
    Code (csharp):
    1.  
    2. public class CustomTableProvider : ITableProvider
    3. {
    4. public string customTableCollectionName = "My Custom Table";
    5.  
    6. public AsyncOperationHandle<TTable> ProvideTableAsync<TTable>(string tableCollectionName, Locale locale) where TTable : LocalizationTable
    7. {
    8.    Debug.Log($"Requested {locale.LocaleName} {typeof(TTable).Name} with the name `{tableCollectionName}`.");
    9.  
    10.    // Provide a custom string table only with the name "My Custom Table".
    11.    if (typeof(TTable) == typeof(StringTable) && tableCollectionName == customTableCollectionName)
    12.    {
    13.        // Create the table and its shared table data.
    14.        var table = ScriptableObject.CreateInstance<StringTable>();
    15.        table.SharedData = ScriptableObject.CreateInstance<SharedTableData>();
    16.        table.SharedData.TableCollectionName = customTableCollectionName;
    17.        table.LocaleIdentifier = locale.Identifier;
    18.  
    19.        // INSTEAD OF ADDIND STATIC VALUES LIKE IN THE CODE, I WOULD NEED:
    20.        Dictionary<string,string> serverData = await Firestore.Collection("TRANSLATIONS").Document(customTableCollectionName)
    21.         foreach(var entry in serverData){  
    22.            table.AddEntry(entry.Key, entry.Value);
    23.         }
    24.  
    25.        return Addressables.ResourceManager.CreateCompletedOperation(table as TTable, null);
    26.    }
    27.  
    28.    // Fallback to default table loading.
    29.    return default;
    30. }
    31. }
    32.  
    However, I cannot figure out how I can use async await in an AsyncOperationHandle<TTable>. Any help would be appreciated.

    --- EDIT ---
    Upon further inspection, I was able to get this working with a custom PostProcessor. I am doing something along the lines of:

    Code (csharp):
    1.  
    2. public class CustomLocalisationTableDownloader : ITablePostprocessor
    3. {
    4.     public async void PostprocessTable(LocalizationTable table)
    5.     {
    6.         Debug.LogWarning($"Postprocess {table}");
    7.  
    8.         if (table is StringTable stringTable)
    9.         {
    10.            // Check if the translation data is cached on the device
    11.            // If no, download the string data and create the table
    12.          }
    13.        }
    14. }
    15.  
     
    Last edited: Nov 13, 2023
  25. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,142
    mfbaer likes this.
  26. mfbaer

    mfbaer

    Joined:
    Oct 22, 2017
    Posts:
    3
    Hi Karl
    Thanks for the info, I will look into it if I find the time, looks promising! Could you give me some pointers on how I would do this with a CustomOperation?

    For now what I am doing is:
    1. Use a Custom ITableProvider to load a table when needed
    2. If the table is cached in my localisation manager Dictionary<tableId,Dictionary<localeId,StringTable>> return it.
    3. If not, create an empty stringtable and return
    4. Using the Custom ITablePostprocessor (which should run after load but before returning strings), download the needed table
    5. If translations are downloaded, then switch languages and reset to original to force string updates.
    This seems to be working for now but its very hacky, until I figure out how the custom operations work, will have to do. So it would be great if you had an example of using a custom operation to download translation tables from the web.

    Here is my hacky postprocessor in case you are interested:
    Code (csharp):
    1.  
    2.  public class CustomLocalisationTableDownloader : ITablePostprocessor
    3.     {
    4.         public async void PostprocessTable(LocalizationTable table)
    5.         {
    6.             string tableName = table.TableCollectionName;
    7.             LocaleIdentifier tableLocaleIdentifier = table.LocaleIdentifier;
    8.            
    9.             // check to see if the location manager contains this table, if yes, return
    10.             if (LocalisationManager.Instance.LoadedLocaleTables.ContainsKey(tableName))
    11.             {
    12.                 if (LocalisationManager.Instance.LoadedLocaleTables[tableName].ContainsKey(tableLocaleIdentifier))
    13.                 {
    14.                     return;
    15.                 }
    16.             }
    17.            
    18.             // if not, create a new table
    19.             Debug.LogWarning($"Postprocess {table}");
    20.  
    21.             if (table is StringTable stringTable)
    22.             {
    23.                 // this is a string table. First, we check if the stringtable data is on the users device
    24.                 // store persistant path in variable
    25.                 string path = $"{Application.persistentDataPath}/i18n";
    26.  
    27.                 LocalisationTableServerData localisationTableServerData;
    28.  
    29.                 // if this directory does not exists, we have no translation data.
    30.                 if (!Directory.Exists(path))
    31.                 {
    32.                     Debug.LogWarning($"Creating directory {path}");
    33.                     // create the directory
    34.                     Directory.CreateDirectory(path);
    35.                 }
    36.                
    37.                 // check if a file with the table name exists
    38.                 string localisationTableFilePath = $"{path}/{stringTable.TableCollectionName}.json";
    39.  
    40.  
    41.                 bool refreshStringDatabase = false;
    42.                
    43.                 if (File.Exists(localisationTableFilePath))
    44.                 {
    45.                     Debug.LogWarning($"file found at {localisationTableFilePath}, checking if up to date");
    46.                    
    47.                     // if the file exists, load it
    48.                     localisationTableServerData = JsonConvert.DeserializeObject<LocalisationTableServerData>(File.ReadAllText(localisationTableFilePath));
    49.                    
    50.                     // check if the timestamp of the server is newer and if so, download the new file
    51.                     if (LocalisationManager.Instance.LocalisationTableUpdateServerData.UpdateDictionary[stringTable.TableCollectionName] > localisationTableServerData.UpdateTimestamp)
    52.                     {
    53.                         Debug.LogWarning($"file not up to date, downloading new");
    54.                        
    55.                         // download the table
    56.                         localisationTableServerData = await GetLocalisationTableFromFirestore(stringTable.TableCollectionName);
    57.                        
    58.                         // delete old file
    59.                         File.Delete(localisationTableFilePath);
    60.                        
    61.                         // and save the new file to the device
    62.                         File.WriteAllText(localisationTableFilePath, JsonConvert.SerializeObject(localisationTableServerData));
    63.                         refreshStringDatabase = true;
    64.                     }
    65.                     else
    66.                     {
    67.                         Debug.LogWarning($"file up to date");
    68.                     }
    69.                 }
    70.                 else
    71.                 {
    72.                     Debug.LogWarning($"file does not exist on device, downloading from server...");
    73.                    
    74.                     // if the file does exist
    75.                     localisationTableServerData = await GetLocalisationTableFromFirestore(stringTable.TableCollectionName);
    76.                    
    77.                     // and save it to the device
    78.                     File.WriteAllText(localisationTableFilePath, JsonConvert.SerializeObject(localisationTableServerData));
    79.                     refreshStringDatabase = true;
    80.                 }
    81.                
    82.                 // iterate through the localisation table server data and add the values in the locale
    83.                 foreach (var localeStringKeyLanguageValue in localisationTableServerData.TableData)
    84.                 {
    85.                     string localeStringKey = localeStringKeyLanguageValue.Key;
    86.                     foreach (var localeLanguageStringValue in localeStringKeyLanguageValue.Value)
    87.                     {
    88.                         string localeLanguage = localeLanguageStringValue.Key;
    89.                         if (localeLanguage == stringTable.LocaleIdentifier.Code)
    90.                         {
    91.                             string localeStringValue = localeLanguageStringValue.Value;
    92.                             stringTable.AddEntry(localeStringKey, localeStringValue);
    93.                         }
    94.                     }
    95.                 }
    96.                
    97.                 // add the table to the loaded locale tables
    98.                 if (!LocalisationManager.Instance.LoadedLocaleTables.ContainsKey(tableName))
    99.                 {
    100.                     LocalisationManager.Instance.LoadedLocaleTables.Add(tableName, new Dictionary<LocaleIdentifier, StringTable>());
    101.                 }
    102.                
    103.                 if(!LocalisationManager.Instance.LoadedLocaleTables[tableName].ContainsKey(tableLocaleIdentifier))
    104.                 {
    105.                     LocalisationManager.Instance.LoadedLocaleTables[tableName].Add(tableLocaleIdentifier, stringTable);
    106.                 }
    107.                 Debug.LogWarning("Postprocess complete.");
    108.                 if (refreshStringDatabase)
    109.                 {
    110.                     // reset the state of the localization settings
    111.                     Locale locale = LocalizationSettings.Instance.GetSelectedLocale();
    112.                     List<Locale> locales = new List<Locale>(LocalizationSettings.AvailableLocales.Locales);
    113.                     locales.Remove(locale);
    114.                     LocalizationSettings.Instance.SetSelectedLocale(locales[0]);
    115.                     LocalizationSettings.Instance.SetSelectedLocale(locale);
    116.                 }
    117.             }
    118.         }
    119.  
    120.         private async Task<LocalisationTableServerData> GetLocalisationTableFromFirestore(string tableName)
    121.         {
    122.             return await FirebaseManager.Instance.GetFirestoreData<LocalisationTableServerData>("TRANSLATIONS/TABLES/TABLE_COLLECTION", tableName);
    123.         }
    124.     }
    125. }
    126.  
     
  27. mfbaer

    mfbaer

    Joined:
    Oct 22, 2017
    Posts:
    3
    After some tampering and reading up as much as I could (resources online seem to be rather scarce), I was able to make a somewhat working solution using a custom operation. I didnt quite understand the section "Terminating the operation", so this might still need some investigating. In case anyone is interested, here is what I came up with:

    The ITableProvider
    Code (csharp):
    1.  
    2. public class NewCustomLocalisationProvider : ITableProvider
    3. {
    4.    public AsyncOperationHandle<TTable> ProvideTableAsync<TTable>(string tableCollectionName, Locale locale) where TTable : LocalizationTable
    5.     {
    6.         Debug.Log($"Requested {locale.LocaleName} {typeof(TTable).Name} with the name `{tableCollectionName}`.");
    7.      
    8.         if (typeof(TTable) == typeof(StringTable))
    9.         {
    10.             // check if the table is cached in memory
    11.             if (!LocalisationManager.Instance.LoadedLocaleTables.ContainsKey(tableCollectionName))
    12.             {
    13.                 // if not, start custom download operation
    14.                 return Addressables.ResourceManager.StartOperation(new AsyncTableDownloadOperation<TTable>(tableCollectionName, locale), default);
    15.             }
    16.  
    17.             // if there is a cached string table, check if the locale is cached
    18.             if (!LocalisationManager.Instance.LoadedLocaleTables[tableCollectionName].ContainsKey(locale.Identifier))
    19.             {
    20.                 // if not, start custom download operation
    21.                 return Addressables.ResourceManager.StartOperation(new AsyncTableDownloadOperation<TTable>(tableCollectionName, locale), default);
    22.             }
    23.          
    24.             // if there is a cached string table with the locale, return it
    25.             StringTable cachedStringTable = LocalisationManager.Instance.LoadedLocaleTables[tableCollectionName][locale.Identifier];
    26.             return Addressables.ResourceManager.CreateCompletedOperation(cachedStringTable as TTable, "");
    27.         }
    28.  
    29.         // Fallback to default table loading.
    30.         return default;
    31.     }
    32. }
    33.  
    And here is the AsyncOperationBase<TObject>
    Code (csharp):
    1.  
    2. public class AsyncTableDownloadOperation<TObject> : AsyncOperationBase<TObject> where TObject : LocalizationTable
    3. {
    4.     protected override string DebugName => "AsyncTableDownloadOperation";
    5.     private string TableCollectionName { get; set; }
    6.     private Locale Locale { get; set; }
    7.  
    8.     public AsyncTableDownloadOperation(string tableCollectionName, Locale locale)
    9.     {
    10.         TableCollectionName = tableCollectionName;
    11.         Locale = locale;
    12.     }
    13.  
    14.     protected override async void Execute()
    15.     {
    16.         // store persistant path in variable
    17.         string path = $"{Application.persistentDataPath}/i18n";
    18.  
    19.         // if this directory does not exists, we have no translation data.
    20.         if (!Directory.Exists(path))
    21.         {
    22.             Debug.LogWarning($"Creating directory {path}");
    23.             // create the directory
    24.             Directory.CreateDirectory(path);
    25.         }
    26.  
    27.         // check if a file with the table name exists
    28.         string localisationTableFilePath = $"{path}/{TableCollectionName}.json";
    29.  
    30.         // if no file exists, download the file and return complete
    31.         if (!File.Exists(localisationTableFilePath))
    32.         {
    33.             Debug.LogWarning("Completing operation");
    34.             Complete(CreateStringTable(await DownloadAndSaveTranslationTable(localisationTableFilePath)) as TObject, true, "");
    35.             return;
    36.         }
    37.  
    38.         Debug.LogWarning($"file found at {localisationTableFilePath}, checking if up to date");
    39.  
    40.         // if the file exists, load it
    41.         LocalisationTableServerData localisationTableServerData = JsonConvert.DeserializeObject<LocalisationTableServerData>(File.ReadAllText(localisationTableFilePath));
    42.  
    43.         // check if the timestamp of the server is newer and if so, download the new file and return complete
    44.         if (LocalisationManager.Instance.LocalisationTableUpdateServerData.UpdateDictionary[TableCollectionName] > localisationTableServerData.UpdateTimestamp)
    45.         {
    46.             Debug.LogWarning("Completing operation");
    47.             Complete(CreateStringTable(await DownloadAndSaveTranslationTable(localisationTableFilePath)) as TObject, true, "");
    48.             return;
    49.         }
    50.  
    51.         // if the file is up to date, return the complete
    52.         Complete(CreateStringTable(localisationTableServerData) as TObject, true, "");
    53.     }
    54.  
    55.     private StringTable CreateStringTable(LocalisationTableServerData localisationTableServerData)
    56.     {
    57.         if (!LocalisationManager.Instance.SharedTableData.ContainsKey(TableCollectionName))
    58.         {
    59.             SharedTableData sharedTableData = ScriptableObject.CreateInstance<SharedTableData>();
    60.             sharedTableData.TableCollectionName = TableCollectionName;
    61.             LocalisationManager.Instance.SharedTableData.Add(TableCollectionName, sharedTableData);
    62.         }
    63.  
    64.         StringTable stringTable = ScriptableObject.CreateInstance<StringTable>();
    65.         stringTable.SharedData = LocalisationManager.Instance.SharedTableData[TableCollectionName];
    66.  
    67.         // iterate through the localisation table server data and add the values in the locale
    68.         foreach (var localeStringKeyLanguageValue in localisationTableServerData.TableData)
    69.         {
    70.             string localeStringKey = localeStringKeyLanguageValue.Key;
    71.             foreach (var localeLanguageStringValue in localeStringKeyLanguageValue.Value)
    72.             {
    73.                 string localeLanguage = localeLanguageStringValue.Key;
    74.                 if (localeLanguage == Locale.Identifier.Code)
    75.                 {
    76.                     string localeStringValue = localeLanguageStringValue.Value;
    77.                     stringTable.AddEntry(localeStringKey, localeStringValue);
    78.                 }
    79.             }
    80.         }
    81.  
    82.         // add the table to the loaded locale tables
    83.         if (!LocalisationManager.Instance.LoadedLocaleTables.ContainsKey(TableCollectionName))
    84.         {
    85.             LocalisationManager.Instance.LoadedLocaleTables.Add(TableCollectionName, new Dictionary<LocaleIdentifier, StringTable>());
    86.         }
    87.  
    88.         if (!LocalisationManager.Instance.LoadedLocaleTables[TableCollectionName].ContainsKey(Locale.Identifier))
    89.         {
    90.             LocalisationManager.Instance.LoadedLocaleTables[TableCollectionName].Add(Locale.Identifier, stringTable);
    91.         }
    92.  
    93.         return stringTable;
    94.     }
    95.  
    96.  
    97.     private async Task<LocalisationTableServerData> DownloadAndSaveTranslationTable(string localisationTableFilePath)
    98.     {
    99.         // if the file already exists, delete it
    100.         if (File.Exists(localisationTableFilePath))
    101.             File.Delete(localisationTableFilePath);
    102.  
    103.         // get the translation data from firestore
    104.         LocalisationTableServerData localisationTableServerData = await GetLocalisationTableFromFirestore(TableCollectionName);
    105.  
    106.         // and save it to the device
    107.         File.WriteAllText(localisationTableFilePath, JsonConvert.SerializeObject(localisationTableServerData));
    108.  
    109.         // return the server data
    110.         return localisationTableServerData;
    111.     }
    112.  
    113.     private async Task<LocalisationTableServerData> GetLocalisationTableFromFirestore(string tableName)
    114.     {
    115.         return await FirebaseManager.Instance.GetFirestoreData<LocalisationTableServerData>("TRANSLATIONS/TABLES/TABLE_COLLECTION", tableName);
    116.     }
    117. }
    118.  
    Dont forget to set the custom provider somewhere when initialising:

    Code (csharp):
    1.  
    2. NewCustomLocalisationProvider customLocalisationProvider = new NewCustomLocalisationProvider();
    3. LocalizationSettings.StringDatabase.TableProvider = customLocalisationProvider;
    4.  
    This is probably in no way the best approach, however, this is what I came up with to the best of my knowledge and it seems to be working for now. Please let me know if you have any suggestions.
     
    Last edited: Nov 14, 2023
    karl_jones likes this.