BG Database (inMemory database | Excel/Google Sheets syncing | CodeGen | Save/Load support)

BG Database is inMemory database with Excel/GoogleSheets syncing, Code generation and Save/Load support.
Home page: http://www.bansheegz.com/BGDatabase
Asset store: https://assetstore.unity.com/packages/tools/integration/bg-database-112262

 

Hi, I saw your attractive assets.
I'll ask you a few questions publicly before I buy your assets.

1. Does this DB run only on main thread? Can this DB run on sub thread?
2. Can this DB communicate with multi-threaded function?
3. How compatible is it with the 2018 version? (Job or ECS)

P.S. I am not an English speaker. I am sorry for poor English.

 

Hi, Cosmoraph, thank you for your interest in our asset!

1) Currently, our DB is not thread-safe (it's supposed to be run on the main thread), but we were thinking about adding multithreading support as an option if there is any demand for this.
2) DB does not call any code outside (apart from event listeners). A multi-threaded function cannot call DB (cause it's not thread-safe )
3) We were thinking about new Unity's Jobs and ECS and how we could integrate with them, but we've run out of ideas. Actually, you can easily create Unity's Entities using information from the database without multithreading, and after you created these Entities, there is no much sense in storing them in the database unless you want to be able to save and load them afterwards. But if you want to save them, you could easily create a thread-safe function for this and call it when user want to save the game. So other than this usecase, there is no sense in using database (maybe we were wrong)

Anyway, if you are interested in using our asset in a multithreaded environment (probably with Jobs/ECS), we could
1) provide you with free evaluation copy (so you could decide if you want to use it or not)
2) If you decide you want to use it, we send you a voucher (so you could download our package for free), and a beta version with multithreading support. Later we decide together what needs to be added.

Please, let me know if you are interested.

 

@BansheeGz,
Thank you for your reply.
I bought it right away and am reviewing your performance.
Before 'multi-threaded evaluation Copy' request, I ask you a few more questions.

I found out that BGID is essential to access Entity on the code. But I need a way to search by a method other than BGID.
If the table is set to a unique name, is there any way I can search by name to get the Row I want?
In addition, I wish I could set up each Row with an ordered number instead of a unique name, and I would like it to be able to search by numbers in this order.
Is there a way to do these? If not, can you add these features?

 

Cosmoraph, thank you for purchasing!

1) In 1.2 version, which went live today several hours ago, there is a method to get entity by name: BGEntity result = meta["entityName"]; Additionally if you use CodeGen you can generate static getters for each entity (if the number of entities <1024)

2) To search by predicate, use

Code (CSharp):

1.  
2. List<BGEntity> result =  meta.FindEntities(entity => entity.Get<int>("damage")>10);
3. BGEntity result =  meta.FindEntity(entity => entity.Get<int>("damage")>10);
4.  

sorry, we forgot to add it to API page
Also, note, we used very bad decision to use temporary list in meta.FindEntities method to get rid of Garbage Collection

EDIT:
3)We've added Comparison<T> parameter to Find/Iterate methods, so rows could be sorted.
I've sent you an updated package.
So, the signatures now are:

Code (CSharp):

1. List<BGEntity> FindEntities(Predicate<BGEntity> filter, List<BGEntity> result = null, Comparison<BGEntity> sort = null)
2. void ForEachEntity(Action<BGEntity> action, Predicate<BGEntity> filter = null,Comparison<BGEntity> sort = null)
3.  

Here are some examples:
1) Without code generation

Code (CSharp):

1.  
2. //find all items with damage>5 and return them in ascensing order
3. List<BGEntity> items = BGRepo.I["Items"].FindEntities(e => e.Get<int>("damage") > 5, null, (e1, e2) => e1.Get<int>("damage").CompareTo(e2.Get<int>("damage")));
4.  

2) The same with code generation:

Code (CSharp):

1. List<Items> items = Items.FindEntities(e => e.damage > 5, null, (e1, e2) => e1.damage.CompareTo(e2.damage));

 

Cosmoraph, with regard to performance: at the moment we are investigating opportunities to improve it, including the possibility to reduce the cost of field value getter(and partially setter) to single access to an array element (at least with generated code).
Probably we won't be able to make it any faster than accessing array element by its index (without using C++)

 

@BansheeGz,
I checked 1.2 version over the weekend. I am satisfied with the update. However, 3) is the 1.3 version? I'll check as soon as updates register with the Unity store. Thank you for your quick updates.

 

Cosmoraph, yes, the sorting delegate parameter (Comparison<T>) is in the 1.2.1Beta now.
I've sent beta version to you in the private message.
If you did not receive it, please, write to our support email, we'll send it by email.
We plan to release 1.2.1 version next week

 

Version 1.2.1 (with performance upgrade) is live.

 

@BansheeGz , With Unity Types, when editing tables, it would be good if you could see a small preview of things like sprites, texture2ds, meshes, gameobjects in the cell? Would this be possible?

How do you store these in the Table databases? As path names to the assets in the Resource folder path?

 

@Duffer123, yes, we will add it. I will send an update to you in a private message very shortly.
Yes, we store it as path names to assets in Resources folder or in a asset bundle.
We want to add support for Unity's addressables in the new version ( https://forum.unity.com/threads/addressables-are-here.536304/ )

 

I would be quite interested in seeing what you come up with if you were to move forward with a multithreaded implementation of this. I have been working on an RPG and actually have been thinking of revisiting my data storage setup.

Even if you don't, I still like what you guys have going on there from what I can see.

 

@MostHated
could you please email us at support@bansheegz.com ?
We could provide you with free beta copy for review and we will be very interested to hear your opinion
I hope it will be ready within 1-2 weeks

 

We've added support for Unity addressables. (BGDatabase version 1.2.3)
We also added Unity assets preview in the table cells and drag and drop support.

 

@BansheeGz ,

Just installed v1.24. I have Windows 10 and Unity 2018.2.10f1.

I am still getting the following (non-breaking) yellow warnings:-

There are inconsistent line endings in the 'Assets/BansheeGz/BGDatabase/Examples/Scripts/Generated/BGGeneratedPlaymaker.cs' script. Some are Mac OS X (UNIX) and some are Windows.
This might lead to incorrect line numbers in stacktraces and compiler errors. Many text editors can fix this using Convert Line Endings menu commands.

Lighting data asset ‘LightingData’ is incompatible with the current Unity version. Please use Generate Lighting to rebuild the lighting data. Realtime Global Illumination cannot be used until the lighting data is rebuilt.

Lighting data asset ‘LightingData’ is incompatible with the current Unity version. Please use Generate Lighting to rebuild the lighting data. Realtime Global Illumination cannot be used until the lighting data is rebuilt.

Lighting data asset ‘LightingData’ is incompatible with the current Unity version. Please use Generate Lighting to rebuild the lighting data. Realtime Global Illumination cannot be used until the lighting data is rebuilt.
UnityEditorInternal.InternalEditorUtility:GetGameObjectInstanceIDFromComponent(Int32)
UnityEngine.GUIUtilityrocessEvent(Int32, IntPtr)


Lighting data asset ‘LightingData’ is incompatible with the current Unity version. Please use Generate Lighting to rebuild the lighting data. Realtime Global Illumination cannot be used until the lighting data is rebuilt.
UnityEditorInternal.InternalEditorUtility:GetGameObjectInstanceIDFromComponent(Int32)
UnityEngine.GUIUtilityrocessEvent(Int32, IntPtr)

Just thought I better flag it.

 

@Duffer123 ,
Oh, I'm sorry, it slipped my mind, thank you for reminding me.
we'll have it fixed soon (I'll send you an updated package just in case you are interested)

Next version (ETA: end of next week) should have new node-based graph editor for building templates

EDIT: this feature is delayed, unfortunately

 

@BansheeGz ,

Whaaat? Tell me more about this graph editor...

 

@Duffer123 ,

We want to build a node-based graph tool to query/update database data without need to write code.
Something like Shader Graph https://unity3d.com/ru/shader-graph , but instead of building shaders we want to query/update data with it.
And these graphs need to be executable at runtime.
At the very least, we want to replace our templates with this tool ASAP, cause they are very basic and not very useful at all.
After the tool is released, any feedback would be greatly appreciated.

 

@BansheeGz , will probably require more tutorial vids

 

@BansheeGz , How's it going with development of this Asset? Anything next in the pipeline?

 

@Duffer123, unfortunately node graph is delayed. We had hard time trying to adopt third party node graph editor and we lost a lot of time. We had to switch to multi threading add-on instead of this feature.
I updated roadmap here: http://www.bansheegz.com/BGDatabase/

 

Ok. Looking forward to it!

 

I just got the invite, I will be sure to try it out. I had a question though about this. I have been working on an RPG but was rethinking my DB setup, my game is somewhat split up so that it has a map server, a login server, etc. I wanted to try and incorporate a "state" server which would essentially act as just an in-memory database and the only thing that actually communicates with the DB is the state server and it would only do it periodically as needed.

The other servers and what not would communicate directly with the state server at all times and the state server would just do something like batch saving every now and then, so that instead of a server having to do queries to a standard db, the info is readily available. I see that this can save and load to google sheets and such, but is it possible to have this act as a state server since it handles the in-memory portion well from what I can see, but then save its data to something a bit more suitable for data storage instead of just google sheets or excel, etc?

 

@MostHated ,
no, unfortunately, BGDatabase is not meant to be run outside of Unity, so it can not be run in any server environment. Even if it did, it would not make much sense, cause we could not compete with traditional databases. It's meant to be an alternative for Unity's scriptable objects, csv/xml files or whatever people are using to store their ingame data. The purpose for multithreading support is to allow to use BGDatabase with new Unity's Entity-Component System. Also GoogleSheets export/import works in Editor only, in Runtime you can update database data on players devices with LiveUpdate plugin, but there is no way to update GoogleSheet data from players devices.
As far as I understood, you want to make a fast proxy server for better performance and to reduce db servers load. Probably traditional inMemory databases is a good choice for this, maybe some specialized software exist. Unfortunately, BGDatabase is a no-go for this scenario.

 

It might still be able to help out with what I am after, or at least make for a good learning experience if nothing else. When I was referring to it being a state server I was meaning that it would still be running as a Unity instance. When I mentioned map server, login server, etc, they are each Unity instances, they are just dedicated to specific tasks, but running in separate binary files so that they could be separated out to run on different physical hardware if need be for expansion and growth, they just run minimal services, such as no physics, no lighting or anything using a customized player loop within the particular Unity instance, so it would still be running within Unity as intended.

As for saving and loading of data, essentially what I was referring to was it would be great to have an instance of Unity running an in memory db which would be the "state server". My game is an RPG, so say someone logs on, ideally when they log in and select a character, it would tell the state server to load up that particular players data, the way I envision it working is the state server would connect to whatever its "cold" or "inactive" data source happened to be and pulls all the user details for that particular user one time and its then there in memory, as the player plays all interaction is done with the state server only so as not to be making time-consuming database queries throughout the play session if they get a new item or a stat changes, etc, in the event that there are hundreds (or possibly thousands) of players. Periodically the state server would take all recorded changes for all players and save that back to whatever the "inactive" data storage happened to be, which it sounds like in this case is just saved locally, which may not be a bad thing, but ideally in my case it would be nice if the inactive and non in use data was saved outside of Unity so say, I have a website with a user account page that displays a users stats similar to something like WoW Armory, once the user logs off, the in-memory data saves a copy to the inactive (or I guess, cold storage?) so it can be used for other purposes but also backed up / replicated in a more enterprise fashion.

While I have not looked at it yet, so I can't say if it would still work for what I am after. This is more of an "in an ideal world I would like..." type of thing. I like the idea of the inactive data being housed in a place that can be backed up and replicated using well known and reliable tools. So if BGD was able to grab data from an outside source in some way as needed (when a character logs in) even if it kept its own copy locally that it used as needed for its operation, then was able to periodically save its current state to inactive storage (even if that is locally) then that data that is saved locally within the Unity instance was able to be exported to another storage medium, or something along those lines, that would be awesome.

Again, I am mostly just throwing out what I think would be useful, lol. So if it has no ability to do anything of the sort, that is ok, I am also making a single player game currently in which I am getting to a point in which I would like to start saving data, so it could very well be useful for that as well. : D I just know I hate the idea of using scriptable objects as an actual database, it just feels so, unobtainable I guess, something like a "black box" in which if something goes wrong you might be out of luck. So anything, even if for peace of mind, ends up being better in my eyes.

 

@MostHated , your idea makes sense to me, but BGDatabase cannot be used in this scenario- even if it can be run technically, it was not designed for this.
I personally don't know any good alternative to cloud database (or some cache server) which is provided as a service and some service on top of it. You'll get
1) Guaranteed high uptime >99.9%
2) Auto upscaling
3) almost zero maintenance cost
I think all these criteria are very crucial to any server and this is not what BGDatabase can provide.
You still can use it on players devices as a replacement to scriptable objects, but not as a replacement to a cloud database, for sure - its completely different type of software.

 

The tool is very rigorous, but it is too painful to manually create each field, and a method that can quickly create a large number of fields is strongly needed. I am a SLG gamer, and we have an unimaginable number of fields. Manual setup takes a lot of time.

 

or you can provide a way for us to build meta from Googlesheets，it must be great！ This is the normal workflow of mark hight numerical games.

 

@Tonyzhai , we'll add support for the scripts in the next version.
I've sent you a prototype in private message.
Please, let me know if something is missing.

Regarding pulling data from GoogleSheets:
Unfortunately, information about types of the fields is missing in Google Sheets.
We could retrieve field names from GoogleSheets, but you still have to fill in fields types.
If it's more convenient for you, please, let me know, we'll add such option.

 

it's cool. very thanks, and i have another question.
why not to be able to use "_" as FieldName. I will work with Python, I often use Python pandas for the analysis of the game number data , I usually use format like "STR_heroName" .So I'm in python can easily call and identifying type.
No matter what, is "_" is commonly used, hope to be able to support.

 

@Tonyzhai I've sent you an updated package with support of underscores in private message.

 

@BansheeGz ,

Happy New Year ! Just to say, really loving this Asset - so flexible and so very useful.

Have left a hopefully nice review on the Store.

In terms of enhancements, I've had a few ideas for you:-

* it would be handy if you could move rows around, reorder them, as you can with List elements - when both tinkering with table schema/meta and when actually tinkering with table rows of data.

* it would be neat if you could quickly re-order rows (both meta/schema and table/data) by clicking on the darker grey column names at the top to sort A-Z or Z-A?

* insert a row above or below? (as opposed to adding a row at the end only)?

* using configuration (for large tables) the ability to hide and unhide selected columns...?

* conversely it would be useful to always keep on the left the 'name' column (optionally - configuration) when scrolling to the right to see other columns in a data table or even meta/schema?

* perhaps an edit box to go to a particular row in a table?

* also, above tables and metas, an indication of the total number of rows of data?

Just some ideas to enhance productivity really.

 

P.S. when I'm done with some BG Database design, I think I am going to look at an integration with GameCreator by Catsoft...

 

P.P.S. - when you create a Relationship-Single, there should be the option to have a a selection from the very same table.

* So imagine you had a table of "Elements" - and you created Fire and Ice. One field/column in the meta/schema could be say "Opposing Elements" - and the Relationship - Single should be able to refer to the table "Elements" as well as all the other ones - so the "Opposing Element" to Fire would be... Ice (and vice versa) - I hope that makes sense....

 

@Duffer123 ,
Happy New Year!
Thank you so much for your support, suggestions and review!
I think all your ideas make perfect sense- I scheduled the changes to version 1.3.4 (version 1.3.3 is already in the Store under review)
I will also send you a preview package as soon as it's ready just in case you might be interested in reviewing it before release.

We were thinking of developing out own assets like inventory/crafting, quests/dialogs, character progression on top of the database for quite a while.
I think using database (with save/load support, GoogleSheets/Excel sync, localization etc. ) as Data Layer would be great synergy for such type of assets.
But at the end of the day, we are not sure if it really worth it.

 

@BansheeGz ,

Excellent thanks (re enhancements).

I certainly wouldn't discourage you developing an RPG Kit (or RPG Kit elements) Assets resting on this Asset. Makes good sense really.

 

@BansheeGz ,

Another idea for an enhancement/improvement - in terms of the database I'm not sure if it's just the byte file or the byte file and the settings json file? Either way, it would be good to have a backup folder/files so that when you update to latest version of this Asset you don't (as I just did - Doh!) overwrite all your database work. (it wasn't much, thankfully)...

Some way to hive off earlier/other versions of the database and settings within the overall project and then pull them back in, in the editor?

 

@BansheeGz ,

Stupid question but is there a field for a List of a Unity Assets. Of particular use would be a field for a List of Resource folder Sprites ... and/or... Resource folder Prefabs ... and/or... Scene GameObjects?

 

@BansheeGz ,

Also wondering about support for Dictionaries? So one of the potential fields/variables with another tied to it - or could you achieve this with lots of mini nested tables? Keys and values.

really useful in games...

 

@Duffer123
Sorry for late response, I did not receive any notifications from Unity forums by some reason, I have no idea why

Do you use the default database file for your own data?
If yes, please, create Resources folder somewhere in your project, copy Assets\BansheeGz\BGDatabase\Examples\Resources\bansheegz_database_default.bytes file to this folder using FileExplorer (copy, do not move it) and rename it to bansheegz_database.bytes
This step will create a new file, which would not be overridden
Or, alternatively, follow the steps, which are described here: http://www.bansheegz.com/BGDatabase/Setup/
You can copy database content by coping database file content.

Currently, there is no "assets list" field. How such field can be used?
I attached little utility, which can scan Resources folder and insert new rows for each found asset if this asset is not already present in the table.
Not sure if it can be helpful
1) Place it under Editor folder
2) In the top menu chose Tools->Asset2Row
3) Chose table, "Unity asset" field and the folder with your assets and click Run
4) Review the changes, click "Save" to commit or "Reload" to revert the changes.

We honestly not sure how to implement it.
We don't support generic fields, all fields have its own predefined type for its value.
And dictionary has 2 generic type parameters(for key and for value). It means we need to create a separate field for each combination and I do not think it's a good solution cause there are a lot of such combinations. And even if we support generic fields, we'd need to use reflection to construct the values, which is super-slow.
We could try to use HashTable instead of Dictionary but it introduces new problems, like not type-safe code, producing garbage due to boxing/unboxing. So I'm not sure how we could do it properly.
Yes, a nested table can mimic the dictionary field. You can iterate all nested records and construct your own dictionary with any key/value

 

@BansheeGz ,

Thanks for the responses above. All makes sense. Figured out new database save thing as well.

Looking forward to implementation of those other things though, particularly moving and inserting rows and being able to reference same table in relationships...

 

@Duffer123 , I've sent you a beta version in private message.
Any feedback would be greatly appreciated

 

Ok. Will trial it asap.

 

Hi, BansheeGz,
I just ran into BG Database and it's quite unique and I would like to ask few questions.

This is my usecase and please let me know it can work.
Basically, I'm looking for a solution that I do most of the parameter tuning work external from Unity and hopefully, it can work seamlessly.

Why work externally? Because of the two reasons.
1. I'm working on a shooter game that needs quite a lot of tunings but Unity Editor is really slow and hard to scroll up/down to find the parameters I want.

2. I'll have to have our parter to fine tuning the parameters, so the tuning must work externally on the built-client, preferably from Excel or GS(Google Spreadsheet)

So, here are my questions.

1. Does the export/import from/to Excel and GSS works seamlessly both from Editor and built-client? I don't need export capability from the built-client though. What happens when there are new parameters are added/deleted or the name is modified?

2. When Excel or GSS updated, will the changes are applied at runtime just like changing the from Inspector? I saw the Event and Binding from the documentation but it looks like it requries a setup for each properties. During the development, I want all of the properties are bounded and updated at runtime without adding all properties are added individually for the conviniences. Is this possible?

Thanks for the help and I'm looking forwad to work with you.

Cheers!

 

@chrisk ,
Hi, thank you for your interest in our asset!
Yes, probably, it may suit your needs
I sent you a demo version with some comments in the private message
Please, try it and let me know if you have any questions

1) Full export/import support from Unity Editor (while game development)
2) No way to export/import to/from Excel from built-clients
3) No export to GSS from built-clients, but you can import from GSS to built-clients with LiveUpdate plugin (this is probably what you are looking for) http://www.bansheegz.com/BGDatabase/Addons/LiveUpdate/

Data will be updated/added/deleted according to the settings

You will need to trigger LiveUpdate plugin to reload the data from GSS (for example with button click). No need to restart the game

Normally, if you want some properties/fields to be bound to database values, you'd need to attach data binders for each such property.
But if you have a lot of properties- probably it would be easier(and faster) to write your own C# script which would inject the values from database to the fields

 

Hi, BansheeGz, thanks for the answer and for letting me try out BGDatabase.
I'll let you know when I have more question.

Replacing Inspector with Excel or GS as the main property editor and having Unity properties to update in realtime is the main focus.

Having said that what you said below is interesting. However, adding/modifying properties are pretty frequent and I'm wondering if there is automate the binding though C# reflection or something. It will help a great deal to write wrapper every time there are changes. Please let me know if that's feasible. Thanks.

Regarding

 

@chrisk ,

1) There should not be any problem with updating parameters. Once databinder is set up- you can change the parameter's name and value- it will work ok. Parameter (row) is resolved by unique id- so you can freely change its name.

2) Adding/removing parameters (rows) takes some additional action.
Currently, there are 3 options:
2.1 Use field binder (the most simple and straightforward way). It injects a single value from the database to another component. It works fine with prefabs.
2.2 Use batch binder. It injects multiple values from database to multiple components and additionally allows to define different GameObjects. It also works fine with prefabs as soon as you do not reference any GameObject outside of prefab. By the way, both binders already use reflection to resolve target field.
2.3 Use custom C# code. With this option, you can code anything you want (probably).

For example, let's say you store full path to GameObject+Component+field( based on names) along with your parameter, something like this: MyRoot/MyGameObject~MyComponent.MyField
In this case, you do not need any binder, cause you can parse the path, resolve GameObject and inject the value to its component.
Also you can add these parameters dynamically outside of Unity and have them injected without restarting the game.
However, I do not think it makes much sense, cause:
1) you can not change GameObjects names and structure
2) you need to maintain the same unique structure across all your scenes, which uses these parameters
3) I do not think that adding this path would be easier than adding a databinder

If you have any suggestions about how this could be set up, please, let me know.

Also, there is an option to not inject any value from the database but use database values directly from your script instead.

 

@BansheeGz , what variable/field types can I put in the hashtable keys and/or values?

 

Following types are supported as keys: int, string, bool, Guid, custom fields.
Following types are not supported as values: Unity assets, relations, hashtable, nullable structs. Null values are also not supported.

 

@BansheeGz , I can't seem to find a way to run a filter at runtime on the database I have set up ( 2400 entries, my goal is to be able to run a filter at runtime?
Im using PlayMaker, so my thoughts were to do a set property on the entity field of the BGEntityGo script like you would in the BG Database.
Seems like the format required to filter in the BGEntityGo component is different than the one that you build in the BG Database window? I havent been able to get one that works.
Second question, is it possible to set a filter at runtime? Doesnt seem to be a public property I can reach through set property.
There's probably an easier way to do what I need it to though?

P.S: What I ultimately would like to do is cycle through all the results of the query and store fields into an fsm variable.