Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

BlobAsset memory leak help

Discussion in 'Entity Component System' started by xindexer2, Aug 30, 2021.

  1. xindexer2

    xindexer2

    Joined:
    Nov 30, 2013
    Posts:
    78
    I have a large immutable data set that I'm working with (financial data). With the help of Coffee Brain Games blog (https://coffeebraingames.wordpress.com/2020/11/29/getting-started-with-blob-asset/) I have successfully created a blobAsset to store the data. However, when I went to create more than one entity, I created a memory leak that I'm not sure how to deal with. Here's my code (simplified)

    Code (CSharp):
    1. namespace Models
    2. {
    3.     [Serializable]
    4.     public struct IrisContainer
    5.     {
    6.         public IrisOption[] optionContainer;
    7.     }
    8.  
    9.     [Serializable]
    10.     public struct IrisOption
    11.     {
    12.         public string ticker;
    13.         public int[] expireDates;
    14.     }
    15. }
    16.  
    17.  
    18. public struct IrisData
    19. {
    20.     public int tradeDate;
    21.     public BlobArray<int> expireDates;
    22. }
    23. public struct Iris : IComponentData
    24. {
    25.     public FixedString32 ticker;
    26.     public BlobAssetReference<NativeHashMap<int, BlobAssetReference<IrisData>>> irisMapReference;
    27.  
    28.     public Iris(FixedString32 ticker, BlobAssetReference<NativeHashMap<int, BlobAssetReference<IrisData>>> irisMapReference)
    29.     {
    30.         this.ticker = ticker;
    31.         this.irisMapReference = irisMapReference;
    32.     }
    33. }
    34.  
    35. protected override void OnCreate()
    36. {
    37.     var loadJson = File.ReadAllText("Assets/Data/Options_2021_08_27.json");
    38.     var irisContainer = JsonUtility.FromJson<IrisContainer>(loadJson);
    39.     for (var i = 0; i < irisContainer.optionContainer.Length; i++)
    40.     {
    41.         irisMap = new NativeHashMap<int, BlobAssetReference<IrisData>>(1, Allocator.Persistent);
    42.         using var blobBuilder = new BlobBuilder(Allocator.Temp);
    43.         ref var irisData = ref blobBuilder.ConstructRoot<IrisData>();
    44.         var expireDates =
    45.             blobBuilder.Allocate(ref irisData.expireDates, irisContainer.optionContainer[i].expireDates.Length);
    46.         for (var j = 0; j < irisContainer.optionContainer[i].expireDates.Length; j++)
    47.         {
    48.             expireDates[j] = irisContainer.optionContainer[i].expireDates[j];
    49.         }
    50.         assetReference = blobBuilder.CreateBlobAssetReference<IrisData>(Allocator.Temp);
    51.         irisMap[irisContainer.optionContainer[i].tradeDate] = assetReference;
    52.         using var builder = new BlobBuilder(Allocator.Temp);
    53.         ref var blobData = ref builder.ConstructRoot<NativeHashMap<int, BlobAssetReference<IrisData>>>();
    54.         blobData = irisMap;
    55.         irisMapReference =
    56.             builder.CreateBlobAssetReference<NativeHashMap<int, BlobAssetReference<IrisData>>>(
    57.                 Allocator.Temp);
    58.         var irisEntity = EntityManager.CreateEntity(typeof(Iris), typeof(Storage));
    59.         EntityManager.SetComponentData(irisEntity, new Iris(irisContainer.optionContainer[i].ticker, irisMapReference));
    60.     }
    61. }
    Is this a good approach to storing my data? I looked at dynamicBuffers but I don't need to change the data once I have it loaded. I also put it into a scriptable object which works but the sheer volume of data makes that solution challenging. My current implementation is to just load the data into an ISystemBase class variable, but then it becomes difficult to access it from jobs and breaks the design goals of ECS. I have installed @WAYN_Games github package (https://github.com/WAYN-Games/Blob) but I can't figure out how to use it or if it's even a good fit for what I'm trying to do.

    This code does store the data and I'm able to retrieve it through a Entities.Foreach loop but I'm getting this error:
    A Native Collection has not been disposed, resulting in a memory leak. Enable Full StackTraces to get more details.

    I think this is coming from the Allocator.Persistent on the IrisMap but if I make that Temp then I can't access the irisMapReference data in my Entities.Foreach Loop. Where do I dispose the irisMap? How do I access it in the OnDestroy method? (this is where I disposed of it when I only had one entity).

    I'm really at my edge of understanding with these blobAssets, but I think I'm close to figuring it out.

    Thanks
     
    Last edited: Aug 30, 2021
  2. WAYNGames

    WAYNGames

    Joined:
    Mar 16, 2019
    Posts:
    988
    Hello,

    Sorry for not answering to the git hub issue, I was not notified for some reason...
    If you are looking for exemples I can try to add some to the read me, in the mean time you can look at the tests it's not really documentation but maybe it will help you understand how to works with it.

    For you particular issue I would suggest running you code without burst and with the full stack trace enabled it will probably give you more insight on where the leak comes from.

    If I get what you are trying to get is to have an entity that for a ticker value has a map with a trade date as key (not sure where it comes from in your code) and a list of expiracy dates so something like BlobAssetReference<BlobMap<int,BlobArray<int>>> correct ?

    To be honest I'll have to dive back in my own code a bit to provide you with more help, I just want to make sure I understand your problem/project first.
     
  3. xindexer2

    xindexer2

    Joined:
    Nov 30, 2013
    Posts:
    78
    My intent is to create one entity for each option symbol (AAPL, TLSA, etc) and attach the option data to it. My hashmap key is the tradedate (Unix time stamp that is loaded via JSON) and the data is the IrisData class (there is a lot more data in there, I just removed most of it for the post). I’m currently using BlobAssetReference<NativeHashMap<int, BlobAssetReference<IrisData>>> ( couldn’t figure out how to add data to a blobarray and then assign that to the hashmap, so im using the reference)

    On startup I’m loading 10 stocks and creating 10 entities with their associated data attached in the hashmap for that day. Once a user selects the option symbol then I run my jobs off of the blob data and create a chart. I will also start another download for historical data for that stock symbol. Ie download 30 more days worth of data and then store that in the hashmap using the Unix time stamp tradedate as the keys. So I will need to be able to add more data to the hashmap as it is downloaded( though I think I will have to copy and destroy instead of add)
     
  4. xindexer2

    xindexer2

    Joined:
    Nov 30, 2013
    Posts:
    78
    I ran the full stack trace and it is the irisMap that is causing the error. I don't know how to dispose of it properly. If I dispose of it in the loop then I can't reference the in the blobasset, if I don't dispose of it in the loop then I get the leak warning.
     
  5. xindexer2

    xindexer2

    Joined:
    Nov 30, 2013
    Posts:
    78
    ok - I did a bunch of experimentation and found a solution. I don't really understand it but it is working as I want it to now. I had to create an array for the irisMap and then dispose of them individually onDestroy.

    Here are the pertinent lines:

    Code (CSharp):
    1.      
    2. private NativeHashMap<int, BlobAssetReference<IrisData>>[] irisMap;
    3.  
    4. (OnCreate)
    5. irisMap = new NativeHashMap<int, BlobAssetReference<IrisData>>[irisContainer.optionContainer.Length];
    6.  
    7. (in loop)
    8. irisMap[i] = new NativeHashMap<int, BlobAssetReference<IrisData>>(1, Allocator.Persistent);
    9. blobData = irisMap[i];
    10.  
    11. protected override void OnDestroy()
    12.         {
    13.             foreach (var map in irisMap)
    14.             {
    15.                 map.Dispose();
    16.             }
    17.         }
    18.  
    why does irisMap need to be individually referenced while everything else is fine in a loop?
     
  6. WAYNGames

    WAYNGames

    Joined:
    Mar 16, 2019
    Posts:
    988
    Where were you disposing of it in the loop ? Last instruction or before assigning the blob asset reference to your entity.
    If you dispose of it with your solution do you still have access to the data after that ?
    The leak warning is happening while in play mode or when you exit ?

    Also if the data needs to change blob asset is not you best option as their intent is to be imputable. You will have to make a new blob and swap the reference on the entity. If it does not happen often it's ok. Otherwise nested unsafe container may be a better fit but you would have no thread safety features.
     
  7. xindexer2

    xindexer2

    Joined:
    Nov 30, 2013
    Posts:
    78
    I was disposing of it at the end of the loop (last line) but if I did that, then the blobReferenceArray would lose its data. The current implementation is working just fine. The system needs to have independent versions of irisMap in order to function properly. Creating the array works - is it the right answer? I have no idea but it's working.I have never had to deal with memory directly before, which is making this challenging.

    The leak warning is happening in play mode - I would get 9 warnings (for a loop of 10) - the first one was fine. Which is what lead me to the array solution.

    The data that I'm using is immutable, but I will need to add to the hashmap just once to get the historical data in. I still have to figure out how to do this, but I know I will have to copy the data out and create a new blob once the historical data has been downloaded.

    I have not looked into nested unsafe containers yet. Once the data is in, I will never need to modify it so it "should" be thread safe. Again - right at the edge of my ability.