Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Displaying big dataset

Discussion in 'Scripting' started by TheNick1140, Jun 16, 2019.

  1. TheNick1140

    TheNick1140

    Joined:
    Mar 1, 2014
    Posts:
    2
    Hey guys, I need your help.

    I have a large dataset of about 180k in a .csv and want to display every one of that 180k as a sphere (with different radius, color, etc..). The player should be able to fly through the 3D space freely and investigate the spheres.

    I know how to read the .csv into Unity, but what is the best approach to display such big data?

    • I was thinking of generating a script to generate random (not overlapping) positions for every sphere around my starting point and then maybe write that to some file, so that I just have to do that once and can read the positions from that file. If that is a good idea, what is the best way to accomplish this?
    • What are the best ways in regard of perfomance?Just load the spheres, if they are in the players range (collider with certain radius) and otherwise deactivate them?
    Looking forward to your help.
     
  2. SparrowGS

    SparrowGS

    Joined:
    Apr 6, 2017
    Posts:
    2,536
    I would draw all of them together with one of the DrawMesh variations for the visual aspect.

    For the colliders id only keep what is near the player (say 10 units)active to save all the physics overhead
     
  3. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,188
    I don't have a recipe at hand, but maybe this can send you in the right direction.

    Instead of generating random positions, you can look into "spatial partitioning" algorithms to create random non-overlapping positions. It makes sense to cache this data in a file, that is true, either at edit time in the Unity editor or if it has to happen at runtime, at least as a file in the Unity player data folder (if the platform supports writing to disk).

    Regarding performance, there are several things to consider. You probably cannot load all spheres into memory, but it should be enough to only load the ones which are visible and destroy them if they go out of view. I wouldn't recommend using colliders. The physics system has too much overhead for this kind of task, I think. You can use manual culling (simply checking the distance or if they are within a box etc) to cull or use the CullingGroup API from Unity to cull spheres behind other objects. To speed up rendering, you can try instancing (they all share the same mesh probably). You may need to dive into material properties and shaders to use vertex colors for the instances for example. Finally, I would say this could also be a case for the new DOTS approach, using the entity component system or at least the Unity job system.
     
    TheNick1140 likes this.