Search Unity

Large Terrain Problem - how to recreate this in Unity?

Discussion in 'World Building' started by ghtx1138, Jan 5, 2019.

  1. ghtx1138

    ghtx1138

    Joined:
    Dec 11, 2017
    Posts:
    113
    This shot is from Google Earth - 13,000 m high looking across the English Channel.

    Can I do this in real world scale in Unity 2018.3? I'm not making a game but a movie. I'm not planning to look at detailed trees - a textured terrain would be fine. But I'd like to be able to move around this area and look at aeroplanes from various angles and see this sort of image in the background.

    I've experimented with large planes 200,000 m square but run into all sort of weird culling problems which I cannot solve.

    If you could point me in the direction of someone who has done this with some tutorials that would be great.

    LargeTerrainProblem.png


    Fewes has done something like this in Unity but I can't find any tutorials on this sort of thing



    Please let me know if I can provide further info

    Thanks
     
  2. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    3,620
    If you need a tutorial just to start you off with it, you really should not do it. You should do smaller things and learn how to do stuff.

    There are many techniques, you can use impostors to fake the stuff at the distance (create low res meshes), you can use simulated stuff if the player can't actually travel there. Or you can use Worldstreamer or Sectr to load/unload pieces of terrain when you need it. And probably you will need to combine these techniques. Plus it's good if you familiarize yourself with the floating origin technique as well.
     
    Last edited: Jan 6, 2019
  3. ghtx1138

    ghtx1138

    Joined:
    Dec 11, 2017
    Posts:
    113
    @Lurking-Ninja thanks for your reply.

    I didn't realise how much smoke and mirrors was involved and thought it might be straightforward and that someone might have made their techniques available. Apparently not.

    I've done some reading in the past few hours and see now that it is definitely not straightforward. I can't have infinite clipping planes cos that causes floating point problems, most streamers are for onground walking/runners not 13 km up in the air where you can see hundreds of kms at once, looks like LODs/imposters are going to be part of the solution.

    Oh well.

    Thanks for the floating origin tip.

    Cheers
     
  4. ghtx1138

    ghtx1138

    Joined:
    Dec 11, 2017
    Posts:
    113
    I hope it is OK for me to put my notes here. If not please let me know mods.

    So it appears there is no one stop tutorial for solving the "Continuous World Problem" so I am going to put my findings here in case others may benefit. My goal is get something working and put the scripts here for the Unity community (if time and my skills permits). My particular interest is in using GIS/DEM data to sim real world terrains.

    We start with this talk from Unite 2013 where Robert Oates talks about GIS Terrain & Unity


    My takeaways from this talk:

    Solving the Continuous World Problem is not easy (who knew?) and the solution is a work in progress.

    Problem: Memory Limits
    Finite memory, lots of content
    Height maps
    Render meshes for terrain and other objects
    Collision structures
    Terrain textures
    GameObjects - agents, props
    Scripts on GameObjects

    Problem: CPU/GPU Limits
    Finite CPU, lots of systems
    AI/ Behaviour - increases complexity by #objects^2
    Terrain object collision every frame
    Object object collision every frame
    Animation
    Scripts
    Culling and Rendering

    Problem: Numerical precision limits
    Transforms use IEEE 754 float
    Have gaps between the 2^32 values on a 2^129 domain
    Least error near 0
    Most error near +/- float.MaxValue
    Numerical instability far from origin
    Visible symptoms at 60 fps
    @ 100,000m float errors are 0.0078m componded up the transform stack at 60 fps
    at 20-40,000 it will look really bad

    Solution: Streaming terrain in any direction
    fundamental ideas - Stream terrain around frustum
    moving tiles
    terrain built at runtime not serialised in scene, dynamically constructed
    database used to persist tiles and objects and stuff
    queried at runtime

    recycle terrain objects
    keep frustum in centre/ keep tiles near origin
    persist/retrieve objects
    stream imagery and heightmaps
    numerically stable
    constant CPU/memory load

    Method:
    generate a n^2 terrain mosiac
    store for later use
    center on origin

    each tile has lat/lon independent of world space
    SW corner is origin with lat/lon

    each tile gets imagery splat texture from disk, cache, remote server
    each tile gets heightmap data
    if tile moves clear imagery

    Database Persistance
    lat/lon coordinate
    orientation?
    query database by lat/lon bounds

    Implementation (2013)
    "good computer" 17 X 17 terrain tiles
    10km X 10 km loaded at once

    Optimisation techniques (2013)
    reduce number of tiles loaded in memory at once
    distribute tile loading across multiple frames using coroutines
    recuse size of tiles
    reduce fidelity of heightmaps or elevation data
     
    Lurking-Ninja and hippocoder like this.
  5. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,218
    Unity's solving this at the same time as you are, for what it's worth: terrain data streaming, ECS (see megacity demo) and so on. If you were going for this, I would be looking at just seeing where ECS takes you as you'll be able to move huge chunks of data around without it taking any time to do so.
     
    Marcos-Elias likes this.
  6. ghtx1138

    ghtx1138

    Joined:
    Dec 11, 2017
    Posts:
    113
    Thanks for commenting @hippocoder

    "Unity's solving this at the same time as you are" lol :rolleyes: - well I guess we'll see who gets there quicker!

    ECS is on my list of things to learn about - several people in my googling have mentioned it.

    Will check out megacity - cheers!

    <edit>Megacity Demo
     
    hippocoder likes this.
  7. ghtx1138

    ghtx1138

    Joined:
    Dec 11, 2017
    Posts:
    113
    Here are some scripts to illustrate the basic concept of "streaming terrain chunks".

    We have 9 terrain tiles with random noise heightmaps and as the camera moves the tiles behind us are recycled and moved to in front of us.

    I've positioned the camera to highlight how the chunking works. It is the same 9 tiles over and over again.

    GifCapture-201901081148219386.gif

    The UpdateTerrainPositionsAndNeighbors() method is the core function here. Note that SetNeighbors is called every time the UpdateTerrainPositionsAndNeighbors() method is called - this makes sense because a tile may have new neighbours when it is recycled.

    The main Update method has two objectives - keep the player on the central tile and keep the player from moving beyond the outer bounds of the 9 tiles.

    Instructions
    Create Terrain in Scene
    Create Terrain Layer, add grass texture
    Attach Extended Flycam script to Camera
    Attach InfiniteTerrain.cs to Terrain object, drag Camera to Player Object slot in script, drag New Terrain Layer to Layer slot in script

    References & Credits
    https://github.com/Meowmaritus/Unity_SpaceGame/blob/master/Assets/Scripts/InfiniteTerrain.cs
    https://www.reddit.com/r/Unity3D/comments/2q4ghg/procedural_terrain_generation/
    http://wiki.unity3d.com/index.php/FlyCam_Extended
    https://forum.unity.com/threads/terrain-layers-api-can-you-tell-me-the-starting-point.606019/
    Texture from Terrain Toolkit
    https://assetstore.unity.com/packages/tools/terrain/terrain-toolkit-2017-83490
     

    Attached Files:

  8. ghtx1138

    ghtx1138

    Joined:
    Dec 11, 2017
    Posts:
    113
    Getting real world elevation data/ height maps into Unity - Show me the data

    Note: This information is presented as a summary of stuff I could find in a couple of hours – it is not an exhaustive survey of all resources and providers. If you know of alternatives please let me know.

    Earth is not a perfect sphere, mapping real-world elevations is not a simple exercise.

    A location on earth’s surface can be referenced with a lat/lon coordinate. This Wikipedia page has a great explanations of lat/lon decimals degrees. A 8 decimal place coordinate would map to a mm – so for our purposes a 4 decimal place lat/lon coordinate will probably suffice.

    Several mapping systems/standards/projections exist - WGS84, Google Mercantor, etc. It is important to know what projection standard the dataset is using because it may impact how the data you acquire will display in Unity.

    There are many real world elevation data providers. We’ll look here at NASA/USGS, Microsoft Bing and terrain.party.

    It is worth noting that depending on your use case higher resolution elevation data might be available for your area of interest – maybe a LIDAR survey. For example ESRI has a nice post about higher resolution (1m in some cases) heightmap data of for particular locations.

    It appears that the Shuttle Radar Topography Mission SRTM in 2000 is the baseline for publicly available real world terrain data/ heightmaps. NASA flew the Shuttle over the earth in 2000 and bounced/measured non-lethal radar beams to get elevation. The first public release was at 90m resolution (STRM 90) and a later release of the same data was at 30m resolution (aka STRM 30, SRTM Version 3.0, SRTM Plus). Bing Maps has 10m resolution for the US but has a 1024 sampling limit (which I have not yet understood with regard to Unity tiles)

    Not all of the earth was mapped in the STRM 2000 mission, the polar regions, for example were not in the release. Derek Watkins has nice visualisations of the data here and here. The STRM data is also noisy and has voids – do not expect perfection. The accuracy is also questioned by some GIS practitioners. Nonetheless it is an incredible achievement and a great resource for Unity developers wishing to get the real world elevation into a Unity project.

    A number of providers including terrain.party and CGIAR have merged other elevation datasets with STRM to address the problems of voids and accuracy. This is obviously great news but could potentially lead to issues when you are asked “Where is this particular datum from?”. Personally I'd be using the CGIAR dataset if I need something canonical. You can read their FAQ here. Be sure to understand the licensing agreements and any costs associated with using the data provider. Also ensure that you get your to understand the licensing agreements with regard to commercial/non-commercial use of derived products.

    The 14,278 .hgt files for planet Earth are available as from NASA EarthData (registration required).

    Each .hgt file represents a 1X1 arc second degree snapshot of the underlying terrain. At the equator this equates to approximately 100x100 kilometres per tile. As we approach the poles the real world dimensions of the tile change significantly.

    The 16bit RAW file is a 2d array with the centre of each pixel representing the averaged elevation over a 30/90m window of a real world lat/lon location.

    Here's a pic from CGIAR illustrating how the SRTM pixels should be interpreted.


    Source: http://srtm.csi.cgiar.org/faq/
     
    hippocoder likes this.
  9. ghtx1138

    ghtx1138

    Joined:
    Dec 11, 2017
    Posts:
    113
    These videos show SRTM data for Earth in Unity 5 from Technische Universität Darmstadt. Apparently 122GB of data - "Calculating all terrains from SRTM and Blue Marble raw data takes round about 7 days"



     
    hippocoder likes this.
  10. ghtx1138

    ghtx1138

    Joined:
    Dec 11, 2017
    Posts:
    113
    Just adding these 2 notes for future reference.

    http://ninetwopro.com/ obviously not Unity but an open source (and textured) world terrain dataset. Not sure if can convert from MBL format to something that can be imported into Unity

    NinjaFreede's Infinity Zoom in Unity. The technique is very interesting: "It's all fake. 5 cameras are set to show one layer each (galaxy, stars, planets, ships) with synced rotation looking at the same target. Each layer "scene" is below 1000 units so no floating point issues. When you mouse wheel, the cameras move in and out in order at different speeds." via Reddit

     
    Rowlan, Adam-Bailey and hippocoder like this.
  11. iichiversii

    iichiversii

    Joined:
    Nov 23, 2011
    Posts:
    104
    Oddly enough im trying to figure this out also, I just want to make a virtual Ireland, not using satellite imagery but apply rules for texturing different heights, then use easy roads with the aid of OSM data to create roads, im stuck at rock bottom however and haven't made any progress as of yet
     
  12. iichiversii

    iichiversii

    Joined:
    Nov 23, 2011
    Posts:
    104
  13. iichiversii

    iichiversii

    Joined:
    Nov 23, 2011
    Posts:
    104
    And you might find this of interest
     
  14. ghtx1138

    ghtx1138

    Joined:
    Dec 11, 2017
    Posts:
    113
    hey @iichiversii thanks for this information. I've put this on hold/ moved on/ given up and am trying simpler stuff in 2D animation.

    When I started this thread I mistakenly thought I could simply generate a big mesh and then run around in Unity. That is not the case - like I say in the posts above there is a lot of smoke and mirrors/ kludges/ workarounds/ creative coding etc to get this effect. And these aren't limits/ problems with the Unity Engine but rather problems like floating point math and our current CPU/GPU architectures or the unsightly terrain topologies using NASA data.

    I ended up purchasing a real world terrain package from the Asset store but that had issues with placing items in real world locations and keeping accurate location at different scales of views. The developer was very helpful but again this was a limit of external third party terrain data and not a problem with Unity or the package.

    If I may give you some advice (and please excuse me if you are an experienced developer and know this already) - start small. Like really small. And then cut that goal into 3 easy pieces and tackle each piece separately. Solve that one problem and move on to the next. Personally I like to script/ automate my workflow so I know I can repeat it - and the cool thing I just did was not just a product of late nights and cold beer.

    Here are a couple of links that might help:
    OSM projects on UnityList https://unitylist.com/browse?search=OSM
    Openworld workflow https://forum.unity.com/threads/realistic-openworld-workflow.593788/

    this looks new and might give you some ideas https://github.com/Microsoft/MapsSDK-Unity

    best of luck and Cheers!
     
    iichiversii likes this.
  15. Adam-Bailey

    Adam-Bailey

    Joined:
    Feb 17, 2015
    Posts:
    226
    :D :D :D

    That's the type of creative solution I love.
     
  16. CityGen3D

    CityGen3D

    Joined:
    Nov 23, 2012
    Posts:
    164
    Hi,

    I came across this thread and just thought I'd mention that I'm working on CityGen3D, which is a real world terrain generator that auto heightmaps and textures terrains in just a few button clicks all within Unity Editor.

    It can also generate roads and spawn other prefabs automatically based on OSM data, as well as having a tree and detail placement system.

    More info at www.citygen3d.com and on work-in-progress thread here:
    https://forum.unity.com/threads/bet...cedural-city-generation-from-map-data.514677/

    It's currenty in Beta so not all the issues mentioned in this thread have been solved yet (re world streaming, etc), but it's already quite a powerful tool.

    Hopefully on Asset Store later this year but Beta already available via website.