Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Dynamic 3D world rendering using offline OpenStreetMap data

Discussion in 'Works In Progress' started by eiskalt, Nov 23, 2014.

  1. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    I would like to share some progress on my open source project: UtyMap. This is a library which provides highly customizable API for procedural world generation based on real map data, e.g. OpenStreetMap, NaturalEarth. Core logic is written on C++11 and can be used on many platforms as it has no dependency to specific game engine or application framework. Demo application is built using Unity3D and it provides examples with interactive world creation at different zoom levels, including globe and ground:



    Full source code is available here: https://github.com/reinterpretcat/utymap
    Latest release binaries: https://github.com/reinterpretcat/utymap/releases/tag/ver1.1
    Released on Google Play: https://play.google.com/store/apps/details?id=com.utymap.demo

    Update from July 2016:
    Initial post referred to ActionStreetMap project which is now deprecated and not supported, but full source code is available.
     
    Last edited: Oct 8, 2016
    SAOTA, John-G and Kirbyrawr like this.
  2. Kirbyrawr

    Kirbyrawr

    Joined:
    Jul 23, 2012
    Posts:
    935
    It looks really cool, nice work!
    Maybe i can help you a bit with the shaders.
     
  3. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Sure, pull requests are welcomed.
     
  4. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Some of upcoming changes inside ASM framework:
    • Reimplemented logic of processing map data: use custom spatial index inside and no need to use splitter tool.
    • Support import of o5m, pbf and xml formats.
    • Reimplemented logic of interpolating height map for elevation mode: it's more accurate now (bilinear interpolation is used).
    • Added support flat map mode: rendering is faster in ~30% and no need in elevation data.
    • Fixed issues with missing elements crossing tile borders: limitation of old map data processing logic.
    • Added simple search for elements with given tag's key/value pair: you can find the nearest bars, restaurants, other places, features.
    • Smart road processing: build road graph, detect road junctions in order to simplify further AI logic.
    • Async loading by Rx: dynamically load map tile without blocking UI thread.
    • Auto downloading and saving of current elevation file from NASA server (or custom one)
    • Auto downloading of map data for current tile from openstreetmap server (or custom one)
    • Added reverse geocoding support by querying of Nominatium server (or custom one).
     
    Yury-Habets likes this.
  5. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Decided to use flat shading as default rendering style:
     
    Last edited: Sep 30, 2015
    Yury-Habets likes this.
  6. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Added pseudo 2D overview mode (see first post updated)
     
  7. Elecman

    Elecman

    Joined:
    May 5, 2011
    Posts:
    1,342
    Two questions:
    -Why flat shading?
    -Are all the object meshes merged into one mesh?
     
  8. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    First attempts were with unity terrain and buildings with predefined textures. But result wasn't nice:
    • Unity terrain introduced some issues with performance. Mostly because heightmap resolution should be high to display terrain regions more accurate. Dynamic creation also introduces some memory traffic and didn't find a way to fix it.
    • In general, OSM data doesn't contain information about facade styles. Only few buildings have information about material/color. It's really difficult to prepare textures for different cities (compare architecture styles of Germany, France, Asian countries...) and apply them dynamically for different areas. I think it may sense only for design time (some products are already doing that). Otherwise user may fell you're cheating: I had some old demo where I tried to apply random textures to buildings in place where I leave and result wasn't nice.
    Terrain consists of chunks which are single mesh (except water in case 3D tile). Buildings, trees are separate meshes.
     
  9. jeffmorris1956

    jeffmorris1956

    Joined:
    Jul 3, 2012
    Posts:
    237
    Is it possible for me to use OSM and terrain data for my city? If so, how can I use the data?
     
  10. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Yes, it's possible in general. Furthermore, choosing any location is main feature of project. Please note, it's not possible so far in current web demo builds due to web sandbox security :(
    But you can download demo project, build it manually and check in editor. I created some simple GUI for choosing location by performing reverse geocoding. It will try to resolve your city name to geocoordinate. After you press Start, app will try to download SRTM and OSM data dynamically (and cache it for future usage) for given location.

    Another approach, change geocoordinate in code once (see ApplicationManager.cs) to skip these steps for regular runs.

    P.S. I prefer to build offline index once for whole region (e.g. Berlin) as it works better so far.
     
  11. unit3ro

    unit3ro

    Joined:
    Dec 18, 2014
    Posts:
    15
    Hello magnificent job I have some suggestions please:

    -Make an internal search in the program such as Google Maps or even more specific. As the ArcGIS; Find related object with same attribute (the object name, location, etc.).

    -a simple editor of 2D polygons to create new buildings by extruding the polygon in the ground surface.

    Please take this into account

    Thanks for your attention
     
  12. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Hi, thanks for your feedback! Actually, it has internal search which is working for internal map index quite good and it's offline which means it doesn't require internet connection. This feature is available by debug console (type help and you will see list of all commands). Search command has the following syntax:

    Usage: search [/h|/H]
    search /q:tag_key=tag_value [/f:element_type] [/r:radius_in_meters]
    search /q:any_text [/f:element_type] /r:radius_in_meter
    -> Just realized this syntax is broken so far


    For example, to get the list of bars in 500 meters you can do the following:

    search /q:amenity=bar /r:500

    So far, you have to know OSM tags meaning in order to use it (not a problem to extend it with some custom aliases to avoid this). This command is used internally for current address resolution
    Another potentially useful command is geocode - it performs reverse geocoding by requesting osm nominatim server.

    The problem that I have no time to build nice GUI to show all potential features.

    Yes, I was thinking to implement something like that. So far, buildings and terrain are different meshes and you can modify their shapes by clicking. Internally, ASM build index for these objects to locate affected vertices. Then you can control what to do with them on application level.

    Thanks for your feedback!
     
    unit3ro likes this.
  13. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Next demo version will contain simple editor tool which allows you to add/remove buildings, barriers, trees, etc and adjust terrain height.
     
    unit3ro likes this.
  14. jeffmorris1956

    jeffmorris1956

    Joined:
    Jul 3, 2012
    Posts:
    237
    I'm able to change geocoordinate for my city in ApplicationManager.cs file and "walked" around my city. I tried to add a car to the scene so that I can "drive" around my city but only the first part of the city was generated. The other parts of the city was not generated. How do I fix this problem? I think that it is possible to replace the faceted skybox with regular one but I'm not sure if this will work.
     
  15. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Is there any log message in editor?. Possible reasons of issue:
    * Issue in ASM related to rendering (most likely)
    * No data for given region
    Could you provide geocoordinate of that place? Then I can investigate what is wrong here.

    Actually, the project is still under development and I'm testing it using only few places. So, arbitrary place may contain some data which is not handled properly and fwk may fail there. Unfortunately, OSM has some ambiguity in data format and that introduces additional level of complexity in implementation and testing.

    My focus now is not rendering and fixing processing of data ambiguity, but new features which may affect internal architecture. That's why building is still not vertically placed in correct way and mapcss has no styles for some objects - it's quite easy to fix in comparison to implementing of editor and multiplayer features (of course, performance should be always measured and taken into consideration).
     
  16. jeffmorris1956

    jeffmorris1956

    Joined:
    Jul 3, 2012
    Posts:
    237
    I used geocoordinate of 40.603621,-74.072954 for my city. When I "walked" around the city using the construction worker, all areas of the city are rendered. I tried to add the car from Unity 5 to the scene and added a camera to the car but only the first area (tile?) is rendered. I think I have to add scripts to the car but which scripts? You used a white material for the ground and buildings. I changed the materials to green for ground and red for buildings but the roads are dark green and I would like them to be dark gray.
     
  17. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    If I understand correctly, it works for default character (construction worker), but don't work for car. In this case, the reason that you need attach Assets/Scripts/Character/ActionStreetMapBehaviour.cs script to your newly created character (car) and remove old one (worker). ASM listens for position changes of character and loads/reloads tiles based on current position.

    Actually, you needn't change any material to do that. I'm using vertex colors and few shaders for all objects (possible to have only one material). What and how to render are defined in declarative way in single mapcss file:

    /Assets/Resources/Config/default.mapcss

    For example, if you want to change default properties of default building then you should take a look at this entry from default.mapcss:

    area|z19[building] { builder: building; height:12; min_height: 0; levels: 5;
    fill-color:gradient(#c0c0c0, #a9a9a9 50%, #808080); facade-builder:empty; material:Materials/Buildings/Building;
    roof-color:gradient(#808080, #606060 50%, #505050); roof-builder:flat; roof-material:Materials/Buildings/Building; }

    It is similar to OSM's mapcss with my custom declarations. Also it allows to change visual representation in runtime without compilation by having different mapcss files (e.g. summer and winter "themes").
     
  18. jeffmorris1956

    jeffmorris1956

    Joined:
    Jul 3, 2012
    Posts:
    237
    Is it possible for you to create the "game" using Unreal Engine 4?
     
  19. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    This question is too generic (I think I can build simple game using UE4 just by following tutorials for beginners). Did you mean "can ASM to be ported on Unreal Engine4"? I just was asked about this on github. Short answer: no, it's not possible at the moment. You can follow discussion here
     
  20. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    For that location I noticed one of the known issues: ASM misses some objects in 3d tiles, but has them in 2d preview tiles. This is a bug of online mode in which ASM is fetching data from openstreetmap.org directly. I have to fix it in future versions.

    Actually, the best way to avoid this is to use offline mode in which map of your region is downloaded and converted from osm format to internal spatial index (but ASM will still be able to fetch data from osm server if you will move outside that region). Offline index has much better performance.

    At them moment, ASM provides MapIndexUtility class (don't like *Utility suffix, so name of the class may change in future) to build spatial index. However, nothing is documented yet..
     
  21. jeffmorris1956

    jeffmorris1956

    Joined:
    Jul 3, 2012
    Posts:
    237
    how do I use offline mode?
     
  22. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    So far, demo project doesn't provide this feature as it requires setting GUI up.

    In general, it's just one call of configured MapIndexUtility class:

    _container.Resolve<MapIndexUtility>().BuildIndex(filePath, outputDirectory);

    OSM map data can be downloaded from http://download.geofabrik.de/
     
  23. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    I updated first post with editor preview video.
     
  24. unit3ro

    unit3ro

    Joined:
    Dec 18, 2014
    Posts:
    15
    magnificent
    It is exactly what I referred, keep up the good work.

    I will continue your work
     
  25. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Thanks for your feedback!

    Actually, framework is not ready for production. So, work in progress.

    P.S. I added simple multiplayer support in demo app:
     
  26. unit3ro

    unit3ro

    Joined:
    Dec 18, 2014
    Posts:
    15
    Hello eiskalt
    I would like to ask you some questions about your project but I think it is more prudent for private messages. You think to sell this project in the assetstore or not have commercial interest?

    This is one of the questions I want to ask you, because I would like to make some changes for my personal use.
     
    Last edited: Jul 8, 2015
  27. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Shortly, there is no decision yet. In general, I would like to have some profit from the project after spending so much time working on it in my spare time (1.5 year at the moment). However, it is still not ready for asset store - various bugs because of data variety, new feature development in progress (e.g. indoor environment generation for all buildings). Also, sometimes I think to make it open source because of complexity which I don't want to reduce. But simple open sourcing won't help and will introduce new problems..

    So, there is a common issue: I'm typical technical guy as it's fun for me to work on complex engineering problems, but everything out of this scope makes me feel uncomfortable.
     
    Last edited: Jul 8, 2015
  28. unit3ro

    unit3ro

    Joined:
    Dec 18, 2014
    Posts:
    15
    I understand perfectly, in a way I also think so. can you tell me what your email to contact you? if you want of course.
     
  29. cmedve

    cmedve

    Joined:
    Jul 18, 2015
    Posts:
    1
    Do you know when you will be making a decision on the release of your code either asset store or open source? This project is exactly what I have been looking for a foundation for a project I have planned. I am fairly new to unity programming and this would give me an amazing start. You have done amazing work... the world can benefit from your efforts.
     
  30. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    I don't know so far, development is still in progress. Probably, I will release some application built on top of the framework. It will help to avoid childish issues in case of publishing in asset store.
     
  31. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Just short demo of next feature - building modifications:



    Actually, the plan is that all generated meshes are modifiable/destroyable.
     
    Last edited: Aug 12, 2015
  32. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Next version will have improved performance, especially for terrain modifications.
     
  33. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    The latest changes in demo project demonstrate various aspect of usage:
    • SimpleMap2D scene loads 2D map of large area with low detail terrain and allows you to skip the certain objects (e.g. buildings, trees) using MapCSS rules for different zoom levels. You can also pan/zoom map.
    • SimpleMap3D scene loads one tile with all details (full detail terrain, buildings, trees) and adjusted tiles with low level of details. You can walk and interact with environment: modify facade of building, make craters. Tiles will be loaded/unloaded automatically.
    • GpsTracking scene demonstrates how to work with Location Services (e.g. GPS). Nmea file is used as GPS signal source to simulate movement.
    • GuiEditor scene demonstrates several features: map editor (create new map object such as buildings, barriers, trees; adjust terrain height), current address resolution (detects where you are), dynamic switch between 2D/3D maps (2D map shows larger area). You may find video which shows this scene in action above.
     
  34. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Next version will support custom textures for different map objects, e.g buildings, barriers.:



    ..and these objects are still modifiable.
     
    Last edited: Sep 29, 2015
  35. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Version with textures is published (first post is updated)!
     
  36. jeffmorris1956

    jeffmorris1956

    Joined:
    Jul 3, 2012
    Posts:
    237
    Where do I enter the geocoordinate of 40.603621,-74.072954? The ApplicationManager.cs in latest version was changed from older version.
     
  37. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    The demo project is slightly changed: now it has multiply scenes (see updated first post). Coordinates are defined via public fields of MonoBehaviour so that you can change them in Unity Editor. Just check behaviors of objects in scene.
     
    Last edited: Jan 24, 2016
  38. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Working on rewriting core logic using C++ to make it completely independent from Unity3D and increase performance. Also support of multiply zoom levels are planned (e.g. render Earth as globe on zoom level one and buildings/all details in 3D on highest level).

    Running on mobile devices is still target.
     
  39. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Just first raw results showing chosen direction - rendered globe at zoom level 1:


    No texturing yet, color gradient logic is slightly broken, but I was able to use coastline data (from natural earth shapefiles) and mesh generation is quite fast.
     
    Yury-Habets likes this.
  40. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Just added future ability to specify color/gradient for each country:

    It will be just one rule in mapcss, e.g.:

    area|z1[sr_geounit=Germany] {
    color: gradient(yellow);
    }
     
  41. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    ..and it is so easy to apply texture to generated mesh keeping assigned country colors:

     
  42. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Another feature: offsetting selected polygons:





    Here, some countries are moved up and water is moved down. All parameters are dynamically configurable via mapcss, e.g.:

    area|z1[sr_geounit=Germany] {
    color: gradient(orange);
    height: 1;
    }
     
    Last edited: Feb 14, 2016
    Yury-Habets likes this.
  43. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    It's not difficult to process some places:



    Here, text and sphere for each place are created on Unity side from geodata provided by the library. Of course, visual representation can be improved: text from opposite side of globe should not be visible, but this is just example.
     
  44. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    I updated first post with the latest information about the project.
     
  45. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Proved: works on Android platform! See first post updated with screenshot.
     
  46. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    I added API for region loading and recorded some video for this feature:
     
  47. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
  48. bombsquare

    bombsquare

    Joined:
    Oct 29, 2014
    Posts:
    8
    Looks amazing! )
     
  49. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
    Not only flat anymore:
     
    Hikiko66 and pixelsteam like this.
  50. eiskalt

    eiskalt

    Joined:
    Jan 10, 2014
    Posts:
    45
unityunity