Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

3D Reconstruction for ARCore [RELEASED]

Discussion in 'Assets and Asset Store' started by lvonasek, Jan 30, 2019.

  1. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    Hi everyone!

    I'm happy to announce that my 3D Reconstruction tool for Android is now available as a Unity plugin, any Android smartphone that supports ARCore (https://developers.google.com/ar/discover/supported-devices) can be turned into a portable 3D scanner that works in real-time using your normal phone camera, works out of the box with Augmented Reality as well, allowing to place virtual objects more accurately than ever before as you're dealing with the REAL geometry and occlusions of your environment rather than generic virtual planes and walls.

    You can get it on the Asset Store right here: https://assetstore.unity.com/packag...reconstruction-for-arcore-android-only-136919 This is my life project and I've been working on it for quite some time, at the time I couldn't find anything like this so I decided to make my own, I believe it's the first of it's kind that comes with a Unity plugin for it, making it very suitable for AR applications and gaming, no more virtual characters clipping or floating through walls and objects!

    I'm currently working on version 1.1, so stay tuned! I'm very curious to see what you can create with it, feel free to share your results here, if you find any problems or have any specific requests, please let me know!
     
    simonpasi_XR, AndroYD and Circool like this.
  2. facebookprofile1

    facebookprofile1

    Joined:
    Jan 31, 2019
    Posts:
    2
    Thanks! It looks great!

    Does your asset store contains the code which is converting feature points into depth map, before it's passed into Tango or it's all wrapped and hidden in the liboc.aar? I would like to buy it and use it but I need to have a full control over the code in case I will have to make any modifications.

    In the package preview I only see ReconstructionManager.cs which could be what I am looking for.
     
  3. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    The conversion from feature points into depthmaps is done inside liboc.aar.

    I am willing to sell you complete source code for $6000 on invoice (except Tango3DR library as I do not have the code of it). If you have an interest then contact me on tridosm@gmail.com.

     
    Last edited: Mar 11, 2019
  4. JediNizar

    JediNizar

    Joined:
    Nov 13, 2016
    Posts:
    111
    wow this is one of the greatest thing I've seen the interaction with the real world is something I've been looking for for quite some time.
    definitely on my wish list and as soon as I finish with my current project . I'll go for it.
     
  5. EstudioVR

    EstudioVR

    Joined:
    Jul 26, 2013
    Posts:
    127
    I am interested in tha plane detection funcion, Do you have an apk for tests?
     
  6. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    The example scenes are avaible here as APKs:
    https://drive.google.com/open?id=1TjKrcyXdbwkOMdc0QoNWzoIRohkBYCcE

    Note that numbers in APK name means plugin version 10 is 1.0, 11 is 1.1.
    Version 1.1 will be uploaded next week into Asset Store and it will be bring better performance.

     
  7. EstudioVR

    EstudioVR

    Joined:
    Jul 26, 2013
    Posts:
    127
    Thank you. I ill try
     
  8. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    The plugin update 1.1 has been uploaded and it should be available during the next week.


    The updated demo APKs are here: https://drive.google.com/open?id=1TjKrcyXdbwkOMdc0QoNWzoIRohkBYCcE
    The link to Unity Asset Store: https://assetstore.unity.com/packag...reconstruction-for-arcore-android-only-136919

    Here is the changelog:
    Meshing
    -better handling of pausing/resuming meshing
    -bug fixing
    -color per vertex added
    -faster walls estimation

    New functions
    -example how to render camera texture on mesh only
    -possibility to store the mesh as OBJ
    -support for Unity 2017.4 and higher
     
    facebookprofile1 likes this.
  9. facebookprofile1

    facebookprofile1

    Joined:
    Jan 31, 2019
    Posts:
    2
    Thanks for sharing an update. I tried the app, 3dr mesh builder 1.1 and I feel like meshing from your previous app, "3D scanner" works much better, covers more area and in a better way. Is it different implementation? I am using the same phone so I would expect similar results.
     
  10. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    It is slower because of vertex coloring, you can disable it reconstruction prefab then it should behave as in previous version
     
  11. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    To the topic performance - I did a lot of improvements which speeds up meshing. I plan to release it as the version 1.2 before the end of February.
     
  12. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    AndroYD likes this.
  13. AndroYD

    AndroYD

    Joined:
    Nov 3, 2014
    Posts:
    28
    You're truly a miracle worker, this update is amazing! It opens the door to even more new possibilities now!
     
  14. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    Thanks, if you create any app or video with it then please share it with me.
     
  15. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    Version 1.3 was uploaded and should be available soon.

    It will bring holes-free meshing and walls detection in plane detection demo.
     
    AndroYD likes this.
  16. AndroYD

    AndroYD

    Joined:
    Nov 3, 2014
    Posts:
    28
    Tried it today, and I'm still not sure if this is real life or just fantasy, I'm in total bliss of how amazing this new experimental texturing feature is! That's the culmination of what I wished to happen since the very first version of this plugin, and now all that it's happening right in front of me like magic, to think that day has finally come fills me with joy!
     
  17. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    This is just a beginning. In 2017 was possible with Chisel library and Lenovo Phab2 Pro do this:

    (the device used depth camera for scene understanding and fisheye camera for motion tracking)
     
    Last edited: Mar 18, 2019
    pachermann likes this.
  18. BinaryBanana

    BinaryBanana

    Joined:
    Mar 17, 2014
    Posts:
    81
    Hi,
    Just wanted to share that soon there will be also a solution that works on iOS. The code is in alpha, so still not perfect but getting better.

    You can find more here, if anyone is interested: http://ARMeshing.com

    Early alpha version video:
     
  19. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    AndroYD likes this.
  20. AndroYD

    AndroYD

    Joined:
    Nov 3, 2014
    Posts:
    28
    The occlusion feature is a great addition to this asset! I'm glad I didn't have to thinker how to implement it, imagine that Google is still TALKING about adding occlusion features to ARCore in the future https://sa2018.siggraph.org/en/attendees/technical-papers/session/22/details , that's what I absolutely love and appreciate of your work, you DO things rather than TALK, unlike other so-called "competitors" who teases you with similar solutions but never actually release anything usable like SOMEONE up here, or worse, have over-inflated egos and don't even think you're worthy of a reply (am I not "elite" enough?), I'd understand that attitude if someone had the most exclusive thing in the universe, but not when alternatives already exists right here right now.
     
  21. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    The situation is a bit more difficult. Google does not care about mobile AR so much (you can see it on the level of support on ARCore github pages). What they care about is the future and that will be AR glasses which should replace the smartphones one day. However their AR glasses need this things:
    1) Great and not so expensive HW (exact opposite of Magic Leap)
    -here Google does a research they don't talk about much (except Google Glass Enterprise)
    2) Many apps, something user can play with
    -that's why we have ARCore, Google needs some content which will developers easily bring on AR glasses
    -also Google Glass Enterprise should help here to bring some serious apps and not just games
    3) Great technologie
    -they need AR persistence, which is their VPS technology (which is delayed for few years and not only because of lack of data), it should be a paid technology which mostly does not work - not a good idea to release it
    -environmental understanding, they have it ready for limited amount of devices (you can see in their paper that it does not work on iPhone), they cannot release it in this state
    -environmental lighting - they talked about it on Google I/O because it is only one thing which is sure that it will work this summer

    However talking about technology brings more customers than doing the technology. I see on my LinkedIn many posts of cheap AR tests which became viral and because of them posts like this becomes invisible.
     
  22. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    The update 1.5 will be available soon. It adds support of ToF cameras using Huawei AREngine and it works currently on this devices:
    Honor View 20
    Huawei P30
    Huawei P30 Pro
    More devices will be added by Huawei soon.

    -One APK can handle ARCore and AREngine easily using ARSelector class
    -Different meshing parameters for active/passive depth sensing
    -Plane detection improvements
    -Support for ARCore 1.10
    -Enabling and disabling AR in the app possible

    For now you can test the demo app:
    https://drive.google.com/file/d/1-M-VJwEwxC4BnRWJIHtsVC2-Sft-oOs0/view?usp=drivesdk

    This update should be available during/shortly after this weekend (depends on Unity's team package check).
     
  23. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
  24. Aaeshah

    Aaeshah

    Joined:
    Sep 2, 2017
    Posts:
    9
    Looks fantastic! I have one question: can I use it on any Android devices or just Honor/Huawei devices?
     
  25. johaswe

    johaswe

    Joined:
    Sep 13, 2018
    Posts:
    28
    I have compiled for samsung galaxy s8 (arcore) without issues. :)
     
  26. Aaeshah

    Aaeshah

    Joined:
    Sep 2, 2017
    Posts:
    9
    Thanks.
     
  27. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
  28. Aaeshah

    Aaeshah

    Joined:
    Sep 2, 2017
    Posts:
    9
    No scene is working properly with me. I am using ARcore with Android.

    I feel there are some objects missing from the scene. I had to add (ARCore device) prefab to make the camera working. The app was giving me "No cameras rendering" before.

    Now the camera is working but I can only throw the cube without any features: No planes detection, No occlusions, No meshes created.

    here how the MeshBuilder scene which I need, tell me what should I add to make it work using ARCore?


    Please help!
     
  29. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    Did you follow the steps in README.txt?

     
  30. Aaeshah

    Aaeshah

    Joined:
    Sep 2, 2017
    Posts:
    9
    Thank you, it works.

    Could you please explain to me the method used to detect the planes and build the mesh briefly. or you could give me a reference that explains the equations used in the code.

    I will be grateful.
     
  31. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    Feature points from ARCore are
    1) transformed into screen coordinates
    2) triangulated Delaunay triangulation
    (there is used an implementation making regular triangles)
    3) transformed back into world coordinates
    4) rasterized in 3D (but only if the triangle shape is not "sharp"
    5) using SDF reconstructed:
    https://en.m.wikipedia.org/wiki/Signed_distance_function

    Note that there more tricks to detect white floors/walls but this is how the basic 3D model is created
     
  32. Aaeshah

    Aaeshah

    Joined:
    Sep 2, 2017
    Posts:
    9
    Thank you, this is very helpful.
     
  33. Aaeshah

    Aaeshah

    Joined:
    Sep 2, 2017
    Posts:
    9
    Hi,

    I was reading this lecture http://www.hao-li.com/cs599-ss2015/slides/Lecture06.2.pdf and it seems like your method combines explicit and implicit reconstruction!

    Is that what going on here!

    why did you use both Delaunay triangulation and SDF

    did you use Delaunay triangulation for creating the mesh and SDF for smoothing?

    Please help!
     
  34. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    Hi,

    the motion tracking of ARCore is very inaccurate and because of it is not possible to create homogeneous point cloud and create surface on it.

    The point cloud is accurate only per frame but if you merge the point clouds together then it is really bad.

    SDF can handle it better but the problem is that SDF requires regular distribution of points. If I use just SDF then I would be able to scan with grid 15 cm and the model would cover the area (except walls and white floors).

    Scanning with grid 15 cm is too smoothed (it means that vertices in range +/- 15 cm are smoothed together). The solution is to do Delaunay triangulation on point cloud from actual frame. Then from the output surface generate a point cloud and use that as an input for SDF. This way it works great for grid 4 cm. It also works with grid 2 cm but there it is a bit problematic for performance.
     
    anticyclone_ likes this.
  35. dimib

    dimib

    Joined:
    Apr 16, 2017
    Posts:
    50
    Hey Ivonasek, your asset looks really great. Do you have any idea how to port this functionality to ARKit ? Or better to say ARFoundation? This would make your solution more generic and long-term supported as Unity puts a lot of work into ARFoundation.
     
  36. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    Hi, it is not possible to bring it to ARFoundation due to their licence. ARKit would be possible but it is not a priority right now.
     
  37. Aaeshah

    Aaeshah

    Joined:
    Sep 2, 2017
    Posts:
    9
    Hi again,
    When I change the cube to another prefab, it is not recognized the plane and object keep falling down from the screen without landing on the floor or any planes. If I remove the gravity from the rigidbody the prefab can stick everywhere even flying on air! how I can solve this problem, please?
     
  38. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    Hi, you have to have on your object collider and rigid body. Take a look into the cube prefab how it is configured.
     
  39. Aaeshah

    Aaeshah

    Joined:
    Sep 2, 2017
    Posts:
    9
    I have everything but still not working!
     
  40. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    No idea, I would have to see the project
     
  41. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    The asset is updated to version 1.6.

    For users of Honor View 20 and Huawei P30 Pro is this update really huge!

    Merry X(R)Mas to you all!
     
  42. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    ixdeveloper likes this.
  43. olli_vrcoaster

    olli_vrcoaster

    Joined:
    Sep 1, 2017
    Posts:
    24
    i just realized i used your project 2 days ago to scan half of my property - works pretty accurate. The dimensions are more then 15m x 15m. Nice work! I'm excited to see where this goes. You'd need to work on your UI design though :D

    Anyhow, i'm interested in what kind of point-cloud data you use for mesh construction. Is it the arcore feature point cloud which is used for tracking as well? Or does it make a second pointcloud for the mesh only?
     
  44. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    It uses ARCore features point cloud (or ToF sensor if available and supported). However I am going to replace it with ML solution which is more robust.
     
  45. olli_vrcoaster

    olli_vrcoaster

    Joined:
    Sep 1, 2017
    Posts:
    24
    which kind of ML ? Does this mean the phone learns how common objects are shaped and remembers them when you scan them and then rebuilds them faster based on the expected shape? Are there APIs for that ? I'd be very interested in the progress, do you have a blog or sth ?
     
  46. lvonasek

    lvonasek

    Joined:
    Jan 16, 2019
    Posts:
    25
    It is going to use Google DepthAPI:


    I do not have any blog. You can join my Slack group - the link is in one of posts above.