Search Unity

  1. Are you interested in providing feedback directly to Unity teams? Sign up to become a member of Unity Pulse, our new product feedback and research community.
    Dismiss Notice
  2. Read here for Unity's latest plans on OpenXR.
    Dismiss Notice

Help Wanted Using MARS for well-defined AR expereinces

Discussion in 'Unity MARS' started by Amyd80, Jun 2, 2020.

  1. Amyd80

    Amyd80

    Joined:
    Jun 2, 2020
    Posts:
    2
    We were kind of hoping MARS is somewhat different than what it turned out to be - but perhaps I'm missing the point right now.

    As far as I can see, most of the recognition & tracking logic is outside of MARS's scope, pushed down to the AR Foundation or lower level. So, if we wanted to create something like environment or scene tracking based on prescanned data (point clouds or meshed geometry), where the AR triggers & experiences are pre-positioned & pre-defined in advance, we would be (right now) out of luck, and would need to use a third-party framework, which probably doesn't quite plugin in MARS (for now) and/or MARS would give us little to no benefit.

    Or am I am missing something basic here?
     
    FutureSystems likes this.
  2. Jono_Unity

    Jono_Unity

    Unity Technologies

    Joined:
    Apr 5, 2016
    Posts:
    15
    Hey @Amyd80, there are a few things in MARS to help you do persistent / location-based AR like you're describing, and some more coming down the pipe.

    If you have a model / scan / other representation of your known space, you can configure that as a simulation environment, and then be working with that space in MARS in the Simulation view. Setting up a simulation environment is covered in this doc: https://docs.unity3d.com/Packages/com.unity.mars@1.0/manual/SimulationEnvironments.html
    To relocalize to your space (get your device lined up with the physical space), today the best approach is using image markers: if you're able to put an image in your space, or better yet use an existing poster or such, then you can configure a simulated version of that marker in your simulation as well. This page covers working with image marker proxies, and setting up simulated markers in your environment: https://docs.unity3d.com/Packages/com.unity.mars@1.0/manual/Markers.html

    Looking ahead, we will be adding support for persistent anchors, at which point you'll be able to relocalize to your known space without the need of image markers. We have the foundation of this support, but currently don't ship with providers for this functionality. It's on our radar as a high-priority ask, and we're building support now.
    We'll also be shipping the MARS Companion Apps later this year, which are phone & HMD apps specifically built for capturing this kind of environment data to bring into the Editor and make this workflow more straightforward.

    Thank you for the feedback, please keep it coming :)
     
    herbrush and FutureSystems like this.
  3. Amyd80

    Amyd80

    Joined:
    Jun 2, 2020
    Posts:
    2
    Thanks, that does sound quite a bit more promising.

    Just two questions: in using scan data in the simulation view right now, I assume it doesn't "automagically" work with third-party frameworks like Vuforia or Wikitude, right? So, you can't actually test the external recognition of those frameworks within the MARS environment? I at least couldn't figure out a way to do that on a cursory try. I guess this is something that they need to implement on their side of things, to plug into MARS?

    Secondy: can you talk a bit more about the persistent anchor functionality? That does sound like something that would be highly interesting for our current projects. Would this be as "simple" as converting an existing point cloud or mesh to a format that the MARS provider can work with (or alternatively using your companion apps to scan/record), or will it be more involved?
     
  4. Jono_Unity

    Jono_Unity

    Unity Technologies

    Joined:
    Apr 5, 2016
    Posts:
    15
    Vuforia and Wikitude don’t currently have provider integrations into MARS, so for example an image marker proxy set up in MARS won’t work automagically using those services at runtime. But, in the Editor, we use ‘simulated’ providers instead of the actual runtime provider anyway, so you can simulate generic image marker tracking in Editor, and then use Vuforia or Wikitude to do your runtime tracking (which would be the typical Vuforia or Wikitude object setup, not a MARS proxy).
    Writing providers for either of those frameworks would simplify this, because a MARS-style image marker proxy would then also work at runtime and not require a separate setup. In some cases, you or we can write these providers; in others, it does require some adjustment from the framework developer. This page gets into writing providers (under ‘Providers’): https://docs.unity3d.com/Packages/com.unity.mars@1.0/manual/SoftwareDevelopmentGuide.html#providers
    Let us know if you’re interested to dig in and we can provide guidance; otherwise, good to know which providers you’d like to see next :)

    About persistent anchors, that functionality is platform-specific and requires capturing the space on the platform - you can’t convert an existing point cloud or mesh into a persistent anchor. So yep, that’s what that feature of our Companion Apps is about: with the app, you scan/record the anchors of your space, and can bring those into your project to relocalize against. Now that our initial version of the MARS Editor extension is out, we’re working on getting these Companion Apps to you as soon as possible (later this year).
     
    jmunozarUTech likes this.
  5. MOlanders

    MOlanders

    Joined:
    Oct 1, 2020
    Posts:
    1
    Hi Jono_Unity,

    You wrote this:
    If you have a model / scan / other representation of your known space, you can configure that as a simulation environment, and then be working with that space in MARS in the Simulation view. Setting up a simulation environment is covered in this doc: https://docs.unity3d.com/Packages/com.unity.mars@1.0/manual/SimulationEnvironments.html
    To relocalize to your space (get your device lined up with the physical space), today the best approach is using image markers: if you're able to put an image in your space, or better yet use an existing poster or such, then you can configure a simulated version of that marker in your simulation as well. This page covers working with image marker proxies, and setting up simulated markers in your environment: https://docs.unity3d.com/Packages/com.unity.mars@1.0/manual/Markers.html

    Can you please explain some more on how to do the relocalize with an image?
     
  6. Jono_Unity

    Jono_Unity

    Unity Technologies

    Joined:
    Apr 5, 2016
    Posts:
    15
    Hey MOlanders, sure thing, here's how we do it in our location-based demo. We have a photogrammetry scan we made of San Francisco City Hall, which we set up as a simulation environment following the process in that doc link. Then we've added to that environment a synthetic image marker (most easily done via Window -> MARS -> MARS Panel, and then under the Create / Simulated headers, 'Synthetic Image Marker', and then configure it to the desired image in the inspector, same as an image marker proxy). We have the marker laid out in a central location in the sim:
    upload_2020-12-9_12-35-7.png
    The implication here is that this marker is really at that exact location in the real location - this is why I suggested using an existing poster or other such 'permanent' marker.

    Now that we have the simulated setup, the other side of the coin is authoring the proxy that you'll actually deploy in your scene/app. Here's the scene we use in conjunction with the sim environment above, with a bunch of content under this marker proxy, so we can position things absolutely around it:
    upload_2020-12-9_12-41-15.png

    That gets you most of the way there -- you should now be able to lay content out relative to the space by positioning it relative to the marker.

    What's missing now is any occlusion from the real environment. This may or may not be important depending on the nature and scale of your app, but we solved it via that child you see in the hierarchy above, 'MemorialCourt_Reference' -- this is the actual photogrammetry scan again, positioned so that it will line up with the real location, meaning that we can then either 1) render a stylized version of the real space over itself, or 2) apply an occlusion shader (write depth only, not color) so virtual objects don't render through real buildings. In a smaller space we'd recommend using a Plane Visualizer to do this occlusion, but plane finding isn't viable in large outdoor scales.

    The Rules (MARS 1.1) webinar a bit ago covers this use case (among others), in the context of the Rules feature - I think I've hit the main points here, but if you prefer video, give it a look :) https://create.unity3d.com/mars-rules-webinar

    Let me know if this makes sense & works for you -- thanks!
     
  7. dariuspowell

    dariuspowell

    Joined:
    Jun 11, 2015
    Posts:
    19
    Hi.

    I'm glad I found this post as it may offer a solution to a problem I have creating an AR trail during lockdown.

    If a 3D environment is loading in using the image marker method, how conisistant will the line up be once the user moves away from the image?

    Is it possible to bring in ARworldmaps into data Mars? Or if not, if I triggered an ARworldmap when image marker is recognised would that help stablise and anchor the 3D environment?

    In a nutshell, I have an accurate scan of a church interior that I want to load in place when user scan image maker. Then as they freely move around in the physical, the virtual space remains aligned.

    Any advice or confirmation would be appreciated!

    Thanks
    Darius
     
  8. jmunozarUTech

    jmunozarUTech

    Unity Technologies

    Joined:
    Jun 1, 2020
    Posts:
    258
    hello @dariuspowell

    Image markers will work well while the app recognizes them. The thing is, if you get far away enough, it will get to a point where the app will stop recognizing them and things might not work.

    To compensate for this, you could use several image markers placed across the environment and if you know before hand the distance from the markers you can still calculate the position and alignment of the environment.
    The important part would be to always have at least one image marker recognized at all times.

    With regards to ARWorldmaps. MARS Will work with ARFoundation; you might want to check the ARF examples (https://github.com/Unity-Technologies/arfoundation-samples), specifically the ARWorldMapController.cs which performs the logic in that example.

    Do bear in mind that ARWorldMap is an ARKit-specific feature.
     
  9. dariuspowell

    dariuspowell

    Joined:
    Jun 11, 2015
    Posts:
    19
    thanks @jmunozarUTech for the reply.

    That's what I thought would be the case but I've also seen what @DanMillerU3D prototyped a few years back, https://twitter.com/danmillerdev/status/1100472988778917888?lang=en

    This seems to show that you could use an image target to launch the AR content and once tracking of image is lost, the world tracking would kick in?

    Also in @Jono_Unity 's example above it shows an image marker as the trigger. The image wouldn't be in view while the user moves around the real environment.

    I feel I'm close to coming up with a solution. Any support would be appreciated.

    thanks
     
  10. jmunozarUTech

    jmunozarUTech

    Unity Technologies

    Joined:
    Jun 1, 2020
    Posts:
    258
    ahhhh ok ok, now I understand what you meant,

    yes you could definitely do it through that approach. Get the initial anchor with an image target, place the augmented content based on that anchor and start your experience.

    The thing is that you might get some drifting depending on the device you use and for how long you do it. This happens since the SLAM (positioning algorithmss) used on ARKit / ARCore are not closed loop (for performance reasons).

    Hence the reasoning on my first post when mentioned that several images would "Re-anchor" your content in case some drifting happens.

    Being said that, try it out. I dont see why it would fail, but do keep in mind that there might be a little bit of drifting :).

    For more info about closing the loop check this out https://blog.rabimba.com/2018/10/arcore-and-arkit-SLAM.html, a little bit old but worth the read if you want to understand whats happening under the hood
     
  11. merrythieves

    merrythieves

    Joined:
    Apr 28, 2021
    Posts:
    5
    Sorry for the dumb question: but what exactly is this demo showing? How is it supposed to be used? More importantly: does it show us how to do something that can actually be done?

    The theory seems to be that a real person in this real version of this environment could walk up and scan a marker that is in the real world. But in the example, the marker is the size of a swimming pool for some reason. Who is supposed to be scanning this code and where are they supposed to stand? Or does this actually only work in the Unity editor player under perfect conditions?

    Assuming a person were to get the code scanned somehow, we'd hope they can walk some yards away to a light pole and see a virtual light correctly aligned to it. But it sounds like from what I'm reading here that this is not possible and you'll need to actually align everything to nearby image anchors. So MARS is providing a slightly better interface for working with all of this, but it doesn't seem like it's actually doing anything more than image anchors under the hood?

    It would be immensely helpful to have a demo of this system working for this use case. This would inform us a bit better than the demo which seems to be more theoretical than functional.
     
    Last edited: Jul 20, 2021
  12. Jono_Unity

    Jono_Unity

    Unity Technologies

    Joined:
    Apr 5, 2016
    Posts:
    15
    Hey samgarfield,
    You're right that the settings on the marker here are not what you'd really want to go deploy this experience - yep, you would for starters want to use a reasonable marker size; it's blown up here mostly for readability in the demo.

    You've hit on something that we've been planning to improve about this demo: for a space as large as the one shown here, really you wouldn't want to rely on a single marker, but use a few markers spaced around the location, and use each marker you find to refine the pose of the matched scene, rather than doing a one-to-one matching like how it's set up right now. Like you pointed out, using a single marker in a large space would inevitably lead to large drift in the content as you move farther away from that marker.

    It sounds like you're working on things which would really benefit from the improvements we've been working towards for location-based experiences -- if you're open to it, we'd be happy to jump on a call and talk through your project and how we can nail supporting what you need in our next updates :)
     
    jmunozarUTech likes this.
  13. CreepyInpu

    CreepyInpu

    Joined:
    Oct 9, 2014
    Posts:
    19
    Hi !

    Is it possible to replicate the functionalities of Vuforia's Area Target ? With Area Target, we can simply import a Matterport scan of a room in Unity and place 3D objects in it. Once the user is inside the room, the tracking works really great and the 3D objects are positionned accordingly to where they we placed on Unity, and no need so scan a marker or something. The only downside is... the price. It's something like 24k$ per year.
     
  14. Jono_Unity

    Jono_Unity

    Unity Technologies

    Joined:
    Apr 5, 2016
    Posts:
    15
    Hey CreepyInpu, alas, we don't support Vuforia Area Targets out of the box in MARS, though if you're particularly motivated, it would be possible to write a provider (http://docs.unity3d.com/Packages/com.unity.mars@1.3/manual/SoftwareDevelopmentGuide.html#providers) for it.
    In a future release (can't give a specific timeframe at the moment), we intend to provide a general purpose persistent anchor workflow which would give you a similar authoring experience to what you're describing.
    For our understanding about what you & others need:
    - Would an intermediate solution which is not cross-platform be helpful to you? In other words, in your project, are you targeting one platform or multiple?
    - Are you currently using Vuforia Model Targets or another solution (Apple WorldMaps, Azure Spatial Anchors, etc)?

    Thanks!
     
    jmunozarUTech likes this.
unityunity