Other WebGL Augmented Reality - mind-ar-js

Hello everyone!

Just want to learn about the interests among the community and gather some initial feedback about a potential package for building webgl based augmented reality app, mainly for image tracking and face tracking.

My name is Kim. I'm the creator of an open source web AR project called MindAR - https://github.com/hiukim/mind-ar-js I would consider it as a successor of AR.js. It's developer friendly with high tracking quality.

MindAR is a javascript library built for browsers. I'm thinking about creating a unity package to port the library into unity. The usage will be similar to Vuforia or ARFoundation Image Target. Correct me if I'm wrong, I don't think there are existing unity libraries that allow us to build web based AR app that can be run in browsers.

However, I don't have much idea on how many unity developers out there interested in web based AR. Please comments and let me know if it sounds interesting to you. Thanks!

 

Hi Kim, it would be much helpful if MiindAR got integrated with Unity. Thank you for your efforts.

 

Hello kim, waiting for it , it'll be so much helpful and time saving, please do.

 

Hi Kim, it'll be so much helpful. Thank you for your efforts.

 

Hello Kim, i'm interested in Web AR, so having Mind AR integrated into Unity would be really incredible. Thank you and i'm looking forward to any updates.

 

I think this is hard to say. Forum is too easily be overflown in new thread and hard to find anything we needed

Currently now I try to integrate mindar into unity by myself because webxr in safari ios is not ready yet, and webxr mozilla viewer did not support image tracking. But to be honest this is very risky because webxr with image tracking might become standard in every browser anytime in near future. Which is overlapped with mindar

 

@hiukim-mindar That would be very much helpful, Kim. @thaianna did you succeeded in the integration? can you share any experience or update about it? Thank you very much.

 

@htetwaiyanaung_unity Sorry for the late because I am not that name. And no I have stop working on mindar integration because I think I could wait for ios implementing webxr in safari, as it was a glimpse of it trying to support in ios 15.4

 

The situation was called for me to really do mindar in unity again. And with a bit let down about accuracy it really possible to just use mindar in unity webgl

First thing first. There is a trick in unity webgl that, we can just override how webcam work in unity webgl build by simply copy

{your Unity Installed Location}\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\lib\WebCam.js

into your project

Plugins

folder and change it to

.jslib

. Unity will just override every function with the same name and use it as that internally

This trick was take a hint from this reply https://forum.unity.com/threads/webgl-android-device-access-rear-camera.755099/#post-5033042

This allow me to just hook the event of start and stop camera (at the function named JS_WebCamVideo_Start and JS_WebCamVideo_Stop). In this WebCam.js file also utilize variable named activeWebCams that store video element. This is the key to sync mindar with webcam in unity webgl

Second. I then making my own webgl template to add script

mindar-image.prod.js

from cdn. And then I just use a class

MINDAR.IMAGE.Controller

you can see how it was used in aframe implementation https://github.com/hiukim/mind-ar-js/blob/master/src/image-target/aframe.js

When you hook the starting event of unity webcam and use the video object to create mindar controller properly and you already compile image target into file

.mind

. It then start giving you the detection and matrix of image target you want to detect

Finally it now only just interop that you get 16 float number and index of image when mindar detect anything. You then need to have some way to send this data into unity to process into matrix4x4 and calculate it as you would wanted. I use dyncall and marshalling and malloc but you can use sendMessage or anything you prefer

I am not sure how I convey this process properly, you can also take a look at this repo.

https://github.com/Thaina/MindARUnity

You can also compile new target and change it at

/Assets/WebGLTemplates/MindAR/targets.mind

and build this project in unity 2022.1.11f

 

First of all, this is great @Thaina. Thank you for integrating mindAR into unity, this will change WebGL AR in unity.

I tried your repo project and changed the targets.mind with my own. I successfully build and hosted it on my server.
I tried it on my xiaomi mi9T android phone but I'm only getting around 5-10fps only, I tried it also on iPhone 13 pro max the experience is smooth but the detection is not working when you first load it, I need to close safari and reopen it to make it work. Hoping for the next updated version of your work.

android browser used: Chrome
Android version: 11
iPhone browser used: Safari
iOS version: 16


btw I built the project in unity 2022.1.12f1

Thanks!

 

Has anyone tried to make Face Tracking work? I'm kind of working on it but I'm going kind of blind stumbling through the scripts, working my way based on what @Thaina said but I doubt I'll be able to work something out.

 

Hi, @NChal do you have success making Face Tracking work? I'm also trying but same, I'm kind of blind stumbling through the scripts

 

@kimvasquez17 quez17 @Thaina Hello how are you, I am new in this world of projects and videogames. And I would like to know how you implemented a not so technical step by step. It's that I need to make a web ar and I see that it works for you but when I mount the project I get errors and I don't know how to fix them. I would like you to explain to me please

 

I recommend you start with standard project without hacky feature like mine doing to understand the basic first

 

I tried to get it from "https://github.com/Thaina/MindARUnity" and run it
"Uncaught (in promise) Error: Requested texture size [0x0] is invalid."

 

I have hard code my own texture there. I think you got error because you might need your own texture

 

How can I fix it to make it run properly?

 

I don't even know what error you really got. Because mine don't. I can only say you need to try to debug in js and learn how these codes work

 

"https://github.com/Thaina/MindARUnity" Download from here and run you can check the error The error occurs inside "mindar-image.prod.js" I find it hard to debug

 

I think you need to block on exception and stack trace up to see which image url was missing. Or did it get camera properly

 

Please help me. I can't get the video to display properly on the MindAR Target. It turns out that the video (and sound from the video) starts before the camera finds the target. How to make video start ONLY when camera finds target, and when camera loses target - video pauses. Also, I need it to be displayed correctly on Android and IOS.

Here's my code:

<html>

  <head>

   <meta name="viewport" content="width=device-width, initial-scale=1" />

   <script src="[URL]https://aframe.io/releases/1.3.0/aframe.min.js[/URL]"></script>

   <script src="[URL]https://cdn.jsdelivr.net/npm/mind-ar@1.2.1/dist/mindar-image-aframe.prod.js[/URL]"></script>

<script src="[URL]https://cdn.rawgit.com/mayognaise/aframe-chromakey-material/v1.2.0/dist/aframe-chromakey-material.min.js[/URL]"></script>

  </head>

  <body>

   <a-scene mindar-image="imageTargetSrc: [URL]https://cdn.jsdelivr.net/gh/hiukim/mind-ar-js@1.2.1/examples/image-tracking/assets/card-example/card.mind;[/URL]" color-space="sRGB" renderer="colorManagement: true, physicallyCorrectLights" vr-mode-ui="enabled: false" device-orientation-permission-ui="enabled: false">

     <a-camera position="0 0 0" look-controls="enabled: false"></a-camera>

     <a-entity mindar-image-target="targetIndex: 0">

       <a-video src="[URL]https://cdn.glitch.global/aac91519-95ff-4996-8909-e5c41f3cfff9/mov_bbb.mp4?v=1680745936939[/URL]" position="0 0 0" height="1" width="1" rotation="0 0 0" id="vid"></a-video>

     </a-entity>



   </a-scene>

  </body>

</html>

 

1 - You should use unity video
2 - video element should have start or autostart attribute, you need to shut it

 

how to do mindAR Video in unity?

 

Video would not related to mindar. You may play video in unity directly or using html video system

 

Hey Everyone. I found out about this asset which, according to its author, is based on both ThreeJS and MindAR JavaScript libraries, fully integrated on Unity. And it's free (for now, at least):
https://assetstore.unity.com/packages/tools/utilities/webxr-plugin-241842