Interactive spatial augmented reality (aka projection mapping)

Hey there,

For my internship and bachelor thesis, I developed a Unity application that does 3D projection mapping. Using a model of the object that's being projected onto, the real time or prerendered imagery is warped so it looks correct after being projected onto a non-flat surface.

The application can output to any number of projectors (as long as the PC has the capabilities for that amount of projectors), and projections can be blended so very large, seamless projections can be achieved. It's also possible to interact with the projection using a laser pointer and a webcam.

Here's a video of a little tech demo I created in a jiffy; you'll have to excuse the programmer art. At the end it shows some stills of a video mapping show (with eight mannequins and a backdrop, using four projectors) that was created using this application, but sadly I'm not allowed to show footage thereof.

 

Thomas, Just saw your reply to my post. Email me alexhodara@gmail.com

 

Hi Thomas,
I reviewed your video on unity app for projection mapping. I have a need for something like this and wanted to know if you are available for consulting. Thanks.

 

What exactly is it you require?

 

I was looking to be able to utilize an application like you develop for a proof of concept. Basically we would have a number of projectors mounted in a fashion to cover a 15'x15' room (project on 3 sides of the room). We would located various objects (styrofoam to represent various objects (trees, facades, etc). We would want to project on to this surfaces a scene to be developed in unity. We would be looking for your expertise in creating unity code, asset development and animation. You can email me directly at js1055_1@hotmail.com.

 

While this is definitely a very interesting project I'd love to be a part of, I'm employed full time and this project is too sizable to undertake after hours. I'm currently in the process of improving my projection mapping code base though, and looking into putting it onto the asset store - it might still be of use to you that way.

 

Hi Thomas, I would be interested in purchasing this, even before you put it in asset store. From your description it provides what I need. Let me know if we could work out an arrangement I would be happy to purchase an early copy.

 

I've given it some thought, and while I would love to help out, I cannot in good conscience sell the project as it is now - it would also require a whole lot of support from my part, for which I sadly have no time. I am working on getting it into a far more refined, optimized, and globally usable state, and I will get back to you then.

 

Hi Thomas, understand perfectly being a developer myself. Let me know when done and I'll be first in line to purchase

 

Microsoft have open sourced room alive which is another option.

 

Oh wow, that's super cool! Thanks so much for introducing me to it, I'll definitely be looking into that.

 

I am trying to work with them to get the unity plugin for it.

I am planning to do a multi projector setup in my labs.

 

wow if you can get Unity plugin working I would be very interested.

 

Hey @TheRaider, are there any news on that Unity plugin for RoomAlive Toolkit?

Thanks!

 

Hi I am trying to work on something like this, and what you did is great man! I was wondering is there a plug-in to do the video mapping or is it a combination of Unity with a video mapping software like Resolume or Madmapper through Spout or Syphon?

Just trying to figure your workflow thank you.

 

Hey Guillaume, thanks for your kind reply. For this project, the projection mapping was done directly through Unity, using my own code. I don't have the time to write up the full work flow right now, but I'll write an article about it on my site when I do, and link it here. You can read about two mannequin projection mapping projects I did with that software there. If you also require interactivity using lasers, that's also something I did myself, using EmguCV in this case. Since then, I have worked out native OpenCV integration, yielding much higher frame rates - I'm also planning on writing about that soon.

For a different projection mapping project, I did go through existing video mapping software, simply because of the much larger scope and the necessity of absolute reliability, as it is a fixed installation, instead of a temporary one. I supplied 2 monitors worth of content to d3 servers to project content onto a massive sculpture (link below).

So, both routes are possible - if you already feel at home with existing software and it is good enough to do the job for you, you can capture Unity's output and send it to your video mapping software. You'll need some kind of capture card for this though, which don't come cheap, especially for higher frame rates. Or, you can directly map from Unity, but since there isn't a plug-in for that yet, at least not to my knowledge, you'll have to learn the ins and outs of it yourself and program it in. I'd love to send some code your way, but the project was made for the company I work for, so I don't have the rights to it.

 

Brilliant