Secondary camera / minimap in VR?

Okay, so I'm having some difficulties with having a minimap, which is essentially a second camera rendered in a small window at the top right of screen. It works fine in regular mode, but not VR mode.

 

What are the difficulties?
Did you set the camera up to a render texture on a plane that's in world space in the top right corner of the VR's view?

 

New to VR (and screen space vs. world space) ... would rendering to a top-right plane in world space mean that it doesn't automatically stay an "overlay" in the view? (I suppose I could child the plane to the camera, such that when the parent camera moves, the plane moves with it...)

 

You'll usually want to avoid GUIs that are fixed to the player camera in VR. Where to put the UI in VR, of course, is really not fully solved. What I personally like best is putting the GUI elements in a sphere around the player's head. The only problem is that usually we don't know in which direction the players are oriented with their body. So what I currently do is a mix of stuff that's put into the world and fixed around the player and a few things that rotate on Y with the player (but those are usually things that appear and disappear so the player doesn't get the feeling that they're following him around ;-) ).

 

I suppose what I mean by "GUI" here is not so much a GUI -- not so much an "interface" with buttons, etc. as it would be an HUD... wouldn't it make a LOT of sense to have that fixed at all times? (health levels, score, etc. -- and, in my case, a 2D mini-map)

 

So, you can make a gui that tracks with the camera with relative ease by making it world space and a child of the camera.
Yeah, it does mean that the canvas would clip through things, but with a shader that changes ztesting you can fix that.

Here's an example project I whipped up demonstrating how I would go a quick mini-map

 

Thanks! I'll check it out...

 

Try it - I find stuff attached to my head extremely annoying in VR. You can do it for a short time if you absolutely have to grab the player's attention (the SteamVR Room Setup is a good example of why this can be useful). But almost always it's better to put the GUI somewhere else, like on a controller or at some fixed location in the world.

A map, to take your example, is something that I really only want to check every once in a while. If that's a child of my head and it's fixed to a position in the view, this is just annoying most of the time.

Instead, you could for example show it when the presses a button on their controller, where the controller is (that, of course, requires motion tracked controllers - but bth, VR without motion tracked controllers is a weird kind of thing, anyways).

A really great example of how UI (including HUDs, of course) can be done is the Tilt Brush Vive demo. This one doesn't really have HUD but it's easy to imagine how adding extra-information (like a map) to that interface would make a lot of sense.

 

I have thought about the map and decided that, you're right, it could be an annoyance/distraction taking you out of the immersive experience. (I decided you can bring up Map and Inventory in my game via simple button press -- such as a tap on Samsung Gear VR)

Other things, however, are critical data that change frequently and you would want to know at all times... things like health/life points. For those, I would think a simple text overlay in the corner would not be too obtrusive or distracting at all...

PS. There are ways to make VR work well without controllers. In fact, controllers are often what make the VR experience nauseating (especially when they control movement); I don't get motion sickness easily, but I wanted to vomit almost instantly when using Dreadhalls with a controller...

People need to rethink games, in general, for VR. For example, there's a big push for FPS games, but anything where you walk/run around really isn't well suited to VR. (It seems to feel more natural and less nauseating when you use a "Look-based" movement, selecting a target with your eyes and then fluidly moving there. The Zombie Shooter in Cardboard is one example, as is Matterport)

People are trying to make running around CoD-style work, but why not simply change some things to make it less clunky and unrealistic? (piloting vehicles, for example, is logical. I also am working on an innovative way to get the FPS experience without walking or running... I'll post the project once I get the first level done, might have to do a Kickstarter if people want a full-fledged game.)

I think a suit/wearable interface makes a lot more sense than the current methods companies are pushing (camera-based and positional controllers held in hands), and I am developing a jacket/gloves for just this purpose (prototype is close to finished.) But that's not going to solve the "walking around" problem....

The reason they are pushing the controllers is so you can walk and strafe, etc. I think it's a bad idea, and is going to turn a lot of people off when they experience how vertigo-inducing it is...

 

What you need to keep in mind about this: Current HMDs are built in a way so that the center area of view is pretty sharp, while when you look to the outer areas of the screen, it's usually blurry (that's at least what the DK2 is like - and it also makes sense to have it that way due to the way our eyes / focus work).

So usually, instead of just using our eyes to look at different things in the scene, we turn our heads. This is not ideal, of course, and it's in fact one of the things that causes motion sickness quite quickly when the framerates are too low. But it's the way things are at the moment.

So, "text-overlay in the corner" - while it makes a lot of sense for traditional 2D screens, it does not make any sense in VR (at least in my point of view). However, having this kind of information in 3D space, to the player's top right or left, so that they can look there (even though they'll have to turn their head to see it), works quite well.

I even have some UI elements that the player only needs every once in a while behind the player. So, instead of having to push a button, or navigate a UI, the player just needs to turn around, which is also working pretty well for the kind of experience where you're in a room that has a "forward" direction (once people realize they can freely move around in the virtual world without having to use a joystick or something similar that just doesn't work well in VR). To make this work in more general situations, we'd probably need some sort of hip tracking or "player direction" tracking which we currently simply don't have, yet. That way, the play could stand and turn their head far left or right and the elements could be attached to the player's main direction.

I wasn't talking about controllers like the Xbox One controller ;-) ... and in general not about controllers the way they were traditionally used for "screen games" (it's not surprising that this "Xbox One comes with Oculus Consumer Version"-thingee was announced at the same time when "you can play screen based games in VR by simply having a screen in VR", which I find ... well ... weird - because any physical HD screen simply is so much better than anything a virtual screen could be until maybe when we have 16K HMDs).

What I was talking about were motion tracked controllers. In other words: Controllers that give you your hands in VR. As humans, we are used to doing things with our hands. In traditional "screen games", there was no way to really have that in VR ... so gamers are used to using controllers / mouse / keyboard instead. But that's actually totally unnatural - and it doesn't work well in VR.

However, if you have motion tracked controllers, the players can more or less naturally use their hands. A lot of things can be done by simply "touching" stuff - and that's totally natural, so it's really the ideal approach.

But then, this is somewhat limited, so having a few buttons that are more "traditional", while it does make the experience a little more abstract is still useful because it gives you that many more possibilities (like, when you push a button on the controller, the controller turns into a multipurpose tool in VR). Oculus Touch controllers certainly area also promising because recognising certain "natural gestures" to do the same stuff in VR will extend the possibilities of what can be done in a natural and therefore intuitive way. A thumbs up gesture ... well ... probably just shows that they're owned by Facebook ;-)

Using a joystick for locomotion, however, really doesn't work in VR. It can work for vehicles, like a car or a spaceship, where you control that vehicle. Again, this is because it's something we're used to.

Personally, I think "real VR" really starts with room-scale where you can at least walk around a little bit in the world (by walking around a little bit in physical space - so once again, just doing what you're already used to do from the physical world). There are, of course, certain kinds of experiences that work with less than that - but to me, that's more like a "360° stereo screen" ... not really VR.

Of course, many people bring in the argument that only few people will actually have a room dedicated for VR, and that's a valid and strong argument. But if it doesn't work any other way (and I'm pretty sure that it won't for a long time), I'd rather go with what works.

I believe there is not a single case of motion sickness reported with the HTC Vive. The way I see it, the reason is because room-scale VR really works while most of the other approaches really are only a 360° stereo screen. And if you try to make it actual VR, it gets quite a few people sick.

 

a) I think the visual limitations you are describing (blur at edges) may be specific to current Oculus setup. I haven't tried DK2 (I am on DK1), but I have Samsung GearVR (mobile VR in conjunction with Oculus), and, even though the graphics capabilities are lower, it is actually a BETTER experience than Oculus in a lot of ways: 1) No wires. 2) The screen/display is better. There is no blur/distortion on the edges like there is on my DK1. 3) Built-in touchpad controller for simple interface

b) Your approach is a very limited approach, which may work for certain types of games. It sounds like it may work for your type of game... what type of game are you making?

However, there are many scenarios where you just can't have objects to look at in 3D space, because you are constantly moving through that 3D space... for example, FPS perspective games... not "fixed perspective" games where you are a 3rd-person observer looking down on a tabletop or something. In those cases, your interface would work well. Another example where it works okay is where it is FPS, but you are only going to be standing in one static location (such as Shooting Gallery VR, or some of the "tower/turret defense" games I have played... Romans from Mars, etc.)

Herobound is an example of a sort of "Zelda-style" VR game that is done VERY well and uses floating icons/overlays on the screen sometimes (such as to indicate where the player is if he is not in your field of view) -- this one uses a standard bluetooth controller, and in this case it's not jarring because it's not first-person view:

Here's another very well-done VR game (Viral) that is full 360, yet uses 2D HUD elements to indicate information at all times (a target/ammo counter):

I've decided to minimize on-screen HUD elements (to bring up inventory or map, you will press button and then select from floating 3D elements in front of you in worldspace), but for things like hitpoints... I don't know. I suppose I could save them for that screen and you'd have to check frequently... or I could try to get creative with visual displays to indicate low health (such as making saturation fade; ie. the closer to zero you are, the more black and white the world becomes)

The Oculus Touch controller IS a joystick controller. That's the point I'm making about it -- there is a thumbstick for each controller. Why? Probably to be able to move around, and that's silly.

I agree with you that you should be able to "interact" with the environment with your hands -- which is why the Oculus Touch is weak. It may track your hand POSITION in space, but from what I can tell, it doesn't know much about your fingers, so you can't really "manipulate" things with your hands. It's not going to be very realistic when your hands are clenched in a fist with your finger on a trigger, but it shows an open hand in front of you on the screen. I also assume it uses IK (ie. it detects the spatial positioning of your hand, and can then adjust arm position accordingly) -- also not entirely realistic, because I could have my hand in the same point in space while my elbow has about 90 degrees of positional rotations it could be in to make that happen. I suppose Oculus Touch would work well for boxing games or shooting games. But that's about it.

They claim you can do a "wave" gesture... I don't see how. If you open your fingers, you drop the controller... fist-wave I guess?

Also, I don't know all of the details, but since it uses IR LED (sort of similar to Kinect), this is very limiting -- you have to have a "camera" / sensor facing your hands to know where they are. This is not virtual reality. This is you sitting at a desk, facing forward with something on your head. What if you have a game where you need to turn all the way around behind you? What if zombies are chasing you and you need to look forward while reaching your arm back to shoot behind you? Not possible with Oculus Touch, or any camera/LED-tracking technology. Kinect worked well for what it was (a screen-facing interface); it is not the right solution for VR. (And Vive suffers from the exact same problems, although at least with that you can turn around backwards and use your hands... IF they happen to be in the headsets field of view. If not, you are out of luck. You shouldn't have to be able to see your hands in order to use them... I'm not looking at my hands as I type this, yet they still function...)


This is why I am saying suit/hand interface is way better than a positional controller. In fact, if you use finger-positioning, you don't even NEED a trigger button -- because pulling your finger (trigger gesture) would work just as well.

I am working on a prototype for all of the above (and some other people already have done similar -- not Facebook, for some reason, and I still don't understand the logic -- maybe it's much faster / lower latency? )



As for movement in virtual space: so far I have not seen an ideal solution to walking around. Yes, you could designate a VR room like you said, but this still limits your VR movement to the confines of... well, a room (probably a small one, if you are at home.) Never mind the injuries that are likely to result. That's why there are companies like "The Void" creating specific environments for this purpose (sort of like "laser tag arenas", but using VR); right now, that's the only real way I can see it working:

There are other people trying to solve this problem, using omnidirectional treadmills with harnesses. This still doesn't seem like a great solution, although being supported while you walk around on a giant roller/trackball is probably as close as it gets right now...

But another solution is to simply avoid use of legs altogether at the moment; have people sit in a hover-chair like Professor X. Or fly like Superman. Or any sort of locomotion that doesn't involve your legs (you could even involve your legs, if you did it via something like cycling on stationary/exercise bike)

 

Notice, too, that in "The Void" game above, they are NOT using silly IR-positional joysticks as the interface... (they are using suit/harness as I suggest)

 

There's several, some of which actually deal with moving in larger areas when you only have room-scale VR. However, I'm currently focussing on Holodance because that has game-mechanics that work perfectly with room-scale (instead of trying to make game-mechanics that require larger scale locomotion somehow work with standing or room-scale). The first is a short trailer, the second is a longer game-play video recorded with the Vive:

I addressed that in my posting. For this to work, you need hip-tracking, or, more generally speaking, proper "player direction" tracking (where the player looks is usually not the actual player direction - "hip-tracking" is one way to estimate the player's direction but it's not fully reliable).

It's up to the developers what they want to do with it. The touchpad of the Vive controllers could also be abused for that - but none of the demos I've seen so far did that. Because as you said: It doesn't make sense.

But you can very well also use a joystick to navigate in a UI that's built around the player's hand. There's a lot of things you can do with joysticks that have nothing to do with locomotion ;-)

That's developers not knowing what they're doing. Not Oculus producing bad hardware ;-)

I'm really not an Oculus fan - but they do support two cameras, and the way the leds are arranged on the controllers is pretty smart. With one camera, unless you hide the controller behind the player, it should work pretty well. With two cameras, it should be pretty hard to break the tracking.

You need to read up on Lighthouse. The controllers are perfectly tracked wherever they are in the room - you'd have to literally cover them to break the tracking. The headsets field of view is completely irrelevant for controller tracking. And it just works.

Read up on Chaperone ;-) I do, however, agree that room-scale VR also has its limitations. Just like using a mobile phone display for both showing what the player sees and controlling what the player does. And yet, there's a lot of pretty awesome games that work that way. The Void is a pretty good approach - especially because they have put the computer into a backpack ;-)

What you're missing there is that our brains are pretty awesome at knowing whether our body moves or not, or in which direction. Or whether we accelerate or slow down. There are potential solutions to this - but they are very tricky. So why not rely on the player's body moving the same way as the player moves in VR? A little too obvious, maybe, and in some ways limited when you are bound to "room-scale".

But sometimes, the simple solutions are the most fun ;-)

 

You may also be interested in a new room-based solution I saw in which a very small area is expanded in 3D space to simulate moving much greater distances (don't have the information available right now, but it was a study at a university or a research paper)

I'm not saying there aren't room-based approaches, and that there couldn't be some great games for it. But the major limitations are that: (a) many people just don't have the space. Here in California where I live, people are spending $3000/mo. for studio apartments (barely room for a bed, let alone an empty room to walk around in); (b) it still limits to a small area, so very popular games where you travel through an immense immersive environment wouldn't really work. (although I suppose you could somehow create some innovative "linking-room" style labyrinths)

I will need to read up on Vive. I honestly haven't done a lot of research on it; I saw the IR sensors and assumed it was interacting controllers off of headset. If you can do a whole-room sensing of IR, then that could work. I understand the idea behind IR-based sensing if the idea is to have room-based motion (since you can easily get whole-body positioning that way, similar to how they have been doing mocap for many years)

 

PS. Cool game. It does demonstrate the type of game I was talking about, in which 3D-space GUI elements work -- a "stand in one place" first-person perspective (like the tower defense / shooters I was describing)

 

Also nice to see that you are using Vimeo for your video hosting purposes (the co-founder of Vimeo, Zach Klein, was in my computer classes with me in college...)

 

BTW... here's my project, and you can see why (a) I need a map functionality (but I'm going to do it differently than originally planned... not HUD), and (b) room-style interface wouldn't work for this (the area is too expansive):

 

BTW, I just got a chance to try out your minimap (other things have been higher priority)... cool! Looks good. Don't think I plan to keep it as a HUD item on the screen anymore, but having said that, this is pretty unobtrusive (since it's off to the side and translucent)