Baking lightmaps with Vray and 3D Studio Max, a tentative guide

Well i've just about figured out how to do this, and having found no real help on the forums i thought i'd make a thread and maybe help people do this themselves, this certainly isnt authoritative and i'm sure i'm probably wrong about a lot of things but i'll gladly take suggestions!

Beast is very good for producing a lightmap within most situations you'd want for games, particularly when you want parity between your baked lights and realtime lights, but for some situations, particularly mobile dev where you won't be relying much on realtime complex lighting, being able to produce lightmaps that produce a visually pleasing environment with the control you'd like in your lighting, it's nice to be able to use outside tools

This post is very much focussed on producing lightmaps for objects with tiled textures also

I'll start from the beginning, so to speak

You'll need to prepare your model, it's best to keep the main textures and lightmapping completely apart, a lightmap cannot reproduce the sharpness of a tiled texture unless your lightmap has a huuuge resolution (And i suppose we're talking basically baking in the entire texture into your diffuse channel here)

(Edit note: make sure your objects use vray materials or you may get strange results)

Assuming your model is broken down into stages where tiling a texture will look good and make sense (Keeping one material per model and sharing materials between models is good, so you can take advantage of Unity's batching), in max, keep your tiling to 1 on all UVW directions in your material, apply a uvw map modifier to your model, set on channel 1, for however you would like to add mapping to your model, do this for all models, keeping in mind not only several models sharing a material but also, and I suppose this has to be found via experimentation, seeing how large, in screen space, a model will be in-game, and how the resolution of your lightmap will affect the quality of the visuals. Set the tiling via the UVW mapping modifier, I find it very useful to keep your max and unity units as close as possible, so look at your units configuration in max to make sure they are in line with Unity's (I think 1m in max = 1 unit in Unity, and also set your grids and snaps to relate to this and how you would like to model), it might be useful to to click the fit button in the modifier panel then select 'Manipulate' and move the boundaries of the mapping so the top left of the model is in line with the top and left edges of the mapping gizmo and stretch the other two edges until it's some nice figure you can do the maths easy with, a gizmo of size 100x100 needs tiling of 100x100, easy!

With the mapping of your object's texture done, add an Unwrap UVW modifier to the object, change the map channel to 2 and click 'Edit' - It will probably look like a mess, right click in the window, select 'face' then zoom out (middle mouse) and select it all with a selection box, go to the 'Mapping' menu and select 'Flatten', the settings here, again, rely on experimentation, but the idea is to get your object flattened to 0-1 UV space and distributed evenly around it, a setting worth noting here is 'Padding' as it governs the space between each unwrapped portion of your model, if they are too close and the final lightmap is low resolution, part of the texture can 'bleed' onto other parts, making strange lighting artifacts appear

You should end up in the Unwrap UVW edit window with a handily arranged bunch of faces of the model, which represents the 3D model in 2 dimensional UV space, which is how you will apply your lightmap, a tiled texture would be very distorted on this mapping which is why map channel 1 is kept to tiling, however as lightmapping doesnt need to be sharp nor rigidly confined by tiling the mapping can be a little more flexible. Be sure that the modifier is using map channel 2, as that is how Unity knows where to apply a lightmap

With your mapping done (I've not gone into more here but you might want to look into optimising your model before all this, changing the topology of your model will affect the mapping and rendering lightmaps is lengthy, you don't want to have to repeat it too many times), it's time to render it!

Vray does it's rendering in linear space, which means it renders with a gamma setting of 1.0, which makes sense, except most monitors work with a gamma of 2.5 or so, which causes a huge problem with the brightness of renders if you arent aware of this - Not knowing about this causes people to try and boost their lighting to overcome it inside max's own frame buffer which isn't automatically adjusted for it, everythings overly contrasty and light falls off too quickly and worst of all, its completely useless for baking lighting, so you'll just need to make sure of a few things

In your max preferenced make sure your system gamma is set to 2.2, make sure this is applied to your colour selectors and material editor too, now assuming you've set vray as your renderer, in the vray options, go to colour mapping, and set the colour mapping's gamma to 2.2 also, there! your lightmaps will now render with the correct gamma, however you wont see this in max's frame buffer, it'll just appear too bright, which can be misleading but i'll get onto that

(Edit: with new versions of vray this has changed a little but my settings are to use linear in the colourmapping settings, make sure 'linear workflow' is off, if youre getting strange little dots on your render, try enabling clamp, set gamma to 2.2 (As your general max gamma correction should be) and from a nice tip i saw in another thread by Filto, if youre finding your lightmaps overbright and resulting in flat results, apparently lowering the dark and light multipliers in colormapping by 0.5 reduces the 'exposure' by 1ev which should hopefully achieve in much smoother graduations in the lightmap (feel free to correct me on this))

Now you're set to render, make sure you've got your vray settings how you'd like them, i'm assuming you know how to use vray and what you like in terms of settings, i think i can just point to a few things that can make a big difference, one is that your lights, assuming you're using vray lights, have their subdivisions set much higher than default, otherwise they'll cast very grainy shadows, another is that while the usual irradiance map is good for your primary bounces in indirect illumination, light cache isnt suitable for secondary rays when rendering to a texture, so you might want to change it to brute force, and you'll find if you use the preset low settings for irradiance map as your primary bounces you'll get very blotchy lighting on your lightmaps, so be aware that the calculation for your irradiance map can be very lengthy to avoid this

With your vray settings done, time to go to 'Render to texture' on the rendering menu. Each object in your scene can have different settings here (Which I quite like compared to some 3rd party offerings) and what I tend to do is have the output folder directly into my Unity assets where i'd have the lightmaps anyway, with the object you want to bake the lighting into, make sure its selected, make sure 'Use existing channel' is selected and choose channel 2 (The channel you used to unwrap the model and Unity's usual lightmap channel). Going down the panel, add an element and make it 'VrayTotalRawLighting', set the resolution as you'd like, set the filename to something you'd recognise and it seems I has to find this out the hard way but you want to save your baked lightmap in a 32bit format for full dynamic range, i'd go with EXR, make sure the EXR is set to RGB, you can delete the alpha channel in an image editor if you save as RGBA (If you don't delete the alpha channel the lightmap will have blue shadows in Unity) but i've had very strange behaviour despite doing this if using certain texture compression formats so best try render it without the alpha element - I tend to set it to save without compression and as scanlines too, no idea what this means but it seems to work

You might want to make sure it doesnt create shell materials too, select 'bake into existing' or something like that but don't select a target map slot, its unecessary and if you create shell materials it can be easy to forget and if you have to re-render it'll render with the shell material and get very confusing

Click render! See what happens. You should have the usual max frame buffer show you it baking the lighting into the model on a 2D plane, 1 you'll notice it's probably too bright seeming (Thanks to the gamma setting) and if you have the tiled textures already applied in max, which you should, to create accurate lighting, you'll see the textures in the render. Don't worry about this, you arent looking at the lightmap, just max rendering 'normally', annoyingly enough you don't see the output of the render element (in this case 'VrayTotalRawLighting') in the frame buffer, just in the resulting file, whcih when the rendering is done, should be an EXR file where you specified it to be saved

Now to get it unto unity! Well, go into your Unity project, assuming you've exported the model from max with its new mapping, presumably in fbx format, it should be ready for texturing and a lightmap when it's added to a scene. Because you did the tiling in the uvw map in max, you shouldnt have to do it in Unity, making sharing materials and therefore batching, straightforward, and now you have the second map channel, or UV2 as Unity calls it, ready for a lightmap. Obviously in the fbx import settings in Unity for your mesh, make sure 'Generate lightmap UVs' is unselected

Go to the windows menu, select lightmapping, select your object, in the lighmap windows, select 'Static' then set the lightmap index to whatever index you'd like that object to have (Make sure it's in the lightmap array bounds, easily set in the 3rd tab along the top of the window), make sure the scale is 1, the tiling on both x and y is 1, and the offset on x and y is 0, then go to the 3rd 'maps' tab and drag your exr that max should have produced from your assets pane onto the suitable slot, your object should appear to have assumed it's brand new vray produced lightmap! But it may look strange, make sure in your import settings for the image it's set to type 'lightmap' and you may want to play with the compression type to see what suits you, and you might want to change the shader for the object to something that suits your taste, ive been using the mobile> vertex lit shader

Haha god i must be bored, that was a rant and a half, but anyways if anyone finds something useful in there then i'm glad, just needed to get that out of my system, and if it was completeloy useless then at least i killed some time, cheers!

 

An excellent post, with a nice background to it. I must say though, one of the areas where beast shines above external lightmap tools is when you have a lot of levels to build out of prefabs, as each prefab will get lightmapped too if static. This has a lot of benefits in performance too, since with the 3dsmax/vray method you're dealing with what is essentially one giant mesh to unity.

However a tip: after lightmapping you can detach elements to separate meshes in an intelligent manner and save it as one file, then once in unity, you can pull these separate meshes out, making them viable targets for umbra, or even fustrum culling for performance reasons.

Thanks for the write-up, I am pretty sure it will apply even for people using mental ray or standard renderers.

 

Ahh yep other renderers simply use different render elements in the render to texture dialogue

I've gone the route of joining meshes where it seems sensible and exporting them individually, making sure they share materials where possible and taking them back off static once the lightmap is applied as i've read things on this forum about a static object rendering multiples of the verts of the basic model for reasons i'm not sure of

I've found while beast is really going to trounce an external program when it comes to games with a lot of realtime lighting, for fairly static lighting the control and quality of renderers like vray and mental ray is really great, I suppose as usual with these things it's a big trade-off

 

Yep, there is no question that VRAY renders just look far better. Don't forget that once its all set up in unity, you can tweak the lighting constantly and just copy the lightmap texture over, which is great for iterative development. Another cool tip is that you can export dummies with lighting information in them. I have tested this by putting dummies where I want my max lights, and they come through fine. You can parse the data of these in various ways for example, the scale is the radius of the light and so on, with the name of the dummy containing the light's colour, strength etc.

Should be able to knock out a quick max script for that sort of match up but unless you have hundreds of lights it shouldn't matter. I wonder if vray can bake to vertex colours too? baking to vert colours has several high level benefits: you can sample them in game for ambient colouring meshes and much more.

 

Regarding vertex lighting, no idea, but i think the benefit of using something like vray is lost when you lose fine control over shadows and global illumination etc, as there's no real analogue in realtime environments, although exporting dummys from max to Unity is a very good idea in terms of keeping your max baking and realtime lighting coherent

When you set up the baking as i described, every time you bake the lighting it'll automatically update when you return to Unity, which IS great, i love that Unity monitors changes to files, but one huge thing against vray is its slow, really really slow, just the indirect illumination calculations can take forever if you dont want blotchy lighting or leaking, i suppose it's all relative because i'm used to waiting 12+ hours for a normal render so 20-30 minutes isnt THAT bad but compared to beast it's a little taxing

 

I stumbled across a vray settings free plugin, which tweaks settings to get 99.5% or so the visual fidelity you have but with a quarter of the rendering time. I wish I could remember what it was called.

 

I'm not sure what Solidrocks goes for, it might be what you're thinking of, not free, mind, but very good, affordable and makes for very speedy renders (that strangely have a whole 'look' of their own, very different from the vray presets), but there are points where things just cant be sped up very easily, i'll reference again shadow quality for lights, in texture baking in so much detailed shadow over such large areas, noise becomes very apparent (Another hint though, when you render lightmaps at 1024x1024, try cutting their res to 512x512 in the import settings, will likely look no less nice and will blur a lot of noise out)

 

Nice guide. Very detailed. I will read it later.

 

Yeah thats it. It's got a free version but costs about 54 euros.

 

Sorry to bring up an old post but im struggling in the last part.

"Go to the windows menu, select lightmapping, select your object, in the lighmap windows, select 'Static' then set the lightmap index to whatever index you'd like that object to have (Make sure it's in the lightmap array bounds, easily set in the 3rd tab along the top of the window), make sure the scale is 1, the tiling on both x and y is 1, and the offset on x and y is 0, then go to the 3rd 'maps' tab and drag your exr that max should have produced from your assets pane onto the suitable slot, your object should appear to have assumed it's brand new vray produced lightmap! But it may look strange, make sure in your import settings for the image it's set to type 'lightmap' and you may want to play with the compression type to see what suits you, and you might want to change the shader for the object to something that suits your taste, ive been using the mobile> vertex lit shader"

When i go to the third tab 'maps' there is no texture and i cant drag any texture either. The only way i can drag a texture is if i bake the element with beast, but it has different UVs.

Please i would appreciate your help.

Thanks

 

Hi LazyGunn thanks for the guide

im struggling in the last part too. I am using Max 2012

could you make some short video please......please....

 

I would really like a video to, Maybe where you bake just a simple object. Would be really, really regarded.

Thanks alot in advance.

 

Ahh hi folks, hmm, well, from what i understand the problems you guys are having is not seeing any slots to drag the lightmaps into in the maps tab?

On that tab theres an 'Array size' value, knock that up and you'll create an array for the lightmaps - These are the indexes you put your own baked maps into in the object tab, you should be able to drag your lightmap from your project view onto the slot

As to a video, i'm not sure,i havent done such a thing before, but i could do that i suppose, as i'm embarking on a little experiment on how to recreate a max scene nicely in unity 4 free

 

Hey guys, I've been doing this lately as well but on full production scenes. I use 3D-IO's Flatiron to bake the entire scene (hundreds of objects) into a couple large EXR maps. One of the huge advantages, aside from baking so many things at once, is that you can preserve instances with it. If you are looking to do this more and don't mind the $350 price tag I would highly recommend it.

Great tutorial by the way, wish I had come across it before I had to learn it all on my own too, haha.

 

Ahh ive had a go with that before (flatiron),its really good but i couldnt get along with the workflow at the time, probably through lack of commitment to it, its a great suggestion though. I suppose what stops me is lack of being sure of what the outcome of a bake will be (especially when i have several different shaders in mind for various objects) and having very limited resources to get the rendering done, its a pain being smalltime. I'll maybe try do some videos using my rather slow process it might help get things going, but if i can look into flatiron id definitely try and show people how to move up to using it for larger scenes, assuming my computer can work them out in timely fashion. Cheers!

 

hi
I can not see the result on my unity,I have done all the things you mentioned, also I success to drag the lightmap to ''suitable slot,'' But I　can not see the right result, nothing changed...Pls help me.

 

well, key things - your model has the uvs for lightmap on uv channel 2, your lightmap is baked and turned to lightmap type in unity inspector, in unity, you've set the object to static, in the object tab of lightmapping window you'e clicked the lil arrow next to 'atlas' and set the index to,say, 0 (its 255 by default), then in the maps tab youve dragged the lightmap onto the slot for index 0? Obviously for later objects you increase the index.

usually when it hasnt worked its that i got the index wrong, still no luck?

 

thanks it works, I did not do "clicked the lil arrow next to 'atlas' and set the index to,say, 0 "
thanks a lot , I spend almost 8 hour on your tutorial... it was very nice....help a lot ....
thanks

 

glad to be of service!

 

Nice Post lazygunn.

We've been using flatiron to create a vray bake and move it over to unity. The scene we're creating is quite large and has large groups of geometry which have all been mapped into say 40 maps.

I was wondering if anyone knows of a way to automate the process of atlas setting, or a quicker way to know which map contains the data for a specific peiece of geometry. As at the minute I'm having to go to each piece of geometry and set it's atlas manually. This is very impractical in a large scene like ours.

Cheers guys

 

On the recommendation i decided to have a look at flatiron and i've found that apart from one annoying bug (You get strange results using single pass render wth vray) it's actually really, really, cool

Are you mapping each piece of geometry individually with flatiron or groups together (Sorry just wanted clarifying)? What i would do would be to name, group, layer (In fact layers sound like the best idea) or use selection sets to group your objects by area or function then unwrap the entire group together with flatiron, then at least you always have a rough idea where something will be using your convention. If you unwrap an entire group you can still render/rerender a single element with single pass turned off in flatiron then i suppose replace it in the original bake using photoshop or suchlike

I suppose i dont know a way to do what you want to short of plain organisation and above but flatiron will automatically unwrap everything you have selected and put it in the same atlas, so you can create very specific groups of unwrapped objects

 

Hi, LazzyGunn!

I'm trying to bake a little scene I have rendered in VRay, but my lightmap outputs are just awful...
Even though I've followed your tutorial, I'm still so confused with the process...
Have you got any video about it?
Other question that came up in my mind while I was reading your post: how can I make cubemaps for my reflective materials from the same scene?

Thank you!

 

This is the scene in trying to bake

 

I've been sayng i'll do some videos for a while... i think i probably should as its coming up more and more

But for now can you show me images of your scene rendered in max and the scene as it appears in the unity editor?

Theres a free script for making cubemaps here http://forum.unity3d.com/threads/57805-Cubemap-Generator you simply select your reflective object and it will create a cubemap from that position.. i was having strange problems with it however and purchased the most recently updated (theres 2) cubemap gens from the asset store. It's not bad but a bit fiddly. Getting a cubemap then that way isnt too hard, i would also recommend looking into a couple of other things too though,one is a script to correct the projection of the cubemap so it reflects accurately in a room, the post is here http://forum.unity3d.com/threads/11...Box-Projection-Correction-Environment-Mapping (steva's shader is very good and you can fix it into much other surface shaders you can think of), and the other is amd's cubemapgen http://developer.amd.com/resources/archive/archived-tools/gpu-tools-archive/cubemapgen/ its discontinued but still available to download and will allow you to blur your cubemaps nicely for glossy reflections, mixed with a normal map it can look very nice

 

well how is it looking in Unity?

 

Hello guys,
Don't mean to hijack but i followed your tutorial, thanks! I have a question about baking textures in Vray and Unity i'm hoping someone can answer.

I have a room with windows and a wood floor. A sun light shines through the window casting shadows on the floor. (the basic setup)

The problem... in Unity im wanting to tile the floor using a simple map and then use a lightmap for the lighting (fine - I Know how to do the basics of this)
What I can't seem to do is get a lightmap with the shadow data in it but with out the texture.

Quick test below shows what i mean... if i use the totalrawlighting I get it all but no shadows from the box or tea pot, this means when its all setup in Unity; 1 the floor is too bright because the shadowed area is not concidered and 2; you dont see the shadows from the windows on my scene floor.
If i use any maps that show shadows, i also get the texture, this kind of works but then causes problems back in unity with the texture being too dark as you have essencially got it on their twice.

The question... Is there any way to create the lightmap with the shadows but not texture? or am i missing somthing in Unity,

Samples of what i get out of vray showing no shadow data in the lightmap
http://imageshack.us/a/img841/2222/testyb.jpg
http://imageshack.us/a/img32/2135/test2dz.jpg

UPDATE:
Just to clarify further these snapshots show the issue:
Vray texture bake - lighting fine, colour fine but resolution not good
http://imageshack.us/a/img94/2609/64051808.jpg

Using a lightmap and tiled texture (lighmap has a texture also to get the shadows) but you can see the floor is dark with both textures but the resolution if good
http://imageshack.us/a/img525/8254/85400941.jpg

Bit of a workaround... using the lightmap (with texture) but then using a tilable (almost bump map) to pick out the details - would not need to do this if I could get a lightmap with out the texture but with the shadow
http://imageshack.us/a/img259/8682/32676659.jpg

 

I'm having trouble understanding the problem - in your first image, you have a vray render that doesnt seem to have shadows.. is that intentional or is that just the result you're getting?

And you're saying you get shadows from the exact same setup if you put a texturemap onto the floor geometry?

Are you using flatiron in the process by the way?

 

Hi,
Yes the top image is a "VrayRawTotalLighting" map and as you can see no shadows on it (so in Unity if i use this map in UV2, as a lighmap and also tile a simple texture onto UV1 it works but i have no shadows and its all a bit overblow because any shadows are missing)

The one under is a "VrayLightMap" and this has shadows but also textures (so in Unity if i use this map in UV2, as a lighmap and also tile a simple texture onto UV1 again it works but this time i get the problems shown in the last three images posted above. i.e darker because of the double texture information)

No im not using flatiron just render to texture.

I think the problem is with the vray outputs.... ive tried all their available elements in 'render to texture' but none output the light with shadows but no textures.

 

Are you using vraymaterials on your objects? Vrayrawtotallighting should be giving you shadows

 

You know what... i'm not sure i was! can't believe id be that stupid. I'm out and about now for the rest of the day but will try again tomorrow and let you know.
Thanks again for the tutorial, it was very useful.

BTW... just to save others from the same mistake in unity... when you create a lightmap array, keep in mind that the index number start from 0 so you're first object index number will be 0 not 1... puzzled me for a good fifteen mins yesterday when going through your tut.


EDIT - It was indeed because I was using std materials.... (feels foolish) thanks

 

Hah, easy mistake to make, but really you should always be using vray materials with the vray system, they're faster, look better and are fully optimised for the system.

 

is this problem only at my end or render to texture doesnt take exposure controls in account when saving exr?
i set my scene with vray physical cam and use vray exposure control to select the phys cam.
in render to texture the tga and jpg render fine but exr is overexposed .. oh well ill set my scene up with max's standard cam.

 

I havent actually tried that, i would have thought setting the vray physical cam as your render node in vray exposure control would adpt for the render to texture too so i find that it doesnt a bit strange and sounds like a total pain in the neck too. I would have kept the vray cam personally and turned exposure control off and dialled down the light multipliers (well, if id gotten into the frame of mind to use the vray cam, i dont actually usually use which may be my bad) but then again, i dont know nearly enough about the exposure control functions as i probably should do

 

using standard cam now, and exr are coming fine..
is there any advantage of using 16bit exr in unity as a lightmap, or the end result is same as 8bit jpeg? there wasnt any different in my experiment. so i suppose the only advantage of exr is to have a fine control over exposure if you dont like the end result... another thing that i find annoying is no curve control in photoshop, i have to open the exr in aftereffect to bring out the right contrast.

 

Hi guys,
Sorry to poke my nose here but i am having a problem. I did everything same but when i try to render i get this error" The following elements do not specify a target map slot:" what should i check or unchecked to get rid off this error? and that texture don't even work in Unity which i get after light mapping.

 

Don't worry about that message; just click OK to proceed.

Since the result is an EXR lightmap so it cannot be used as a texture...

 

Thank you very much <lazygunn> for this helpfull post...
Also Thanks to all guyes for sharing their experiments...
After 2 days with many tries working on INTERIOR scene, I discovered that if I want to use [GAMMA] color space inside Unity then I have to save the EXR file with override gamma [2.2]...

 

Hi Guys

Though i am late by a few years...thanks a lot lazygunn for the detailed step by step guide for light baking in vray.

This might seem a little off topic. But, I am trying to bake a huge scene in unity. But at 50 texels, the number of lightmaps crosses 255, which is the array size. So i thought to bake it in 3d software and then import to unity. But this method too involves adding the lightmaps to the same array. So finally i end up using the same limited array of lightmaps that unity allows. Is there a way around this problem??