My Texture Atlas Nightmare (With Tips!)

I have been determined to squeeze as much performance out of Unity as possible so I just wanted to share my nightmare journey with texture maps so some other poor soul might not experience the same issues that I have so far.

Why Use Texture Atlases?
Why not? Anything that can be used to increase performance is worth it, right? Well maybe not. Whilst there are many benefits to using multiple atlases (less draw calls, less changes to the render state, no calls to get/bind textures to objects, etc)


Why You Might Not Use Texture Atlases
- You can't tile textures.

- You need a good idea of the scene/section your are building to minimise wasted UV space (I've watched a lot of AAA talks and read a lot of papers that seem to hold a standard of 70 - 85% (so only 15% - 30% space wastage).

- Authoring new gameobjects can be a hassle due to the requirement to expand your texture atlases.
- Basic stuff just doesn't work anymore. Normal maps? needs custom work every time. Terrible mip map island bleeding throughout the mipmap chain.

- You can't have a metal texture and a wood texture on the same atlas if you want decent results as they react differently to light and require different material settings.

- Edge cases... ie - is it worth even putting on a texture atlas? (a single box that appears once in an entire level) etc.

- You need to be very specific about your texel/pixel ratio for assets/textures which can be laborious and time-consuming.

Issue #1: Mysterious Seams
A common issue I experienced is seams showing up on "seamless" textures in the scene view. This was quite confusing as I was 100% sure the UV mapping was correct in Blender (using snap to corner).



The solution to this is quite simple. In the inspector, "Alpha is Transparency" must be enabled for the texture.



Ok, great. No more seams! Let me just zoom out and get a better view.

Issue #2: You Got Your Concrete Texture In My Wood Texture!



This was another issue for me that took a while to reach the solution. The problem is mip maps and how they work (or specifically how Unity implements them).

The texture atlas is 4096 x 4096. This means there should be 11(?) mipmaps in the chain. You will never be able to stop bleeding at certain levels since mipmaps go all the way to 1x1. However, I think the "important" chain sections are the next two iterations from the "main" texture.

So, for a 4k atlas the only concern is the mipmap quality at 2k and 1k (I'm sure there is a reason to care about even lower tiers but I think two levels down is a good start).

How do I figure out how much space to use between islands? I spent a lot of time researching this issue and ended up watching a lot of GDC talks by AAA studios, and also read some papers on texture atlases which led me to a paper that used a very specific (yet simple) equation.

Texture Atlas Size / 215 = Island Space

4096 x 4096 = 19 pixels between each island
2048 x 2048 = 9 pixels between each island

(Note: This was specifically in reference to a texel density/pixel ratio of 512 pixels/meter. I will talk about this further down the post).

Issue #3 Texel Density/Pixel Ratio
For a long time I was used to just scaling a texture on an object until it "looked right" because it was practical and simple. Trying to transition to using an atlas presented a new challenge... mainly, how do I know what size textures to use?

The solution to this is to designate a specific density/ratio and then stick with it. For example, 512 pixels per meter. This would mean a 2m x 3m door would require a texture that is 1024 x 1536. For a 2m x 2m floor you would need a 1024 x 1024 texture.

This works better for modular sets. I'm still unsure how you would go about using an atlas for a huge object (like a ship that is 100 meters long) but not modular.

The logical conclusion would be that you can fit 16 floor textures onto a single atlas. However, that is not correct. You need 19 pixels space each side (95 pixels per row), which means you only have 4001 pixels to work with.... not 4096.

Just because you have less space doesn't mean you can't get value for money. What I ended up doing was, for each 1024 x 1024 seamless texture, was to just resize the image to 1000 x 1000 pixels. This results in still being able to fit 16 textures within a single atlas whilst providing adequate space between islands.



Issue #4: Normal Maps & Ambient Occlusion Maps
A huge issue for texture atlases (specifically when using tilable/seamless stuff) is most generated normal maps will cause the seams to show. Why does this happen?

In my experience what happens is the normal map is never flat around the edges of a texture. This is fine for non-repeating objects (tables, a cabinet, a door) but looks terrible for modular assets. How to fix it? I have not found any method quicker than manually modifying the normal map and AO map to essentially set the weight to 0 around the edges, essentially skipping the normal map process for all seams/edges.

Final Result (using a different texture atlas than the one posted above)



I would be happy for anyone to point out something I may have missed or contribute as to how it's possible to get better results using atlases and how they fixed the problems I mention above.

Hopefully this might help someone else as it has been quite a headache to get to this stage.

 

You now that Unity's standard shader supports metalness/smoothness maps so you can indicate different PBR properties on different objects using the same underlying material.

 

I actually didn't know that was possible and just found the Unity docs on it by googling "metal smoothness texture" so I'm grateful you pointed that out.

"In this example, a Metal/Smoothness texture map has been assigned. The buckle now has a high metallic value and responds to light accordingly. The leather straps are shinier than the leather body of the box, however they have a low “Metallic” value, so it appears to be a shiny non-metal surface. The black and white map on the far right shows the lighter areas for metal, and mid to low greys for the leather."

so I just have to create a custom black/grey/white texture to indicate metal or non metal surfaces.

 

yeah, for the metalness workflow, the alpha value (opacity) indicates how shiny the surface is (more opaque=more shiny), and redness indicates how metallic the surface is.

 

I was actual considering trying to hack it using an ambient occlusion map. The idea being that metallic stuff would just be 100% white to up the amount of light that gets reflected but knowing this now I'm going to experiment to see what results I can achieve with a custom texture.

 

Mip maps are a pain, I have the same problem with lightmaps, haven't been able to solve it sadly


https://forum.unity.com/threads/strange-mesh-seam-artifact.879337/

 

Yeah I experienced similar issues so ended up using realtime lighting to avoid dealing with any lightmap issues. I'm still experimenting with texture atlases (and lighting) so if I find anything that could help I will post it in your thread. Hope you find a solution eventually. It can be incredibly frustrating to get stuck without a clear way forward.

 

Just my experience, but save your optimization until end of your project. Just make the art look right first. Work fast and dirty. At the end you'll know for sure what is going into the game and what isn't, so you can safely consolidate your resources then.

 

This is an interesting perspective. Have you ever run into issues by waiting until the end to optimise? I will admit trying to pre-plan everything can be a nightmare if you change/scrap/add stuff in the future. To get around this I just try to lock-in what assets will be in which section and go from there.

My only worry would be that you would need to go back and redo 100+ assets. I read a whitepaper about doing things "quick and dirty" and then, for example, using a custom atlas packer to generate atlases at runtime or on-the-fly. I've just not had any great results trying to do this myself.

 

Avoiding redoing things is key reason for waiting until the end. Too many variables involved, its impossible to plan for. It's not until you basically complete the game that you understand where all the important interdependencies are. At that point its safer to start doing "destructive" work.

It's not a hard rule, just something to keep in mind when making decisions. You'll get burnt out by doing complex organizational work just to scrap it when you discover something new that forces things to change.

That's just speaking from my own experience, but plenty of horror stories about this sort of thing from major studies. People quitting major projects because somebody changed their mind about this or that which means redoing work for everybody else. Or some technical limitation is discovered so everybody has to redo a bunch of S***. If the big teams with money and experience face the same problem, you might as well plan for it too.

One downside is if you were making a game that is pushing hardware boundaries, then you need to figure out your limits sooner, right? But that is a good reason to not make games like that unless you either have tons of experience to be able to make accurate predictions, or have a team of experts at your disposal. Otherwise, just stick within reasonable scope, work fast and dirty, then at the end consolidate things as best you can.

Some of the programmers here can probably say something about automated texture packing. For me I just do things the dumb manual way. In general, I stick with industry standard practices developed by environment artist for that sort of thing. In general that means packing materials by type, making use of trim sheets, etc. But in order to get to playable, shippable status sooner and not waste time doing work they has good chance of needing to be redone, I just give everything a unique material to get the ball rolling. Inevitably I might discover that I don't use this or that model, this or that material, so if I took hours to plan out organization of texture sheets then that is all wasted.

 

Thanks for your input @BIGTIMEMASTER it's definitely shown me I still have a lot to learn in terms of development. I'm certainly not trying to push graphical boundaries but maybe I am a little too focused on graphics rather than the overall picture.

(latest attempt - Normal + AO)



This is really just an experiment in terms of level/environment design methods though but it has shown me that if I wanted to make an actual game than this might not be the best way to do it in terms of efficiency. Modelling, uv mapping, stitching atlases together has been a lot more labour intensive than I originally thought.

 

didn't mean to deter you. It is good to experiment and learn what can be done, of course. And only way to know is to try things out, post about it, talk about it, etc. So you are doing everything right it looks like to me. I just making a point to not do this too early in development. Its good to have an idea about how you will do it though, and so taking time to figure out the general workflow is good idea.

 

It's fine I'm not deterred. I just think that I had a "one method fits all" mentality when it comes to level/env design. For example, I watched a few interesting GDC talks about linear level design where they essentially split the game into sections, and it allowed multiple people to work on different areas.

The other benefit to this was their ability to compartmentalise sections where "Level Section 1" could be fully complete, textured etc, but "Level Section 2, 3, 5 etc" were in various stages of development. You would start with the first section of your game, essentially do all the models, texture work, gameplay elements etc, then move onto section 2.

This is what I was experimenting with.



I was able (through meticulous planning) achieve a pretty decent result. The entire section uses a single atlas material. The only exception are the stairs which brings up another weakness to atlases.

The stairs are 4M x 6M which according to my texel density spec would require a texture of 2048 X 3072. This would be (imo) too much space to dedicate within a texture atlas for a single object. Eventually I decided to use a 512 x 512 texture and tile it under the impression that it's actually more efficient due to

1) The stairs being a single object and not likely to be rendered "constantly".
2) the 512 x 512 texture is better than a possible 4k texture just for a set of stairs.

and it doesn't appear to make a difference in terms of quality.



If I were to continue then I guess all of the props (tables, desks, chairs) could probably fit into a single atlas too. Essentially using only 3 materials for the entire section leaving more resources for better lighting/effects/shaders etc.

 

You should also ask and post your results at the polycount forum. Lots of environment artist there who probably have plenty to share regarding this sort of stuff. Good luck!

 

this is exactly what im looking for at the moment as im trying to make my project so optimized as possible due to it being a VR world and an open world at that.
I would like to know how you add the normal and heightmaps, emission and the other atlases. If there is gonna be any atlases for the other parts of the material.
I cant seem to find a good explanation to how you implement an atlas for every material part.

 

This is a great thread to bump, I always just kinda manually tweaked my texture padding, but knowing a formula is going to make my life a lot easier.