Looking for advice on building large, complex cave level

Hi, I am currently planning a project based around a single, large cave system and am looking for advice on the best approach to constructing it. I'm used to using the basic Unity terrain system, but it won't be sufficient alone for the project due to the need for overhangs/tunnels.

The level will have a mix of very large (1000 units across) caverns connected with smaller passageways. There will be features such as stalagmites/tites, waterfalls, holes above to the sky and down into an abyss etc. The closest comparison I can name are the expansive cavern systems in Subnautica.

The level will ideally be constructed as a single level without loading between cave sections. In order to do this well, I think I will need to purchase additional assets/tools but most of the tools are $50+ and I don't want to waste money play with strategies.

There are two potential ways I can see to do this :-

1) Use Unity terrains for the cave floors, and build the ceilings, walls and rock features as meshes in a separate program (I use Maya, but I feel there might be more organic feeling tools for this). Then purchase some kind of splat tool to paint the mesh rock surfaces. The downside to this is there will be a seam were the terrain and rock meshes meet, but this may look fine if the terrain is clearly a "soil" surface sitting in the bottom of the cave.

2) Use a voxel terrain engine to craft the entire level. There are several available, and I am not sure how I decide between them. I DON'T need the terrain to be editable at run time, so I don't know if this is overkill, but from what I have seen this might be a one stop solution.

Does anyone have help to help me decide between the different options? Easy of creation, particularly organic sculpting tools are a big plus for me, provided they don't leave me with performance issues later on.
Also any tool recommendations would be appreciated!
Thanks!

 

What i did for Descent was building an advanced modular system.

Here's a basic workflow to try out:



Foreword: The picture above shows how it looks before and after the process. This can be automated with Houdini - for you, let me stick to an more simple workflow.

#1 Make planes with lets say 20x20, 10x10, 10x20, 5x10, 5x5 meters. Give them a tesselation that
a) is equal on all planes
b) good enough to get the "rough" details you need for having a good looking base cave environment.

#2 UV's or not? Yes, this question depends on which route you're going later. There are two approaches:

A) Each "plane" is using a texture. Then you need to UV Unwrap having in mind if 20x20 is your biggest piece,
you need to make sure that all others are using smaller UV space which will be "tileable" when the tiling is on 100% space on the smallest part. Sounds complicated but its not. You want to have an equal texel density everywhere. With that in mind, the smallest part must show a tileable texture. Big downside: You may see seams here and there later which might be fixed with custom vertex paints, decals and greebles. Upside: Its faster than option B.

B) The quicker, better quality at cost of performance way: Leave the UV's as they are, don't touch em. Use a Triplanar-Shader setup. Downside: There might be some difficulties creating a shader material that is supporting vertex map and with correct world normals. We did this in UE4 and its eating GPU's, not sure about Unity here.
Upside: Looking nice!

#3 Optional: Openings. Take a look at the pivot in the picture. I used a Cylinder and four 5x10 planes, then connected these. After that, i deleted the planes to get the cylinder with an quadratic shape surrounding it. (sorry for bad english).

#4 Create your first cave module!! In our game, we have a lot of them. Sort of 100 for any purpose we need to create caves. No, don't create 100 Pieces now. Let's start with one. Copypaste Planes, place these to make it look like in the picture above for example or to whatever cave "part" you like.

#5 WELD and FOLLOW THE RULE: What.. rules? Meeh. Yes but they're important for the next step

A) Weld the pieces together.
B) Make sure that you follow two rules here:
#Rule1: Don't have any plane overlapping the other.
#Rule2: Don't have any planes inside the cave or unconnected. If you look at the picture, each plane is connected with another. There is nothing inside the cave left.
#Rule3: Quads. Avoid tris.

#5 Once you're happy with the cave, make a copy for backup and safe it, hide it. You may reuse it for an simple Automap for example or to make changes later. The whole process now is modular with modifiers in 3dsmax. Not sure this can be done in Maya as well.



#6: You want to keep the openings clean to make them "connectable" to each other. Select the openings, invert the selection and add modifiers to make your "cave" shape. I used a push modifier, relaxed it, then added noise.





#7: Looks not too shady.. now we need to make sure the section between openings and cave are looking good so select the parts which may look odd and give them a relax modifier.



#8: Congratulations, you made a cave module! Well. In most cases you can just use the mesh as collider or create your own more simple collision shapes out if it. If the cave looks too blocky, you can start to smooth out some parts or soft-select areas and move them to a shape you want.

Notes:
- Don't make modules that are too big. Go smaller and make them usable like Lego pieces.
- You can make slopes, but they might have tris in the end. They may mess up this workflow so make sure that you follow the rules that all polys must be connected. No extra cutting juice here or you may end up with holes during the "noise and smooth" process.

 

That's a really helpful guide, thank you for taking the time to go through your workflow. I can see how sticking to regular quad only planes really makes things much simpler in the long run, with welding together and not getting broken geometry.

I'm definitely leaning towards using a tri-planar shader over worrying about UVs. There seem to be several suitable options in the asset store in Unity, although I'm going to stick to free ones until I work out exactly what features I need.

I don't need the same graphics fidelity as you have there, but I definitely want to keep things as simple and as problem free as possible, will try your approach and see how it comes out!

 

The good part about having a system like this is that it might be even compatible with dungeon generators by using sockets on the openings. Aside this, with just 3-4 different parts - you can build a rough Cave system from scratch. It looks boring and does need extra assets to fill it with life. Like Stalagmites, Stalagtites, little Water holes, etc. It could be possible to automate this as well but might not work if you want to rotate a cave module by lets say 90°. Important note about rotations. You may rotate certain cave parts like you want, but keep in mind that you may loose the ability to connect the following parts with an earlier opening by leaving "The Grid".