Question Projecting texture onto world from camera to add detail.

Hi All,
I am trying to create the world for an isometric pixel art game in 3D to avoid dealing with sorting orders and design large spaces quickly.
I have accomplished the task of making the world appear 2D and stable with no pixel crawl.
Ignore the ugly test environment.


However, the terrain lacks detail and I don't want to have to texture the entire world traditionally.
Ideally, I would like to be able to capture a screenshot of the world from the angle of the camera, draw on that image and then project it back onto the terrain as a texture.

The camera orthographic, a fixed 270p resolution and its view angle doesn't change so I believe this could world well if it is possible to do.

Any ideas would be great thanks

 

It is also of note that the camera is on a 60-degree angle and the view matrix is scaled by 2 on the y-axis and 1.155 on the z-axis to achieve the desired view angle.

 

For starters, just wanted to say I love the look. I did something similar a few years ago but i really pushed the low-poly aspect. The way you've made it look like authentic 2D pixel are is very nice though.

I suspect you could do something clever with stencil buffers but I really don't know enough about that kind of thing. If it were me I'd probably encode some kind of id into the material data and use that to decide what screen-space details to render in the shader itself.

 

In the tool you're using to make your 3D assets, you should be able to do a planar projection from any angle / scale, and you should be able to match the resolution and angle your camera is using in Unity. Then it's just a matter of painting the texture as you want.

Alternatively, if you really to, you could calculate the new UVs in Unity and update the meshes with a c# script. You kind of already have the transform you need from the camera's projection matrix, just scale it up to some power of 2 resolution. If you're already modifying the camera's matrices you should be able to figure that part out.

You can also think of this projection as a 2D plane that you can use as "world coordinates". If you've already handled the pixel crawl, presumably you're doing some kind of pixel snapping with the camera? That pixel "position" is on that projected grid.

The "hard" part is how to handle what I assume is going to need to be multiple textures to cover the entire world. Either you'll need to automatically group parts together based on if it's contained within the bounds of each texture you use, or do this manually. You will have needed to do this manually if you setup your UVs in an external application anyway, but there's a lot of ways to choose. And you need to make sure you never have a single mesh that's bigger than your largest possible texture size. Otherwise you'd need to slice up the mesh. Speaking of which you'll probably end up with a lot of overlapping textures, and that's fine.

For taking screenshots, you could make a tool in the editor that renders to a render texture and saves it as a .png file. Maybe named with the current 2D coordinates the camera is at. You can also then write a tool that can read a texture, decode the coordinates, and copy the relevant parts to the textures the game actually uses. Since all of the meshes are using the same projection, you can again use that 2D projection to figure out which meshes were visible, what the textures are, and what pixels of those textures are visible in the screenshot.

I know I'm being somewhat nebulous here, but that's because there's no "here's exactly how to do it" that doesn't involve someone basically writing all of the code for you. How you want to solve it is kind of arbitrary. There's nothing built in that'll help you either. You're going to have to write it all yourself.

If you get stuck on specific parts, come back and we can help.

Yeah, no. Stencils are super unnecessary here. That's a red herring.

 

Well, I'm not sure if I'll be able to get the workflow to be efficient enough to be useful but exporting the probuilder terrain to maya to project the UV map from the correct angle and then drawing over a rendered image definitely seems to be possible.
Your proposal for how a custom tool could automate much of the process would make it much easier and then the only problem is the possibility of not having much flexibility once I have started drawing details by hand.
Thanks for the help bgolus

 

Ah I see now. I totally misunderstood what you meant by 'details'. I thought you mean some kind of procedural detail map-like texturing applied to the materials. You actually meant specific pixel drawing-like details in specific areas.

Hmmm, assuming you used your projected camera to draw the raw pixel art onto a series of textures could you perform some kind of meta-texture style streaming to apply it as kind of post process layer drawn after the geometry? Shifting and streaming new textures in and out as needed of course as the camera pans around the scene.