HELP! struggling learning concepts behind shaders

Its really easy to find a ton of tutorials and examples that focus on the syntax as well as concepts behind shader creation, but I am struggling to find resources (even non unity based ones) that focus just on the theory/concepts and do away with the hard to do math / syntax.

A good example:

Lots of tutorials will talk a lot about the actual syntax for sampling a texture "sampler2d etc etc" but not a lot about why and when you would want to do this.

I guess what I am saying is I am struggling to work out why people make a lot of choices for shader creation, and perhaps having a resource where the actual thought behind planning the shader is emphasized over the actual syntax and writing of it.

I would love for instance to go "I have an effect in my mind" and at least know what techniques to start with. Right now I know what a lot of buffers etc do but not when or why I should use them.

How did you wizards get past this part of learning?

Any good exercises I should do to improve this aspect?

EDIT: I understand why and when you sample a texture, I was just using something very basic as an example

 

By working with people with more knowledge than I, reading a lot of presentations on shader optimization, and trial and error testing on my own. Still a lot I'm terrible at and much that simply isn't documented.

Generally though when it comes to why someone does something in a shader, it's because that's the way they thought of to do it and it's often not a lot more complex than that. For more experienced devs there are of course elements of understanding of the hardware and mathematics for what is faster, but that seems a bit further than what you're asking.

Do you have a specific example?

 

Hi thanks for getting back. I really enjoyed your triplanar article on medium, very informative even for the novice!

I dont really have a good example, perhaps a better question is:

What are some good things to "play around with" in shaders that will help me learn aspects of shaders and graphics programming?

An example of what I mean, to understand texcoords I started saving arbitrary data into them to understand what I could and couldnt keep inside them, it made me understand what I could achieve using them and just how flexible they are, and in the process accidentally learnt more about UVs.

Are there any other things that are reasonably easy to experiment with like this? Or exercises that will expose some of the big pitfalls etc that everyone "should make the mistake of" at least once? Just struggling to fill my knowledge holes with the very unfinished documentation.

EDIT: is there a reason people dont comment shaders much? is there some kind of performance hit for extra lines or is it just people dont like to comment and explain their tricks?

 

Thank you.

Shadertoy has some great stuff for learning fragment shader tricks. For stuff that includes vertex shaders there's a few things on Emil Persson's site, but otherwise it's strewn haphazardly around the internet in random GDC talks, academic articles, and the like. There are also plenty of examples out there here on the forums or on Github of people doing various interesting things. Keijiro Takahashi's github page has some amazing stuff.

From my personal experience I would say learning to understand the basic chain of how matricies convert from local mesh space to positions on the screen, and the basics of rasterization (the magic that happens between the vertex shader and the fragment shader) are extremely useful to understand. I also found learning how to do basic trig with dot products to be invaluable. But both of those things might seem kind of math heavy, and they are, but it's more about understanding the various steps then understanding the actual math.

One last thing to think about that most people don't, precalculating data and passing it from the CPU to the GPU, or even from the vertex to the fragment shader, may seem like good practice, but there's a cost to moving that data. With modern GPUs (even on mobile) it can frequently be much faster to recalculate something for every pixel than pass it in from the vertex shader, even if it seems wasteful.

There's no performance penalty, but I'd say most of the time when you're writing a shader you don't know what something is "called" well enough to document it even for yourself. Especially when it comes to doing some effects where it might feel like you're just randomly adding and multiplying stuff together to get what you want. How many times can you use the term offset or gradient? I have a hard enough time thinking of variable names sometimes, let alone document what I'm doing.

 

Thanks so much, this is literally invaluable information. Looks like I am on the right path, guess there is no quick route!

So far I have been following the rendering tutorials by Jasper flick (catlike coding), alan zucconi's blog, minionsart for some cool toon shaders, and then I follow you and the creator of microsplat.

To be honest, the most relieving bit of info to hear is that experienced people "randomly add and multiply stuff together". I think this is the main reason I have found it so dense a subject, it seems like a lot of it is trial and error and now that I know that, I wont be so hard on myself when I have to take that approach to work the stuff out.

Looks like practice is the next thing on my list then!

I've just started writing a surface snow build up shader based on a tutorial, then ill try and translate that into a vert / frag one, before trying to get the snow normal mapped and textured so I can use it for other effects. Ill just keep taking shaders that I understand as a starting point and practicing by building and experimenting on them, one day hopefully Ill know enough about what I am doing to help someone else out!

Thank-you for your time! Also please keep up the medium articles, terrific reads! Link them on unity connect too

 

So, I have taught a few shader classes over the years. These are the concepts I use to get my students over the hump in writing shaders:

1. I am a pixel.

People think about shaders as if they are working on a texture or image, but this is incorrect. You really need to think of a fragment shader as "I am a pixel, what color should I be?", or a vertex shader as "I am a vertex, where should I be positioned?". When you see mario brothers written in a shader on shadertoy, what you are seeing is every pixel computing the full game of mario brothers to determine what color it should be. This is a total mind F*** when you are used to writing more traditional code, but once you start thinking this way everything becomes much easier.

2. Numbers == Colors == Math

Artist in particular tend to think of numbers and colors as different things. They think of a texture as an image, not an 2d array of vector4 values. They think of UVs as vector2 values on the vertices, not as a gradient mapped over the face of the triangle, or colors stored on each vertex which are interpolated across the face for you. Once you start realizing these truths, you start to look at textures in a whole new light, start stuffing data into UVs, distorting UVs with texture data, and start using the data you already have to generate new effects.

3. The math is not actually that complex

Lots of people think writing shaders means intense math, but this is simply not true for many shaders. Most of the math happens in a 0 to 1 or -1 to 1 space, and is fairly easy to understand. I was thrown out of highschool before I learned algebra 2 and went to music school - I taught myself many concepts on the job, but I am not someone with a traditional education in math. Most of it is simple linear algebra, and if trig comes into plat you are likely doing it wrong anyway (dot > trig).

4. It's about building a bag of tricks.

Everything in graphics is a hack. Even as lighting, etc, become more realistic, it's still a bunch of hacks on top of hacks. Shader writing is about building up a bag of tricks, and approaching things from a GPU centric performance perspective. For instance, you might have an art problem which would traditionally be solved with bones/skinning/etc and realize that by doing the work in the shader it's going to have almost no performance overhead and actually be much simpler to setup than dealing with all of that overhead. Writing shaders is about simplifying your workflow as much as it is increasing what you can do.

Some examples I give my students, via tutorials or follow along exercises:

1. Procedural ripple effect
2. Distorting UVs using a texture
3. Distorting UVs using procedurals
4. Generating value noise on the GPU
5. Basic lighting
6. Normal Mapping
7. Destroyed surfaces
- Blending between multiple textures based on world space noise functions
- Shadows and dirt in transition areas
8. Procedural Planets from Star Trek Timelines
- Gradient mapping a height field
- Packing textures (normal + height in one texture)
- Procedural water on the surface
- adding Shorelines, city lights, and other effects

 

I love that the 2 shader wizards I follow both commented back !

There is a lot to get my teeth into in this post, thank you so much for taking the time to write it. I am at work right now (VR developer day job) but when I get home and fire up my personal unity installation Ill start having a go at some of these suggestions.

This is exactly what I was looking for, thank you and i hope this helps others!

EDIT: just wanted to say that the examples at the bottom were spot on, just what I need to give my learning some direciton!

 

Like @jbooth I am not a programmer or mathematician by training, I'm originally an artist and went to college as such. Ironically I dropped out of college just one math credit shy of getting a degree. I learned most of this on the job. These days there are tons of ways to get up to speed on the mathematics, like Khan Academy, or Wolfram Alpha.

One other thing that I've learned is there are plenty of devs and academics out there who are happy to help if you reach out to them. Many have personal web sites with comments or even a forum, or just a twitter or e-mail address that you might find on some article. If you have questions, try contacting them. Just be polite, make sure you're asking relevant, direct questions, and don't expect a response immediately (or possibly at all) you might be surprised what you get back. I've read some academic papers that did something interesting and may offhandedly mention something, or only provide partial code for a technique they're using, and asked the authors directly if they could expound on a detail or show a more complete example. Often times they're happy to respond, and just didn't include them for brevity or because they didn't feel the code was good enough to publish. Many are just happy someone is trying to use their work (it's usually why they published it to begin with).

One of the things I spend the most time doing to learn is take someone else's example and go through it, turning stuff on and off or changing lines to try to understand what it's doing. I would say the vast majority of my knowledge on shaders comes from looking at code and trying to puzzle out why something works.

 

Just wanted to point out that its only been a week and I have already manged to write some basic shaders I have a use for, as well as helped someone on reddit make a foiliage shader for underwater plants

Currently playing around with fractals

Thanks for the morale boost to both of you!