How to improve your Shader documentation

Necessary Background Information

On my current prototype project, Sara the Shieldmage, I was working earlier today on writing a modified version of the Particle/Additive shader that performed backface culling, and had a Rim Shader effect.

Just so we're clear where I started, here's an image of what it was I was trying to fix. It may hurt your eyes.


Naturally, as such a shader does not come pre-packaged with Unity, I took a delve into Shader programming for the first time. Now, a few things. First, I am a complete Shader novice. While looking at the code for the Particles/Additive shader, I saw some things like 'float' and 'float4' which seemed reasonable enough, given my prior programming experience.

But, I hit a brick wall - I could not understand how to modify the Particles/Additive shader to have a rim-lighting effect. After spending a few minutes looking up backface culling in the Unity Documentation - to no avail - I posted that topic.

In the ShaderLab topic I linked above, a guy answered, but part of his answers led back to the documentation. I tried to blend in the relevant parts of the shader code for the Rim Lighting example earlier, but it completely broke the shader. I tried coming at it from the opposite angle, blending in additive effects to the Rim Lighting shader, but that's also not working at the time of this topic.

Problems

I wound up just walking away to do something different, because I'm getting increasingly frustrated with trying to figure out how to write a shader in Unity's ShaderLab system.

The first problem, is that your documentation appears to be written for someone who already knows what they're doing - not someone like me who doesn't have the slightest clue about how to write a shader. The details about rendering tags for the primary subshader are pretty much non-existant - I can't find anything about them, period. Another problem is that you seem to rely on the examples to impart critical information about the structure of one of these shaders. My research into your documentation showed me that the main 'form' is something like this:

Code (csharp):

1. Shader "SubMenu/Name" {
2.     Properties {
3.         _SomeProperty ("Interface Name", InterfaceType) = (Color/Number Tuple) | "random string?" {}
4.         // etc.
5.     }
6.  
7.     Subshader {
8.         Tags {
9.             // Other than 'opaque' and 'transparent', what are the other valid render types?  Your docs
10.             // don't say.  What's more, in the Particles/Additive code I Googled, I noticed other tags that
11.             // the documentation does not even talk about, period.
12.             RenderType = "Some Predefined String"
13.         }
14.  
15.         CGPROGRAM
16.         // The pragma metastatements seem reasonably well documented, unusually enough.
17.         // The documentation could be made easier for a novice, but it was one of the few parts
18.         // I actually feel reasonably secure talking about.
19.         #pragma surface surf [Shading Model] [other directives]
20.         #pragma target [Shader Model Number, e.g. 2.0]
21.  
22.         // Your documentation says there are default values for things in this,
23.         // But that there are some optional values too.  I can't make sense of
24.         // this, based on how your documentation is written.  Do I need to include
25.         // all the values in this struct?
26.         struct Input {
27.             // Stuff?
28.         }
29.  
30.         void SomeMethod() {
31.             // I noticed submethods like this scattered throughout your examples.
32.             // If I may, and with all due respect, what in the *balls* are these for?
33.             // What calls these?  Alternatively, could you be any more vague with
34.             // what this code actually does?  I don't think you're nebulous enough.
35.         }
36.  
37.         ENDCG
38.     }
39.  
40.     // Something else your docs don't talk about.  In the code I downloaded
41.     // for Particles/Additive there were Passes, and additional subshaders for
42.     // other cases?  Your documentation does not talk about these.
43.     Subshader {
44.     }
45. }

One other note: this is not C# code, this is ShaderLab code. Fix your code tags too.

You assume a great deal of knowledge for people who are writing shaders. This doesn't help me figure out how to write them. I don't feel I'm far off of the mark assuming that the reason no one has been able to give me more help, is because they don't know how your shader system works either. If your documentation is anything to go off of, I can hardly blame them. Apparently your shaders work based on magic principles, not anything technological.

If I seem a bit upset, well, I am. Normally I enjoy researching technologies I know nothing about, and pulling together disparate scraps of information to build something useful.

I expected learning how to write ShaderLab shaders to be no different, except it has been one of the most frustrating experiences, because the documentation has the quality of most of Lightning Returns: Final Fantasy XIII's writing (that's a nice way of saying, complete S***.)

But, wait! It gets worse!
However, I didn't stop, I kept on reading, hoping to find obscure threads of information that I could string together into something useful. I'm not averse to research, I do it often.

Of course, it doesn't help when an empty sight like this greets you.

Seriously, Unity? An empty documentation page? Maybe your shader language is so constrained, that advanced applications are impossible. That, or your really need to re-evaluate your documentation.

How You Can Fix It
First, instead of assuming that everyone who reads about shaders is a seasoned graphics programmer, assume that we know nothing and have to write about some effect that we have no idea how to implement. Instead of starting off saying, 'Oh there's three different types of shaders' (useless information), give us something helpful: 'A shader is a program that operates on your graphics card to modify either pixels, fragments, or other information, in pursuit of attaining a specific look for the associated asset.' Show us the standard structure of a shader program. Introduce us to the format of a shader program. Spell out even the obvious stuff like, 'a float4 means that the value stored is a tuple of four floating-point numbers.'

When you've taken the time to explain how shaders work - you know, the same way you do for the scripting languages - then get into the more advanced stuff, like the examples. This helps us by showing us those useful fundaments in action.

Conclusion
Unity Tech, I am pissed off at you. Your shader documentation sucks. It's utterly unhelpful. It's probably the worst documentation I have yet read from anyone. I didn't expect it to be this way, because you generally take great pains to make things (like Game Development, period) accessible. I don't know what possessed you to make such a cesspit of what seems like an extremely useful skill.

First, help me fix my problem so I can get back to something I actually enjoy - writing games, not shaders.

Second, fix your damn documentation. In its current state, it's inexcusable for a noob (me) to walk into it, and nearly have a heartattack from sheer frustration because it says nothing useful.

Third, I'm saying this because I care - Unity has helped me learn to be a better programmer, a better game developer. There are some cool, and seriously skilled people with and around you. I am astonished that you could screw up this badly. It's quite shocking.

And finally, if your smart-ass reply is, 'well, go download StrumpyShaderEditor/buy a visual shader editor from the Asset Store', I have a few choice curse words for you. StrumpyShaderEditor's resultant shaders do not work in the current version of Unity, and I don't really want to pay a company that can't properly document how to do something, to buy an editor so that I can remain ignorant. I've bought assets when I needed them before, but I'm not going to pay you money to do something that does ship with the software, but has no coherent instructions.

Fix your crap, Unity, and pull the Strumpy Shader Editor from the asset store. It's worthless.

I'm done. If suing for emotional damages wasn't a dick move, I'd totally do it, because this is so frustrating. What the hell, Unity?

 

Hello! Sorry to hear that you are disgruntled. Whilst Graham, who works closely with the Documentation team, will probably respond in due course, I can help you with one particular point:

If you look carefully, this is a parent-page, introducing you to a series of sub-pages. On the menu on the left, you can expand this parent and view a drop-down list that has all the links provided. Ive even attached a screenshot proving that they exist.

 

That makes much more sense. If I can offer a suggestion, change the text to make a better note of that. "Read those to improve your ShaderLab-fu!" does not tell me to look to the left and click the plus sign. In fact, a little more text would likely make it easier to realize that there are sub-topics, I thought I hit an empty page (of course I was already a bit disgruntled by that point, which likely doesn't help.)

I'm sorry for the harsh words Andy, but this documentation was not helpful. As I said, I feel you guys are better than this.

 

Why is it worthless? I just tried it out for the first time and it seems to be working for Unity 4.6 Beta 17. It does seem to produce a bug in a specific location of the shader code every single time though if that is all it shouldn't be difficult to fix.

 

As a basic test of the Strumpy Shader Editor, I used a default graph, which produces a Rim Shader. When I exported the raw graph to a Shader (I named it, "Test/Strumpy Rim Shader",) Unity refused to compile it. Also, I was already mad at that point, and it seems about as clear on how to use, as the documentation is clear on how to write a Shader.

Of course, I also don't know what I'm doing. My frustration is that I don't know, I want to know, but it's like everything is set up to be as unintelligible possible. It's like a massive 'f*** you' on as many fronts as possible. As noted before, I have no aversion to research, but to have such huge barriers to understanding is proving unusually frustrating. Prior to today, I didn't know I could get this pissed about what's simultaneously a coding problem and learning a new language. But, I guess I can.

That, or I'm simply not intelligent enough to write shaders. It's a possibility I don't like, but it could be there.

 

I assume that the text was written before the documentation switched to the current layout; so that the old layout had the sub-pages just beneath the text and that 'Shader-Fu' line made alot more sense to read. It'll probably be changed when the documentation team see this thread!

 

Open up "ShaderTemplate.template" under "StrumpyShaderEditor\Editor\Resources\Internal\" and modify the vert function to look like this. Found the solution under Unity Answers if you want more details.

http://answers.unity3d.com/questions/754151/vert-output-parameter-o-not-completely-initialized.html

Code (csharp):

1.  
2. void vert (inout appdata_full v, out Input o) {
3. UNITY_INITIALIZE_OUTPUT(Input, o)
4. ${VertexShader}
5. ${VertexShaderMods}
6. }
7.  

 

Asvarduil, I can assure you that you and everyone else can write shaders in unity.
You just need to invest in it.
Would it be reasonable to expect that you can create a rigged textured character without prior knowledge in a day ? Or an A* pathfinding solution ?
Take your time and you will make it.
Unity docs do not suck, I find the tree structure unappealing too (those plus signs are so chm like), but it is there, it is ok, you have to go through it.

 

First, thanks for the vote of confidence. As someone who wants to make something that is good, I appreciate it.

The bit about 'Unity docs do not suck', though...I think it's safe to assume you didn't read my wall 'o text - particularly the 'structure of a shader' bit - that talks about all the stuff that the documentation does not talk about.

I learned a lot of stuff by downloading the Particles/Additive shader from Unity Answers, but still not enough for me to actually create my customized variant of the Particles/Additive shader. And, this is OK - research involves going out and finding crap yourself. I don't mind that part at all.

The reason I say the documentation on the Shader system sucks, is because it doesn't help you to figure much of anything out, especially if you're the type of person who needs it (that is to say, someone who does not know what they're doing. For reference, see Asvarduil.)

The documentation neglects to tell a user even the most basic aspects of ShaderLab, and instead assumes that you have detailed understanding of it, so much so that it launches straight into admittedly useful information about how certain things like the Input structure have certain information, or that there are actually three types of shader program you can write.

I'm sure it will make Unity rather happy to find out, I figured out where the rendering tags were hidden - inside the tree that Andy Touch showed me. The problem is, this is a really basic part of writing a shader. It is not an advanced topic at all. I think that's yet another part of the Shader documentation Unity needs to rethink.

The problem is, the useful information is only useful if you're capable of using it. I'm still not exactly sure how those shader methods are getting called, which makes me a bit worried. Is there some sort of system similar to Monobehaviors where you have to predefine certain Shader methods in order to hook into events in some back end? I don't know; the documentation won't tell me how the mechanics of this system work. The documentation won't tell me anything to actually use it. My best bet at present is just to download other shaders and pray to the deity of your choice that I get a flash of understanding, because the goddamned official documentation won't.

Ippokratis, I'm glad you know how to write shaders, and that you find the documentation useable if not awesome. In light of that, it's a shame you don't seem to be active on the Shader forum, where I'm asking other people for help, but I appreciate your feedback on my feedback. Would it be cliche to ask you to share your knowledge with the rest of the class?

 

All you need for a start is this: http://en.wikibooks.org/wiki/Cg_Programming/Unity

 

I'd suggest taking some time to start and have a read/take some tutorials on shader programming, HLSL, Cg, etc in general. You're getting too hung up on Unity's implementation. Once you understand what you're trying to program it will all start to fall into place (ie: those magical 'methods')

 

In a nutshell :
-Start here : http://docs.unity3d.com/Manual/SL-Shader.html
-Read all the subpages
-Then go http://docs.unity3d.com/Manual/SL-SurfaceShaders.html and do the same

You will find out all the stuff regarding Properties, Tags, etc.
Download the builtin shaders zip, open the additive particles shader source
Tweak the code and see what happens.
If you do not understand what the code does, ask. Chances are someone who has the time and the patience will help you. Daniel, Equil, Martin, Auberge, Holmes are always around.

If you browse the Shader forum, you will find a couple of shaders that I have created and shared.
I remember a force field shader, a IOS distance field shader for bmfonts that exists in most relevant packages without attribution, a plane cut shader, etc. In my signature there is a free Toon Shader with source included.

I remember my despair for the lack of tutorials, now there are plenty, on youtube, the cgcookie, as well as a book, unity shaders cookbook, Martin's wiki pages Vic20 mentions. Even I contributed with a small Cg to Unity Cg tutorial in the wiki.

Relax and take your time.

 

Right, so I've had a good night's sleep. So, no more Pissy Boss-Mode Asvarduil. I'm just my normal bumbling incompetent self again. The way my temper works, I emotionally go supernova every couple of years, so we're good for a long time to come.

@VIC20, thanks for the guide on HLSL/CG programming. Not only does that make the presence of the 'CGPROGRAM' and 'ENDCG' statements make more sense, I didn't realize that the ShaderLab format was actually just encapsulating a whole different language. Due to my lack of knowledge I was assuming that Unity had created a wrapper for low-level shader languages similar to what was done with scripting languages. Now that I know that's not the case, I've got some reading to do.

Another suggestion for Unity - link to what VIC20 gave. Ideally before going out to write Shaders, one would want a good overview on HLSL and CG programming. That may be a fundament even more basic than knowing what tags go where, if someone wants to write shaders. I don't think it would be reasonable - even by 'me yesterday' standards - to expect Unity to detail the minutiae of a completely separate set of languages that Unity does not/should not support, but the documentation giving a noob a place to start is an absolute necessity.

And, Ippokratis, thanks for your info on what I should do. I'm not going to do Shader programming for a while, because I really need to read about how the shader languages work, but when I do get back to writing this shader, I'll PM you for help. Thanks for the guidance, and I apologize to you for getting pissy.

 

You have raised pretty good points, though. The first thing I'd ask if I were a designer looking after this stuff is "why was the user so frustrated, and how can we reduce frustration in the future?" Just having the info present isn't enough. Like, why didn't you know that ShaderLab was a bunch of Unity-specific stuff built on top of Cg? That's pretty fundamental, and if I were working on this it'd be one of my goals to make that as clear and easy to learn as early on as possible.

I remember similarly feeling that there were a lot of gaps in so far as "This went from A to C... where was B?" I don't remember what they were about as it was ages ago, and there's every possibility that they've since been addressed. And I know that some of them were general, fundamental shader knowledge. Now, I don't think that the Unity docs should cover Cg fundamentals, as it's out of their scope and there's plenty about that already available. But making you aware of where Cg and ShaderLab each start and stop, and referring you on to appropriate sources for the Cg stuff should be pretty front and center I'd have thought.

So, arising from all of this, a concrete suggestion is to have an "Expected Knowledge" listing in the intro to the shader section, referring people to where they can go to learn that stuff.

Next, just like coding Mono in Unity involves a lot of engine-specific knowledge to get stuff done, so does programming shaders. In both cases the language used is just one piece of a puzzle. You also need to understand what else the system is doing around you. A lack of that understanding is what causes people to call things "black magic" and such. I don't know what the docs on this are like, but I seem to have got a reasonably solid understanding of it from somewhere. I don't have a concrete suggestion for this, but I guess just make it clear somewhere how the shader code we write fits into Unity's pipeline as a whole.

 

I agree that this post should not simply be treated as "crisis averted, now back to business as usual." Like @Asvarduil , I consider myself competent using C# and would be interested in learning how to write shaders in Unity; I have also begun reading the documentation and likewise quickly felt out of my depth. I don't mind needing to start smaller with other resources, but because there was no reference to the skills I would need to begin learning ShaderLab, nor even that I needed them, I just threw up my hands and filed shaders permanently on the back burner. Where, to pursue the metaphor, they quickly incinerated.

More helpful documentation on shaders would be great.

 

Have you watched the tutorials on CGCookie? I'm just curious at how efficient they might be because having little programming knowledge and experience those videos were/are going to be my goto tutorials when I get to a point of playing with shaders.

 

Actually, the documentation for shaders does specify that you use hlsl:

"You write surface shaders in a couple of lines of Cg/HLSL and a lot more code gets auto-generated from that."

Thats from the first page, and is mentioned a few times, and if you google hlsl, there is loads of information, Unity uses C#, UnityScript and Boo, they don't document the full syntax of these languages as like HLSL, they are 3rd party languages, so you refer to the original documents, just like how there is no documentation for the System.Linq namespace in Unity docs...but you can still use it.

 

First: I wanted to only quote the middle two paragraphs, but really...all of it matters.

That's the thing that frustrated me so badly yesterday. As someone who writes code for a day job, I wind up saying 'RTFM' quite a bit, it's a valid piece of advice. It's emotionally painful when documentation obstructs your progress, due to being unhelpful/withholding information from you, due to either content or poor organization.

Speaking of both, that brings me to a good point...

Ippokratis is correct, much of the questions on the ShaderLab code structure (and, some of the most bare-bones basics of Cg/HLSL) I had are in fact there, but I never would have thought to approach them in that order. For basic concepts, they're remarkably well-hidden, wouldn't you agree?

This shows another problem to the list with the shader documentation: your tree is too deep, and stuff that is vital is hidden too deeply. What's more, the stuff that you need to know before tackling an advanced concept isn't where it should be (near the beginning, in a shallow branch of the tree.) The detailed information is what should be 2+ levels deep in the tree.

I have not. I will, though. Thanks for that. That might be another good resource for those who want to learn CG/HLSL.

 

Sure, but that's only talking about one part of the job. Then it's all wrapped in ShaderLab.

I just had a refresher skim, and it looks like the info is indeed all there, but it does seem a little scattered. I was super confused for a moment when I clicked a link in the Shader Reference's intro section that took me to the intro of the Shader section of the Graphics Overview.

Reading that page, my immediate feedback is that ShaderLab is really glossed over. There's a high level intro to the different types of shaders, which get a paragraph each, then ShaderLab gets just one sentence and an example before booting us off to the syntax reference.

If I open a shader file ShaderLab is the first thing I see. If I want to write one, I start with ShaderLab. Why is it not the first thing? It also always seems to get mentioned at the end of something else as an afterthought, despite the fact that you need to understand it in advance to have context for whatever you just read.

Having said all of that, it did all seem to make more sense when I just skimmed it compared to when I started out with it years ago. So it's certainly improved somewhere. And I agree that some of the issues could be helped a lot by updating language to better match the new navigation.

 

Yes! Yes! See my above post. (By the way, I cut that post down a lot 'cause I was suggesting so many changes it just seemed to be ripping what's there to shreds, which really wasn't my intention.)

The docs consistently put ShaderLab last, despite the fact that it's where a learner should start.

 

So, is it too much to ask that learning to write shaders should be so easy even an artist could do it? Is that an unreasonable place to put the bar? On another note, is there any Learn tutorials involving shaders, because at last check that seems to be unrepresented.

It is kind of odd that shaders end up being the only aspect of unity that doesn't have much of a learning curve (more of a cliff).

 

I can't answer that, but it is still a form of programming. And while individual shaders are fairly light in terms of architecture, they often involve reasonably complex math, so that's something you'd have to understand as well, though some of that math you'd have at least a partial understanding of from your other 3D work.)

Have you tried different visual shader systems? They might achieve what you need.

 

I don't know, ripping it to shreds might be doing Unity Tech a favor.

It was sort of like when I was trying out the Not Another Fantasy RPG idea - it was fundamentally flawed, had fundamentally bad design, the graphics were wrong, the writing is probably worse than what's currently in The Hero's Journey...when people said, hey, this is a bad idea, you really need to rethink this, I was happy to, because they were right, they explained to me how and more importantly why it was bad. It led me to make better games.

I don't think hulking out like I did on Saturday afternoon may exactly be the way to go, but as someone who deals with this on a much more intimate level than someone like me, who frankly doesn't know what in the good Lord's green earth I'm talking about...I see serious potential to help Unity make the docs friendlier for noob and pro dev alike.

And, now that I'm on a much calmer emotional footing, I apologize even to Unity for hulking out, even though I still feel that you deserve this, because that's what reading the documentation felt like.

And, no epilepsy warning for you either. If you want one, don't write painful documentation!

 

I can't say I agree with this tbh.

A Google of 'unity shader' brings up the overview that tells you it is HLSL as the first result (for me).
As does entering 'shader' in the unity manual search box.

I can't see what more can be done there...

If you were to ask for an easier to use shader creation system (some node based system perhaps), I could probably understand, it's certainly where the rest of the industry is heading.
But shader language hasn't changed in years and as has been pointed out before there are a lot of resources available for learning CG/HLSL.

 

My point is not RTFM actually.

I agree that the order of presenting shaders documentation as well as the tree structure of the current unity manual implementation can cause problems to someone who starts on this subject. This is why I tried to point out the steps I would take if I was starting out now (having the obvious advantage of going through the material many times before). Actually, for every new version of unity, I try to go through the parts of the manual I find useful and shaders are one of them.

I refer to those pages weekly for the past 2-3 years and still I find the new layout difficult to navigate, for my taste.

The info though is there and if someone wishes to learn, it is possible. This is what I wanted to point out with the previous posts.

But it takes time. Cg, HLSL, GLSL, ShaderLab fixed shaders, surface shaders, DX11 features, image effects, shadows and illumination implementation "hidden" in cginc files, Unity 5 unified shaders... It is not going to happen in a weekend. Hence the "take your time" suggestion.

I wouldn't try to apply rim shading / matcaps / fresnel on a flat surface. It has no normals to support it. Perhaps a glowing effect does the trick better in this case.

@graham : Great !

Kind regards,
-Ippokratis

 

Or make their own node-based shader editor. Buying up Shader Forge or hiring the developer would probably work too.

 

In fact, there is absolutely nothing about shaders in the tutorial section.

 

I've found this guys stuff to be an invaluable resource: http://scrawkblog.com/category/water/ look through the different categories on the right. He provides project files from all those and I learned a lot from poking through them.

Also nvidia has good cg documentation. http://http.developer.nvidia.com/Cg/ If you start opening up shaders you can find, they'll make a lot more sense when you look through the docs.

 

I appreciate that Graham. Thanks for at least looking at this. Changes would be most helpful -

I wish there was an unlike button, but I have to settle for liking the fact that you brought that up...

 

Hi I'm C# programmer but i need to say that the unity shaderlab documentation sucks, please make it as you have a awesome scripting documentation (the best that I've looked), please down to UnityEngine and UnityEditor add ShaderDocumentation, with examples and showcase. We will love it and if you add some decal system which let us export on 3d model format will be awesome