Official Package requirements in ShaderLab

Thanks for sharing, this is really cool!

It looks like the package requirements are AND requirements - what if I have a pass that works for URP and HDRP, should I just set Core as requirement?
What if I have a pass that works for URP and built-in, but not HDRP? Duplicating the pass (or many passes, rather) doesn't sound ideal.

 

So is the plan for just growing infinitely more complex shaders rather than actually providing a usable abstraction?

This basically allows you to stuff an HDRP7.x, HDRP10.x, URP7.x, URP10.x (and maybe built in?) shader into one file, but that just means that every release of a breaking change to any SRP makes the amount of passes needed grow. Next year the file will contain all the previous years passes plus HDRP13.x and URP13.x, increasing the total complexity further and making maintenance harder. Unlike Unity, anyone providing shaders to people has to support more than one version of an SRP.

For the majority of shaders, the user either wants to modify the inputs to the lighting equations, or the lighting system itself. Rarely are the inputs to the lighting system changed, and right now these systems are so tightly coupled causing everything to break when either side changes. Simply cramming more and more complexity into the same file doesn't seem like forward motion on fixing this issue.

 

I'm trying to think of a use case for this that isn't about stuffing multiple SRPs into the same file. Correct me if I'm wrong, but the only case where you need to do this is if the package has an include file you need to use, which would cause an error if the package wasn't there.

Originally I thought ok, maybe there's a custom pass needed for some package, so you want to have that pass in the shader when the package is present. But if you just add a custom pass to a shader, it's not going to be used unless the rendering uses it, so that isn't really a use case.

So I'm back at thinking the only use case for this is stuffing multiple SRPs into a single file again - which is really just a road to INF complexity.

 

This is a little disingenuous, isn't it? This isn't actually shipping "one shader that works with multiple pipelines" but rather "multiple shaders shipped in one file" where the number of shaders shipped within that file is going to balloon over time. Am I missing something?

This seems a bit like trying to slap a bandaid on a bullet hole.


Also, is there a way to specify a pass that isn't reliant upon an SRP? I.e. is there a way to create a multi-shader file that supports BuiltIn, HDRP, and URP at the same time?

 

That can already be done:

https://github.com/slipster216/ShaderPackager

It reminds me of FBX, where they were like "Here's a universal format for Maya, 3dsMax and Motion builder to read and write too" and inside you find 3 files stuffed in the same file. Technically it's not an invalid statement, but not in any way that's really useful.

Since Built in shaders still compile under SRPs, I think you get this for free- at least until they pull Built in.

 

Umm... what are the exceptions here? What is beyond the 'extent' that you imply?

Some context would help.

 

Why are you assuming that URP 13.x shaders will be incompatible with URP 10.x shaders? Has that proven to be the case? Are you using private calls from the URP shader libraries?

 

Because that is the pattern set so far. In fact, URP 7.18 and URP7.21 broke compatibility from each other, and they were like 2 weeks apart and both part of an LTS release.

It's actually more likely that there will be incompatibilities between 10.x, 11.x, 12.x, and 13.x than there won't be. For instance, deferred rendering was added to URP at some point along that line, so you have to add new passes, etc. I use 7.x, 10.x, and 13.x as examples because I only support LTS versions in my products for SRPs, the actual incompatibilities are much more widespread. The only way this stuff won't change is if Unity stops developing or adding features to the pipelines, which is the whole point of advocating for an abstraction layer.

I think there are some advantages and disadvantages to both:

Package references in subshaders
+ Doesn't require packing step (less error prone when distributing)
+- Shaders must share property blocks, fallbacks, etc.. Both a benefit and a detriment.
- 5x to 7x times the shader code in one file for support of 2-3 years of pipelines (or more)
- Shaders know about packages (not bad, necessarily, but seems a bit odd for them to know about packages)
- We now have a matrix of stuff to deal with- passes based on SRP versions, #ifdef's based on SRP versions- trying to keep code shared and concise will be a challenge, since you might have 5 versions of a pass for a given SRP in a single file.

Packer approach
+ Can pack shaders from any tool that creates them
+ Only dealing with one shader at a time
+ Shaders don't have to share any properties, fallbacks, etc
+ Much smaller file to process when importing shaders (no idea how much time is spent here)
- Needs better detection of *active* SRP instead of installed SRP (can be fixed)
- Would work better as official unity code instead of shipped with every asset
- Requires a new file type

 

Looks rly bad..

 

Yeah I have to agree with what has been said so far that this feels like a band aid to the real problem and this is why people are frustrated because as usual Unity (as a whole) misses the point.

To a user making a game, they 'should' never need this feature. They should pick a single Pipeline and support just it. They wont ship a build with multiple pipelines because that is madness and I hope also impossible.

So, this feature is mainly for the asset store/tools developers then. Great, fair enough. But... the problem is that you don't address why this is even needed. From the start of this SRP madness Unity have been frustrating by doing things like labelling SRP as production ready and trying to push users towards them when they don't know any better. Horribly deceptive move. I don't care if its just the marketing department trying to big things up, it should never have happened.

Anyway, this extends that. We are now in the middle of this horrible technical mess and each time we see a small feature like this we can't help but think, are you trying to say this is the new normal? Is this what you present as the solution to this madness?

Without addressing stuff like this from the bat, it shows that Unity is not focused on the bigger picture, or at least it feels like that from the outside.

So today you have given us a new feature which gives us more choice to be cleaner (depending on perspective, I do prefer one file over multiple) whilst sharing code which is of course a good thing. But without sharing how this fits into a bigger picture we will always be frustrated because from our perspective, it feels like Unity is either lost or doesn't care about this mess because it always tries to paint the positive picture.

What would have helped me is: "Here is this new feature. It's not going to solve these big problems and we are still working on them, but this was a fairly quick thing for us to make to help make things easier whilst we are working on X as a long term solution".

Without having a clear and definitive big picture you can share with us, we will always be frustrated seeing a small piece because we don't know how it fits in.

Without you sharing how this fits into a big picture, we don't know that there is one. "continue improving the shader area" is not good enough, its too generic.

For me, admitting that SRP are not production ready and that eventually the plan is for them to change no more than twice a YEAR would be the Start of a long term solution. But I feel stupid even typing that which shows how far away from reality that is (and how deceptive labelling them as production ready more than a year ago was).

Aleksandrk I'm sorry you are bearing the grunt of this. I fully appreciate that big companies are bogged down by bureaucracy, too many cooks etc. etc.

 

Would be pretty easy to solve that use case though, simply by checking the active SRP at import. Just because I'm not currently handling it does not mean it's a flaw in the approach.

Keep in mind that I've likely spent less man hours creating an actual shader abstraction that Unity has spent in meetings talking about not providing one at this point.

I fail to see how that's any different? How can it check the shader's validity if the package isn't installed?

Not quite - I'm packing N shaders into a single file, your system provides 1 shader with N subshaders based on dependencies. So in my case I am talking about shaders.

Kinda a + or - really, depending on your structure. For instance, I distribute Better Lit as a single packed file with all (non-unity) includes inlined into the file- this makes it extremely portable, but means the internal shader code is verbose (this is not a problem for me because it's being generated by Better Shaders).

One potential nice change about the new package dependencies that this raises is the ability to have includes which don't have to be pathed in the project. Because Unity only supports absolute or relative paths, it can be difficult to move shaders around if they have dependencies- but now you can install a package with your includes and access them from anywhere without requiring the user to have a specific organizational structure with their asset folder. For instance, I could switch MicroSplat to use includes for some of the places it inlines shader code right now, because I could depend on the MicroSplat-core package and find them from that. Although I guess I could do this without this feature too..

No, shaders don't have to share properties, but subshaders in a shader do. Lets say I'm making a shader for HDRP/Lit and Built in. In HDRP, it uses some extra textures and properties for some HDRP specific stuff. Since I only have one set of properties, unless I write a custom editor, I end up with a bunch of HDRP only stuff in my editor. With my system, each can have their own properties, and thus each their own interface.

Anyway, most of these points are fairly semantical. However, the discussion did bring up one non-SRP use of the new package dependencies, which I look forward to being able to use one day (I'm actually moving MicroSplat into packages now).

I think the issue here is that it's hard to not see something like this and have a PTSD like response to it. Especially when the obvious use case is to stuff all the render pipelines into one file rather than addressing the core issue. I've personally been told by every person at Unity I've ever talked to, including your CTO, that the SRP issue would be fixed, running on five years now, and this is literally the ONLY feature to be introduced in that time that looks like it's attempting to address it (and again, not addressing the real issue). Meanwhile, I had to write my own surface shader system just so I could make maintaining my existing projects viable and not have my revenue cut to a third.

Last year over 60% of my development time was spent on managing SRP shader issues, so imagine if your boss walked into your office and said "Hey, I'm going to cut your pay by 1/3rd, and to make it back you have to spent 60% of your time working on someone else's mess to hopefully earn it back instead of working on actual product ", and then the marketing department put out press releases every week saying how great it was, and then the actual developers you needed to get information from to do this job decided that they weren't going to document any of the code you actually needed to understand to do the work, and then decided they wanted to take away your depot access so you couldn't see changes they made. Then they decide that your a pariah for being upset about it, and tell you to "calm down". I could go on, but it just raises my blood pressure at this point, but if your actually wondering why this is such a hot button issue, this is it.

 

As a point of reference - the initial development time for Better Shaders was about 2 weeks. There's obviously been plenty of work done in addition to that, and tons of development time before that abstracting MicroSplat to use the the SRPs, but this is mostly reverse engineering HDRP/URP and figuring out how to template that, something that Unity doesn't need to do since they wrote them in the first place.

Writing a text based shader abstraction is not rocket science, nor difficult. It's pretty obvious that Unity has repeatedly chosen to not solve this issue from some kind of stance, not from the actual cost of doing so. The reasons against doing this I've been given do not make any sense (One day we'll have mesh shaders!). Any changes to that abstraction layer for future changes in technology pale in comparison to what we are dealing with right now.

Further, I think Better Shaders has not only proven that these things can be abstracted, it's proven that there is a much nicer workflow for shader authoring than what currently exists in Unity and other engines. Being able to stack functionality from multiple shaders together makes integration of shader based assets trivial, allowing users to combine different solutions from different sources easily. Being able to write new lighting templates and mix and match them with existing shaders is super powerful, opening up new potential markets for stylized rendering that just works with your existing shaders. Being able to have one set of functionality written in text, while another is written in a graph, allows artists and programmers to work together in ways that haven't been possible before. Having something like this standard in Unity just makes sense.

 

Ok so if you are fully aware of the problems we face and fully agree that there is potential for nicer workflows, then what is Unity's official stance on this subject?

Are there real improvements planned?
If so what are they and when are they scheduled to be delivered?

 

I think this is specifically why we're not excited about the feature announcement. This "feature" feels like a bandaid that enables you folks (Unity) to continue hobbling along with your current practices; to continue to ignore the actual problems.

Does anyone at Unity actually dog-food this stuff? What Unity Technologies asset on the Asset Store properly releases updates that support multiple versions of Unity simultaneously? I don't mean the assets that receive updates that simply state "Compatibility with Unity 2021.1" because that's just someone opening the asset, making sure things work well enough, and then uploading that package with a bump. I do mean an asset that receives actual feature updates that work across all Unity-"supported" LTS versions (at least).

The feeling from us-your-customers is that no one at Unity Technologies actually understands what it's like to try to professionally support this stuff. Therefore, features like this feel like dog bones handed out to distract from the real issues.

 

The answer is no, they don't. Unity has never actually done what asset developers have to do, which is support older and newer versions of a shader in an actively changing code base. Once they ship a unity version, they might go back and do fixes, but they don't have to continue developing new features with that as an active target. And they certainly haven't even tried to do it across SRPs, versions, or between pipelines. The Snaps assets only work with 2018 HDRP, and a few were upgraded to HDRP2020, but then removed support for 2018 and skipped 2019 entirely. On top of that, they only support a single pipeline in every shader asset they have released. There's no question why these have a 2 star rating.

Unity doesn't see publishers as it's customers- there have been numerous statements that show they think their customers only work in one pipeline, so why would they need to support things across pipelines? The fact is I pay unity more than what enterprise support for a studio costs every year via it's 30% cut of my sales (which is now high for this industry), and most of what they do is make my life harder. The downloading system for assets in Unity doesn't even work on half the versions of Unity people use, and they refuse to let users know about this and push off the support costs to publishers who have to explain to every user that the package manager lies about the version that's installed and you have to manually delete a folder if you want it to actually upgrade your asset.

Had they actually had someone dog fooding this stuff for these use cases, I'm pretty sure we would have heard reports of an employee going on a post office style murderous rampage by now.

Having to rewrite all my shaders once to move to a more modern platform, that allows Unity to move graphics forward- that I could have accepted. Having to rewrite them on a regular basis, whenever Unity changes anything, for 3 different rendering engines, with no documentation, while Unity promises over and over to improve things but does nothing, thats unacceptable.

 

Fully agree with its own thread, although I imagine there are many touching on this subject overal. Jasons #SRP life one comes to mind.

What we need is Unity to make an official thread or layout a proper roadmap again.

Also you not being able to talk about this is Unity failing miserably Either:

1: You don't know of an official plan, which I doubt and which would be utterly terrible on management.
2: There is a plan, but its still not sure if its a good one so the official line is "don't talk until we are sure".
3: There is a plan, but it looks like it will take so long to finish it that the line is "not to talk about it to avoid bad press on the severe wait".
4: There is no plan at all other than "keep hacking at URP until its stable and we can get rid of BIRP".

None of these scenarios are good considering how long SRP have been around.

Another thing Unity fails to miss the point on is that without the asset store, Unity wouldn't be as popular as it is. Sure its not what's making them their big bucks but it is what makes the engine accessible. Beginners can drop a few $ to grab an asset to plug the holes in their experience and every so often one of them becomes the next hyper casual success which if they properly got their customers into the Unity ecosystem of ads etc is where the money starts coming in (if I remember rightly from the last public report I skimmed a summary of).

Anyone who has been a publisher with more than 5 sales can tell you how painful it is to support the store since its full of so many beginners, and this is without the SRP madness.

 

When unity creates such a thread we’ll be more than happy to move there. But in the past this hasn’t happened (at least a dozen “this isn’t the right place, we should have a thread on it” that I can recall), so it’ll continue to inhabit every thread that touches on the issue until that happens.

basically until unity takes an active part in addressing this issue and moving forward, the community’s anger and disillusionment will only grow.

 

Was about to say something almost exactly the same. Also the "feel free to start one" doesn't work since Unity never chimes into the user started threads on this (such as the #SRP life one and likely many others kicking around)

 

I strongly disagree - a proper abstraction layer doesn't need this functionality, it simply compiles the user code to all three pipelines on demand. If this was actually true the shader graph would have needed this functionality to cross compile between HDRP/URP, and it doesn't because it does the same thing I do in Better Shaders.

What DOES need this functionality is simply sticking 5-7 different versions of a shader into one file. In essence, this change points to unity doing exactly what everyone fears- retaining the current complexity but covering it up so users don't realize how incredibly hard it is to maintain compatibility with SRPs.

Given the lack of communication around this issue, the blatant ignoring of it for several years, "Trust Us" isn't exactly going to be heard coming from Unity on this matter.

 

Ok so to summarise and quote myself:

"4: There is no plan at all other than "keep hacking at URP until its stable and we can get rid of BIRP".".

Forgive me if this doesn't fill me with confidence, especially since its been 2 years since Unity claims that SRP are "production-ready" and I would guess at least 4 years of total development time: https://blogs.unity3d.com/2018/02/2...er-pipeline-optimizing-real-time-performance/

Plus as you say it will be another year before improving handwritten shader workflows is even possibly looked into.

This entire time BIRP has been massively neglected in favour of SRP, and looks like it will be at least 2 more years before URP is even close to a state to consider using it. Although more realistically it will be 3 years since it will be 1 year to potentially get parity, 1 year to maybe have scripted shaders again and then the final year for these features to stabilise judging by a generously forgiving time it takes Unity to stabilise features (E.G Burst, ECS etc).

Lets say everything goes great though and its only 2 years. Well this still means I am working on a pipeline that is now 5 years out of date (assuming it has been neglected for the past 3 years of SRP development as seems to be the case).

Do you not see how horribly gaping this flaw is? Games technology evolves incredibly quickly and 5 years of neglect is HUGE.

My options with Unity are utterly terrible. I either stick with an outdated but sort of stable platform. I could move to URP which doesn't have the feature parity I need or move to HDRP. HDRP would be right for our target market (high end PC) but its just as unstable. Now lets ignore the fact that I do not want to work on a platform still undergoing major stabilisation changes, it also seems to have too high of a performance cost (from reading other users testimonials and from quick tests). However to properly test this myself I would need to ditch or rewrite every shader I have bought from the store, from scratch. I would have to rewrite all my own shaders and its not just a rewrite but a conversion to a node based system which I hate because complex shaders are an utter mess in node based systems. One of the includes files we have for a water shader is 1300 lines long (uncompiled). I don't even want to begin to think how that would look in a node based system.

All that, just to start testing if performance is good enough for our project.

 

As mentioned previously, if you could direct me to an official thread monitored by relevant Unity staff that is appropriate to discuss this then I would be happy to do so.

What did I misunderstand? Ali said that investigations/discussions had started on improving hand written shaders but they wouldn't be done by 2021. I had then assumed this meant that something would be done to improve them in 2022, or possibly later, otherwise the investigations/discussions are utterly meaningless.

Sadly, I fully appreciate that this means nothing and that the end result could be that no improvements are made even though there are clear alternatives/options available.

 

While true, @ali_mohebali just basically said a real abstraction isn't being worked on, or even looked at any time soon, thus debating that this "might" be part of a future shader abstraction layer is disingenuous as clearly it's NOT part of something your not even remotely working on at this time. This feature IS about stuffing multiple SRP shaders together, the use case of potentially putting things in different packages did not come up until I theorized about it - had that been the actual use case, it would have been in the original post.

If you had just said "yeah, this is about stuffing multiple SRP shaders into one file" we could have skipped over half this thread.

And this will continue to happen until Unity actual engages people on solving the real issues with the mess they have thrust onto it's users. The more it gets talked around, the less direct answers to things like "Hey, is this just about stuffing multiple SRP shaders into one file?", the worse you make it. If the answer is "Yes, it's about stuffing multiple SRPs together in one file", then say that. I might not like the answer, I don't have to, but at least it's a truth I can deal with instead of some theoretical exercise in justifying the feature without admitting that. Unity's reaction the SRP backlash has mostly been to avoid it, justify it, brush it off, or shoot the messenger. That isn't a good way to deal with issues.

With that said, I do need to point out that I do appreciate you two actually responding, because the silence on this has been deafening, and only causes me to raise my voice louder, and as much as it might seem like I enjoy this trust me I don't.

 

Agreed. You two are currently bearing the grunt of my frustrations (which isn't personal at all and I still try to keep respectful of course) precisely because it feels like there is nowhere else Unity will properly listen short of pinging Aras on Twitter which is not fair to put everything through him and thus something I try to avoid.

 

Ali did explicitly say "delivering this year".

Granted I was thinking in years rather than specific Unity versions so I see your point but I don't think it disproves my estimated timelines since they are aimed to land within their named years.

Also, ECS was first released 2 years ago and is still in preview. I think Burst was also two years ago and is only just now coming out of preview.

My estimate for a year to complete a big feature was very generous given the current track record so I think my timeline stands strong to say it could very likely be 3 years before we have:

1: Feature parity with BIRP
2: A system to write shaders with code and not shader graph
3: A conversion system to auto switch between pipelines
4: Each feature out of preview

Do you disagree and think it will be done sooner?

P.S Point Light Shadows, which is expected in any modern games engine, took 3 years to be delivered.

 

Out of curiosity, how many Asset Store developers actually support multiple SRPs with hand-written shaders?

My understanding was that the limiting factor for supporting multiple SRPs wasn't so much getting the right shader connected to a material (there are various ways we've worked around this so far), but rather that there is, like, zero documentation for writing custom shaders to begin with [without ShaderGraph]. This lack of documentation isn't limited to proper interfacing with SRP-specific features, but changes that occur between SRP versions (even minor ones) which makes upkeep a nightmare.

Am I missing something? Has documentation improved to the point where Asset Store Developers are now writing lots of hand-written custom shaders for URP/HDRP?

 

Still zero documentation, people have just reverse engineered it, particularly in URP which is simple compared to BIRP even.

In terms of how many assets use hand written shaders, there are a lot more on BIRP/URP than on HDRP due to it's complexity. Quite a few asset developers have moved over to Better Shaders at this point- off the top of my head Space Graphics Toolkit, Paint in 3d, Vegetation Studio, and of course my assets. Quite a few are using amplify as well (pretty much everything that works on all pipelines, since Unity didn't support BIRP with their shader graph until just recently). Also, quite a lot of assets just don't support SRPs or in some cases just pulled their assets rather than dealing with the headaches any longer (CTS).

Amplify is basically doing the same thing I'm doing- running after every release and creating new templates to abstract the pipelines. It's basically a massive amount of wasted work being done with what I can only describe as anti-help from Unity. If Unity had to port this stuff themselves, or anything really, we'd have had an abstraction years ago.

 

That aligns with my perception. I was hoping that someone at Unity could chime in with their understanding as it may help better contextualize why this was selected as a useful feature to add. If they, for instance, have data that suggests that "hundreds of assets on the asset store have multi-RP packages with hand-written shaders and this would help all of those people", then, cool.

 

There's clearly publishers that are making tools/shaders by hand and want to support multiple RPs, but the much much louder group is the 3d art content publishers who can't support multiple RPs because there's no base abstract shader that works on all pipelines.

This is a huge drawback to SRPs, the system is completely horizontal. Pushing a feature like this in the local space is a nice thing, but in the larger sense it's a band-aid on a system that needs proper abstraction and structure. To be honest I think that unless that gets changed then things like this are just adding to the mess and ultimately a waste of time.

 

And you repeatedly ignore/avoid the important points/questions

 

Then do you not think perhaps its time to start that official thread as has been asked for many times? Or perhaps to direct someone who could comment to this thread?

Is @ali_mohebali someone who is in a position that can comment?

I'm pretty shocked I even needed to say this....

 

I mean this has been the pattern for ages - it flares up, we're told this isn't the right place to have that conversation, then the right place never appears. It's just playing politics at this point. I'd much prefer them come out and say "No, we don't plan to ever fix this, suck it up" than playing this game as it feels dishonest. I was told all kinds of things last year in a private meeting, and none of it materialized. There is no plan, just doubling down on ramming these pipelines through and covering up the artist facing issues with them.

 

So, recently I've moved some of my assets to packages on the asset store- we figured out a way to get them to install into the packages folder instead of assets, which better matches the new workflow.

So I converted MicroSplat to that format, which has a bunch of advantages, including asmdef support.

However, the asset store has decided that we're not allowed to have more than 4 packages in an asset.

If one goal of this feature is it allow us to have shader includes in packages, then that's going to be pretty hard if the asset store is going to place an arbitrary limit on how many packages we're allowed to have in a single asset.

Left hand, are you sure what right hand is doing?

 

Request for clarification: can you expand upon what this means? When you say "4 packages in an asset" do you mean that the Package Manifest that can optionally be included can have no more than 4 dependencies? Or that you can only include a maximum of 4 .unitypackage files in the asset? Something else?

 

A maximum of 4 packages - you can have dependencies on as many of them as you want. For instance, the MicroSplat Terrain Collection is 11 modules, which are also sold separately. Currently when you install the terrain collection, it installs all of these modules together as if it's one thing, rather than forcing you to download and update each one separately. This is obviously a great convenience, especially considering how many issues the package manager has had.

However, if I switch to packages, I will be forced to turn this into the new 'bundle' style, where the user has to go to all 11 modules and buy them (for free), then download/install them all separately, because the asset store will reject any asset that has more than 4 packages in it.

So if you wanted to logically break your asset into reusable packages of code, making it easy to share common code between assets, but then want to ship them in a way that doesn't require the user to manually install all of these dependencies, then you're limited to the arbitrary max package count of 4. There is no technical reason for this limit.