Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Official Package requirements in ShaderLab

Discussion in '2021.2 Beta' started by aleksandrk, May 12, 2021.

  1. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,948
    Hello Unity users!

    TL;DR:

    In Unity 2021.2.0a17 makes it possible to specify package requirements for SubShaders and Passes directly in ShaderLab. A Pass or a SubShader that does not meet the requirements is excluded from further processing and compilation.

    (EDIT 14.05.2021): This functionality is mainly aimed at asset store and tool developers that need to target multiple SRPs.

    (EDIT 28.07.2021): 2021.1.17f1 and 2020.3.16f1 and later will also support the PackageRequirements block.

    Example code:
    Code (CSharp):
    1. Shader "MyShader" {
    2.   SubShader {
    3.     Pass {
    4.       PackageRequirements {
    5.         "com.unity.render-pipelines.universal":"[10.0,10.5.3]"
    6.       }
    7.       ...
    8.     }
    9.     Pass {
    10.       PackageRequirements {
    11.         "com.unity.render-pipelines.universal":"[11.0,11.2.4]"
    12.         "com.my.other.package"
    13.       }
    14.       ...
    15.     }
    16.   }
    17. }
    The first pass is compatible with URP 10.0 to 10.5.3 inclusive, the second pass is compatible with URP 11.0 to 11.2.4 and requires "com.my.other.package" to be installed.

    There are multiple ways to declare package requirements:
    • “<package name>”: specifies that the SubShader or Pass works with any version of the package.
    • “<package name>”: “<version restrictions>”: specifies that the SubShader or Pass only works with a subset of package versions.
    • “<package name>”: “unity=<version restrictions>”: specifies that the SubShader or Pass only works with a subset of Unity versions and requires a package with the given name.
    • ”unity”:”<version restrictions>”: specifies that the SubShader or Pass only works with a subset of Unity versions.
    Version restrictions define a set of version ranges. If the installed version of a required package is not inside any of the ranges, the package requirement is not met. Similarly, if a requirement specifies a set of Unity version restrictions, the same applies to the current version of Unity.

    Specifying version restrictions

    In ShaderLab's package requirements, a version uses the major.minor or major.minor.patch format. If you only use major.minor, Unity uses 0 for the patch. Package versions can also include a -preview or a -preview.n postfix where -preview is equivalent to -preview.0. Preview versions come before non-preview versions, so 1.2.3-preview.4 comes after 1.2.2 but before 1.2.3.

    There are multiple ways to specify a version range. Each one provides a different behavior. They are:
    • <version>: includes the version you enter and all versions after that. For example, 1.2.3 includes all versions starting with 1.2.3;
    • [<version>]: specifies the exact version. For example, [1.2.3] only includes version 1.2.3;
    • [<version1>,<version2>]: specifies a range between <version1> and <version2>. Using square brackets and round brackets causes the range to include or exclude to version respectively. The opening bracket affects <version1> and the closing bracket affects <version2>. Example: [1.2.3,2.3.4) includes all versions from 1.2.3 to 2.3.3.
    You can also specify sets of version ranges for a single package. To create a set of version ranges from individual ranges, use a semicolon as a separator. For example, [2.0,3.4.5];[3.7];4.0 includes versions from 2.0.0 to 3.4.5, version 3.7.0, and version 4.0.0 and above.

    When you set the versions for a package, be aware of the following:
    • Versions, version ranges, and sets of version ranges cannot contain any extra characters.
    • Version ranges cannot be empty.
    • Sets of version ranges cannot have intersections.
    If the syntax does not adhere to the above, the version restriction is invalid.
    If you define package requirements that can never be satisfied, the shader import process fails with an error.

    Stay tuned for more updates!
     
    Last edited: Jul 28, 2021
  2. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    802
    Thanks for sharing, this is really cool!

    It looks like the package requirements are AND requirements - what if I have a pass that works for URP and HDRP, should I just set Core as requirement?
    What if I have a pass that works for URP and built-in, but not HDRP? Duplicating the pass (or many passes, rather) doesn't sound ideal.
     
    LooperVFX likes this.
  3. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,948
    That's correct.
    This is more about what packages/versions are required for the pass to compile. For example, when you're using an include file from URP, you have a dependency on URP being installed. The same applies when you have a pass that works with URP and builtin.
     
  4. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    So is the plan for just growing infinitely more complex shaders rather than actually providing a usable abstraction?

    This basically allows you to stuff an HDRP7.x, HDRP10.x, URP7.x, URP10.x (and maybe built in?) shader into one file, but that just means that every release of a breaking change to any SRP makes the amount of passes needed grow. Next year the file will contain all the previous years passes plus HDRP13.x and URP13.x, increasing the total complexity further and making maintenance harder. Unlike Unity, anyone providing shaders to people has to support more than one version of an SRP.

    For the majority of shaders, the user either wants to modify the inputs to the lighting equations, or the lighting system itself. Rarely are the inputs to the lighting system changed, and right now these systems are so tightly coupled causing everything to break when either side changes. Simply cramming more and more complexity into the same file doesn't seem like forward motion on fixing this issue.
     
    DrViJ, funkyCoty, Edy and 3 others like this.
  5. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    I'm trying to think of a use case for this that isn't about stuffing multiple SRPs into the same file. Correct me if I'm wrong, but the only case where you need to do this is if the package has an include file you need to use, which would cause an error if the package wasn't there.

    Originally I thought ok, maybe there's a custom pass needed for some package, so you want to have that pass in the shader when the package is present. But if you just add a custom pass to a shader, it's not going to be used unless the rendering uses it, so that isn't really a use case.

    So I'm back at thinking the only use case for this is stuffing multiple SRPs into a single file again - which is really just a road to INF complexity.
     
    Edy likes this.
  6. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,948
    The idea is to make it possible to ship one shader that works with multiple pipelines and doesn't fail to compile if a package is missing instead of shipping several shaders separately and having the end user to find the right one.
    We'll continue improving the shader area :)
     
    brn likes this.
  7. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,083
    This is a little disingenuous, isn't it? This isn't actually shipping "one shader that works with multiple pipelines" but rather "multiple shaders shipped in one file" where the number of shaders shipped within that file is going to balloon over time. Am I missing something?

    This seems a bit like trying to slap a bandaid on a bullet hole.


    Also, is there a way to specify a pass that isn't reliant upon an SRP? I.e. is there a way to create a multi-shader file that supports BuiltIn, HDRP, and URP at the same time?
     
  8. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    That can already be done:

    https://github.com/slipster216/ShaderPackager

    It reminds me of FBX, where they were like "Here's a universal format for Maya, 3dsMax and Motion builder to read and write too" and inside you find 3 files stuffed in the same file. Technically it's not an invalid statement, but not in any way that's really useful.

    Since Built in shaders still compile under SRPs, I think you get this for free- at least until they pull Built in.
     
    atomicjoe and SonicBloomEric like this.
  9. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,948
    Rather multiple SubShaders inside a shader.
    Yes, there's nothing preventing you from doing that. The SubShaders are still filtered by tags for the current pipeline.

    To some extent.
     
  10. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,083
    Umm... what are the exceptions here? What is beyond the 'extent' that you imply?

    Some context would help.
     
  11. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    273
    Why are you assuming that URP 13.x shaders will be incompatible with URP 10.x shaders? Has that proven to be the case? Are you using private calls from the URP shader libraries?
     
  12. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    Because that is the pattern set so far. In fact, URP 7.18 and URP7.21 broke compatibility from each other, and they were like 2 weeks apart and both part of an LTS release.

    It's actually more likely that there will be incompatibilities between 10.x, 11.x, 12.x, and 13.x than there won't be. For instance, deferred rendering was added to URP at some point along that line, so you have to add new passes, etc. I use 7.x, 10.x, and 13.x as examples because I only support LTS versions in my products for SRPs, the actual incompatibilities are much more widespread. The only way this stuff won't change is if Unity stops developing or adding features to the pipelines, which is the whole point of advocating for an abstraction layer.

    I think there are some advantages and disadvantages to both:

    Package references in subshaders
    + Doesn't require packing step (less error prone when distributing)
    +- Shaders must share property blocks, fallbacks, etc.. Both a benefit and a detriment.
    - 5x to 7x times the shader code in one file for support of 2-3 years of pipelines (or more)
    - Shaders know about packages (not bad, necessarily, but seems a bit odd for them to know about packages)
    - We now have a matrix of stuff to deal with- passes based on SRP versions, #ifdef's based on SRP versions- trying to keep code shared and concise will be a challenge, since you might have 5 versions of a pass for a given SRP in a single file.

    Packer approach
    + Can pack shaders from any tool that creates them
    + Only dealing with one shader at a time
    + Shaders don't have to share any properties, fallbacks, etc
    + Much smaller file to process when importing shaders (no idea how much time is spent here)
    - Needs better detection of *active* SRP instead of installed SRP (can be fixed)
    - Would work better as official unity code instead of shipped with every asset
    - Requires a new file type
     
    Last edited: May 13, 2021
  13. BattleAngelAlita

    BattleAngelAlita

    Joined:
    Nov 20, 2016
    Posts:
    400
    Looks rly bad..
     
  14. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,948
    What exactly?
     
  15. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,948
    I looked at the link @jbooth provided, and he states it clearly that it doesn't work in certain cases. At the same time, from my point of view, having a built-in possibility to have a single asset (shader) work with multiple SRPs at the same time is beneficial.

    Let me go over your points one by one.
    That's correct. More than that, we provide error checking.
    We seem to talk about different concepts here. A Shader in Unity is an asset that has a specific structure inside. SubShaders and Passes always share properties, fallbacks, and other Shader-level things. The addition of package requirements doesn't change that in any way.
    Otherwise you'd have to do that in multiple files. So the amount of code doesn't really change much, and now we provide a possibility to have all of that in one place, instead of being scattered between several files.
    Packages are there, and people can ship assets inside. Shaders are special in terms of dependencies - the code is scattered across several files. If your shader and its dependencies are split into multiple packages, and a shader cannot function without a specific package, there's an implicit dependency between the shader asset and this package. With the new syntax you can make the dependency explicit.
    Would it be different if it were 5 different files? This makes it possible to keep them all in one place, but doesn't require to do so.

    Any tool that creates shaders can use the new syntax.
    Again, this is about terminology. One shader at a time - do you mean a SubShader? A Pass? Anyway, I'm not sure it's actually a +.
    Shaders still have to share them. You're probably talking about SubShaders, but then one can have multiple .shader assets still.
    Yes, the asset is larger, however, the import time will not be affected much. Figuring out package requirements is one of the first things we do.
     
    PutridEx likes this.
  16. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    Yeah I have to agree with what has been said so far that this feels like a band aid to the real problem and this is why people are frustrated because as usual Unity (as a whole) misses the point.

    To a user making a game, they 'should' never need this feature. They should pick a single Pipeline and support just it. They wont ship a build with multiple pipelines because that is madness and I hope also impossible.

    So, this feature is mainly for the asset store/tools developers then. Great, fair enough. But... the problem is that you don't address why this is even needed. From the start of this SRP madness Unity have been frustrating by doing things like labelling SRP as production ready and trying to push users towards them when they don't know any better. Horribly deceptive move. I don't care if its just the marketing department trying to big things up, it should never have happened.

    Anyway, this extends that. We are now in the middle of this horrible technical mess and each time we see a small feature like this we can't help but think, are you trying to say this is the new normal? Is this what you present as the solution to this madness?

    Without addressing stuff like this from the bat, it shows that Unity is not focused on the bigger picture, or at least it feels like that from the outside.

    So today you have given us a new feature which gives us more choice to be cleaner (depending on perspective, I do prefer one file over multiple) whilst sharing code which is of course a good thing. But without sharing how this fits into a bigger picture we will always be frustrated because from our perspective, it feels like Unity is either lost or doesn't care about this mess because it always tries to paint the positive picture.

    What would have helped me is: "Here is this new feature. It's not going to solve these big problems and we are still working on them, but this was a fairly quick thing for us to make to help make things easier whilst we are working on X as a long term solution".

    Without having a clear and definitive big picture you can share with us, we will always be frustrated seeing a small piece because we don't know how it fits in.

    Without you sharing how this fits into a big picture, we don't know that there is one. "continue improving the shader area" is not good enough, its too generic.

    For me, admitting that SRP are not production ready and that eventually the plan is for them to change no more than twice a YEAR would be the Start of a long term solution. But I feel stupid even typing that which shows how far away from reality that is (and how deceptive labelling them as production ready more than a year ago was).

    Aleksandrk I'm sorry you are bearing the grunt of this. I fully appreciate that big companies are bogged down by bureaucracy, too many cooks etc. etc.
     
    cxode, Edy, cecarlsen and 2 others like this.
  17. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,948
    This is correct. It is a small feature mainly aimed at asset store and tools developers to ease their life.
    It's not going to solve any other big problems, and is not aimed at doing so.
    I suppose I can be more explicit about that and not just say "hey, here's this new functionality, try and guess, what we wanted people to do with it". Point taken :)
     
  18. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    Would be pretty easy to solve that use case though, simply by checking the active SRP at import. Just because I'm not currently handling it does not mean it's a flaw in the approach.

    Keep in mind that I've likely spent less man hours creating an actual shader abstraction that Unity has spent in meetings talking about not providing one at this point.

    I fail to see how that's any different? How can it check the shader's validity if the package isn't installed?

    Not quite - I'm packing N shaders into a single file, your system provides 1 shader with N subshaders based on dependencies. So in my case I am talking about shaders.

    Kinda a + or - really, depending on your structure. For instance, I distribute Better Lit as a single packed file with all (non-unity) includes inlined into the file- this makes it extremely portable, but means the internal shader code is verbose (this is not a problem for me because it's being generated by Better Shaders).

    One potential nice change about the new package dependencies that this raises is the ability to have includes which don't have to be pathed in the project. Because Unity only supports absolute or relative paths, it can be difficult to move shaders around if they have dependencies- but now you can install a package with your includes and access them from anywhere without requiring the user to have a specific organizational structure with their asset folder. For instance, I could switch MicroSplat to use includes for some of the places it inlines shader code right now, because I could depend on the MicroSplat-core package and find them from that. Although I guess I could do this without this feature too..

    No, shaders don't have to share properties, but subshaders in a shader do. Lets say I'm making a shader for HDRP/Lit and Built in. In HDRP, it uses some extra textures and properties for some HDRP specific stuff. Since I only have one set of properties, unless I write a custom editor, I end up with a bunch of HDRP only stuff in my editor. With my system, each can have their own properties, and thus each their own interface.

    Anyway, most of these points are fairly semantical. However, the discussion did bring up one non-SRP use of the new package dependencies, which I look forward to being able to use one day (I'm actually moving MicroSplat into packages now).

    I think the issue here is that it's hard to not see something like this and have a PTSD like response to it. Especially when the obvious use case is to stuff all the render pipelines into one file rather than addressing the core issue. I've personally been told by every person at Unity I've ever talked to, including your CTO, that the SRP issue would be fixed, running on five years now, and this is literally the ONLY feature to be introduced in that time that looks like it's attempting to address it (and again, not addressing the real issue). Meanwhile, I had to write my own surface shader system just so I could make maintaining my existing projects viable and not have my revenue cut to a third.

    Last year over 60% of my development time was spent on managing SRP shader issues, so imagine if your boss walked into your office and said "Hey, I'm going to cut your pay by 1/3rd, and to make it back you have to spent 60% of your time working on someone else's mess to hopefully earn it back instead of working on actual product ", and then the marketing department put out press releases every week saying how great it was, and then the actual developers you needed to get information from to do this job decided that they weren't going to document any of the code you actually needed to understand to do the work, and then decided they wanted to take away your depot access so you couldn't see changes they made. Then they decide that your a pariah for being upset about it, and tell you to "calm down". I could go on, but it just raises my blood pressure at this point, but if your actually wondering why this is such a hot button issue, this is it.
     
  19. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    As a point of reference - the initial development time for Better Shaders was about 2 weeks. There's obviously been plenty of work done in addition to that, and tons of development time before that abstracting MicroSplat to use the the SRPs, but this is mostly reverse engineering HDRP/URP and figuring out how to template that, something that Unity doesn't need to do since they wrote them in the first place.

    Writing a text based shader abstraction is not rocket science, nor difficult. It's pretty obvious that Unity has repeatedly chosen to not solve this issue from some kind of stance, not from the actual cost of doing so. The reasons against doing this I've been given do not make any sense (One day we'll have mesh shaders!). Any changes to that abstraction layer for future changes in technology pale in comparison to what we are dealing with right now.

    Further, I think Better Shaders has not only proven that these things can be abstracted, it's proven that there is a much nicer workflow for shader authoring than what currently exists in Unity and other engines. Being able to stack functionality from multiple shaders together makes integration of shader based assets trivial, allowing users to combine different solutions from different sources easily. Being able to write new lighting templates and mix and match them with existing shaders is super powerful, opening up new potential markets for stylized rendering that just works with your existing shaders. Being able to have one set of functionality written in text, while another is written in a graph, allows artists and programmers to work together in ways that haven't been possible before. Having something like this standard in Unity just makes sense.
     
    Edy, a436t4ataf, JoNax97 and 2 others like this.
  20. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,948
    Yes, you're right. I mixed it up in the text, it's the other way around :)
    Still, you cannot have a single shader file and have several property blocks in there. You can create several shader assets in this case.

    I'm glad the feature is useful after all :)

    No, I'm not :) I'm fully aware of the issue you're talking about.

    I fully agree with that.
     
    SonicBloomEric likes this.
  21. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    Ok so if you are fully aware of the problems we face and fully agree that there is potential for nicer workflows, then what is Unity's official stance on this subject?

    Are there real improvements planned?
    If so what are they and when are they scheduled to be delivered?
     
  22. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,948
    I'm not in a position to talk about that, sorry.
    I also think making such statements deserves a separate thread.
     
    pachermann and SonicBloomEric like this.
  23. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,083
    I think this is specifically why we're not excited about the feature announcement. This "feature" feels like a bandaid that enables you folks (Unity) to continue hobbling along with your current practices; to continue to ignore the actual problems.

    Does anyone at Unity actually dog-food this stuff? What Unity Technologies asset on the Asset Store properly releases updates that support multiple versions of Unity simultaneously? I don't mean the assets that receive updates that simply state "Compatibility with Unity 2021.1" because that's just someone opening the asset, making sure things work well enough, and then uploading that package with a bump. I do mean an asset that receives actual feature updates that work across all Unity-"supported" LTS versions (at least).

    The feeling from us-your-customers is that no one at Unity Technologies actually understands what it's like to try to professionally support this stuff. Therefore, features like this feel like dog bones handed out to distract from the real issues.
     
    hippocoder and Edy like this.
  24. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    The answer is no, they don't. Unity has never actually done what asset developers have to do, which is support older and newer versions of a shader in an actively changing code base. Once they ship a unity version, they might go back and do fixes, but they don't have to continue developing new features with that as an active target. And they certainly haven't even tried to do it across SRPs, versions, or between pipelines. The Snaps assets only work with 2018 HDRP, and a few were upgraded to HDRP2020, but then removed support for 2018 and skipped 2019 entirely. On top of that, they only support a single pipeline in every shader asset they have released. There's no question why these have a 2 star rating.

    Unity doesn't see publishers as it's customers- there have been numerous statements that show they think their customers only work in one pipeline, so why would they need to support things across pipelines? The fact is I pay unity more than what enterprise support for a studio costs every year via it's 30% cut of my sales (which is now high for this industry), and most of what they do is make my life harder. The downloading system for assets in Unity doesn't even work on half the versions of Unity people use, and they refuse to let users know about this and push off the support costs to publishers who have to explain to every user that the package manager lies about the version that's installed and you have to manually delete a folder if you want it to actually upgrade your asset.

    Had they actually had someone dog fooding this stuff for these use cases, I'm pretty sure we would have heard reports of an employee going on a post office style murderous rampage by now.

    Having to rewrite all my shaders once to move to a more modern platform, that allows Unity to move graphics forward- that I could have accepted. Having to rewrite them on a regular basis, whenever Unity changes anything, for 3 different rendering engines, with no documentation, while Unity promises over and over to improve things but does nothing, thats unacceptable.
     
    funkyCoty, Edy, cecarlsen and 2 others like this.
  25. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    Fully agree with its own thread, although I imagine there are many touching on this subject overal. Jasons #SRP life one comes to mind.

    What we need is Unity to make an official thread or layout a proper roadmap again.

    Also you not being able to talk about this is Unity failing miserably :) Either:

    1: You don't know of an official plan, which I doubt and which would be utterly terrible on management.
    2: There is a plan, but its still not sure if its a good one so the official line is "don't talk until we are sure".
    3: There is a plan, but it looks like it will take so long to finish it that the line is "not to talk about it to avoid bad press on the severe wait".
    4: There is no plan at all other than "keep hacking at URP until its stable and we can get rid of BIRP".

    None of these scenarios are good considering how long SRP have been around.

    Another thing Unity fails to miss the point on is that without the asset store, Unity wouldn't be as popular as it is. Sure its not what's making them their big bucks but it is what makes the engine accessible. Beginners can drop a few $ to grab an asset to plug the holes in their experience and every so often one of them becomes the next hyper casual success which if they properly got their customers into the Unity ecosystem of ads etc is where the money starts coming in (if I remember rightly from the last public report I skimmed a summary of).

    Anyone who has been a publisher with more than 5 sales can tell you how painful it is to support the store since its full of so many beginners, and this is without the SRP madness.
     
  26. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,948
    Since we agree on that, let's keep the discussion in this thread about the feature that just became available.
     
  27. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    When unity creates such a thread we’ll be more than happy to move there. But in the past this hasn’t happened (at least a dozen “this isn’t the right place, we should have a thread on it” that I can recall), so it’ll continue to inhabit every thread that touches on the issue until that happens.

    basically until unity takes an active part in addressing this issue and moving forward, the community’s anger and disillusionment will only grow.
     
    funkyCoty, hippocoder, Edy and 5 others like this.
  28. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    Was about to say something almost exactly the same. Also the "feel free to start one" doesn't work since Unity never chimes into the user started threads on this (such as the #SRP life one and likely many others kicking around)
     
    Edy and jbooth like this.
  29. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,948
    I thought about this over the weekend. This may feel like bandaid, but if you look at it from a different perspective, it's a needed piece of functionality at the low level. It would be required to address larger issues mentioned in this post anyway. It also makes lives of asset store and tools developers better.
     
    fherbst likes this.
  30. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    I strongly disagree - a proper abstraction layer doesn't need this functionality, it simply compiles the user code to all three pipelines on demand. If this was actually true the shader graph would have needed this functionality to cross compile between HDRP/URP, and it doesn't because it does the same thing I do in Better Shaders.

    What DOES need this functionality is simply sticking 5-7 different versions of a shader into one file. In essence, this change points to unity doing exactly what everyone fears- retaining the current complexity but covering it up so users don't realize how incredibly hard it is to maintain compatibility with SRPs.

    Given the lack of communication around this issue, the blatant ignoring of it for several years, "Trust Us" isn't exactly going to be heard coming from Unity on this matter.
     
  31. ali_mohebali

    ali_mohebali

    Administrator

    Joined:
    Apr 8, 2020
    Posts:
    119
    Thank you @jbooth and everyone else for the feedback, I just wanted to chime in here as well.

    This feature makes supporting multi pipelines/single pipeline across multi versions with handwritten shaders more streamlined. We have tried to make sure we reduce breaking changes in shaders as much as possible this year. But we also understand users are still utilizing older versions of Unity and SRP packages, and it takes time for users to migrate. So this feature is built to address developer pains with these specific problems. Simplifying the development of handwritten shaders across SRPs is another problem space we are looking at addressing; however, we are not addressing that in this year’s release cycle.

    The goal this year has been around making sure each pipeline is successful in its goals, stabilizing them, and improving feature completeness and quality of each pipeline. Specially bringing URP closer to parity with built-in, improving the upgradability from built-in, and preparing it to replace built-in as the default rendering pipeline. These goals will address the concerns of a large number of our users. Having said that, we appreciate your feedback, and we are not ignoring other gaps that users such as yourself are raising. We have also started improving the foundation for cross pipeline workflows this year and are planning to reduce the burden of working with and supporting the multi pipelines. We are also looking at and discussing improvements from the perspective of handwritten shader workflows. However, we will not complete these goals in 2021. They are not part of the main goals we will be delivering this year.
     
    cxode likes this.
  32. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    Ok so to summarise and quote myself:

    "4: There is no plan at all other than "keep hacking at URP until its stable and we can get rid of BIRP".".

    Forgive me if this doesn't fill me with confidence, especially since its been 2 years since Unity claims that SRP are "production-ready" and I would guess at least 4 years of total development time: https://blogs.unity3d.com/2018/02/2...er-pipeline-optimizing-real-time-performance/

    Plus as you say it will be another year before improving handwritten shader workflows is even possibly looked into.

    This entire time BIRP has been massively neglected in favour of SRP, and looks like it will be at least 2 more years before URP is even close to a state to consider using it. Although more realistically it will be 3 years since it will be 1 year to potentially get parity, 1 year to maybe have scripted shaders again and then the final year for these features to stabilise judging by a generously forgiving time it takes Unity to stabilise features (E.G Burst, ECS etc).

    Lets say everything goes great though and its only 2 years. Well this still means I am working on a pipeline that is now 5 years out of date (assuming it has been neglected for the past 3 years of SRP development as seems to be the case).

    Do you not see how horribly gaping this flaw is? Games technology evolves incredibly quickly and 5 years of neglect is HUGE.

    My options with Unity are utterly terrible. I either stick with an outdated but sort of stable platform. I could move to URP which doesn't have the feature parity I need or move to HDRP. HDRP would be right for our target market (high end PC) but its just as unstable. Now lets ignore the fact that I do not want to work on a platform still undergoing major stabilisation changes, it also seems to have too high of a performance cost (from reading other users testimonials and from quick tests). However to properly test this myself I would need to ditch or rewrite every shader I have bought from the store, from scratch. I would have to rewrite all my own shaders and its not just a rewrite but a conversion to a node based system which I hate because complex shaders are an utter mess in node based systems. One of the includes files we have for a water shader is 1300 lines long (uncompiled). I don't even want to begin to think how that would look in a node based system.

    All that, just to start testing if performance is good enough for our project.
     
    cecarlsen likes this.
  33. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,948
    Technically speaking, this depends on how you build this layer, and on what foundation. The fact that some designs do not need this doesn't imply that no design needs it.
    Right. It makes it possible to ship a single shader asset that works with several pipelines, instead of having to ship multiple shader assets, and users having to choose, which one to pick. If you organise things properly in include files, you don't even have to keep a lot of code in the main shader asset, if you prefer it this way.
    It also addresses an issue that has been around for a while. Shader assets are quite special, because they can be spread across multiple files, thanks to includes. The presence of the concept of packages makes it possible for a logically single asset to be split across multiple packages, and this is completely orthogonal to the render pipelines. This feature makes those cross-package asset dependencies explicit.

    This is not what @ali_mohebali wrote :)
    I still urge you to discontinue writing posts that are not relevant to the discussion of the feature that this thread was about originally.
     
    pvloon likes this.
  34. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    As mentioned previously, if you could direct me to an official thread monitored by relevant Unity staff that is appropriate to discuss this then I would be happy to do so.

    What did I misunderstand? Ali said that investigations/discussions had started on improving hand written shaders but they wouldn't be done by 2021. I had then assumed this meant that something would be done to improve them in 2022, or possibly later, otherwise the investigations/discussions are utterly meaningless.

    Sadly, I fully appreciate that this means nothing and that the end result could be that no improvements are made even though there are clear alternatives/options available.
     
    Edy likes this.
  35. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    While true, @ali_mohebali just basically said a real abstraction isn't being worked on, or even looked at any time soon, thus debating that this "might" be part of a future shader abstraction layer is disingenuous as clearly it's NOT part of something your not even remotely working on at this time. This feature IS about stuffing multiple SRP shaders together, the use case of potentially putting things in different packages did not come up until I theorized about it - had that been the actual use case, it would have been in the original post.

    If you had just said "yeah, this is about stuffing multiple SRP shaders into one file" we could have skipped over half this thread.

    And this will continue to happen until Unity actual engages people on solving the real issues with the mess they have thrust onto it's users. The more it gets talked around, the less direct answers to things like "Hey, is this just about stuffing multiple SRP shaders into one file?", the worse you make it. If the answer is "Yes, it's about stuffing multiple SRPs together in one file", then say that. I might not like the answer, I don't have to, but at least it's a truth I can deal with instead of some theoretical exercise in justifying the feature without admitting that. Unity's reaction the SRP backlash has mostly been to avoid it, justify it, brush it off, or shoot the messenger. That isn't a good way to deal with issues.

    With that said, I do need to point out that I do appreciate you two actually responding, because the silence on this has been deafening, and only causes me to raise my voice louder, and as much as it might seem like I enjoy this trust me I don't.
     
  36. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    Agreed. You two are currently bearing the grunt of my frustrations (which isn't personal at all and I still try to keep respectful of course) precisely because it feels like there is nowhere else Unity will properly listen short of pinging Aras on Twitter which is not fair to put everything through him and thus something I try to avoid.
     
    Prodigga likes this.
  37. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,948
    That's about Unity versions. 2021 is in late alpha right now, and, as @ali_mohebali said, it wasn't part of the focus for this version. I'm not saying it will be the focus for 2022, though :)

    Given that shaders don't even compile if an include comes from a package that is not installed, don't you think it's an obvious thing? The main and immediate use case we had in mind for this was to help developers on asset store which target multiple SRPs, sure. I stated that explicitly here and at the same time made an edit to the first post on the thread. But that doesn't mean that we didn't think about this when planning this feature.
    Just for comparison: when we talked about keywords in the thread right next to that, I immediately admitted that we didn't count the behaviour change we did as an important one, and promised to fix it.
     
  38. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    Ali did explicitly say "delivering this year".

    Granted I was thinking in years rather than specific Unity versions so I see your point but I don't think it disproves my estimated timelines since they are aimed to land within their named years.

    Also, ECS was first released 2 years ago and is still in preview. I think Burst was also two years ago and is only just now coming out of preview.

    My estimate for a year to complete a big feature was very generous given the current track record so I think my timeline stands strong to say it could very likely be 3 years before we have:

    1: Feature parity with BIRP
    2: A system to write shaders with code and not shader graph
    3: A conversion system to auto switch between pipelines
    4: Each feature out of preview

    Do you disagree and think it will be done sooner?

    P.S Point Light Shadows, which is expected in any modern games engine, took 3 years to be delivered.
     
    atomicjoe likes this.
  39. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,083
    Out of curiosity, how many Asset Store developers actually support multiple SRPs with hand-written shaders?

    My understanding was that the limiting factor for supporting multiple SRPs wasn't so much getting the right shader connected to a material (there are various ways we've worked around this so far), but rather that there is, like, zero documentation for writing custom shaders to begin with [without ShaderGraph]. This lack of documentation isn't limited to proper interfacing with SRP-specific features, but changes that occur between SRP versions (even minor ones) which makes upkeep a nightmare.

    Am I missing something? Has documentation improved to the point where Asset Store Developers are now writing lots of hand-written custom shaders for URP/HDRP?
     
    Edy, cecarlsen, Neonlyte and 2 others like this.
  40. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    Still zero documentation, people have just reverse engineered it, particularly in URP which is simple compared to BIRP even.

    In terms of how many assets use hand written shaders, there are a lot more on BIRP/URP than on HDRP due to it's complexity. Quite a few asset developers have moved over to Better Shaders at this point- off the top of my head Space Graphics Toolkit, Paint in 3d, Vegetation Studio, and of course my assets. Quite a few are using amplify as well (pretty much everything that works on all pipelines, since Unity didn't support BIRP with their shader graph until just recently). Also, quite a lot of assets just don't support SRPs or in some cases just pulled their assets rather than dealing with the headaches any longer (CTS).

    Amplify is basically doing the same thing I'm doing- running after every release and creating new templates to abstract the pipelines. It's basically a massive amount of wasted work being done with what I can only describe as anti-help from Unity. If Unity had to port this stuff themselves, or anything really, we'd have had an abstraction years ago.
     
  41. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,083
    That aligns with my perception. I was hoping that someone at Unity could chime in with their understanding as it may help better contextualize why this was selected as a useful feature to add. If they, for instance, have data that suggests that "hundreds of assets on the asset store have multi-RP packages with hand-written shaders and this would help all of those people", then, cool.
     
  42. LaneFox

    LaneFox

    Joined:
    Jun 29, 2011
    Posts:
    7,442
    There's clearly publishers that are making tools/shaders by hand and want to support multiple RPs, but the much much louder group is the 3d art content publishers who can't support multiple RPs because there's no base abstract shader that works on all pipelines.

    This is a huge drawback to SRPs, the system is completely horizontal. Pushing a feature like this in the local space is a nice thing, but in the larger sense it's a band-aid on a system that needs proper abstraction and structure. To be honest I think that unless that gets changed then things like this are just adding to the mess and ultimately a waste of time.
     
    SonicBloomEric and jbooth like this.
  43. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,948
    Well, this just means it's unlikely to happen during 2022.1 alpha.
    You seem to mix "time taken do develop" with "time between something identified as needed and it being available to the users" :)

    To be honest, I don't know. AFAIK when we planned the feature, we had enough requests from users.
     
  44. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    And you repeatedly ignore/avoid the important points/questions :)
     
    atomicjoe likes this.
  45. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,948
    Right, see post #22 :)
     
  46. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    Then do you not think perhaps its time to start that official thread as has been asked for many times? Or perhaps to direct someone who could comment to this thread?

    Is @ali_mohebali someone who is in a position that can comment?

    I'm pretty shocked I even needed to say this....
     
  47. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    I mean this has been the pattern for ages - it flares up, we're told this isn't the right place to have that conversation, then the right place never appears. It's just playing politics at this point. I'd much prefer them come out and say "No, we don't plan to ever fix this, suck it up" than playing this game as it feels dishonest. I was told all kinds of things last year in a private meeting, and none of it materialized. There is no plan, just doubling down on ramming these pipelines through and covering up the artist facing issues with them.
     
    Last edited: May 20, 2021
  48. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    So, recently I've moved some of my assets to packages on the asset store- we figured out a way to get them to install into the packages folder instead of assets, which better matches the new workflow.

    So I converted MicroSplat to that format, which has a bunch of advantages, including asmdef support.

    However, the asset store has decided that we're not allowed to have more than 4 packages in an asset.

    If one goal of this feature is it allow us to have shader includes in packages, then that's going to be pretty hard if the asset store is going to place an arbitrary limit on how many packages we're allowed to have in a single asset.

    Left hand, are you sure what right hand is doing?
     
    nasos_333 likes this.
  49. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,083
    Request for clarification: can you expand upon what this means? When you say "4 packages in an asset" do you mean that the Package Manifest that can optionally be included can have no more than 4 dependencies? Or that you can only include a maximum of 4 .unitypackage files in the asset? Something else?
     
  50. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    A maximum of 4 packages - you can have dependencies on as many of them as you want. For instance, the MicroSplat Terrain Collection is 11 modules, which are also sold separately. Currently when you install the terrain collection, it installs all of these modules together as if it's one thing, rather than forcing you to download and update each one separately. This is obviously a great convenience, especially considering how many issues the package manager has had.

    However, if I switch to packages, I will be forced to turn this into the new 'bundle' style, where the user has to go to all 11 modules and buy them (for free), then download/install them all separately, because the asset store will reject any asset that has more than 4 packages in it.

    So if you wanted to logically break your asset into reusable packages of code, making it easy to share common code between assets, but then want to ship them in a way that doesn't require the user to manually install all of these dependencies, then you're limited to the arbitrary max package count of 4. There is no technical reason for this limit.
     
    Ruchir and SonicBloomEric like this.