Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Official Package requirements in ShaderLab

Discussion in '2021.2 Beta' started by aleksandrk, May 12, 2021.

  1. ali_mohebali

    ali_mohebali

    Administrator

    Joined:
    Apr 8, 2020
    Posts:
    119
    We can definitely start a dedicated thread specifically around the topic of Surface Shader like solution support in SRPs. Is that what you are requesting? Let me emphasize something though, we have gathered a lot of feedback around this topic, and we understand lack of this feature is a pain point for many of our users. In our update last year, we acknowledged that we are aware of the gap and started planning for shader abstraction for programmer workflows. Addressing this gap is one of our top priorities for SRPs, and we are actively discussing and working on delivering a solution. However, as I mentioned here in this thread, it will not land as part of our release cycle this year (21.2, 21LTS). Our goals this year include other high-priority items.

    I think it makes sense to start a dedicated thread so we can gather further feedback, hear your thoughts and for our team to engage and validate ideas as we are working towards the solution. I will start this dedicated thread soon. However, if you are after commitment for a version/timeline from us, we can not give you at this point. We are still in the development and delivery phase of our 21 release cycle goals. And we are moving towards planning for 22 goals. Once we have locked down the roadmaps for future releases, we will update you on the timelines.
     
    Edy, Ruchir and ccjay like this.
  2. Deleted User

    Deleted User

    Guest


    Hey @jbooth ,

    I just took a look into this. We've decided it makes sense to allow for more than 4 packages in a "Bundle."
    While there is no technical limit to how many packages you can add using this technique, we would not like to have a user experience where too many packages are imported for the user at once without setting expectations.

    We will keep the limit at 4 for now, but we will accept up to 25 for products that are described as "Bundle" in the title.
    We are open to changing all of these limits in the future as they make sense for users. For now, we are carefully reviewing assets that are being submitted as packages to make sure that we set a fair expectation with users.

    @jbooth
    Please resubmit your updates with the other changes we have requested as well being the type set to tool in the json.


    Do note that if you are a publisher who wants to upload in this fashion, your asset will be held to specific standards which we are testing and building internally for 3rd party packages on Asset Store. We ask for your patience while we finalize messaging on this.

    Our goal and intention is to partner with the community on this as we bring the package format to Asset Store.

    Kind Regards,
    Salvatore
     
  3. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    So I need to change the name of the Terrain Collection to Terrain Bundle then?

    I get the desire to not have someone install hundreds of packages. In fact, I’d definitely prefer to have all of mine under a MicroSplat folder if that was possible, but I don’t think that is right now?

    in the long run, it does seem like the right answer is 1 or N, with 1 grabbing everything via dependencies, but I realize that’s a bit out of scope for where things are right now.

    [EDIT] Ok, finished the submission. I left it as terrain collection for now, because that's what it's been known as for ages. But if it has to change I will change it.
     
    Last edited: May 20, 2021
  4. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    @ali_mohebali thanks for the links and chiming in.

    Ok so skimming through that thread shows exactly what the problems are. So many people have commented on it or asked specific questions and its been completely ignored by Unity employees, including yourself, for almost a year now.

    Its worrying that something considered top priority (Surface Shader like solution) isn't even on the roadmap.

    Also what is worked on in the roadmap is pretty confusing. I see 2 days ago you mentioned Enlighten support has been added to URP, I thought Enlightened was depreciated? Global illumination also has its own tab and Enlighten is in progress (guessing its just not updated yet) but there is nothing in there about replacing it entirely which is what I thought the plan was.

    Honestly after looking at the massive thread and seeing how it was abandoned I don't see the point in starting a new one now. Nothing will change and frankly promises from Unity feel emptier every day.

    Then this, right here, is the core problem. You have no idea when URP will be ready to take over BIRP whilst hitting all the goals mentioned in that master thread. If the SRP were still treated as in production then I really couldn't care less about this, they could take as much time as they needed to get to that point.

    HOWEVER. Whilst all this is going on the core rendering of Unity is getting more and more neglected/outdated and that is not acceptable. Its basically been a year since that thread was started and there is still a long way to go to get to that finish line.

    What I would love to see is another update like that thread. A year later, what we achieved and what we still need to do. Then as a part of that thread Unity really needs to finally acknowledge that SRP have been a hot mess from the start. The vision is still good, but to get there will either take very serious changes or a Long time and ultimately the realisation that it will take too long. So in the meantime more resources need to be re-tasked back to BIRP to get it back to more modern standards. I won't go into specifics too much since Unity should be keenly aware of them with the work on HDRP but the big 3 for me are:

    Improved shadow options/support
    Realtime GI (since I'm told not to use Enlighten)
    Raytracing (this is even on the asset store, I see no reason why it can't be in built in).
     
    Edy and atomicjoe like this.
  5. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    Overal, my issue with SRP is it feels like Unity is trying to shove them down my throat when they are not ready yet. I would like that to stop.
     
    atomicjoe, LaneFox and knxrb like this.
  6. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    I'm actually perfectly fine with not having a strict timeline yet- that's most likely going to be wrong until you actually start coding and running into the issues, and then people get angry because the timeline gets changed. I can also understand if it wasn't considered a priority for the 2021 cycle- I don't agree with it, it would be a 2017 feature if I was running things, but at least it's an actual answer. (God retrofitting this is so much more work than starting out with it)

    My frustration primarily comes from these being marketed as production ready, forcing us to adapt to keep our products relevant, with what can only be described as anti-help from Unity, combined with a complete lack of communication on the issue. Most of the burden of these decisions falls on the asset developers, be that in having to reverse engineer everything, or users taking out their frustrations of using SRPs on us, and having crazy expectations of how fast such a transition would take and no expectations of paying for it. A lot of my public persona of the last year(s) has been to make sure they know exactly how bad this transition has been and how much of a nightmare it is to counter balance those expectations.

    Even without an abstraction layer, had Unity come out with documentation and helped developers with these transition issues, the tone would be very different. Had I not been personally told lots of things from people in charge, non of which came true, my tone would be different. Other than a few employee's who deeply wanted to solve this issue but were prevented from doing it, the only post or communication previous to this thread hijack which really acknowledges the issue at all is one from almost a year ago, which only really happened because I had started a very large and vocal campaign about it right before the IPO.

    Unity could solve this issue in moments, and there are a number of options to do it- from acquiring Better Shaders and making it free, to completely ripping off better shaders in every way possible, to writing their own custom solutions, etc. I don't really care which, I just want to see forward motion on it and enough velocity that I think there will eventually come a day when I'm not spending all of my time chasing after SRP issues and changes and running 5 copies of Unity so that I can check changes in what are effectively 5 different rendering engines.
     
    hippocoder, Edy and Jingle-Fett like this.
  7. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    1,083
    [EDIT] Please disregard my previous questions in this space. I hadn't noticed that there was an entire "Page 2" of responses that already provided answers to those questions.

    Derp.
     
    Last edited: May 20, 2021
    Deleted User likes this.
  8. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    A future where shaders could depend on packages and set a define when they exist would be super powerful. A pain point for shaders right now is integration between assets. For instance, I have integrations with Digger, Enviro, Weather Maker, Polaris, Bakery, Vegetation Studio, Curved World, and likely a few other systems. Stackables in Better Shaders did a lot to help with this (add the stackable, done), but I still end up shipping a lot of, say, bakery code with Better Shaders. Now, it could use an include from Bakery, but this would require an absolute path to the cginc file, which breaks if the user wants to organize their directories differently. If a shader could set a define when a package is available, it would solve several of these issues. First, the path would be relative to the package, instead of absolute, so include files become a real possibility again. Second, the integration could be reduced to #if _BAKERYINSTALLED DoWhatever() #endif in many cases. While there would still be potential issues there when the API changes, it's reduced and can support multiple versions, and no code needs to get duplicated, which is nice.

    Anyway, I might be deep into stuff which only affects a small percentage of people, but it's a pretty big challenge to support asset dependencies for asset authors, especially in shaders which are not super modular in nature, so it's something I thought a lot about when writing Better Shaders. If shaders can have dependencies on packages, optional ones which set a define could greatly help with integrations and reduce code duplication.
     
  9. Deleted User

    Deleted User

    Guest

    Cheers @jbooth

    Thank you for working with us on this one! We are excited to see user feedback on the use of your packages. I find this to be a great opportunity to improve the quality of life for users while Unity completes our full 3rd party package manager solutions.

    Kind Regards,
    Salvatore
     
  10. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,952
    We have this in our backlog :)
    But there's a cost to such functionality: if a package does this, each time it gets installed/updated/removed we would need to reimport and, potentially, recompile all shaders. Even with caching, this may not be a very nice experience.
    It's also likely it will be limited to having simple macro name definitions, even without values. Otherwise there may be a collision between packages (one says FOO is 1, another one says FOO is 2).
    And then there is the question of "what happens if there is a shader that says #pragma shader_feature FOO BAR, and we have FOO declared by some package".
     
  11. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    Recompiling all shaders on package import doesn't sound like fun.

    I suppose you could accomplish this now with some scripts that set a multi_compile keyword, it would just be a bit hackier. With the keyword count being essentially unlimited, not such a big deal to have an extra one per package. Maybe tie the name to the package name, since they can't collide anyway. So if com.jbooth.microsplat.core is installed, COM.JBOOTH.MICROSPLAT.CORE is defined.
     
  12. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,952
    multi_compile wouldn't work - the includes that are behind user defined keywords are parsed anyway and the import will fail if it doesn't find it.
    It doesn't matter, how many keywords a package defines (and the engine defines based on package presence) - there may be some token in some shader or include that wasn't treated like a macro name, se we need to run at least the preprocessor to figure out if things changed :)
     
  13. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    559
    can't you do it like asmdefs do for conditional defines?
    a shader can declare "if package 'foo' is installed, then define FOO for this file", and then it can do "#if FOO" or equivalent. (FOO should remain local to the current file)
     
  14. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,952
    We'll figure it out when we go and implement it - I'm just saying that a generic "add shader defines through packages" is not that simple :)
     
    SonicBloomEric likes this.
  15. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,457
    And as predicted, no forum thread on a surface shader like abstraction has appeared.
     
    atomicjoe and PutridEx like this.
  16. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    2,109
    Hi @ali_mohebali. I would like to know when the dedicated thread will available?
     
    atomicjoe likes this.
  17. ali_mohebali

    ali_mohebali

    Administrator

    Joined:
    Apr 8, 2020
    Posts:
    119
    In terms of understanding the need and collecting feedback, we have been doing that actively. I personally have collated and documented all your feedback from different sources, including various forum threads. It is a priority for us. At the same time, we acknowledge that we can do better in terms of transparency and keeping you in the loop about the state of this initiative. To make it clear that we are actively working on this, we have added the item (here, and here) to our public roadmap. You can use those cards to give your feedback.

    Based on the earlier discussion we had in this forum, my understanding was that you would like a thread with a more concrete/tangible update from us to have meaningful two-way communication. We want to also use this thread as an opportunity to share our plans and technical solutions to get your feedback on and validate it. Before we can start this thread and share the initial update, I need a bit more time together with the team.
     
    cxode, Edy, cecarlsen and 6 others like this.
  18. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    Will this include a timeline for complete feature parity to built in and if not, then a commitment to ensure built in doesn't become even further out of date?
     
  19. marcte_unity

    marcte_unity

    Unity Technologies

    Joined:
    Sep 24, 2018
    Posts:
    17
    This conversation isn't about SRPs as a whole - we know built-in parity a concern, but we need to have specific discussions around shader authoring so that's what we're focusing on in the thread Ali is proposing.
     
  20. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    Ok, and I can't believe that I have to specify this, again... can we have an official update/conversation that "include a timeline for complete feature parity to built in and if not, then a commitment to ensure built in doesn't become even further out of date?"

    If you want it as a separate post then that is fine so long as the massive elephant in the room is actually addressed. Shaders are just a part of this bigger overarching problem.
     
    knxrb likes this.
  21. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    I can see why everyone ignores/avoids my questions around a timeline for feature parity when it takes more than 2 weeks just to start a conversation.......

    Its this which leaves me with no confidence in the future of Unity. Ignoring the blatant lies (Enlighten licences) which is already terrible, it seems the big company slow to do anything affliction has fully spread. Which means whenever new tech comes out Unity will take forever to catch up.
     
    april_4_short, knxrb and atomicjoe like this.
  22. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,869
    Let me quote this so it stays relevant:
     
    Edy likes this.
  23. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,869
    Again, let me quote jbooth:
    I don't know what you actually do at this point. Seriously.
    I think I have reached a point were I have to just stop believing Unity will magically fix all this mess and find solace somewhere else instead.
     
    april_4_short, Edy, matjumon and 2 others like this.
  24. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,952
    Small update: support for package requirements will be available in 2021.1.17f1 and 2020.3.16f1.
    2019.4 will get an update that skips the PackageRequirements block. I'll post here when I know the version in which it will appear.
     
    ali_mohebali and JesOb like this.
  25. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    Meanwhile its been 6 weeks since the request to start a conversation was acknowledged and its still not delivered...
     
    Edy likes this.
  26. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,952
    2019.4.30f1 will no longer throw errors when the parser sees the PackageRequirements block.
    However, it will not skip subshaders or passes based on the data in this block.
     
  27. DEEnvironment

    DEEnvironment

    Joined:
    Dec 30, 2018
    Posts:
    437
    Some feedback
    I see a few posts about api incompatibility and wish to bring up a specific case for lessons learn

    talking about 7x and 8x and what happen, it’s a topic to be left in the past for me as I focus on 10x and higher now..

    7x had a few breaking points that started in 7.4.2 and then again 7x became incompatible with 8x if you wished to use xbox or play station due to 8x never getting the correct shader tags..

    if you make a system to build the super Uber shader please consider how to deal with Unity core API supporting tags in one version but not found in others..
     
  28. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    698
    FYI, only took 4 months but the conversation did finally get started:

    https://forum.unity.com/threads/focusing-on-shaders.1171444/

    Now we just have to wait a year for it to release and another year for it to stabilise...
     
    funkyCoty, cxode, Edy and 2 others like this.
  29. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    700
    I am. And I want to die. link here.

    I'm only on reading this thread because I was hoping I could drop the PackageRequirements{} blocks in my URP specific shaders and call it good. Looks like that's not an option yet though. I'm still not clear what versions of unity have this, all the LTS releases don't seem to have it yet.
     
  30. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,952
    Everything after 2021.1.17f1 and 2020.3.16f1 should have support.
    WDYM by "don't seem to have it yet"?
     
  31. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    700
    On Unity 2021.1.28f1. I'm trying the following in a Lit Shader (copied from urp, with some modifications).

    upload_2022-1-10_20-2-9.png

    I get the following error:

    upload_2022-1-10_20-2-22.png

    This happens no matter where I put the PackageRequirements{} block, no matter the Unity version I've tried.

    Also; is there a solution for *.raytrace files? I have some .raytrace files for URP only, but not sure how I can wrap them in package requirements.

    I really wish I just had some define i could #ifdef and be done with the whole thing.
     
  32. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,952
    @funkyCoty Put the PackageRequirements block as the first thing in the pass, it will work. That's a limitation of the parser we use. We'll fix it, but not right now.

    No, there's no solution for *.raytrace or compute right now.
     
  33. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    700
    Alright, thanks. Is there a plan for raytrace and compute? Seems like half a solution right now without those.
     
  34. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,952
    Eventually, yes.
    But those two are a bit more involved, as there's currently no structure there like in .shader
     
  35. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    700
    Could we just get a define available for packages? When compiling, just include "#define {package_name}" and "#define {package_name_with_version}" etc. This would be enough for my use cases in all shader types.
     
  36. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,952
    If we did it this way, we would need to reimport and recompile all shaders on any package config changes.
     
  37. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    700
    Package config changes seem pretty rare, so that seems okay to me? Or maybe packages could either have a flag saying it contains shader data (so don't recompile otherwise), or it could be auto detected to only recompile if that package has shader data?
     
  38. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,952
    There are certain possibilities, for sure, but implementing this would still be more involved than providing PackageRequirements block support to compute and raytracing.
     
  39. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    700
    Are there any plans to make this actually work in 2019? We added URP and HDRP support to one of our asset store plugins, and we're getting support emails about users on 2019 getting errors because of this package requirements block not fully working in 2019 still :(
     
  40. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,952
    No. The changes between 2019 and 2020 were too drastic for a full backport.