Official Changes to the Graphics GitHub repository

At very least, you'd have to make sure actual fixes would stay in separate commits so people sticking to some older version of the SRP / Unity engine would have possiblity to cherry-pick fixes that land on future versions.

 

strengthen our programs and processes around source code and sensitive information

What sensitive information benefit is there to hide the source code if the packages contain it anyway?

 

Have access to all branches you can see the progress of a feature being working on whether it's progressing smoothly, on hold or abandoned.

 

Thanks for taking the time to open a dialogue, more often than not, people feel like they're shouting in the wind.

Yes, the roadmap itself is fairly static and only covers major topics. A concrete example is I now see the Render Objects feature it getting a Rendering Layer mask option, more specifically that it's being added to URP 10. I suspect that's something that's going to be mentioned in the changelog, but it's nice to know it's coming, or actively being worked on, a head of time.

Breaking changes to shader code are fortunately few, but they're not covered anywhere. Since URP 9, the
LightingPhysicallyBased suddenly required an additional argument, which wasn't mentioned anywhere. I suspect largely because people are expected to use Shader Graph anyway.

When URP 7.2.0 released Philip Lira shared a shader upgrade guide, describing the changes need to take the new cascades shadow maps into account. This was super valuable, and I was hoping to see this trend continue. This sort of communication does take extra effort, but definitely doesn't make us feel like Unity is plowing through with blinders on, and we're left to pick up the pieces.

I'm personally okay with digging through the repository to track down why my shaders/scripts break in a new release, or peering into the state of new features, it's a bit of a sport really. But I'm generally opposed to taking that privilege away, which fortunately is no longer the case! If moving the repo is necessary, then yes, we would need a more detailed and active roadmap and changelogs.

 

Mainly I track the main and release branchs.
But when I'm trying to make a custom SRP, for example URP with shadow mask, or point light shadow, or framebuffer fetch postprocess, looking around the other develop brunchs is important.

 

Unfortunately, the package changelogs rarely tell the full picture, for all the packages (there are some notable excuses like the Input System where the changelog is awesome).

Information completeness goes something like this:

1. PR / commit information (source repo level, like the Graphics repo)
2. Release diffs (package release level, like Mirror)
3. Package changelogs (on the package website / in the package)
4. Package documentation (on the package website)

On each step, a considerable amount of information is lost, or just hidden, either accidentally or by design.

The more complex a system is (with Graphics being high up the chain), the more complete info we as developers and
publishers would like to have to anticipate and react to changes.

100% this. A lot of the valuable information about the SRPs is not in the docs, but from ambitious individual developers. Things like the UniversalShaderExamples or HDRP-ShaderGraph or ... Of course, Unity doesn't do this officially because you don't want to put the burden of maintaining this on yourself.

The Product Boards are far too high-level and too rarely updated to know what's going on. A recent example on how this type of communication is not enough is the "Master Stack" in URP10; this sounds like a nice feature on the product board while in reality the only debugging option shaders had was removed without thinking twice. Of course this only becomes clear once that ships. There's no "Removed: ability to debug shaders" in the changelog, release notes, or anywhere, it's hidden behind a fancy-sounding "added master stacks".

 

All in all is defintly impossible to work with a simple change log, plus even if was detailed enough would still make development much slower than look at code and much harder to debug

 

I'm mainly tracking HDRP changes from the git repo and at least with HDRP the productboard - although it being new for it - hasn't really told much about upcoming things, it has merely told what's already in current releases and preview releases making it kinda pointless as it is now.

I'm super skeptical in any roadmap kinda approach for delivering updates to users in general, they've never really worked well on almost any piece of software where Unity doesn't make any exception. In fact Unity has been notoriously bad on updating their roadmap webpage in past (no need to go further into that as everyone knows the state it's currently at) and seeing how HDRP productboard has worked so far, it doesn't exactly shout confidence that this is the right way to deliver news to the userbase. I feel like these kind of roadmaps are a nice on idea level but really difficult to maintain and keep up-to-date in practice.

I do admit that I don't really follow URP or SG productboards as actively so if they do it better there, I don't really know much of it but so far for the HDRP the best source of information has been the git repo.

Just to make a point, this is only card in HDRP productboard right now that's not in any actual release already: https://portal.productboard.com/8ufdwj59ehtmsvxenjumxo82/c/186-material-variants. I do appreciate we have this card and I've seen some work on it but there are tons of other things going on with HDRP atm that are not communicated through the productboard at all.

 

I usually follow everything to have an overview of the development pace and direction, but focus is more on features or fixes that are important to my projects.

Seeing PRs give much better context too, and even allows some of us to fix our own issues while waiting for the official fix to land.


In general, having access to as much of the source as possible, as well as the steps in-between if possible, will often have us developers using what Unity has to offer more effectively. If I can understand and prepare for changes on systems I'm using, there's a real reduction in issues derived from not having enough knowledge about the tech itself.

Working with more transparent systems and development processes is just a lot less stressful to me.

 

Similar to what @rz_0lento does with HDRP I do with URP. Since I have forked URP in order to add all the missing features I need it to have, I use the git repo to properly communicate to me what Unity is actively working on as well as implementation details that may not make it to the changelogs. It's extremely useful for me to know the changes being done as it allows me to properly schedule the work on my end instead of downloading latest package and watch everything blow up.

The product boards tend to be too high level, usually devoid of any implementation details, and often are slow to be updated. To me that makes them mostly useless although I could see it helpful for a subset of people.

As for SG, we don't use it and rely on Amplify instead as artists like it more. Plus with Amplify we can more easily add new shader templates and at this point we don't really use any of the built in URP shaders.

 

Another thing to consider is the question of "is it worth introducing any more roadblocks for people transitioning to SRP". Transitioning developers to new systems, especially such large ones, is already hard on such a large scale, even more especially when SRP lacks in feature parity. I imagine making developing using SRP even 5% harder needs a pretty good reason to do so, and I'm not entirely sure what removing the day-to-day updates (and therefore the best patch notes a man can ask for) does in terms of gain, to be quite honest.

 

For us we do not use the github directly for the source code.

We read the comments that are within the github. It is interesting to read those because it gives a sense of issues and more importantly solutions that Unity engineers run into when making changes, bug fixes, adding new features, etc.

The comments do not necessarily have to be exposed through github. Potentiality a simple forum post every month or couple of months would be enough for us.

 

Hi, thanks for clarifying things. I've been porting a large project to HDRP over the last several months. I'll try to break down how different aspects of the public git repository have helped that process and ongoing development of the game I work on. If I didn't have access to the git repository during the past year, I might've had to rethink using HDRP at all because the effort would be significantly higher and the benefits fewer.

• Full commit history of all release branches. I've effectively forked the Graphics repository and have several changes and fixes applied to it. Without commit history of all release branches it would be much harder to navigate the codebase to apply fixes, harder to reapply changes to new versions of HDRP+Shader Graph+etc when moving to new Unity versions, harder to evaluate what the internal changes are in a new version, harder to know what the practical benefits and drawbacks are when changing versions... the list goes on.

• Ability to view feature branches in real time. Being able to see features and their implementations as they're developed not only helps me understand how I might need to change my code and what information I can give to artists on my team, it has also been a huge help to how I prioritize what work I should do myself. Sometimes the needs of my project are larger than what HDRP currently provides - but sometimes the HDRP / Graphics teams are already working on an equivalent feature or change. And in some of those cases the feature doesn't quite match up with what my project needs, so being able to judge for myself based on the real code has been very useful. On top of that, sometimes features aren't merged into a release version or might not be merged for months (at which point the version they're merged to might not match up with mine) for a variety of reasons. Having the option to integrate those features myself (only when it makes absolute sense, of course) is a real benefit.

• Being able to see pull requests from the Graphics team. Similar to the above point, seeing a summary of a feature or change or bugfix in the Graphics repo as it's finalized - as well as whether comments are left indicating it might need more development time before being merged - helps my own development process.

Now, let's say theoretically I had continued access to all that just for my project, but parts were taken away or limited in various ways from the general public. This would be ok for my project in the short term - but in the long term it would hurt the overall Unity ecosystem (third party packages and tools, bug reports and fixes, discussion of feature direction, and more would all hurt by limiting access). So if others don't have the kind of access they do now it still hurts my project, regardless of what I have.

 

Which information you claim as "sensitive" in this case? Do you mean source-code transparency or what?

 

@smcclelland Missing out on the PRs would be a huge deal breaker because they bring essential context to new features and breaking changes where often roadmaps, release notes and documentation are lacking, sparse, or where a public API does not yet exist and may not for a few years or more. And with all the advancements and changes I don't see this changing for several years even if Unity throws huge teams of people at improving the documents and APIs.
Or ever assuming Unity is planning on staying on the cutting edge with new features and advancements. Which I think is one of the growing pains Unity Technologies is going through right now with all these ambitious projects that honestly, are the main reason I moved to Unity as my primary engine a few years ago.

Putting too many cooks in many of these kitchens will actually slow down the development process and add additional confusion and disparity between the actual development process and the documentation of it. Much simpler and straightforward for all parties internal and external to just have a fully transparent development process and let us see the PRs, Branches and Commits, etc a full mirror. I'm not sure if there is some sentiment internally at Unity to keep PRs private or if there is a technical challenge to overcome with mirroring the PRs from the GitHub Enterprise instance?

Anyway, In short, I think we all want Unity to stick to this statement in the Graphics repo README.MD

"Unity is committed to an open and transparent development process for SRP and the pre-built Render Pipelines. This means that so you can browse this repository to see what features are currently in development."

It's already limited enough that we don't have full source available

By the way, thanks for unlocking this thread and addressing this issue directly. It is much appreciated.

 

In a purely hypothetical idealistic sense (everything up to date at the speed of light and 100% accurate and complete info) It would be maybe ~80% as valuable as having visibility to the Pull Requests. But in reality with human error or misunderstandings, mistranslation / miscommunication, omission and obfuscations (intentional or just oversight) and just plain out of date information due to delay in time to publishing docs and roadmaps, changelogs etc I estimate realistically only expecting this being at best ~60% (EDIT: other devs estimate as little as 5-10%) as valuable as simply having visibility into the PRs. (for software developers.) yet it will take more human resources and busywork from development and knowledge management teams to attempt to capture and recontextualize the level of detail already in the PRs elsewhere. Not that the higher level resources aren't helpful but for the most part they're already detailed enough for the less technical team members that benefit from them, not those of us tasked with writing the code that needs to weave a lattice together into a continually advancing and evolving Unity Engine and SRPs, and other packages.

 

Considering how lackluster the existing URP docs are I could never imagine a situation where I could be happy with the repo being restricted, I have 0 faith in Unity to adequately document such a complex feature group. Currently, the Scripting API docs may as well not exist, they explain almost nothing. The Manual is an incredibly light touch that does little more than a brief onboarding.

When starting with URP I had to commit a great deal of cargo cult'ing, copying any shred of an example I could find. Most of the time this resulted in replicating horrible mistakes that others had also made due to the lack of clarity in the API (ScriptableRenderFeature.Create is called similarly to Monobehaviour.OnValidate, meaning you can very quickly leak managed resources to the tune of GBs).

I really doubt Unity has the resources to duplicate enough information quickly enough to replace the public repo.

I also wonder what that means for public PRs, would this be the end of users contributing fixes to the projects?

 

I liked @slime73's post because I agree to most of it and because there were multiple things mentioned which would be practically impossible to do if we didn't have current graphics repo on git. I now mean things like:

- Be able to see upcoming things so we don't do our workarounds for nothing. Roadmap would help on this but like mentioned, there's very little faith in we getting a roadmap that would cover all current development and possible developments in the future - we know Unity doesn't want to overpromise things so any potential roadmaps we might get would not even list all things being developed currently.

- Be able to merge small improvements or sometimes even fixes that are not coming to version we rely on ourselves - there's a long history on "we are not going to backport feat x and y" or in case of fixes, it could just take over month to see them land and arrive - if they even do land. Now that SRPs are in core, this also means the fixes will NEVER arrive to the version we are currently using and we are forced upgrade the engine to get SRP fixes the official route (and this means we get new bugs along with it - this is a very bad in general and I hate it).

- Be able see PRs and the internal discussion on them. I realize this isn't exactly a thing we are supposed to see but more like a side effect but PR descriptions usually tell the actual intent and reasons behind the upcoming changes. There's never going to be a doc or roadmap that would explain all of these reasons. Instead docs just explain what the currently available features do. In case there's some optimization PR, usually the PR description gives nice timings so we can understand what kind of improvements we are getting from that change or PR could give nice programmer art that's not documentation ready that shows what the change did visually (which almost never makes to official docs either).

- Be able to understand what it takes to implement some change in SRP codebase ourselves. This is a huge for me. If I didn't see individual PRs and files they had to touch to make it happen, I probably wouldn't have been encouraged to make similar changes myself locally to tweak HDRP do things my way. This is a huge learning resource that would go away in case we lost the full repository access.

 

Those merged mega PR's are the biggest issue here. We can't merge individual fixes or feats if all we get is commits that have 10 PRs squashed together.

But PRs in general are really useful - like explained on my previous posts last 2 points. Being able to see how things are being done by Unity for SRPs is just really valuable resource in case you need to do custom work on these yourself.

 

I think the most frustrating thing is that the reasoning for closing off this information is so weak from the end user's point of view. We're losing access for no benefits to ourselves and a very vague procedural change for Unity

 

@smcclelland This is my thinking too. Can you clarify what internal problems are being solved by this? Maybe if we understand your pain points better then it will clarify why you want to change things at all.

Everyone on this side of the fence seems happy with the repo situation staying as it is and it's not clear why you can't just leave things in place.

 

I'd like to chime in one this one with a general remark: In development, in general, the "closer" pieces of information are together, the better. Roadmaps, release notes, documentation and changelogs are all useful in their own right, so I wouldn't abandon them - but they cannot replace access to PRs. And PRs probably cannot replace access to a complete commit-history ("probably" because PRs include commit history ... but sometimes, a pure chronological change history is simple a better way of navigating than via PRs). IMHO, changelogs are probably best replaced with access to commit history. Release notes give the higher level abstraction of what has changed, the commit comments provide the rationale behind each change, and the file diff then provides what exactly has changed.

It's the same with bug reports / feature requests: I understand that Unity has QA processes in place, and it creates friction when there are different bug reporting systems through which bug reports come in. But IMHO, usually, for everything related to a specific package (or even group of packages), the infrastructure provided by GitHub is actually much more practical to work with than Unity's general purpose systems.

 

I feel like my previous post and a few other people's posts already answered that, but I have a question - if you're confident limiting access is acceptable, wouldn't it make sense for an internal team at Unity to develop a full game (with a development time spanning multiple Unity releases) using HDRP or URP with the same limitations in place that you're suggesting, and see what that team thinks?

Maybe that's a bit extreme, but the general idea of dogfooding is something I wish Unity did more frequently, at least.

It's probably also worth pointing out - one reason why lack of source / commit history / feature development access in the legacy pipeline was tolerated (albeit without much choice, for most people) was because it didn't change much at all. I'm very happy that HDRP and URP aren't stagnant, but it also means that there are magnitudes more changes made which directly affect every project that uses them. Being able to deal with those changes is critical to the health of Unity's user ecosystem.

 

If talking of fully closed source then this is what Unity was about 5 years ago. But game development scene has changed, without competing game engines giving full source codes to their users, I doubt we would have even gotten Unity's c# package sources here. It's still a LOT less than what Unity's competition does and going more back to the old Unity direction is just lose-lose for all.

Of course we users don't have much say on this (although we do hope to get listened) and all we can do is voice our concern or start looking at the other alternatives.

 

Нou gave people access to the source, now you want to take it back. Of course people will complain about this.
If you want charge $$$ for source access - just write about this.

 

You mean now consoles, right? Afaik both UE and CE gives you source code access to these platforms engine code as well once you prove them you are a verified console developer but this is not super relevant to this discussion. We are now talking about main engine functionality here and not specific platform SDKs.

Any source access is better than nothing of course. It would "work" but like countless reasons already mentioned on this thread, it would make many things a lot worse, more tedious and for some things straight up impossible. There's still quite little reasons given here what are the true benefits to Unity for such change but there are tons of explanations given here what users would lose.

 

No current plans to contribute those back and depends on work allowing us to share it.

The other thing is that our min spec, as far as mobile goes, is much higher than what URP is trying to support. That means a lot of our solutions rely on compute shaders, texture2D arrays, or other modern features. So it would be somewhat incompatible with what URP is currently trying to do.

 

Yes a valid reason for keeping experimental development branches private would be if platforms are under NDA, and they need to be referenced in code. Access only to those branches to who signed the platform NDA is that an option?

But if they are so much affecting the main code I can see why thats hard.

 

Yes. Actually I wish all the other packages Unity owns can have public repo so people can submit PR. I ady see quite a number of people want to submit PR but there's no public repo to submit yet.

 

Yeah that would be great to still be able to do that.

 

To be fair, I think regular users have been able to send PRs to Graphics and SRP repo before that but it hasn't really gotten that common (probably because people haven't realized they can do that). I think I've sent like few minor fixes to old repo and nothing to current one.

I think it's more about the culture we have here where the assumption is that Unity isn't accepting PRs. If you look at other engines, they have tons of user PRs coming in all the time.

 

SRP is currently in massive flux due to being early in production as well as changing goals over time. The documentation is very lackluster and out of date, and I know you've mentioned "what if it was better", but I think that requires it to actually be better for a few months to a year before the discussion can go in that direction.

 

With all due respect idk what "products" you are basing this on. Certainly not publicly available game engines in 2021. Unity's largest and most direct competitors all offer generous full source code access. And they benefit from being more readily extendable and flexible because of it.
So let's compare apples to apples here.

i.e. I still can't extend Visual Effects Graph (A "Graphics" package) with basic extra functionality that Unreal Niagra VFX already has like sorting and searching particle neighbor attributes because even with the package source, the buffers are actually (needlessly) set from Unity's C++ source. And while considered on the roadmap, there is no clear timeframe on when this be resolved or when there will be a major API refactor or really ANY public APIs available or what features will actually make the cut maybe 2 years from now. Yet when needed I can extend Unreal's Niagra VFX today or fork the entire engine. But you don't see me or anyone else selling some bootleg Unreal Engine just because I have access to source under license. Curious what some folks at Unity apparently think they are gaining and not losing by obfuscating and obstructing their developers from being able to use their products to build software. We are building increasingly complex software that requires equally increased transparency to integrate with.

This isn't Adobe or some company primarily providing product/services aimed primarily at non technical end users that can function with "no source access or repository level access" and a few meager plug-in APIs and SDKs in today's landscape. And if executives at Unity don't realize that, it'll result in reduced market share and penetration from companies instead choosing Unreal, CryEngine, Godot, etc because they didn't have to break their door down and start dumping in tens of thousands of dollars just to get a source license and the chance to talk to a Unity ISS rep about maybe getting access to the resources they need to efficiently flexibly implement and extend a real-time engine.What gives Unity most of its value and ubiquity is a function of what the community has access to understand and create resources for and spread knowledge about.

Anyway, thanks for hearing us out and gathering this feedback.

 

This! All of it +1!

I don't want to be that guy who always brings up UE4 in a Unity forum, but leading a company with two teams - one Unity based, one running Unreal (B2B client reasons...) I think, I can offer an angle here too.

What @landonth posted made me think, if that is actually part of the problem here and with certain trends in Unity in general. Maybe it is the comparison with the wrong competition. I completely agree - Unity should never, ever look to Adobe or Autodesk when trying to figure out how to be profitable. Yes, their premium services, subscription models etc looks enticing from a busines point of view. But while both companies do really well in terms of stock value and shareholder returns, their customers suffer heavily from those decisions and their is a huge grief building up in those communities. This opens potential entries for new competitiors to completely take over their markets.
Unity faces the same issue - from my perspective, because it increasingly runs into the issue of being ground up between Unreal from High-End side of things and now Godot for Indie/lower end of the spectrum.

All meaningful competition of yours, the one you should actually look at, is becoming more transparent - the whole creative industry is, if you look towards the huge increase of support for Blender.
Yet here we are, with Unity backpedaling the few steps they took rather recently in becoming more open in terms of codebase and development. If it's about plattform SDK risks (consoles), build NDA-walls, that eligable users can easily overcome...if that helps?

If you ask, if more current roadmaps and better changelists would help, you completely miss the point to begin with. Users want to be more involved in the whole thought process of the development. Software development became so complex in the recent years, that just SDKs and APIs don't cut it anymore. And if you commit yourself to an engine as a plattform for your whole business, it should be one, where you have a good grasp of their potential development.
The problem with Unity here is, that a lot of trust was wasted away in the recent years with extremely premature pushes towards new tech while abandoning the old stuff. You ask, if we would be ok with top notch documentation instead...sure...but show me one user in this forum, who sincerely believes that Unity is capable in offering that. The track record so far does not offer much to base that trust on. The SRPs are still lackluster in some areas in comparison to built-in and users need motivation to invest a lot of time (and thus some sort of money - business-wise) to make that transition. It should be Unity's best interest to offer increased value for making that transition. Open repos were a strong one...and the readme in the graphics repo says, that at some point, there was a strong internal belief in that, too.

I also don't get it really. I also would like to chime in with the others and turn the questions around: what does Unity gain from that move and why is it taken now? What are these policy changes and what are their goals?

 

I am also wondering this - why was this choice made, and at what level made it?