Official IMPORTANT: Upcoming changes to the Graphics repository

I would like to voice my dissatisfaction with this decision.
As many have said before, being able to see the commits is absurdly important, especially with Unity's track record and even moreso with the Graphics package!

We are all developers here, so we can reason a lot about the development, struggles, goals, etc, just from being able to follow the commits and see the actual progression.

Having access to the code itself is of minimal importance, since we can just take a look at the local package, and decompile any DLLs (as many of us have been doing for years to make sense of Unity's internals).
Also, let's not forget that Unity killed the previous GitHub repository that hosted Unity's decompiled source and then proceeded to ignore the replacement they created themselves for years.


I implore you to reconsider.

 

Can you please explain why this is the case? Specifically, how does this move enable you to "develop features and improve packages faster"?

Without that piece of the explanation, this sort of sounds like a bunch of meaningless marketing speak. I.e. why was the public repository holding your team back in the first place? As a software developer, it is difficult to understand why retargeting your git remote from one address to another would impact development velocity.

This statement is simply incorrect, though, isn't it? Aren't your feature branches being squashed? Cherry-picking changes, then, ends up with a completely different granularity than before. Depending on the number of changes in a given feature branch, it may actually be faster to simply find the relevant lines and copy them around. What's more, individual commits can frequently reveal lots of useful information, especially when the code itself lacks any useful comments. This also requires your teams to have excellent commit discipline, of course...

Can you please explain why this delay exists in the first place? One would presume that such a system would be 100% automatable. Having a "goal" of "not longer than three weeks" is far, far less helpful than having a rule. We cannot trust a goal as goals are frequently missed.

Additionally, is that three weeks delay the delay between feature branch merging or actual Editor release? If the latter, then have you considered how this may affect Asset Store Publishers who have to integrate lots of stuff with the SRPs? This would potentially imply that publishers wouldn't have access to changes until "up to three weeks post-new-Editor-release if the goal is met". That opens a window where Unity's and Publishers' mutual customers will get really upset at Publishers due to assets not being fully compatible on day one (or possibly even month one).

 

Sadly, that's far from enough, especially with the implication that this is not automated.

I'm sure we'll have many developers voicing similar opinions in this thread, and I hope you'll take those into consideration to, at the very least, keep a full (every commit), automated (and I can't stress this enough) mirror of the monorepo visible to us developers.

 

What ever happened to "Democratizing game development"

 

Anyone who's spent longer than 3 months in Unity ecosystem who is remotely shocked by this is deliberately burying their head in the sand.

 

I always felt Unity should be higher performing, more reliable, cohesive and be somewhere ahead of where it is right now. If this change helps that, then what more could I want?

 

Git has the option for Submodules it isn't a technical problem to have the module public.

It gives the feeling that are trying to hide bad development.

 

I support faster development! Whatever gets me display HDR in URP faster! XD

 

Provided that the mirroring solution actually works as intended, it seems like we will (mostly) not lose anything aside from increased latency with the ~3 week delay and decreased granularity with the commit squashing. Is my understanding correct?

I maintain our in-house custom SRP and I rely heavily on the GitHub repo's staging branches for inferences to otherwise undocumented APIs in lieu of any actual documentation (inline or other). If this suffers from this change at all it is taking an already very inconvenient workflow and making it painful.

 

Could you explain why making these changes does any of these things? What about the current workflow is preventing say automated testing? What workflow issues does using Github have that internal repos don't?

 

Im smelling a new built-in renderer...

 

So you cant manage your own development environment without making it private, but expect us to nitpick stuff out of squashed commits? Plus, it is obvious that this "somewhat open code base" idea is not well maintained as it took ages for unity editor csharp code to be updated to the latest editor releases.

The year is 2022. Instead of making the whole engine code public, it's getting more and more internal and hidden. The granularity of gfx repo was very useful to track individual WIP features and test them on our environment before it lands. So that it is easier to adopt new features and come up with fixes and PRs possibly.

Honestly, I am not happy with this. Why do we have to be constantly disappointed in using Unity instead of Unreal?

 

Many of the existing PRs/commits targeting stable versions of HDRP are themselves already a big group of sometimes unrelated changes - for example this one https://github.com/Unity-Technologies/Graphics/commit/3179cc5e90c0bc23f00d0489f1f48f664f13614a . With the current setup it's already kind of annoying to trace actual changes through those monolithic commits (doing so has been necessary many times for me as part of making my large Unity project shippable as well as for submitting bug reports that are actionable by Unity), but since those monolithic commits link back to individual smaller PRs at least I'm able to do that tracing myself.

It sounds like the new setup might lose useful information like that (e.g. PR links or commit hash information for the original changes a monolithic backport PR is comprised of) – is that an accurate guess, or do you have a plan to make those at least as usable as they are currently?

Also, will existing/old PR and branch information be preserved once the mirroring begins? Losing the old ScriptableRenderPipeline repository a couple years ago meant a bunch of history links got broken and it got harder to debug issues, I'm hoping something similar doesn't happen again.

Unfortunately it sounds like the development of the Unity projects I work on will be hurt by this change because at least some of the workflow points I brought up the last time around will still be impacted: https://forum.unity.com/threads/cha...github-repository.1029502/page-3#post-6699961
For example I've been keeping a close eye on the HDRP water system pull request content and comments (not just commits) recently and it's helped guide where I spend time, and now I'll be losing some of that.

 

Thanks for the clarification! At the very least I feel much better about this than the previous attempt a year ago.

 

You realize that a "conservative estimate" is "one in which you are cautious and estimate or guess a low amount which is probably less than the real amount", yeah? Did you perhaps mean a "liberal estimate" here? One where you would estimate "plenty of extra time over what you actually anticipated"?

This is, of course, part of the issue from our end - that your "three weeks" will actually be something like 3-8 weeks.

I'm still a little confused/concerned here. Did you perhaps mean to say that "By the time a new Editor version is released, all Graphics Repo-related content will be available in the repository"?

 

I have to wonder what it is it people are making that requires such cherry picking from source, that would cause this to be a big concern so I am very curious about what it is people actually need from the live repo...

Last thing we tried (and failed to do) was merge the HDRP water shader. It was such a failure we wondered exactly what the point of this all was vs making local modifications?

 

Speaking for myself and my project, it's a larger game that runs on multiple platforms and doesn't have the exact same release schedule as major Unity/HDRP versions. Most of what I've cherry picked has been important fixes to specific bugs in HDRP rather than features (waiting for months for the backport -> release process isn't always feasible and has a lot more risk than pulling just the change we need). There are also features introduced in beta HDRP versions which aren't too massive like HDR tonemapping support, that can be important for a project targeting HDRP 12 for example.

Aside from those, figuring out where I should spend my time implementing something myself versus waiting for Unity features versus backporting something from Unity has been invaluable. Being able to track development as it happens is a huge resource for me.

 

In my opinion the git repository need a bit of a cleanup

 

Speaking from experience in non-game-related industry: Git with submodules is a pain. You are adding lots of dependencies that way when you want to merge things and it makes having good continuous integration harder as well because you can end up with cyclic dependencies with the automated tests.
For a newer, equally massive project as before, my employer has decided to go with singular monorepo.

 

I haven't yet had to do this with Graphics, but I did have a similar problem with the Render Streaming package. Render Streaming is still in a very rough alpha state. We developed some code to extend and work with Render Streaming - and then in the next version update of Render Streaming, the Unity team did some refactoring that completely broke all of our code. Because Render Streaming is a public repo with full commit history on Github, I was able to look through the individual commits, determine what had changed that was and wasn't important to us, and cherry-pick specific changes we wanted while forgoing those that would break our custom code.

As frustrating as closed source and lack of transparency can be for us that are using Unity's products, we should keep in mind that open source and transparency do create their own headaches for Unity's team. It's easier to experiment and iterate in a private repo where you don't have to worry about the rest of the world scrutinizing and criticizing your temporary WIP code.

 

This (specifically) is also trivial to do while maintaining the open source repo. You simply have a separate downstream private fork where you do all of your experiments and, when ready, merge your branch upstream. It's pretty straightforward.

I personally don't mind keeping experiments and early proof-of-concept code private. That's better private. What I find disconcerting is commit-squashing and lag time between when a release occurs and when the relevant code makes its way into the public repo.

 

@AljoshaD Two things:

1. That doesn't actually change the amount of time between fix-and-release, correct? I.e. what you're saying here is that the public (us) simply no longer have any insight into the changes that are coming in the next release of Unity until they land, yes?
2. You understand how it is actually preferable to some users to be able to see changes coming before a release of Unity is publicly available, yeah? Asset Publishers, for instance, will now have zero chance of identifying breaking issues for their assets until the same exact day as end users who update and find that things broke.

 

wow will the new LTS come two weeks early

 

Got it, thanks for the clarification.

That does make even tighter for Asset Publishers who track changes in the repo to understand the effects on their assets.

Out of curiosity, do you happen to know what the new minimum delay between landing in the repo and shipping in Unity will be?

This is all somewhat academic until the public mirroring solution gets worked out. If public mirroring has any meaningful delays then it would simply be faster to go look at diffs over on Needle Mirror and start figuring out changes manually. (This also assumes that PRs are kept at Feature-scale and not at ReleaseVersion-Scale because, you know, squashing...)

 

Hi, I just have a question about the github version tag. I have seen recent changes while the latest tag is still 10.8.1. Is that possible for you to still maintain the version tag and make it consistent with the pkg version from package manager. I think it will be so helpful for us to compare, upgrade or migrate features.

 

First, I do appreciate the level of communication in this thread.
That being said, three weeks is too long. I know you guys are aiming for something faster than that, but I just wanted to pitch in and say it really needs to be way more often than 3 weeks. I realize it takes time, just worried with this thread going inactive things might slow down and it ends up losing steam, never to happen.

 

The amount of bugfixes I could have done without full PR history would have been dramatically different. In fact I don't think I could have solved most of them without the access or it would have been super tedious as you can't really find the moment things break without full history and only alternative then is to reverse engineer the root cause which is always way more difficult.

 

I'm also saddened by this decision because it really leaves out all the WIP branches that Unity never merged that still had useful work put into them. I don't expect people who don't follow the development as closely to understand this but there's a real vault of information on these WIP branches that help you better understand how SRPs are used. If you only ever use whatever is given to you as is, of course it doesn't matter for you but people who have to figure the inner workings of these systems themselves, now one valuable resource is way less valuable.

Couple this to the fact that none of the internal structures of SRPs are documented in any way. In fact there's often a clear aim to keep as much code as possible internal to reduce support and documentation amount.

While getting full PR's as single commits is better than nothing, having PR's individual commits around would still tell a whole lot more of the code and what really happened there.

 

I think what you're missing here is these bug fixes instead of taking a month to fix could come out in a week without you having to fix them. In the time it takes for you to look over the repo to try to figure out and fix the problem someone who knows the software can read a bug report and likely have it out by next release, which won't take a minimum of a month. To me that's a much broader benefit.

Sure the previous process made making fixes yourself easier, but also meant that unity releases were much slower.

And at least for me the reason people use an engine like unity is to not deal with issues, and have issues that may occur get fixed by unity as soon as possible.

Just about everyone can agree unity's development has been slow, and if there's was a legit month wait period as described anything removing that is a good thing even with the draw backs.

People complain the SRP keeps changing but at the same time development is so slow no standard can really be formed. And by the time the next major release comes out everything changes. I would hope if the goal is better standardizing the way SRP's work the issues related to your understanding of SRP's from what you're talking about could be less nessecary. And those standards can be formed and built on faster so there's less to relearn.

I'm just saying man look at the long term.

 

If it worked like that, yes that would be fine. But on my own part, I used to be able to fix some of the issues myself in few hours instead of waiting for weeks for Unity to maybe fix the issue. In my own experience the chances of Unity actually fixing some issue is way more likely if I can pinpoint the source of the issue to them and recently even that hasn't always worked. And now I can't even do that.

Also in past when I've reported issues I have often seen dev branches for the fixes. Therefore I have been able to test the fixes early while they've been still being developed and have often given Unity devs feedback at this stage. This is not an option anymore either.

I'm really not buying the "private repo leads to faster delivery". They could do that even with public repo if they wanted to and I don't even think Unity truly tried to communicate they needed private repo for this. It's just additional thing they are doing now. I do welcome faster delivery though.

As a bonus this new approach means that they will not mirror things immediately, so getting hands to the source code history of the URP / HDRP version you are currently using is going to get even less likely if you like to keep your engine up-to-date.

 

One thing I'm seriously concerned here is that we will only get final PR's mirrored as commits and Unity has actually moved to UE style setup where they pick a whole bunch of PR's into single commit when they merge them to main, this has been actually a standard practice for HDRP PR's for a long time already so being able to separate the changes for individual PRs is pretty much impossible already (if the change you want to examine falls under such commit).

 

I think we have a few years of evidence this simply isn't the case, I doubt this is exclusively the problem but likely a big part of it. I doubt the painfully slow development for the last few years was just to excuse this change. Just about everyone can tell faster development is needed to keep up, not just in graphics, new technology and formats, still no native USD support, etc.

The problem seems so prolific Unity has been releasing what might be considered new features as packages entirely outside of Unity itself (Like the LookDev tool), to me it looks like they've been trying to deal with this for a while but it's too messy.

I mean I don't exactly think you should need to do this in the first place. I understand why you do it, and if I considered myself a programmer I would do it too. But to me this is a service issue that shouldn't be considered necessary, just a bonus at best.

I would imagine this faster process also means faster iteration of those fixes so personally I'm not concerned here.

 

How does this affect the process for contributions? I'd like to contribute some code; namely, I'd like to make dependencies on certain built-in modules optional. (For example, my project doesn't use the 3D physics module, but it *does* use post-processing.)

 

The new structure that landed no more has History of previous changes. That is quite cumbersome. Some of my saved PR web links still work tho. Is there some way we can see the old structure to navigate the previous history in the github webpage? I haven't forked the repo.

Is there a tag or branch that corresponds to pre-structure change? There's one branch that looks like it and it's recent, but its named 10.x.x/release, which is not the latest number at all.

 

All previous commits are there on my end, which branch are you looking at specifically?

Nobody is going to maintain two branches for each version from now on. Like mentioned on post above yours, this change only applies to master, 2021.3 + 2022.1 staging branches and whatever staging branches come after this. If you are a branch for older version it does not apply to you. If you are on newer affected version, just read those detailed guidelines on how to work with this.

If you really have to get to last version prior this change, just checkout the branch with commit prior the restructure PR.

 

I'm checking on master, on webpage: https://github.com/Unity-Technologies/Graphics
The history is gone.

Like I said, I don't have a fork.

I just want to be like to be able to navigate the History on the webpage like I have been doing for the past 3 years. To inspect PRs and do stuff like Blame on files.

They surely must have saved a backup in a branch or a tag so that we can inspect the previous PRs. Nobody needs to maintain that branch.