Unity Changes to the Graphics GitHub repository

Yes for us in particular, this would be great and better than going to the github.

But this is hard to do, documentation is one of the hardest things to keep up to date in any company dealing with any size software projects. I know this from experience I had over 50 engineers working on software in an industry that requires deep audits and certification for software and it was a pain. I retired from that industry and decided to make to games instead.

So the point, do not promise documentation, especially right now since the Unity Engine has gone through a lot of changes and still is (the github did give us a sense of direction and light at the end of tunnel) and that's not even taking into account the year 2020 covid work from home kids out of school...

A potential solution for us would be to keep it simple. For instance a locked forum post in each srp forum that only unity can post in that would just let us know what they are thinking, working on problems etc.

Unity does have the roadmap, but you have things like this Point Lights
This card has not been touched since 2018 but yet the feature is available right now 2021.1 and there are several other cards like that too.

There is no easy solution but I do recommend keep it simple

To end I have been really impressed with what Unity has accomplished with the SRPs and the direction they are going!

 

It's 2021, and we are already seeing another Unity disappointment. I can't believe in 2021, Unity developers are questioning why developers want to have access to the game engine source code. This isn't a digital design suite or a website SDK. Unity is a complex game engine, where developers have to create very complex interactive experiences which run at a consistent framerate on multiple platforms. You're talking about expanding Unity's offerings to large enterprise organizations, where it's even more critical they have complete source code access, and currently have to pay $$$$$$$$$ to access the Unity Source Code. While all the competing game engines, like Unreal, CryEngine/Lumberyard, and even free engines like Godot, all offer the grand majority of the engine code entirely for free.

Unreal might omit one or two platforms, while Unity omits the whole freakin game engine. There is absolutely no comparison between Unity and Unreal when it comes to source code access for developers. Not to mention, indie developers don't have to pay royalties when releasing a game until their organization makes a million dollars. Even Apple's new small business for developers plan isn't that generous. And Unity still charges premium prices for Pro Licenses to developers who make $200,000 or more.

Every new feature which gets released for Unity in the last five years has been labeled "Premium". Premium means you have to pay more than the Pro license fee to access said feature. And you can't charge for a Premium feature if it's source code is openly available on Github.

This whole Github Enterprise debacle is a complete PR disaster. Just stop! We know how corporations run. They are designed to make a profit. With Unity going public, and big money is involved, they will do whatever they can to nickle and dime they're royal customers as much as they can.

We, as Unity's primary customers, have to tell Unity this is unacceptable. Don't support them by buying Pro licenses or unnecessary Asset Store purchases. Demand they do more for the developers. Demand the engine's game and editor performance is decent on moderate hardware, and doesn't chug when doing simple tasks. Demand they release the source code for all developers. Demand promised features (DOTS, Render Pipelines, Visual Scripting, Physics) are finished in a reasonable time period.

 

(I mean we’re going to get sidetracked, but I’m pretty sure he means stuff like Pixyz, which is a paid add on, even though some sort of automatic LOD is kinda a basic thing that should come with the engine)

To get back on topic, maybe I missed it, but has there been an answer on what the rationale is behind the source code policy change?

You said

but those words contain very little actual information.

 

I'm not really sure but it seems like the goal of product manager team does not align with graphics team. Does product manager team aware of this thread (https://forum.unity.com/threads/wha...nity-with-scriptable-render-pipelines.924218/) and read through it? The thread mention that graphics team will continue to develop in the open Graphics repo. Not even 1 year yet this Graphics repo suddenly become read-only and migrate from public to private repo decision has been executed. It really makes people worry about what promise will break next.

 

Fair enough. But could you please answer this SINGLE question we have asked a minimum of 14 times?

 

What are you talking about? 18 months? Surely you've misread and you're thinking of something else? That thread is from last summer and it's the first time you've actually tried to reply to our concerns regarding the SRP's.

 

That thread is 6 months old and if it's outdated, please provide us with another, newer thread / blog / post with information on what has changed.

 

Just to add to this:

If Unity's commitment to foundational tech development (the likes of SRPs) can change that much (going from an open, transparent approach to "let's do it all internally and just ship packages) in just about a year, you can probably understand the tension and confusion in your userbase. We are not talking about changing a logo or an icon...it's foundational tech, that Unity wants it's users to switch to and build upon. The initial releases were rather lackluster and not exactly stable - which can be acceptable for preview-tech. But it should at least have foundations, that can be taken for granted. Being able to build "together" with the Unity graphics teams instead of just "using what we are given" is quite a difference in approach. If that is the new deal, it was - again - very badly communicated. Which in turn leads to a certain distrust, when we are promised "better documentation and roadmaps and communication"

P.S.:
I'm not intending to offend you personally. I really, really appreciate your efforts in coming here and openly discussing with us. Thanks a lot for that, because you are only the unfortunate target of a lot of build up tension, that build up exactly because of this severe lack of direct communication over the last months. I know, a lot is not to be blamed on you, maybe not even on the graphics team itself. It seems like quite a few top-management decisions recently have not been the exactly in alignment of what users hoped/wished for - and this is the result now.

 

I think your responses are starting to grate, there are key issues that everyone here has been dealing with for months or years, and what you are suggesting is making it harder for us while providing no action to improve our quality of life.

We need vision into the repo because it is horribly documented. Yes, if you take it away, we will all survive, we will develop other tools and processes to diff the repo and reverse engineer what's going on. At the end of the day, most of us here are salaried to solve these issues for our respective teams. The biggest issue is that we have to deal with this procedural fluff in the first place! The life of a Graphics Programmer in Unity currently is not one of only developing breakthrough visuals. Significant time must be allocated to fighting the tools, debugging surprise issues, constantly doubting whether the approach is correct. I frequently doubt my current interpretation of the API, especially when upgrading.

We are in the dark and this has not, and seemingly at this rate will not change.

Where is the documentation? Why is it so poor currently?
Why is so much of the rendering still occurring in C++ (hello per object unity_LightData et al.)
Why is the URP forum a wasteland with very very little interaction or input from developers? Many questions go unanswered there.

 

I also want to add that the communications nightmare that is working with Unity takes a toll, I do not think the next project I work on will be in Unity, but Unity will not see this falloff immediately. We are all working on long-term timelines, over the next 2-3 years unless serious change happens those of us that can shift anywhere else will. At this point, it is a better commercial decision to take a punt at working with something less polished but more open and configurable than continuing on the closed source Unity death march

 

Have to agree with CianNoonan on this. Graphics programming is not too ideal right now in unity.
SRP by itself is a wonderful thing and I absolutely love how it abstracts rendering into high-level commands. I also really, really like SRP Batcher, which makes the performance of scheduling draw-calls to the driver second to none. However, there are certain, very crucial things on the internal C# and C++ sides of the engine that I wish I could modify. I would've liked to change how the culling system schedules its jobs, I would've liked to remove overkill safety on CommandBuffers and ScriptableRenderContext, I would've liked to add suport for secondary light probes that would also integrate with the lightmapper, the editor and Renderers (since using MPBs breaks SRP Batcher), but I can't.

In my opinion, you can't move everything neatly to C#, no matter what abstraction you come up with. C# + Unity are too ill-equipped to do that optimally. So I'm 100% looking forward for the engine's source being available at more lenient terms. IMO it would boost the engine's popularity by a ton, many programmers still consider Unity to be a pure delight to code games in. If the paid source code access is UT's largest pfofit source, then maybe it's time to rethink the pricing model?

For me, this whole decision of trying to move the graphics repo to an internal server away from the eyes of people that use them is very strange and inconvenient. I don't understand UT's mindset on this and I wish to know what benefits you thought it would bring at the expense of... well, the entire thread speaks for itself.

 

Do companies really still do that? Certainly not the ones that I enjoy working with.

But more seriously: It depends on the use case.

Like, personally, I'm not even interested in having to deal with Unity's source code (the engine, a huge monster of C++ ... something ;-) ). That said, it would be very useful to report proper bugs for all the crashes that I'm getting with Unity 2019.4.17. Those bugs currently get stuck in QA because I usually don't have time to create a small repro-project - but I might have time for a debugging session to pinpoint where exactly it happens.

And here's where source code repository access gets interesting: Like, one of the bugs that I did this with was a Unity UI bug (that was when Unity UI was still "proper open source", back on Bitbucket). Turned out the bug was a regression due to a change that I could find in the version control history. Access to that history also helped me understand that it was a fix for an issue that wasn't relevant for me, so my quick work around would not cause side-effects in my own project - but it wasn't a change that could easily pushed back via pull request (because then, a better solution for the original fix would have to be found).

So, basically, if you offer something that's boring, is rock solid, works perfectly for everyone and will never change: Give me a DLL and don't waste my time with your source code.

But if you do something that's actually interesting, might be too complex to be bug-free, and if actively being worked on, with frequent changes: Give me access to version control!

 

They have to due to certain platforms imposing NDAs. But I can tell you that people under the same NDAs do get access, and there are certain areas where this was a huge competitive advantage that those other engines had over Unity. So much, in fact, that some people chose those other engines over Unity for projects where those areas were relevant.

 

Since this hasn't been elaborated on yet, I'll try to interpret 'sensitive information' in the context of the Graphics repository and some of the ways it's already dealt with there, as well as some potential steps Unity as a company can take to avoid actual issues with sensitive information around Graphics without making Unity as a product worse.

As far as I understand, the Unity Graphics repository is pretty similar to UE4's source in terms of source access. The Unity Graphics repository already has NDA content for consoles separated out cleanly so only people with legal access can grab that content and view its code. You can see that documented here: https://docs.unity3d.com/Packages/c...versal@10.2/manual/Building-For-Consoles.html (I thought it was documented elsewhere as well but I can't find anywhere else, maybe I'm just searching the wrong key words.)

So sensitive code in main branches is already dealt with in the Graphics repository. As for code in other branches - anything that's meant to eventually be merged into a main release shouldn't have any sensitive information in its code to begin with. If there are any branches that have internal R&D work not related to actual Graphics features (for a future demo or whatever) and they absolutely shouldn't be public, those branches can be pushed to a Unity internal enterprise github repository instead of pushing to the public one, without having to move main Graphics work to the internal enterprise repo.

As for sensitive information in pull requests - I believe the vast majority of PRs should have no information that shouldn't be public (because their changes will be public one way or another). If details from Enterprise customers or NDA'd first parties somehow need to be in the pull request description rather than a Slack conversation or whatever, the PR could be done in that Unity internal enterprise repository (again without moving every PR there).

I haven't been able to think of other situations with sensitive information but if you have concrete examples not covered above maybe we could help figure out a path forward?



I have to say, it's pretty disheartening to have to advocate to Unity staff on behalf of Unity in order to prevent Unity from losing key features (and thus negatively affecting everyone from Unity staff to its users such as myself to the company I work for and its products). I don't think it should have ever got to the point where this was publicly discussed - Unity staff working in the Graphics department could have pointed out everything said by users in this thread and more on their own, and that should have been enough.

 

Hi smcclelland,

Thanks for taking the time to respond to folks on this thread. I have a couple of thoughts and suggestions based on our experience using the Unity packages. For reference, we are attempting to build games at a AAA/AA quality in Unity, and make heavy use of the current Git repositories for the SRP, Input System, and Timeline.

More detail is not just a little better, it can save days or weeks of work.
Both PRs and Git History have saved us an incredible amount of time over the past year. Let me give an example:

Last year we moved to a new version of the Input System package. Upon upgrading, we noticed there were a few weirdnesses with XR Devices in the new version. I opened the Github repository for the Input System and found an open PR for the next preview version, fixing the issue we had seen.

Being able to see that this fix PR was in-flight (and read the feedback of the team lead) gave us the confidence to move forward with the package, instead of reverting and spending engineering time finding a manual workaround. It meant we didn't have to spend time filing a bug report (which requires submitting a minimal-case project). Even better, we were able to regularly check back in on that PR, to see how it was progressing.

I don't believe this work-flow should be handled by the Unity Bug Reporter. The bug reporter is great for crashes and clear problems, but it takes time to submit a minimal-case for a report, takes effort from Unity QA team to repro it (and in many cases they can't), and then the best you get is that the issue is being looked at. It's a process that works great for Unity Engine bugs, but I don't think it's a fit for packages.

It's much more efficient for everyone (us, and the Unity teams) to be able to simply follow along with development, and merge in the fix early. It gives us the confidence to use these packages and means we have a much higher tolerance when issues do pop up.

Unity Documentation and Open-Source Packages fulfill different roles
I've seen you wonder if some of the needs we have might simply be satisfied by better documentation or changelogs? The logic goes something like "If there's a piece of information that's only mentioned in the PRs, maybe that should be documented?"

The issue with this logic is that much of the information we find useful is fundamentally unstable. I doubt the teams will want to document these changes or features because, often-times, they're not sure of them yet themselves. It would be a mess to document some new features onto a roadmap, only to yank them away down the road.

So there's a tension here: Unity wants to present the idea that there is a clear and reliable roadmap that developers can trust, but at the same time many developers deeply depend on being able to see the bleeding-edge changes as they evolve (even if they never release).

What I like about the open-source package approach is that we can have both. Most developers just use the default verified packages, but for those that need it, access to the development of the SRP allows us to dip our hands in the molten metal, before it is casted into the final Unity Engine. It's not suitable for all developers, but in order to ship AAA quality games with Unity (which we are actively attempting), developers desperately need this ability.


Suggestions
It's hard for us to suggest best routes forward, because I don't really understand what type of information you're worried about protecting. Could you elaborate? I can understand worries about security vulnerabilities accidentally being disclosed, or widening the potential attackable surface for the company. Or is this mostly about competitive secrets (competition?).

I think we could help direct what would be best for us and the security team at Unity if we knew a bit more about what the worries were.

That said, I think there are great examples of open source development from large companies that achieve both, so I firmly believe it is possible to work in the open, without compromising security.

 

@smcclelland Perhaps you could put some of my anxieties over this at ease.

We maintain a custom SRP (that started as a branch of then-LWRP but has since diverged quite drastically). We still largely depend on the CoreRP package, will we still be able to do this once the repository is closed off again?

Secondly, we have a custom branch of ShaderGraph that has several modifications for extensibility. We support things like adding Custom Passes that pull from their own template files, additional slots to Master Nodes, things that don't seem to be on the road map (or anywhere near a priority, but necessary for our productions). Being able to merge our changes when we do decide to pull a new release in cleanly through Git has become an invaluable part of our workflow. I really fear losing it because I don't really have a clean workaround.

I get that there isn't really a question in the second bit, but maybe you have suggestions or perhaps I'm misunderstanding the situation and my fears are misplaced?

Any additional information would be greatly appreciated.

 

depreviewUnity

 

Dont forget to tell us when this time actually will come, so we can be sure to make such posts or not.

Lets talk about facts: basics in other engines things on poor level(such as visual scripting or editor extending(or editor at all?)) or require to bought from asset-store. I was on asset-store yesterday and what ive seen in recommendations was total nonsence, asset which proposing bad developing practices(programming in inspector, lol) and require to buy things from one to another(which presented like, wow, extensions), so you are pushing begginers to spend money on what they dont need at all. If you does not agree with me about such assets, then are you even a game developer or what?

Sorry for offtopic, but thats all result of Unity politics which sounds: "Money over peoples" and thats what we are trying to tell you from different angles.

Who are responsible for doing such weird things, like PR poor and useless assets, closing repositories and other? Who benefit from sells in asset-store? After this and other things you trying to tell us that Unity is free and does not require anything(like Pro)?

 

We had a private fork of HDRP that had a lot of modifications including a custom PBR sky (the current HDRP implementation was just too slow). This required changes to the sky systems, including lighting and fog, and we wanted to be able to respond to upcoming changes. Being able to see feature branches, PRs, and full commit history was extremely valuable. We would audit the active branches daily for features and PRs coming down the pipe. It allowed us to know what the HDRP team was working on, and let us know what to not waste time on. Having to work off of only release branches would have been a lot harder.

If the SRP packages were much more mature, this wouldn't be as big of an issue, but in their current state we want to be able to see what Unity is actively working on, in detail, and the reasoning behind it. I imagine asset store publishers who deal with SRPs feel the need for this even more so.

In just the past few months HDRP has been moved towards a render graph, big changes to the light loop, and many more. These are all significant changes if you have a custom fork.

Feature branches, commit histories and PRs give us insight to what the graphics teams are thinking. A roadmap or changelog tells us very little.

Yes there are closed source SDKs, but that doesn't mean we like them. We can generally deal with a lot of Unity being closed source because we aren't trying to modify core features. Graphics are a different beast and the SRPs and related packages are immature. We don't want to be surprised with a merge nightmare every release.

It's very concerning that Unity didn't see the value in this.

 

I can see, that you are angry about quite a few things, but I don't really see your point in this discussion.
If I may suggest...if the whole business model of unity upsets you that much (I'm not judging it myself here and now), have a look if maybe one of the competitors offers more of the things you need and want?

Godot is completely free and open source, unreal has solid offers and advantages too.

I don't really see the point in having this kind of conversation here and now, because it offers such a nice excuse to close or abandon the thread before we actually get an answer and/or a fruitful discussion about the current issues.

So if I may ask for it, please put that issues aside and see, if there are other solutions for them. I'm still hoping to learn more about the policy changes and all that and I hope, that this thread will not lie dormant for weeks as others did...

 

How many times do we have to ask for an explanation on what these policy changes are, and why they are being made? Also, is the promised blog post still happening, or was that canceled?

 

Thanks for advice, im looking for Godot too and my first post in this thread was about Godot. Am i angry? Yes a little bit, because politics like this makes my life harder everyday in different spheres of life, and of course not me only.

The reason why im here is because im not ready to fully migrate to Godot or even UE because of my work, from other side i getting bugs from Unity which im not able to fix and someones thinks its my fault. Try to understand situation like mine. And yes its fully reffered to this thread about source-transparency at the end.

Thanks. Regards.

 

Sounds like they're going to take some time to figure things out -- they've restored things back to normal for now.

 

If that's the case, then they should say so, instead of ghosting us.

 

... After a while, you'll get used to that pattern. It's kind of their signature move by now

 

I should be numb to it by now but my blood boils every time.

 

Just let u know the thread posting date is at Jul 2, 2020. Anyway kindly provide the new updates like forum post or blog post. Don't go into complete silent again just like that thread.

 

To echo @Kleptine, my guess is folks at Unity Technologies had a blog post all ready to go doubling down on closing off some or all visibility into the repo which, in response to the unanimously negative feedback.... they put on hold and now are reconsidering their direction and messaging on. Then once they decide what they are gonna do, they will rewrite the blog post and/or make a new forum post.

Shawn McClelland @smcclelland (Product Lead, Developer Services @ Unity) was recently very active in this thread in collecting more detailed information on this of which we covered a wide gamut and made our case on developer needs very clear with no stone left unturned. So, presumably, Shawn is now focused on representing us in internal discussions and presenting our feedback to the executive powers that be on these matters. It would be nice to have those next steps clearly outlined out for us, but that's generally how I understand these things tend to work in large organizations.

On top of that, the repo was reverted back to the state that it was. i.e. providing maximum benefit to the Unity developer community and (in the long run) Unity Technologies the most. Actions speak louder than words --reverting the repo to the public state sent us a message that they've heard us loud and clear.

I suggest we "pump the brakes" a bit, let folks at Unity reflect on what just happened, with an expectation that they are clear that the Unity developer community demands Unity Technologies maintain or expand their commitment "to an open and transparent development process for SRP and the pre-built Render Pipelines" (and with clearer messaging thereof) going forward.

 

What will happen is someone from Unity will make a post saying “we hear you” and then 6 months later someone else will tell us “what? That? That was 18 months ago and things have changed”.

It has happened way too many times at this point.

 

This is what worries me most about Unity in recent years. None of their actions feel well thought out or planned in advance. Here they're objectively trying to screw us over and we tell them that would suck for us and this was enough for them to re-evaluate? Like how could that come as a surprise to them? We're not even talking about them listening to our ideal wishes here, we just want to maintain the bare minimum of insight needed to work with the mess they've created for us.

 

That's the thing. That is all speculation, and the community is currently very untrustworthy of Unity nowadays. We have NO idea what is going on, and can only guess - guesses that range from "URP is getting shut down" to "Unity has actually listened to us". If URP's goals can completely flip on its head in 6 months, and then presumably flip again once we get mad, what else will happen? I agree that actions speak louder than words, but without words, we cannot even try to hold them accountable. Communication. Did they originally have a blog post and then have decided to revert and think about it? Tell us! Tell us that they are listening to us! That's free brownie points. I think it's quite possible they aren't changing anything, but instead planning on implementing their goals once the fire has calmed. That is also a speculation, something we are prone to do without any information whatsoever. If this isn't the case, then they need to tell us. If they don't know yet, then they need to tell us that as well.