Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Have a look at our Games Focus blog post series which will show what Unity is doing for all game developers – now, next year, and in the future.
    Dismiss Notice

Official IMPORTANT: Upcoming changes to the Graphics repository

Discussion in 'General Graphics' started by ali_mohebali, Feb 14, 2022.

  1. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,357
    Yeah basically what @sabojako is seeing is correct. There is no other commits for that specfic path yet. To see the past history, you have to look at the commits from the root folder or from commits prior this move. All history is there, this is just how it's presented on github.
     
  2. sabojako

    sabojako

    Joined:
    Jul 7, 2017
    Posts:
    48
    @superpig For convenience, could you ask the team to create a branch from the commit before the folder move? Pretty please :) Like I said, nobody needs to maintain it, just don't delete it.
     
  3. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,357
    There's no need for separate branch, just checkout the commit before the move's commit. For example on master it's this:

    https://github.com/Unity-Technologies/Graphics/tree/056eccb89d
     
    sabojako likes this.
  4. valarus

    valarus

    Joined:
    Apr 7, 2019
    Posts:
    399
    This is more a Quality of life change request.
    Please reduce size of HDRP 14,15 both API and library. This change will speed up library import and editor responsiveness.
    When you are making new scripts and changing them oftenly, does HDRP packages affect script (re)compilation time?
     
    EagleG likes this.
  5. merpheus

    merpheus

    Joined:
    Mar 5, 2013
    Posts:
    190
    So? Did they finally start mirroring? I see no major commit recently.
     
    Lymdun and PutridEx like this.
  6. AljoshaD

    AljoshaD

    Unity Technologies

    Joined:
    May 27, 2019
    Posts:
    123
    We have not mirrored yet due to a combo of a small bug in the mirroring service and Easter vacations.
    Assuming no other issues pop up then we expect to start the mirroring on the 26th.
     
  7. Liderangel

    Liderangel

    Joined:
    Jul 8, 2018
    Posts:
    94
    Hey I'm wondering about that invisible time gap between committing to master and being on a Unity build.

    For instance we are waiting for this feature to be delivered. It was committed to master on this branch.
    That was over 2 months ago. So what happens between that commit and a Unity release? Since releases happen every week we are confused as to why there have been 8 Unity releases and no delivery of said feature.

    And the other thing that I would love to know is where features land. Meaning, Unity 2022.1, 2021.3, etc.
     
  8. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,357
    Afaik this feature shipped on URP 14 already. You can use it if you use latest 2022.2 alpha.
    And here we get to that alpha again. Unity doesn't land features to already released versions. Unity gets to feature freeze stage at some point when the version is at beta and then only major / high prio feats can sneak in.

    Unity doesn't typically backport features either, mainly SDK updates and bugfixes but there are some exceptions every now and then (for example we got FSR added just few weeks ago to 2021.2).

    To answer to your main question. New features land to whatever version Unity is working towards on their internal trunk. The version they are working with is afaik few weeks ahead of the alphas we get - if there is public alpha available. If there's no alpha then chances are we are not getting the feat until there is.

    Current 2021.3/staging etc branches are for maintaining the older versions, meaning we get the bugfixes and whatever gets backported to these and master is where the normal development is at.
     
    AljoshaD likes this.
  9. Liderangel

    Liderangel

    Joined:
    Jul 8, 2018
    Posts:
    94

    You are correct, it's on URP 14 hidden in the changelog. Thanks!

    It would be great to know which features will get backported (like the one you mentioned).
    Because of course it's quite a huge risk to upgrade the project to an alpha/beta Unity build, and we need that feature quite badly (perhaps it seems like a small insignificant one but when you make a black & white game and all the sudden yellow pops out of nowhere then everything falls apart), so it would be much appreciated to know Unity's plan on everything or if we need to do something ourselves.
     
    AydinDeveloper likes this.
  10. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,357
    The safe assumption is that majority of the new features won't get backported. Knowing how little Unity wants to tell us when upcoming features will land, I seriously doubt they would be willing to start making promises of specific feature backports (which like mentioned are rare cases anyway).

    Don't get me wrong, I'd LOVE to get more transparency on the feature roadmap ETAs myself. Unity used to give them but they got tons of backlash from the community when they didn't meet the targets - so it kinda makes sense that they don't tell specific ETAs anymore.
     
  11. YuriyPopov

    YuriyPopov

    Joined:
    Sep 5, 2017
    Posts:
    211
    Im just sad I can no longer track the development of the gpudriven branch. In my opinion this was the most exciting feature to be developed for the srp's and can actually be a game changer. Im even more sad by the fact that it relies on some internal c++ side api's so I cant really pull it in and make it work for our case. I really hope this festure gets the love and attention it deserves.
     
  12. AljoshaD

    AljoshaD

    Unity Technologies

    Joined:
    May 27, 2019
    Posts:
    123
    Hi everyone,
    another update.

    Today we are mirroring the first set of PRs. We first mirror them to intermediate branches in the public repo with an
    internal/
    prefix and then merge them into the release branches.

    However, we have not merged the (public) intermediate branches into the release branches yet. We'll wait a few days to allow you to pull in an internal branch and provide feedback. The intermediate branches that we mirror to are named like the release branch with an internal/ prefix:
    You'll see that the first new commit changes a lot of files.
    This is because our internal repo excludes .png image files from LFS so they are now committed.
    Also, the reference images in the test projects are deleted. We had to do this for internal security purposes but it also helps to keep the repo small(er). You can generate the images using the tests projects.

    With the current setup we expect the average time between a PR landing in our internal repo and the public repo to be 1 week. This is well ahead of when a (patch) release ships that includes the changes.
     
    Last edited: Apr 29, 2022
    valarus likes this.
  13. slime73

    slime73

    Joined:
    May 14, 2017
    Posts:
    101
    It might be ahead of a patch release, but it's visible in the target branch to us a week later than it used to be for typical PRs (only a small percent of graphics PRs rely on separate Unity source changes).

    And now overall visibility of any PR will happen significantly later than before, since a PR can take days, weeks, sometimes months before it's merged to a release branch.

    And as mentioned above, a lot of work used to be visible that is now completely invisible to us, because experimental changes (such as the gpudriven branch) don't always have PRs. And we won't know to ask for visibility.

    Do you have plans to address those issues with the new approach? It's really frustrating to lose things that were important factors in major choices we made.
     
    PutridEx likes this.
  14. AljoshaD

    AljoshaD

    Unity Technologies

    Joined:
    May 27, 2019
    Posts:
    123
    In absolute terms it will get to you earlier, not later, because of the increased velocity we get with our optimized process (optimization is ongoing). I've explained this in more detail in one of my earlier posts. In relative terms, it's on average a week later in the public repo than the private repo. For example, if something would have landed May 1st in the public repo, it will now land April 23 in the public repo and April 16 in the private repo.


    We are looking into an easy way to share work in progress branches in the public repo. We realize that it is a change, and we only did this to significantly speed up feature development and the quality of what is released. We'll definitely look for other approaches to get the benefits you had with seeing the work in progress. The good thing about the release branches is that anything that lands has gone through a much higher level of testing than what is in development branches. The good thing about our alpha and beta patch releases is that you can see what has landed months before an actual tech release and LTS release.
     
    Last edited: Apr 29, 2022
  15. slime73

    slime73

    Joined:
    May 14, 2017
    Posts:
    101
    It appears this hasn't happened yet. For example: https://github.com/Unity-Technologies/Graphics/commit/8c6a242300e292fd7e2e16f0d72b71b7a0b6de40

    Is there an ETA for when that will be the case? How can I figure out the details of the intent of those linked changes? This type of thing is exactly what I was afraid of - it already sucks to lose the details of those fixes, so now the best I can hope for is it will be contained to commits made in the transition period rather than an ongoing issue.
     
  16. AljoshaD

    AljoshaD

    Unity Technologies

    Joined:
    May 27, 2019
    Posts:
    123
    We are indeed still rolling out guidelines and fixing a number of issues due to this large change of process. You can see from the other PRs/Commits that we aim to land logical pieces per PR and not batch unrelated changes. The batching you see in that example is something we'd like to avoid but devs might batch a few minor (single line) bugfixes in a PR to be efficient. What I would want to see in that batched commit is more details in the message about the changes and not internal links. We're now in a difficult situation with some fallout from the process change and the deadline in a few weeks to land features for the 2022.2 release. Things will stabilize from end of May.
     
  17. slime73

    slime73

    Joined:
    May 14, 2017
    Posts:
    101
    I've seen more batched commits since my post a couple weeks ago - one had a collection of one-line descriptions (still not great but better than the earlier one I posted), and one didn't. I'll hold off on poking y'all about it until the end of May (if it's still happening by then), but since the repository isn't getting retroactive improvements to its commits, any commit with a usability problem adds a permanent usability problem to the repository. So the longer things are switched to the new process without being 'stabilized', the more issues the repo will have forever.
     
    Last edited: May 13, 2022
  18. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,357
    Bigger fear right now is - knowing past actions from Unity - is that they may just silently drop the whole thing. I don't want to jinx it but I have my concerns.
     
    SonicBloomEric, m0nsky and PutridEx like this.
  19. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,357
    is there some reason why some packages got left behind in the root folder? this just makes the repo look messy now
     
    m0nsky likes this.
  20. AljoshaD

    AljoshaD

    Unity Technologies

    Joined:
    May 27, 2019
    Posts:
    123
    The postprocessing package and the templates have not moved to the internal repo currently. They remain where they are until we find the best place to put them.
     
  21. slime73

    slime73

    Joined:
    May 14, 2017
    Posts:
    101
    3-4 months later, this doesn't appear to have materialized yet. The repo is several weeks out of date and doesn't seem to have some of the changes that are listed in the release notes of newly released versions of Unity.

    Can you give us an updated ETA of when the Graphics repository mirroring step will have that 'days at most' timeframe? Even if you're still doing manual verification the several week delay is pretty extreme.
     
    Lymdun and PutridEx like this.
  22. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    963
    I was thinking the same, it's been a long while since the last mirror. Last one was over 15 days ago I believe.
     
  23. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,357
    last repo sync happened 17 days ago, commits here: https://github.com/Unity-Technologies/Graphics/pull/7453/commits

    I think majority of 2022.2.0a16's commits are here @slime73 but I do agree it's not great sign to see just few merge attempts from internal repo since the announcement.

    Would love to hear more comments from Unity staff what's the hold up and what's the CURRENT plan. Please don't tell you plan to sync the repo once a month now because we can already see it going to 2 months then, and then once half year etc.

    All I take away from this is that this change just somehow makes it less convenient to merge internal branches when people at Unity is busy doing other things. By the sounds of it you are now doing this all manually but we all know that's not going to work in the long run. You should totally automate this as much as possible.
     
  24. AljoshaD

    AljoshaD

    Unity Technologies

    Joined:
    May 27, 2019
    Posts:
    123
    Hi,
    the repo is synced roughly every two weeks. This results in an average delay between the private and public repo of 7 days.

    We don't land PRs every day in the private repo. We just synced the repo today and you can see that the average delay is only 5 days for the new commits between landing in the private repo and landing in the public repo.

    We are now investigating a solution that would allow us to mirror PRs when they are done but before they land in the private repo release branch. So this will get the delay from -7 to -14 days, meaning, PRs will be mirrored before they even land in the private repo.
     
    Last edited: Jun 10, 2022
    Jes28 likes this.
  25. slime73

    slime73

    Joined:
    May 14, 2017
    Posts:
    101
    Thanks for updating the repo.

    I think using averages for this sort of thing is kind of misleading, because a 7 day average doesn't communicate that half the commits will be visible 7-14 days late compared to if they were pushed immediately. i.e., max is a lot more useful here than average IMO.

    Progress on mirroring speed is good to hear. The numbers and branch situation is a little confusing to me. Here's my understanding of how it was and how it is, let me know if I got anything wrong.

    Previously:
    • Non-public: Some experimental commits were made in the internal repos not visible to users.
    • Public: Many experimental commits were pushed to publicly visible branches before any PR is made (between 0 days and several months before a PR is made).
    • Public: A PR is made in the public repo with individual commit history within the PR and may be publicly visible for somewhere between a few days up to a few months before being merged.
    • Release: After a PR is merged, it may take 1-3(?) weeks for a new Unity release that contains a version of HDRP/URP/etc with the change.
    Currently:
    • Non-public: Experimental commits are very rarely made public, if ever (a change may be majority-complete for several months before it's publicly visible).
    • Non-public: A PR is made in the internal repo and may take between a few days up to a few months before being merged to the internal staging branch. A supposed goal of the current strategy is that this specific step may be a bit shorter than before.
    • Non-public: The staging branch will only be merged into a release branch after some time has passed (a week? more?)
    • Release: A new Unity release including the release branch changes may happen before some of the items below. because Unity releases and public repo syncs happen independently from each other.
    • Non-public: Up to two weeks will pass after that merge.
    • Public: The release branches will be pushed to the public repository after those delays.
    • Release: Very recent changes that got pushed to the public repo will make it into a Unity release after that push.
    Your potential future change:
    • Non-public: Experimental commits are very rarely made public, if ever (a change may be majority-complete for several months before it's publicly visible).
    • Non-public: A PR is made in the internal repo and may take between a few days up to a few months before being merged to the internal staging branch.
    • Public: Some internal staging branch commits may be pushed to the public repository (at what cadence?)
    • Non-public: The staging branch will only be merged into a release branch after some time has passed (a week? more?)
    • Public: depending on push cadence, more staging branch commits may be mirrored to the public repository.
    • Release: A new Unity release including the release branch changes will happen.
    • Non-public: Up to two weeks will pass after that merge.
    • Public: The release branches will be pushed to the public repository after those delays.
    Again the wording has been a little confusing to me so I'm sure I got something wrong in that summary.
     
  26. slime73

    slime73

    Joined:
    May 14, 2017
    Posts:
    101
    It's been 17 days since the last sync - will it be synced again soon?
     
    m0nsky likes this.
  27. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,357
  28. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,357
    @AljoshaD Is there any chance we could get access to ALL commits that get merged to master? For example on the last PR one of the commits was this https://github.com/Unity-Technologies/Graphics/commit/aee83ab636866275bf8edb3b16eb1489ea3b30fc which is just useless if you need to cherry-pick some individual fix from here. There's no way to separate the fixes anymore since the original commit history is gone.

    I guess this would mean Unity also mirroring extra staging branches like in this case I assume we are missing the commits because these were staged at HDRP staging which isn't mirrored now.
     
  29. AljoshaD

    AljoshaD

    Unity Technologies

    Joined:
    May 27, 2019
    Posts:
    123
    We have built a plan to mirror out an additional staging branch. With the current setup internally for landing PRs, it's very inefficient to land very small changes individually directly to a release branch. With mirroring out this staging branch we aim to get two benefits for you:
    - you'll have visibility on PRs (still as single squashed commits) a few weeks earlier
    - you'll see small PRs, ie, more fine grained changes, that you can cherry pick. For example, this small hdrp fixes will land individually in the staging branch and will be mirrored out individually.

    For the master branch, our devs will land PRs in a staging branch that will be mirrored. Also the release branch will continue to be mirrored so you can see when something actually lands in the release branch. Obviously, the PRs from the staging to the master branch will be large squashed PRs but you'll see the individual PRs in the staging branch (as single squashed commits just like now). For the older release branches we will not use a staging branch but aim to backport the PRs directly to the older release branches.

    We're still aligning the resources. I'll give an update ones the work is planned to set this up. The actual setup might change a bit.
     
    m0nsky and rz_0lento like this.
  30. AljoshaD

    AljoshaD

    Unity Technologies

    Joined:
    May 27, 2019
    Posts:
    123
    Hi everyone,
    we decided to move forward with the staging branch I mentioned in my previous post. We'll need to do some experimentation. In the next weeks, likely 6-8, you'll see bigger squashed PRs land in the master branch.

    While we experiment, we won't be able to mirror out the staging branch. However, we expect at the end of this to be able to mirror out all the individual PRs separately in the staging branch retroactively. So the squashed PRs in master will be mirrored out later as individual PRs. Thank you for your patience on this.

    This process will allow us to have smaller, more detailed PRs for you to cherry pick and it will allow us to mirror these smaller PRs to you weeks earlier, before they even land in the release branch.
     
    merpheus, Le_Tai, slime73 and 6 others like this.
  31. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    963
    What's happening with the mirroring process? It's been over a month since the master branch received anything from the internal repository.

    Last post here was months ago. Any updates?
     
    wwWwwwW1 likes this.
  32. m0nsky

    m0nsky

    Joined:
    Dec 9, 2015
    Posts:
    199
    Some communication from earlier in this thread:





    - Review stage for the "first months" has now been going on for 8 months
    - 3 weeks for every review stage has slowly turned into 10 weeks per review stage
    - "few weeks of testing" where Unity patch releases are ahead of the public repo has turned into 7 months
    - PR's are not mirrored as single commits at all, but squashed together in a huge single commit
    - Original PR description and messages are lost

    The "few days at most" delay has turned into an unknown amount of time with no communication at all.
     
    Last edited: Oct 27, 2022
    JoNax97, Lymdun, slime73 and 4 others like this.
  33. SugoiDev

    SugoiDev

    Joined:
    Mar 27, 2013
    Posts:
    395
     
  34. Prodigga

    Prodigga

    Joined:
    Apr 13, 2011
    Posts:
    1,085
    Though to be fair who could've seen this coming..!!
     
  35. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    963
    That's the issue, a lot of people in this thread were afraid of exactly this. Promises, then it all goes quiet. Everyone forgets about it, and no progress is made. Seems to be the case so far, it's been months since the first post, and months after the last post with seemingly no progress, in a way it's worse than before as the master branch received nothing for over a month now, not even manual commits from the internal repository.
     
    slime73 and m0nsky like this.
  36. slime73

    slime73

    Joined:
    May 14, 2017
    Posts:
    101
    Yeah. On the one hand I'm disappointed and not surprised, because it's a familiar thing across Unity.

    On the other hand, this impacts my job quite heavily (I was at a point in my project the past few months where I didn't have to care about it very much, but that's changing). The way this has shaken out is honestly unacceptable for a professional tool and service like Unity.

    EDIT: This was a pinned thread until recently, and now it's not despite major parts of the changes supposedly still being WIP. What is going on? Combined with everything else I'd almost think Unity is trying to bury it. I hope that's not true.
     
    Last edited: Oct 29, 2022
    m0nsky and timmehhhhhhh like this.
  37. AljoshaD

    AljoshaD

    Unity Technologies

    Joined:
    May 27, 2019
    Posts:
    123
    Hi everyone,
    another update on our progress.
    Our internal developer services team has accelerated (and largely completed) two large multi-year internal migration projects. Our internal repo is now also switched from mercurial to git for all release branches and the CI and test infrastructure is being unified for packages and the Unity editor. This will tremendously simplify future process improvements. However, the changes knocked out the mirror service and delayed new improvements a bit since the team was fully occupied. We are now looking into getting the mirror service back up and running and get back to the regular cadence of mirroring roughly bi-weekly.

    The plan is still to mirror out a staging branch. We need to set up more elaborate CI and automated test coverage to ensure a certain level of quality with the staging branch, both for our developers and for you if you want to cherry pick PRs. The automated CI depends on the unification of the CI between packages and the Unity editor. With the large migrations going on it's hard to predict when this will be completed exactly.
     
  38. AljoshaD

    AljoshaD

    Unity Technologies

    Joined:
    May 27, 2019
    Posts:
    123
    quick update, a very large file landed in our repo that is blocking the mirroring of the master branch to the public repo. The other branches have recently mirrored. I hope we can mirror master in roughly 10 days.