Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Dismiss Notice

Official Trim your repository size with Purge

Discussion in 'Unity Version Control' started by vitor-unity, Jun 22, 2023.

  1. vitor-unity

    vitor-unity

    Unity Technologies

    Joined:
    Feb 13, 2023
    Posts:
    14
    Real-time 3D projects can grow substantially in storage size. The iterative nature of development, trying as many ideas as possible, can generate multiple revisions of large files that, by design, are kept safely stored in Unity Version Control. Although keeping this behavior as the standard we also want to provide our customers with flexibility so they can choose how to better manage their data.

    We are excited to introduce Purge. You'll have the power to permanently delete those old, unused revisions and trim your repository data leaving room for the ideas that lie ahead. You'll be able to select a specific period of time and file extensions you want to remove, and in a few steps, you get your repository shrunk.

    Important note: this operation is destructive and irreversible. While we want to give users the flexibility to have the ability to control their repositories, we also don’t recommend purging your repositories. Unity Version Control is designed to keep the history of your project fully registered as long as it lives, and the use of purge means that you are losing data.​

    A feature with safety features in mind
    While this operation is destructive and irreversible, we designed it to preserve the most vital revisions. This ensures that you will always have stable checkpoints inside your repository.

    The purging process calculates the scope of your requests skipping those revisions loaded at the head of a branch or if they exist in a changeset with a label. These simple safety features guarantee you can always return to any label or branch head.

    Try now on your CLI
    Purge is only available through the command line interface, where you can calculate, execute and consult the history of your operations. Don’t forget to set permissions to it and keep it as an administrator-only functionality.

    First, you will need to request a purge. This is done using the “cm purge register” command, specifying the file extension and the ‘before’ date when to apply the purge.
    Code (csharp):
    1. >cm purge register ".png" "2023-Jun-21 13:40" --repository=prince
    2. A7919afc-45af-4c58-a5b8-c54c32f1c0ea
    The register command will return you a purge GUID, this value can be used to check its status via the “cm purge show” command:
    Code (csharp):
    1. > cm purge show a7919afc-45af-4c58-a5b8-c54c32f1c0ea
    2. PurgeId: a7919afc-45af-4c58-a5b8-c54c32f1c0ea
    3. Repository: prince
    4. Author: manu
    5. Status: ReadyToPurge
    6. Purge size: 77 revisions, 290.06 MB
    7. .png: 77 revisions, 290.06 MB
    Once you are sure you want to run the purge operation, you will need to use the “cm purge execute” command, specifying the purge GUID you want to execute.
    Code (csharp):
    1. >cm purge execute a7919afc-45af-4c58-a5b8-c54c32f1c0ea
    2. PurgeId: a7919afc-45af-4c58-a5b8-c54c32f1c0ea
    3. Repository: prince
    4. Author: manu
    5. Status: ReadyToPurge
    6. Purge size: 77 revisions, 290.06 MB
    7. .png: 77 revisions, 290.06 MB
    You’ll need to confirm the request to safeguard the process.
    Code (csharp):
    1. This operation cannot be undone or canceled. You will permanently lose the revisions of these files.
    2. Tip: if you want to inspect all the affected revisions, do NOT continue and run this command:
    3.  
    4. $ cm purge show a7919afc-45af-4c58-a5b8-c54c32f1c0ea --verbose --server=localhost:8787
    5.  
    6. Are you sure you want to execute the purge? (y/n)
    7.  
    8. y
    9.  
    10. The purge action is now running.
    Lastly, you can use the command “cm purge history” to check the specific status of one purge or previously executed purges.
    Code (csharp):
    1. > cm purge history
    2. a7919afc-45af-4c58-a5b8-c54c32f1c0ea PurgeCompleted
    Check the complete CLI documentation and learn more about Purge
    Since we are talking about CLI, we recently updated and brought the documentation to the web, making it easy to everyone to find, search and learn about every available command and the extensive list of capabilities from Unity Version Control.

    Important note: if any purged revisions are selected in the graphical interfaces you’ll get a message displaying that the specific revision was purged and is not available anymore.​

    How to Access
    You can visit our webpage and download the latest version right now. Please note that this feature is only available through the Unity Version Control CLI and for cloud customers. Only new Cloud organizations will have the new ‘purge’ permission enabled by default for the “Administrators” group, if you want to start using this feature make sure you grant the ‘purge’ permission first.

    If you are not a customer or want to start using Unity DevOps for free you can subscribe here.
     
    DevDunk likes this.
  2. KristofMorva

    KristofMorva

    Joined:
    Mar 22, 2023
    Posts:
    4
    Hey @vitor-unity!

    First off, great feature! I'd have a few feature requests if you take them currently:
    • Most importantly, it does seem to ignore the existence of Unreal Engine a bit, where everything is an
      .uasset
      , and as far as I understand, this only works with file extensions. Would it be possible to add a feature to pattern-match files (i.e. with regex or Plastic .conf matching format), so that we can purge files like
      *_BuiltData.uasset
      ? Would also welcome the option to purge old revisions of all files.
    • More minor, but could "before_date" be optional (defaulting to "Now")? So that we can easily purge all revisions of PDB files for example, excluding the most recent one (since that's the HEAD).
    • Also minor but would be great if we could also use Revision Count instead of Date (i.e. keep only the last N revisions)
    • And probably the most long-term request, could we set up purge rules in the dashboard so that they execute automatically every month (i.e. before the billing period)?
    And more of a confusion:
    • The changelog says "If you want to start using this feature, make sure you grant the ‘purge’ permission first". However, even for a cloud repository, Unity Dashboard -> User Groups -> Administrators -> Permissions has no "Purge" in the list.
     
    Last edited: Jun 25, 2023
  3. matiasges

    matiasges

    Joined:
    Jan 24, 2021
    Posts:
    139
    You have to go to 'Repository Server Permissions' to find 'Purge'

    I also don't know the extension, the example is pretty confusing. It didn't work for me, it just shows me the instructions after typing "cm purge register".
     
    soleron and valentin56610 like this.
  4. vitor-unity

    vitor-unity

    Unity Technologies

    Joined:
    Feb 13, 2023
    Posts:
    14
    Thanks for the feedback guys!

    The purge permission is new, so organizations created before this feature do not have it set for the Administrators group (unlike the new ones). The appropriate thing to do is to add that permission for the Administrators group, which can be done through the ACL subcommand or through the GUI. We'll improve that in future iterations to add support from the web as well.

    Feedback noted, definitely we want to support all file extensions that are relevant for users, and we'll increase support over time. The date was a design decision we need to create guardrails to prevent a repository from being mistakenly purged, same applies to the automation you described. We'll consider all of those for future iterations as we get more customer feedback.

    Cheers!
     
    KristofMorva likes this.
  5. valentin56610

    valentin56610

    Joined:
    Jan 22, 2019
    Posts:
    145
    Hello there,

    Little (salty) feedback. My project is currently using over 125 GB (2 years worth of commits) and it makes no sense to keep 2 years old change sets, I am therefore looking into deleting those, which led me here

    Question number 1: Why is it available only through CLI? Doing that through some decent UI was not possible? Year is 2023, not 1990...

    Question number 2: The CLI arguments are way too specific and .. strings? Like, how do I know which format is accepted for the date? I am french and we write dates differently, so how do I know which is accepted or which is correct? And per file filter, really? I thought I was going to be able to delete entire changesets! Not cherry pick each file type! So how do I do to specify that I want to delete the entire thing? Not just PNGs or JPGs...

    And finally, question number 3: Why does it have to be so damn complicated to simply delete stuff that is over 2 years old?

    August 2023, still isn't clear how you can 'simply' delete stuff. Please keep things simple. Seriously. I don't have hours to sink in learning how to simply delete stuff.

    "A feature with safety features in mind" yeah you got it, we can't even delete anything due to it being overly complicated

    Update: Over an hour has passed since I started messing with this command line thing and I still can't get it to register anything. It throws at me the same thing over and over again (instruction on purge usage bla bla bla)
    This is SO frustrating, seriously. If someone ever told me that to make a multiplayer game, the hardest part would be deleting some files, I would have never believed him.

    I guess I'll just delete the entire repository and create a new one. At least this is available for the common people. And will be even faster!
     
    Last edited: Aug 11, 2023
    soleron likes this.
  6. soleron

    soleron

    Joined:
    Apr 21, 2013
    Posts:
    512
    CLI don't cut it. It's not the 90s.

    We need Archive, Purge, Delete, that works consistently, from the Desktop client.

    Also, the Administrator should have all options on by default in organizations old and new.

    What is the reason not to?
    What is the benefit of not making sure this is enabled in all organizations?

    (As someone with product manager experience, I know what, I just want to see what you might respond.)
     
    Last edited: Aug 21, 2023
    valentin56610 likes this.
  7. soleron

    soleron

    Joined:
    Apr 21, 2013
    Posts:
    512
    That would be awesome.
    Although I am pretty sure accounting and product managers will not like the idea
     
    Last edited: Aug 21, 2023
  8. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    10,902
    Is there a (rough) timeline on when we can potentially expect this command to make into the GUI in some form?
     
    valentin56610 likes this.
  9. carlosalba1985

    carlosalba1985

    Unity Technologies

    Joined:
    Jul 19, 2021
    Posts:
    799
    Hi, I'm afraid it's not planned for the following months.
     
  10. valentin56610

    valentin56610

    Joined:
    Jan 22, 2019
    Posts:
    145
    Can't help but notice how easy it is to make changes to get more money from users but so hard for them to stop paying money to use services... :))

    But it must be just a coincidence!
     
    Marc-Saubion likes this.
  11. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    10,902
    I managed to register a purge command, but it included a lot more stuff in it than I was intending, so I am trying to unregister it and I get:

    You are not the purge author. Only the purge author is allowed to execute or unregister it.​

    Any ideas?
     
    Last edited: Sep 30, 2023
  12. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    10,902
    Also, am I correct that I can't apply the purge commands to specific files or specific paths? Do I have to apply it to all files with the same extension?
     
  13. carlosalba1985

    carlosalba1985

    Unity Technologies

    Joined:
    Jul 19, 2021
    Posts:
    799
    I'm afraid I cannot reproduce the issue. I replied to your support ticket with some ideas to debug what could be the problem.

    If you want to purge file paths, please use the "cm archive --help" command instead.