20GB project - how to put in source control?

Hi,

We have a project that is 20GB. I am curious what other people use for source control? Should I put git on my own server so I can have more then 1 GB of space? All the free accounts only have 1GB (github, bitbucket). I would rather put the code on a hosted, with backups server and not my own.

Also, is it best practice to put even the large files into code control? If not, it would be helpful to know where others are putting them as a backup.

Any recommendations are greatly appreciated.

Thanks in advance,
Clint

 

I'm curious what methods people use as well.

 

You have to pay for the repository or self-host it. No other way around it.

Git lfs could handle binary files. Another way to go about it is to separate source and binaries. Put source code under git control and store binaries elsewhere.

Beware of storing stuff (like models) on Google Drive, though. This thing has a nasty habit of nuking files you're working on.

 

or you can use microsoft team service its free and unlimited in space, i have a 12 gb repo which work pretty well

 

I would actually question how much value you get out of version controlling binaries. Sound, model and texture files don't generate sensible diffs. So putting them in version control doesn't really give you anything more then having version 1, 2 and 3 in a folder somewhere.

If you only version control code and text assets then you should come well within repo limits.

 

We've hosted massive projects on GitHub. Just get a paid account, no problem.

 

Why is your project so big? I have a project that is around 6gb and I am using Gitlab for source control. If you want you can try using Gitlab. Although it supports up to 10gb of disk space per project but if you use git ignore to ignore those unnecessary files, I'm sure you'll be able to upload it for source control. You can give it a try if not then you'll have to use your own self hosted server.

 

Because, Assets.

 

Visual Studio Online. Free and unlimited size git. Works perfectly fine for my 40Gb project. You can even attach it to cloud build with a little hack.

 

Because you'll be able to revert the whole project to previous working state instantly. If binaries are not under same version control, you'll have to manually revert them to a working version, which may be trouble, if you, say completely changed your object/component setup meanwhile.

It is not about having sensible diffs, but more about having sensible version history.

 

Are there any sensible Mercurial (hg) offers other than Bitbucket? I really prefer using mercurial because it's just easier.

 

We use Plastic SCM, it's specifically designed for big projects. Can still have everything in one repo with just code assets version controlled.

 

There is source revision control asset store tool I wrote called Chronos Time Machine, It handles big projects and treats your multi-gigabyte art assets the same as it does code assets. It uses unity's own project view for status and control. It truly extends Unity into a first class Source control system. It has true collaborative file lock (will lock even unity out of editing assets and meta files). It is good for both Single and Team user environments. Just point it to a remote server and the repo is managed (and served up) by Unity itself. You can from within Unity roll back/forward a file or folder, either partially or fully. It is not limited to code assets but will do this in a sensible way for whatever type of asset it is. And of course you can diff compare your assets against any commit.

 

You're best off buying an external USB drive and running automated incrementals daily unless you have a fibre optic internet connection and money to spend on cloud storage, even then a local USB drive with incrementals is much more convenient and faster.

 

I'm still looking for a not-in-my-home-backup solution that can store a backup of my main data harddrive. Currently that's 2.4 TB though, because for most art jobs I create 5 to 50 gigabyte of data in versions of Photoshop files etc. per project. And I need to archive all that stuff because even ~10 years later a client might want me to do something based on the old source files. It has happened. And yes, I could delete more than half of that data because I don't need all 10 progressive versions of a file, but the time to sift through that is much more expensive than buying bigger hard drives.
Does anyone have experience with storing harddisks in bank safes or something like that?

 

I use VisualSVN Server.

With TortoiseSVN as a client.

Has the added benefit of being UCB compatible too.

I host my own server and use a free SSL cert from StartSSL.com.

My SVN repo is about 300GB right now. No issues.

 

Simply use SVN, like every large company

xp-dev.com is a good host, but there are many

You won't get a free plan if you need 20, 40, 100 gb, you'll have to pay.

On Mac, use VersionsApp SVN and on Windows use Tortoise SVN

 

It's not just assets that need to be stored, but also the development data used to create the asset. It's not uncommon for a single Gimp/Photoshop project file to be a 1Gb alone just for all the layers used in the creation of a texture/NRM used by a material. I don';t know about Photoshop, but Gimp projects can explode in file size very quickly.

This. Disk space is cheap. Labour is not.

 

Since this almost 2 year old thread got necroed anyway I can now answer my own questions (hopefully for the benefit of someone stumbling over this in the future). Hard disk in bank vault works and isn't that expensive, but don't kid yourself, you'll update that think only twice a year or so.
And I tried Backblaze: it can realistically utilize about 15% of my 10Mbit upstream because I'm in Germany and all their servers are on the US westcoast it seems. Support said there's nothing they can do for me, so I uninstalled it. Next thing I'll look into is a "Hetzner Storage Box" because they are hosted in Germany:
https://www.hetzner.com/storage-box

 

this is why everyone just uses SVN. xp-dev is actually in the UK and if you ask they'll host you IN GERMANY! (on 1and1 I believe - world's biggest data center)

enjoy!

 

When you have a large project, obsessively there are none code assets that are taking that space, like graphics, sound... so you don't usually often change those files... so I recommend adding the large files to a cloud drive, and remove them from your version control (git) by Using gitignore. and if you want to clone the project you have separate repos. a clean small one that only has the code and the project settings, and a separate one that only has the assets, that's what I learned at the company that I worked for.

 

@Rabee_Kiblawi , you know, everyone just uses subversion (svn) - which is literally made for exactly this job.

It is used by all film companies, media companies, audio companies etc, and indeed game companies for this purpose. Because they deal in huge files. A Unity project is a whole lot of huge image, video, texture, audio etc etc files, that's what a Unity project is.

(The actual c# files are nothing, a few kb.)

We do Unity projects for the largest corporations and startups, I've never seen anything other than svn for Unity projects. It's precisely what svn is for.

(Sometimes you might also use git - only for the few c# files - so that you can do PRs and the like. But really it rarely comes up.)

I've never done an iOS or Windows project that wasn't git, and, I've never done a Unity project that wasn't svn. It makes it extremely easy, as it's for the purpose, which is why it's so widely used for Unity

(Mind you, Unity have their own sharing system now - which I have not tried yet.)

 

Can you elaborate more details

 

If you want to go this approach, you are better off using git LFS.

Git LFS is essentially what Rabee_Kiblawi described except it is all handled for you via normal git commands.

 

we used git lfs and it is taking hours to clone.

 

I just host my own SVN server. You basically get to just ignore the file size issues.

 

If you are solo or 2 people use SVN not git, it makes it 10x more complicated for no gain until you want branching
If you are a team use perforce not git, git struggles with art assets and can't even exclude things
Git is for programmers pushing some text scripts

 

@EricFFG cannot agree on a single point you make there. I use Git as a Solo Dev, in small teams and in giant teams. It's the only thing keeping us sane. And yes, in the old days Perforce and SVN were the go-to places. That was around 10 years ago though for us and I never looked back. I think Git has a learning curve but once you know what you are doing it becomes a tool me and most people I know would not want to miss again. I use it even for the simplest of Unity projects because then I have my automatic backup, local and in the cloud.

 

looks like people tend to use version control tool which they already have experience in

 

I do use dev.azure.com, it's free and has unlimited space!

But there is a catch. You won't be able to do the first commit of those 20gb at once. It will give an error with a message not very explanatory.

Try doing 4 commits of 5gb each and you should be able to upload everything.

 

I do the same @wagenheimer and it's great! and yes it really depends on your upload speed, I try to keep my commits below 2Gb indeed