Resolved Git and Unity Scene files larger than 100mb

Git has a 100 MB hard limit on files. What do you do when your scene becomes larger than that? I have been searching for an answer but can't seem to find a (practical) one.

The only thing I can think of is adding them as LFS, but wouldn't it stop tracking changes within the scene?

If you have any info, I'd really appreciate it, thanks!

 

LFS is a workable solution if you're fine with not being able to merge changes. I personally don't mind because those rarely work well. I use the Git LFS file locking feature to avoid working on the same assets as someone else - here's a tiny batch script shows who has locks on which assets:

Code (csharp):

1. @echo off
2. echo Loading git locks...
3. :start
4. git lfs locks > %temp%\unity-git-locks.txt
5. cls
6. echo Active git locks:
7. echo.
8. type %temp%\unity-git-locks.txt
9. goto start

You'll also be consuming a rather large amount of LFS storage with each scene change you commit, and if you're using a cloud host you might eventually run out of storage - maybe consider self-hosting GitLab on your own hardware, could be cheaper in the long run.

 

What's contributing so much to the scene size? I would first check if you can save any of the scene contents as separate assets in your project. That way, even though the scene still contains those assets, it's just a reference to an object, and doesn't significantly contribute to the size of the scene.

For example, some tools allow for mesh creation in scenes, such as tools that allow you to fracture/break a mesh. By default, those generated meshes tend to be saved directly in the scene, and that can result in very large scene files. But if you can save those meshes as assets in your project, it significantly reduces the size of scenes that contain them.

This probably isn't an option for all kinds of large objects, but it's worth considering if you haven't already.

 

That's an excellent suggestion. You can try opening the scene file in a text editor and looking around - previously I've had issues with certain components taking up huge amounts of space (eg. the ParticleSystem component has a rather heavy serialized representation). Polybrush is also notorious for making scene sizes explode as a result of meshes being serialized as text.

 

From my point of view scene files should be kept as small as possible. Sometimes, large scenes are a result of buggy code or just really bad implementations of some tools. If you're stuck with that, I'd still say Git/LFS isn't the best choice for game development because of these reasons. In my experience, scene merging works really well, if you set up SmartMerge. Perforce or PlasticSCM, for example, don't have any issues with files larger than 100MB, and can be configured to still merge such scenes via Unity's merge tool, so I'd recommend these tools over Git.

 

Thanks for this!! When I first saw this I was wondering the same thing, but I thought this was normal. I am using MapMagic, I think it might be the culprit -It keeps terrain data in the scene probably. Maybe I can save it as a prefab. Do you know if the prefabs contribute to the size? or are they considered data in the project?

 

Thanks all for the replies! I will look to see what's taking up so much space in my scene. It could have been MicroSplat terrain blending objects, which I think creates a new object or it could have been MapMagic.

@Xarbrough Thanks, I recently learned about PlasticSCM, it looks so much better for game dev (and I think Unity recently bought them out). I didn't know about the SmartMerge, I will check it out!

P.S. I was also considering splitting these scenes into multiples (terrain, props, etc..) but managing them would be combersome.

 

It depends. Just saving something as a prefab might not be enough. The "big" part of the object would generally need to be saved as a separate asset outside of the scene. If you create a standard Unity Terrain object in your scene, you'll notice that Unity creates a separate terrain data object asset in your project that holds all the terrain's data. That means you can add a complex terrain to a scene without making the scene all that large. (The terrain asset will get pretty big instead.) I don't know if the tool you're using can be made to save the data to a separate asset, but that's probably ideal.

 

Thanks, I tried tested making a prefab and it didn't work. Yeah, I am aware of the terrain data file. It was MapMagic that that was taking up the memory. It keeps an active instance of the terrain in the Scene file (not in the project as you mentioned). I guess I can export the terrain and make it a conventional terrain.

 

Amen Xar, Amen.

Another thing to look into @FlightOfOne is to break the scene up and load things additively.

F'rinstance, here's some rando-scribblings I wrote on the topic:

Additive scenes are one possible solution:

https://forum.unity.com/threads/right-way-for-performance-divide-scene.1023673/#post-6630961

https://pastebin.com/Vecczt5Q

https://forum.unity.com/threads/problem-with-canvas-ui-prefabs.1039075/#post-6726169

Other notes on additive scene loading:

https://forum.unity.com/threads/removing-duplicates-on-load-scene.956568/#post-6233406

Timing of scene loading:

https://forum.unity.com/threads/fun...ject-in-the-second-scene.993141/#post-6449718

 

@Kurt-Dekker

I am already splitting the scenes and loading them additively. But the scene with the terrain is what's causing the issue and looks like there is no way around it unless the asset author does something to save the data into the project -I have yet to inquire.

Thanks for those links! I will take a look.