How to properly merge Scenes using SCM repos

I am sure there is a solution but I sure cant find it. We tried many different repos and they all fail to merge the scene files properly. Currently we are using Plastic SCM with a large project that has been four years in development. It fails to merge the scene file every time and requires a manual merge.

Surely large studios are not doing manual merges with 20 devs working on scene files?

Does anyone have success merging ~100MB scene files with a merge tool?

Plastic SCM is basically saying unity can't handle the file complexity and we should do it manually. We have tried multiple third party merge tools at this point. Love to hear how others are making this work.

The project has roughly 11GBs of data, builds to about 3GB and the Scene files in questions are roughly 80-120MB.

Thanks in advance all.

 

There's no one clear answer to this.

Unity scenes / prefabs / assets are stored in YAML, which cannot be merged confidently.

Unity offers a YAML merge tool but I've never had good luck with it.

Breaking scenes up is always the win.

Multiple additively-loaded scenes make the area of overlapping concern much smaller.

A little bit of runtime code to rejoin the parts of a scene after loading is time well-invested.

Keeping different developers out of a scene is also helpful. This gets harder as your branches live longer.

Chunking pieces of a big scene out into prefabs is also an option, but I find it impossible to reason about what changed, since every single change now causes dirt in a scene AND in prefabs. I do NOT recommend use of any prefabs that exist in only one place. You're just asking for trouble. Keep it all in scenes and subscenes.

A long-lived "next version of the world" branch may benefit from an actual copy/pasta clone copy of the scene, ultimately with abandonment of the original scene, perhaps with some cherrypicking of bug fixes that happen before the merge.

When you think about it, it's no different than if you had X number of developers working in one giant C# class that was 100mb in size and had to be merged regularly. It's just that with C# code it tends to be more confidently mergeable, but certainly not infinitely mergeable!

 

Thanks Kurt. Great answer and clear and I do appreciate it. I'll be honest though, for such a world class asset as Unity portends to be, the fact they cant manage their Scenes in a way that allows multiple developers to work simultaneously seems ... Mickey mouse. Your suggestions do make sense but are really kludged work arounds for a poor control structure of the scene. Generally a group of 20 devs would not be working simultaneously on core changes to a critical class. But every single thing we tend to do in Unity touches the scene object. How can a team of 20 devs or hey, even 3 devs, execute a workstream without having to constantly fight unsuccessful merges from the other devs. I can't see a professional studio even attempting this.

Anyone else had any luck or know how professional studios do this?

Kurt thank you again for your insight. I'm sad that it re-inforced my belief that Unity is really lacking in this area but your responses do align with what I've encountered so far. Wouldve been nice 4 years ago if Unity had a disclaimer that said dont try and make a scene bigger than 5 MBs or it wont merge.

Certainly makes a clear case for switching to Unreal.

 

Clear case? What makes you think that UE can merge stuff any better?

https://answers.unrealengine.com/questions/708829/git-conflict-resolution.html

At least Unity is YAML text nowadays... Unity used to be binary files.

 

Most of the "professional studios" (and any sane developer who has any idea how merging works and why) don't allow multiple people to poke around in one file. Recipe for a disaster. You can't really merge binary files, scene files or 3D models. But even CSV files are very sensitive.
It's the perforce-way: lock the file while you're working on it, submit your changes, release the file. IDK if SCM has locking feature or not, you can probably simulate it with proper work assignments though.

You can't even merge code 100% reliably, how can anyone expect to merge scene files? It's beyond my understanding. But sure, run for Unreal, you're up for a very bad surprise.

 

For anyone reading this question in the future, SCM does support file locking. As far as the hate on Unreal Lol. Okay. This terrible scene management that Unity uses is just another in a long litany of reasons to think twice before using the engine for a serious development. But to each their own. Thanks again for the responses.

 

You're utterly mistaken. No one hates Unreal here. You were the one, who threw the temper-tantrum that "okay, then I go to Unreal", which wouldn't fix your problem, but you failed to research it before you lashed out.

You are also failing to hire anyone who actually knows how this whole thing works, you supplement the expertise from the Unity community forums and then you're whining here that you're going to Unreal. Okay then.

No problem, if you leave your attitude at the door, we happily help you whenever you need it.

ps: and as you can see if you don't leave at the door, we still help you, but we're grumpy about it.

 

hey @Spartikus3, we have quite a bit of experience with Plastic and Unity using basic cloud repos and fairly simple workflows.

First, obviously set up Unity to use text serialisation so scenes and prefabs are stored as YAML text.

Next, make sure you have set up Plastic correctly to use the UnityYAMLMerge tool for .scene and .prefab files as outlined in the Unity docs - and make sure both of these entries are above '$text' in the Plastic Merge tools settings. Merging scenes does have clashes now and again and will pop open a 3-way merge in your merge tool of choice (Apple Merge, Beyond Compare, etc) but often these are basic choices to make because UnityYAMLMerge has done a bunch of merges before you're presented with the clashes. For us, often this 3-way merge clash is just a floating point rounding error between Mac and Windows for scene lighting colour settings.

Finally, as mentioned above, split your scenes up where you can and load them additively, and make use of prefabs so you can avoid scene churn and potential clashes as various parts of your scene and objects within them are edited by your team. You won't have a lot of fun merging single 100+meg scenes with low use of prefabs and a few people trying to edit them on various branches.

 

Hey Greg. Thanks for that. Great info in there. I was reading up on the proper tweaks for YAMLMerge tool. I haven't embarked on it yet. Our project is very fortunate to only have a small number of devs working but we are making huge core changes constantly as we get ready to go to a public alpha phase and the frustration of tripping over each other is really slowing down productivity.

Based on the responses here I am taking this weekend to make some pivotal changes to how we do the project.

1. I am going to convert all UI root objects to prefabs. Based on what I am reading this should make the UI available for coding work without massive changes to the scene. (Just transforms on the root I hope)
2. I am going to prefab all of our placeables (well over 8000) at their root so again, just transforms for scene changes allowing devs to work on changes to the structures inside the prefabs
3. Our world zone is definitely "too big" at 4k by 4k with the amount of placeables we have in it (and I know we will triple that as we do final touch ups for asthetics) but we have almost 2 years invested in scene development to get to where we are at today and rebuilding villages, towns, NPCs, quest triggers, mob pathing etc. at this point is counter productive. That being said, future world zones are going to be much smaller based on the challenges we are having here.

I think that is a solid start. We have been looking at moving to Unreal for about 2 years now but were hoping to be able to get into public alpha for proof of concept to see what kind of a player base would be interested in the product before starting over. One dev has been slowly migrating code over to Unreal and it is coming along well. My "tantrum" was actually just yet another kick in the crotch from Unity which gets added to a fairly extensive list of reasons why we are working with Unreal in the background.

This thread isn't to bash Unity though and I really appreciate the constructive responses so that future developers might find this helpful as they run into the issues.

 

Locking, planning and using a structure where a single object (scene/prefab/etc) should not need to be touched by more than developer at a time. Additive scene loading works as well. Addressables/asset bundles also work. It's not specifically a Unity flow but just a logical team flow. Regardless of engine/toolsets, not building modular will always bite you in the ass. For our larger Unity games at Disney, they were comprised of multiple projects broken up asset types. It was a custom ECS framework that had one scene that just loaded fired up a script. Nothing was wired in the inspector or anything like that. (except in sibling project that generated asset bundles). Lock the stuff you are working and no merge problems. (at least nothing that wasn't user error).

In smaller teams we worked differently, heavily relying on prefabs. (and locking). But, really if you have regular situations where more than one person is needing to touch something at a time, it should probably be a lot more modular. Tools shouldn't be used to solve design problems. Locking can help prevent merge problems, but if you have too much crammed into a single place, it's just hiding the problem, not fixing it.