Feedback and experiences from using Collab

I tried out Collaborate over the last weekend for the global game jam.

We were 5 people working on a project, and 4 of us used Collaborate to share files.
I've been following the Collaborate project with interest. Trying to build a replacement for git or other scms is... a daunting task indeed, so I wanted to see how it worked. I figured my experiences using the system might be interesting for Unity/other users, and I'm waiting for fbx imports, so here goes.

In my day-to-day, we use Git for sharing, mostly through SmartGit, so that's my baseline for SCM. We're at the most 8 people who touch the repo in a day.

The good things:
Collaborate was really fast to set up. Just click some buttons and you're sharing a project. Simple stuff.
Collaborate was really easy to learn. Non-programmers who had very little experience with Unity managed to use it just fine.
The servers seemed to be plenty fast.

The bad things:
The interface was pretty lackluster. The tiny window for sending and receiving changes was way too small, and was a strain to use on a laptop. I could also just see two or three files at a time when looking through the files to be commited. This should be a normal editor window! Sharing files is important stuff. I'll probably want to use an entire monitor for it, not a tiny rectangle!

The history was also not quite there. 5 commits per page does not give a good sense of the history. Ditch the pages! SmartGit allows me to scroll through 3 years of history with the scroll wheel, and dynamically loads older commits as I scroll. That's an infinitely much better experience.

- Conflict resolution was not good. The "take theirs" button has a yes/no prompt, but the "take ours" does not. I kept hitting "take theirs", tried to move my mouse over to the "yes" button, but had the cursor slip below the button (becasue all of these buttons are way too tiny), which caused the prompt to disappear, and I ended up clicking "take ours" instead, as it's placed on the same spot as the "yes" button. Both of the "take" buttons should have prompts, and the prompt should be a popup. Clicking the wrong thing here deletes work, the interface does not need to be smooth, it needs to be safe.

In addition, all of the third party merge tools I tried were utter garbage. I could not easily see what the conflict was, and merging was hard. I'd really love to at least have the choice to spit out a git-style merge file which shows both versions at the same time. You can't count on third party freeware tools being able to solve every code conflict, and for those instances, it's nice to be able to fall back to a text editor as a last resort.

- Unity kept getting in a state after pulling changes where it would never quit compiling. I had to right click a script and reimport it to trigger a new compilation to be able to do anything.

The collab button and the collab window kept getting out of sync. The collab symbol could switch from "you can push" (blue) to "you need to pull" (yellow), but clicking the button would still show the push window for a couple of seconds before it switched back. If the button got into the red error state due to eg. losing network connection, it had a tendency to not go out of it again.

The really bad thing:
We lost work. The code merging did NOT work.
Several times we programmed up a feature, saved the code, tested it, and went to push it. We had to pull down changes from the server first, and found that those overwrote the freshly made feature. This happened to both of the programmers several times.
I talked to a coworked today that participated at the same jam, who also used collab, and he experienced the same thing - random loss of code after pulling down changes.

Philosophy differences:
It's been a couple of years since the last time I used a centralized SCM, and oh my. Having to be in sync with the server to get changes from my team is the worst. I wanted to grab some levels from the level designer to check out a bug he had found, but since the other programmer had made commits, I had to merge the shooting code changes to see those levels.
That doesn't make any sense!
I realize that you're not trying to recreate git because it's hard to use. And I agree. But it's hard to use because Linus Torvalds is horrible at naming stuff, and leaks his abstractions all over the place (see "checkout"). It's hard to use because rebasing is confusing as hell. It's hard to use because all of the error messages assume that you know how to use man pages.
The distributed workflow on the other hand is very, very intuitive. "Jessica wrote some code I want, so I get that code from Jessica" is a much better way to work than "Jessica wrote some code I want, so I get everything from everyone". It's probably too late, especially since you've gone down the route of forcing us to host the repository on your servers (... why?), but if there's a possibility, the option of having a distributed workflow would be a big improvement.

As a result of the centralized model, pushing was a nightmare. I kept trying to do it, but while I was writing the message, somebody uploaded to the server. So I had to pull those changes down, and resolve conflicts. And then I tried to push, but now a different team member had pushed changes. By the time the server was clear, I had forgotten what I was intending to push!
Having commiting and pushing as two seperate actions enables the user to first make a decision about what to commit when they've got that fresh in mind, and then handle the pulling and pushing. The concept of a staging area is possible in your centralized model, and enabling the user to ready a package of changes to push before they have to pull down changes is a big quality of life improvement.

Not being able to push a subset of all changed files was also pretty bad, but I understand that feature is on the way.

To sum up:
Collaborate has some major bugs, which caused lost work. That's a gigantic problem, but fixable.

In addition, the user interface is not where it needs to be. All the menus are super tiny and shows very little information at a time. Unity has very powerful tools for creating great editor windows, but it seems like you've bypassed that entire system and instead made your own GUI, and it's not good. Collaborate needs a proper window to do things, not small popups.

The system has potential for small teams and small projects. Without a distributed model, I'm very vary of using it for projects with more than one programmer, however. How programmers and artists need to share files in a game project is very different. Git is a bad fit for artists, but I fear that Collaborate is heading towards being a bad fit for programmers. The lost progress might have led to a bad taste in my mouth, but since Unity has some problems with assets (like several-minute import times for large .fbx models), having to pull all new assets from the server when I really only want to pull code will cost a lot of time.

I hope my feedback could be some help. I really wish you well in this - a simple-to-use SCM would help a lot for people picking up Unity, game jams and prototype work. Maybe you can make it good enough to be useful for large productions as well, but that'll require a lot of resources and probably some major changes in direction. We'll be sticking with git for the time being.

 

@Baste, thank you so much for the feedback! We really do appreciate your team trying out Collab for the global game jam and we're sorry to hear that it didn't quite live up to your expectations. We understand that different teams have different needs and we try to address as many of those as we can. Thank you all the same for these well thought-out notes and I'll be sure to pass them along to the team.

I'm very sorry that you lost work at any point during your workflow . That's obviously a huge deal to us and I'd like to get more info. Which merge tools did you try? Were more than two people on the team ever working on the same feature? Would teammates typically pull updates right away or wait and pull several commits at once?

 

We tried both SourceGear and TortoiseMerge. Maybe we also tried one other? Can't quite remember. I can't remember which one was active when we got the issue either, probably SourceGear as we ditched TortoiseMerge after trying to resolve an issue with it once.

But the work lost was not lost when Collaborate recorded a conflict. It should have recorded a conflict, but instead overwrote one person's code with the other. Do you use the third party tools for merging, not just conflict resolution?

When this happened to both of the programmers, we'd been working for about half an hour or so without syncing, so we probably grabbed several things at once. Game jams have a rather fast pace after all! In all three (I believe) cases, we had worked on the same code file at once.

It might be that the code that was overwritten had not been committed yet (ie. what would be "unstaged changes"). I'm 100% sure it had been saved to file, though, since we had tried to run the code and seen it work.

Looking though the collab history, I can see commits where we complain about the code having dissapeared ("collab, plz"), but without any way to see the diff of a certain commit, it's hard to see where it went away. Of course, if it was uncommited code, it won't show up in the history, so that won't be much help. I can give you the ID of the project if you're interested in looking at it.

 

TortoiseMerge is not playing nicely with Collab at the moment so I get why you would stop using it (we do have a work item for that). SourceGear's DiffMerge is one of the diff tools that I use regularly with my testing so I'd be very interested to hear if you ran into any issues using it. But we don't use the external diff tools for handling non-conflicting merges.

This is great info, thanks! I just tried this myself (making changes to a script that no one has pushed yet and then updating from one user that finally does) and I see that it does detect that there are conflicts but any attempt to launch an external diff tool fires an error saying that it couldn't resolve the conflict. This isn't quite what you saw but it does show that this edge case (resolving conflicts with unstaged changes) needs to be better accounted for.

What would really help us out is if you could send a bug report from this project (in the Editor, go to Help->Report a Bug). You can add your project ID in the bug description so that we can investigate this bug both from the client side and server side. When you get an email from our bug reporter, can you post your fogbugz number here so that other devs can track it to you and get in touch in case they need more info? We'd really appreciate it!