Question Too many open files

I'll write a few things here about the problem which has been plaguing our CI Jenkins and our macs for some time now, crashing Unity before it even opens.

Basically Macs can't run process that use many files/many processes/many files per process/many processes per uid

basically because of that, as Unity tries to access the next file, say a dll or a shader it crashes and the error says something random like "thread crashed" or "shader compiler failed" basically leading us to random conclusions.

Some useful commands that I have used to see the problem ( but still no solution )

Code (CSharp):

1.  
2. shows you current limits: ulimit -a
3.  
4. turns on server mode to boost performance: sudo nvram boot-args="serverperfmode=1 $(nvram boot-args 2>/dev/null | cut -f 2-)"
5.  
6. shows processes that use most files: sysctl -a | grep ^kern.max
7.  
8. edits maxfiles usage till next reboot: sudo launchctl limit maxfiles 204800  204800
9.  

you can edit max limits on startup for a Mac creating a sysctl.conf ( latest OSs ) on the /etc/ folder with the following settings:

Code (CSharp):

1.  
2. kern.maxfiles=204800
3. kern.maxfilesperproc=245760
4. kern.maxprocperuid=25600
5. kern.maxproc=25000
6.  

apparently each mac depending on it's specs ( one mentioned based on RAM ) have different max limits but even though I've managed to edit them all the problem still appears at around 1700 files.

Anyone that has managed to solve this please give us a hand <3

 

I can't believe no one else has had this, it happens 2/2 big projects for us.

 

Seems to be FileSystemWatcher which is a tool by Microsoft. It's normally a 4k buffer but now appears to be 8k by default. You can set it in code to increments of 4k but no higher than 64k. If it loads too many files in too short a time then it overwrites its buffer and crashes! This is obviously trash software but hey it's free. Also Unity claims to fix it in 2022.1.0a12 but, nope, it still fails [https://unity3d.com/unity/alpha/2022.1.0a12 macOS: Fixed Unity shader compiler crashing on macOS Monterey when using Apple silicon editor. (1361979)] Fails on my M1 Max 64gb memory with 2tb SSD.

I also upped all of the mayflies and other nonsense that's really low to no effect.

It really seems to be something in FileSystemWatcher and it's in several packages such as JetBrains Rider (removable from Unity Registry), PlasticSCM (not removable), and some Lump libraries.

It also only happens on import for macOS (Monterey or other versions) where it doesn't need to use FileSystemWatcher as, according to others, it's only needed for Windows. It also seems to happen when you have around 100+ assets.

Works fine up to version Unity 2021.1.28 but anything from 2021.2 or 2022.0 fails on import.

Still seeking a solution as well. Currently stuck on 2021.1.28.

 

We've had this problem from 2019 and 2020 so maybe it works only on 2021 but it's not even LTS

 

This has been a really bad issue for me as well, and every solution that i can find online doesn't seem to apply for the M1 Macs running Monterey. It looks like i can get MacOS to tell me the file limit is raised, but when it actually comes down to opening a project it still fails.

I'm unfortunately stuck on Unity 2019.4.x

I've had to abandon projects and the new ones that i'm starting i feel as if they're on a time limit until i can't use them anymore.

 

@zdragon5556 interesting that it's a problem even with M1 Macs, I was hoping to buy a new mac and test whether the issue persists. That's terrible for every big project.

 

Yeah 2-3 times minimum over the years, this post isn't the first one.

 

@Alex-Thibodeau A month ago I wrote in this thread:

https://forum.unity.com/threads/cannot-open-project-on-2021-2-0f1-apple-silicon-editor-m1.1188772/

According to the conversation in that thread, this bug is fixed:

https://issuetracker.unity3d.com/is...emwatcher-is-used-on-a-large-number-of-files?

In our case it is not true as of 2021.3.5f1 LTS, we have several projects that fail on 2 macs of different generation and type (iMac and MacBook).

 

I've lost mine over the years but the question is whether your project has a lot of files in, like a full production project type of size 30k files for example

 

I can't share it but I imagine it should be easy to reproduce if it has to do with the amount of files regardless of what type they are. Making a console application that generates thousands of cs files inside assets and try to open it later.

 

@Alex-Thibodeau I have a repro project, always crashes

CASE IN-10056

 

Take a look at it as soon as you can because from QA they have answered me and as you will understand, it is not the answer I expect since the person who has reviewed it has not even noticed that I am reporting it in a version higher than the one that mentions that issue as fixed.

 

You have tested it on mac I guess? Have you opened it several times? It's random, it doesn't happen 100% of the time.

 

@Alex-Thibodeau makes sense what @KaroCerv says? I mean, in 2021.3.6f1 has something been modified that may have fixed this problem? Because reading in changelog, it doesn't seem so.

Personally I haven't tested to open it in 2021.3.6f1 (I'll test it and let you know) but it sounds weird to me for that reason.

https://unity3d.com/unity/whats-new/2021.3.6

 

Same problem here...

 

It's quite an obscure one, sometimes it's not just the amount of files a project has, it's the cache, whatever unity creates on the background, the amount of meta files there are etc. But generally it all points to file count in a mac.

 

In my case, my problem is on intel macbook, not m1 (we don't have any).

 

there was another person with M1 that had it too. It's very weird, I have tried moving files from a project to another and it opened, but after first try it stopped then removed 1 file and it worked but broke again later. It's like because too many files are open through the editor ( anything in the library ) the Mac chooses which ones to run, even at random thus the weird behaviour

 

@Alex-Thibodeau They have marked marked report IN-10056 as "Not Qualified" and as you can see in this thread, it is still happening.

I understand it's hard to reproduce on your side, but for us it's still a problem, right now we have 2 macs unusable for Unity because of this bug.

 

Yeah, I've had this problem way before 2021 when I did this post but it's still here. Also 2 macs that are unusable and I'm hoping to buy 2 more when M2 come out

 

What I have done to reproduce is this:
- Create a new project or take the example people have sent you
- Keep duplicating the files so that meta files and the actual files ( potentially including library files ) reach a big number ( ~30.000 )
- Restart the unity editor as you are doing this.
- Observe that eventually the editor will not be able to process all the open files, start creating issues and even throw the errors we've sent above.

It's surefire, I've tried it in different projects, empty projects, etc. Our mac is used a server / build machine so the only other thing that could be running is sourcetree ( not always on ) and jenkins. But in smaller games it never happens.

 

Given the fact that unity team could not reproduce the problem, I should not hope for a fix/workaround since the last post, right?

 

I mentioned this before, this has been going on for many years so probably no

 

Thanks for the update! I cannot know (I am not sure how to look for it) if we share the exact same symptoms, although, judging by your description, it seems that the outcome happens for similar reasons.
In my case I just try to open an empty 3D scene with the URP package from Hub and I don't have a big project like you mention. The crash happens in unity versions higher than 2021.1.28, while importing the URP package components, so it looks like that the problem lies with URP v12+ (v2021.1.28 gets the URP v11). I have done an extensive trial with different versions of unity editors and I found out just recently that it was possible to create 2D URP projects, so probably is not the URP itself. Still trying to figure out why, but I don't have the skills to do it properly.
I also reported an issue report some days ago and still waiting for further assistance.

 

the reason is that packages also count towards the total file count and i guess urp too

 

The thing is that URP v11 is imported normally (or other packages too). What can be that different, considering file count, with package v12?
As a side note: I can import URP v.11 on Unity 2021.1.28. I don't know if I can do the same in higher versions of Unity, since the package manager automatically tries to retrieve the latest 12 version. Which, as I said, it was imported properly in a 2D project (URP version 12.6 I think).

 

it's just a file count being open by a single application, nothing to do with urp versions. This issue has misled many ( as you can see since the post is quite old ) you could open up an empty project and it would build ok, import 10 files and it would break.

 

Could graphics driver be responsible to any extend?

 

I doubt it as there was a gentleman also saying above that it happened on his M1 mac mini too. I am running a 2014 mac so it being older should be a cause but it's happening to many others too

 

Ok, thanks for replying! I am using a far older 2009 MBP with patched Mojave and I noticed that the graphics driver is not Apple provided. I also have speculated that this is probably not the reason behind, but I decided to ask nevertheless. Other than that, I don't see other potentially problematic aspects on my system's part.

From your sayings, I understand that it is not a problem specifically related to package installation (and probably mine is another issue after all), but I was wondering if you can see reasoning related to what you have figured out and the fact that the URP package will install properly with a 2D project, but it always fails with like 10 different versions of Unity Editors higher than version 2021.1.28 and a 3D project.
I figure it could be a random symptom, but if I had to address some repeating behaviour, is that the the crash happens when compressing some image files, during the package installation. Then the affected project would not reopen (instead will crash the editor shortly after trying to open it), unless I delete the files in the Packages folder (manifest and packages-lock), so that the project would reopen bypassing the urp files.

 

It's an OS issue basically and how Unity has been created

 

Keep getting this trying to build for iOS on 2021.3.14. I assume this has become a problem because of the multithreaded importing, as we never had this on 2019 and we have just upgraded. I have yet to manage to get a batch build to work for iOS on 2021.

 

I would suggest that it's not that and that it's that the project got bigger or that the unity's engine files became more and more

 

No, nothing to do with the project getting bigger, I can still do builds off the 2019 build configuration fine from the same repo node fine.

It is that almost definitely multithreaded importing that does it, it crashes while importing project files and fundamentally having multithreaded importing *will* result in more files being open at the same time. To be precise it will involve 3 times more files open on that machine (75% worker count on a quadcore machine) and the default maxopenfiles on macOS is 256.

I fixed it by increasing the maxopen files in both launchctl and the ulimit for the user the batch build is running as.

 

If you check all the dialogues above there were issues with people on 2019 as well, myself included. Also it seems some older macs don't change the maxopen files even if you mess with the settings :'(

 

I'm sure there were, with 256 being the max that limit will be hit for multiple different reasons. In this case it will have been multithreaded imports doing it, because it was the same versions of the same assets being imported by different versions of the engine, after package dependencies etc. were resolved.

Edit: Especially since in this case it was 100% reproducible rather than intermittent.

I wouldn't be hugely surprised if that's the case but the thing to note is that Apple appears to have changed the means of setting max files practically with every new version of the OS, so it's a complete guess whether the fix will work for any version of the OS you use and whether it will continue working post-upgrade. I had to change the limit in two out of the 3-4 places the OS has for it. The kernel already had a *very* large limit set automatically (this one depends on the system apparently), ulimit had 2048 and launchctl had 256.

 

Hi @AlkisFortuneFish! It's my bad, but I don't really get your sayings. Do you suggest that there is a workaround? Do you also suggest that depending on the OS it might work or not? Anyhow, is it easy enough to share the exact steps need to be taken or alternatively point me to the direction to find relative information?

 

I tried changing the file limits workaround (not sure if applied properly), but did not solve the problem. As I speculated before however, the source of the problem in my case, as it comes out, has been different. The good news is that a user's (@Uphage) suggestion in this thread did the trick! Although brilliant in my case, I am not capable to say if it is related to the issue described here...(seemingly not, since the symptoms look a bit different, but I wish otherwise)

 

In our case we do not have the URP package installed, so the problem you mention must be different.