Search Unity

5.2.0p1: P4 version control hangs editor due to fstat spam bug

Discussion in 'Editor & General Support' started by OneThree, Sep 18, 2015.

  1. HanfordL

    HanfordL

    Joined:
    Jan 4, 2014
    Posts:
    4
    This exact problem is happening to me in Windows 5.6.1 ... apparently it returned in 5.6.0? Only happens when unity is connected to my Perforce server (which is remote.)

    I am getting a lot of FSTAT messages in the console (I have the unity plugin set to Verbose) and every time I click on something it hangs while it does an FSTAT. Sometimes when I switch out of the editor and return Unity hangs up for close to a minute and when it's done there's a half dozen FSTAT messages in the console window.

    Anyone else seeing this? Any way to fix it?
     

    Attached Files:

  2. JakeTurner

    JakeTurner

    Unity Technologies

    Joined:
    Aug 12, 2015
    Posts:
    137
    Hi

    We need to investigate. This is the first report of this problem on 5.6 there was a similar report for a user using a custom plugin not Perforce based it might be the same common problem though.

    To help identify the problem can you:
    a) switch the plugin logging mode to "Notice" or "Fatal" : do the hangs still happen?
    b) Give a rough idea of how big your project is in terms of files in version control under the "Assets" folder e.g. 100 files, 1,000 files, 10,000 files ?
    c) Is your Perforce server hosted locally or remotely? i.e. trying to get a rough idea of a ping time between you and the Perforce server.

    Rough answers to b) & c) should let us recreate the problem locally to let us debug it.

    Thanks

    Jake
     
  3. HanfordL

    HanfordL

    Joined:
    Jan 4, 2014
    Posts:
    4
    >a) switch the plugin logging mode to "Notice" or "Fatal" : do the hangs still happen?

    Yes.

    >b) Give a rough idea of how big your project is in terms of files in version control under the "Assets" folder e.g. 100 files, 1,000 files, 10,000 files ?

    7,000 INCLUDING .meta files. (3500 files plus meta)

    >c) Is your Perforce server hosted locally or remotely? i.e. trying to get a rough idea of a ping time between you and the Perforce server.

    remotely, on assembla.com

    To give you more info, I upgraded from 5.3.4, and it did not have this problem. Going back and trying it on 5.3.4, it appears the Fstat commands in 5.3.4 happen asynchronously, so when I click on an object in the scene, the object is highlighted and the inspector changes right away, and a second or two later the Fstat output appears in the command line. Unity is responsive the whole time.

    In 5.6.1 (and in 5.5.4, I just tried it) when I click on an item, it highlights but the inspector does not refresh. During this period unity is not responsive. When the Fstat output appears in the console is when the inspector changes and Unity is responsive again. It appears not to be accessing the DB asynchronously.

    Sometimes when I switch back into Unity in 5.6.1 it takes forever for it to refresh.
     
  4. JakeTurner

    JakeTurner

    Unity Technologies

    Joined:
    Aug 12, 2015
    Posts:
    137
    Thank you very much for the information. We will investigate the problem. There was a change in 5.6 to fix a problem when checking out files to make sure the Version control status was up to date and not using unknown version control results. It might be related to that change.
     
  5. Homer-Johnston

    Homer-Johnston

    Joined:
    Nov 26, 2012
    Posts:
    46
    We just updated to 2017.1.1 from 5.4 and one of us is noticing intermittent freezing as well now. So far it seems like it could be related to ping - one person has a steady ping to our server of 150ms and does not have issues. Another person has a ping that is usually about 40ms, but sometimes goes to 300-400ms. It seems like if their ping is higher (300ms+) their editor is frequently freezing up for minutes during fstat commands. When their ping is low, they have no issues. Still investigating.
     
  6. KB73

    KB73

    Joined:
    Feb 7, 2013
    Posts:
    234
    We're seeing this on 2017.X, do you know when a fix will be ported in? We can confirm with Work Offline enabled, the editor on Windows runs properly, with it enabled, the editor is bordering on unusable for some people.
     
    jimmyjamesbond likes this.
  7. KB73

    KB73

    Joined:
    Feb 7, 2013
    Posts:
    234
    * Our last test with 5.5.4 on the same machine, this issue does not occur.
     
  8. jimmyjamesbond

    jimmyjamesbond

    Joined:
    Mar 17, 2015
    Posts:
    19
    +1 for our team as well. 2017.2.0f3 - Working offline the editor is buttery smooth. Connect to perforce, sad.

    I don't see a lot of fstat spew in verbose. Just significantly slower when using VC.
     
    Last edited: Oct 13, 2017
  9. Meceka

    Meceka

    Joined:
    Dec 23, 2013
    Posts:
    423
    Edit: Issue started to occur less often. Not sure what fixed it but it seemed to me like it was fixed after resetting unity layout to default. Or after submitting with Plastic SCM Gui.

    I started having the same problem after updating from 5.6.3.p1 to 5.6.4.f1
    I am using Plastic SCM, and setting verbose doesn't show any logs, but I see in unity version control window that it is doing this;
    "Performing changeStatus operation. Please wait.."
    or
    "Performing Status operation. Please wait.."
    This happens without modifying any files in project, even happens in play mode testing the game, unity runs very poorly(about 2fps) for about 10 seconds. From task manager I can see that initially Plastic SCM gets CPU usage, then Unity Editor. After it finishes unity performs well for a few seconds and the cycle repeats.
    It sometimes hangs when I switch from unity to another application and back in windows. Hang takes about 15 seconds.

    When I enable work offline from version control settings, the issue is gone.
     
    Last edited: Nov 20, 2017
  10. wightwhale

    wightwhale

    Joined:
    Jul 28, 2011
    Posts:
    397
    I have to deal with this issue all the time. My network usage spikes to 100mb (max) usage on my local network for perforceplugin.exe. Sometimes I have to wait a full minute or longer for the editor to respond to me clicking a different gameobject in the project window. This issue is extremely irritating to me and my team. The view highlights but the inspector hangs while it's figuring out whatever is going on.

    Hangs still happen on Notice and Fatal. My project is 60gb and about 130,000 files. I'm running unity 2017.1.1f1. I have a local server sitting right next to me. This problem happens for everyone on my team and goes away when working in offline mode.
     
  11. KB73

    KB73

    Joined:
    Feb 7, 2013
    Posts:
    234
    *bump* Anyone from Unity able to comment on when this issue is likely to get addressed? thanks
     
  12. Meceka

    Meceka

    Joined:
    Dec 23, 2013
    Posts:
    423
    I will share how I work around this issue with Unity version control and Plastic SCM. In post #59 I explained the problem I had.
    For me, issue was caused by moving a controlled folder with content into another. Plastic team said that Unity has a bug and it doesn't send plastic's plugin the necessary info for the move operation.
    Issue happens while renaming folders as well.
    I think that, as plugin doesn't know about move/rename operation, it causes some stuff to go out of sync between the plugin and unity, causing unity to go into a refresh loop and reduce editor performance, making it un-usable for me.
    Plastic team recommended me to submit using their own GUI (Plastic SCM GUI) instead of unity version control window, as it can already detect move/rename operations submit works just fine.
     
  13. wightwhale

    wightwhale

    Joined:
    Jul 28, 2011
    Posts:
    397
  14. JakeTurner

    JakeTurner

    Unity Technologies

    Joined:
    Aug 12, 2015
    Posts:
    137
    Ok for the problem with large pings we have a change in development which (an option that can be enabled) to allow for asynchronous status updates on files which should relieve the Editor stalls. We have been testing this with Perforce backend.
     
  15. JakeTurner

    JakeTurner

    Unity Technologies

    Joined:
    Aug 12, 2015
    Posts:
    137
    If there is a problem with moving files/folders in PlasticSCM please report that. We recently tested this and it was verified to be working.
     
  16. DavidLFunkitron

    DavidLFunkitron

    Joined:
    Oct 12, 2017
    Posts:
    2
    Our company is seeing this issue reproduce in the latest, released version of Unity, which is 2017.3.0f3 (as of this writing).

    Any idea on when we'll see the fix land, in Unity?
     
  17. JakeTurner

    JakeTurner

    Unity Technologies

    Joined:
    Aug 12, 2015
    Posts:
    137
    Hi

    Please can we discuss how you are recreating the issue as we have trouble recreating the issue.

    Also please can you share what setting you have for your "Version Control" LogLevel : the advice is "Notice" or "Fatal" only. Any more verbose log levels can cause a lot of log spam and cause performance issues.

    Thanks

    Jake
     
  18. allornothingJM

    allornothingJM

    Joined:
    Feb 3, 2015
    Posts:
    20
    We're having some truly dreadful performance in the Editor with Perforce connected on LAN. Project is quite big, but working offline has everything running super smooth. Simply selecting an item in the hierarchy hangs the editor for anywhere up to 30 seconds, and saving a scene takes several minutes (with no ui to tell you it's working or not)

    Seems the same as everyone else in here, but still here in 2017.2.0p4. Is there any advice on this? It's unusable for many of our staff and causing a lot of issues!

    Perforce Log set to Fatal and not seeing spam in the console.
     
  19. JakeTurner

    JakeTurner

    Unity Technologies

    Joined:
    Aug 12, 2015
    Posts:
    137
    Can you tell me if you have the "Version Control" window open?
    I have been investigating performance issues when the "Version Control" window is open.
    Perhaps contact me via direct message and we can do some investigations. I want to investigate any Perforce related performance issues and work together to find solutions
     
  20. KB73

    KB73

    Joined:
    Feb 7, 2013
    Posts:
    234
    * We switched to 2017.2.2f1 and it seemed to ease the issues for the people experiencing problems on windows. Duplicating an asset used to take 5-7 seconds, now it takes 2 or so.....

    Scrolling the project window is still 'slow' but everything else has improved.
    I never experienced this problem on the Mac.
    We were on 2017.1.X prior to this.
    5.6 never had this problem, it only seemed to appear since we went to 2017.
     
  21. JakeTurner

    JakeTurner

    Unity Technologies

    Joined:
    Aug 12, 2015
    Posts:
    137
    Just to check : what "Version Control" "Log Level" setting do you have? For best performance I would try "Fatal"
     
  22. syscrusher

    syscrusher

    Joined:
    Jul 4, 2015
    Posts:
    1,104
    I have replicated this problem with Unity 2017.4.x, 2018.1.x, and 2018.2.x over the past couple of months. I have "background sync" enabled, which the manual says is supposed to keep the editor from being unresponsive. I'm connected to the server over a WAN because I am working remotely for a client project.

    For me, this isn't an issue of "takes a few seconds". It takes several *minutes* to perform some simple operations, and exiting from Unity editor after making any changes is so excruciatingly slow that I actually thought the editor was hung, until I started checking Windows Task Manager and found that the Perforce plugin is pushing a lot of network traffic.

    My Internet connection is 54 megabits down and 18 megabits up, and the customer has an extremely fast dedicated pipeline to the net. I realize that the network is going to limit performance if I'm moving a lot of data, or even metadata, but here's the thing: As a test, I tried the same operations in Unity and in the standalone P4V client for Perforce. Things that take 20 minutes in Unity take 20 seconds in P4V. Yes, it's *that* dramatically different.

    As I write this, I tried to exit the Unity editor after deleting one small folder with maybe 20 C# files from my project. That was about 30 minutes ago. Unity just finished exiting, as I typed the sentence right before this one.

    I'll try it again with the version control tab closed, to see if that makes a difference, and I'll turn down the logging. My local disk is an NVME SSD, so I don't think logging is the problem, but I'm certainly willing to be proven wrong. :)
     
  23. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,770
    When logging is set to Verbose, the editor logs each and every version control operation it does (including status queries, etc.). Looking at the log might reveal whether it's for example continuously trying to query status of the same files, or what.

    An actual bug report would be helpful, as usual! A whole bunch of version control related fixes went into Unity 2019.x releases, if you could try that (e.g. 2019.3) that would be great as well.
     
  24. syscrusher

    syscrusher

    Joined:
    Jul 4, 2015
    Posts:
    1,104
    I haven't tried changing the logging level yet, but with the version control tab closed, things are indeed quite a lot better.
     
  25. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,770
    Right, so one issue that having Version Control window being open, is that it was "fully refreshing itself" (checking incoming/outgoing changesets and their contents) after each.and.every.version.control.operation :facepalm:. I think we fixed that in 2019.3.
     
    syscrusher likes this.
  26. syscrusher

    syscrusher

    Joined:
    Jul 4, 2015
    Posts:
    1,104
    I wish I had the option of upgrading, and if it was my own project, I'd consider it. But this is a customer's project, and I literally just got done moving them from 2017.4 to 2018.4. They want to stay on the LTS versions only because they're close to release. (I made the case for the 2018.4 upgrade, which was done incrementally, based on the EOL date for 2017.4 and the undesirability of maintaining a published product with the pre-Package Manager versions of TMP, Cinemachine, etc.)

    All that said, if time permits I might try importing a throwaway copy of their project into 2019.3, with every expectation that some things will break, but just to see how the version control subsystem behaves.
     
  27. syscrusher

    syscrusher

    Joined:
    Jul 4, 2015
    Posts:
    1,104
    I'm willing to submit a bug report, of course, but I haven't done so because of the size of this project. Not counting all of the automatically-generated contents of Library, this project is circa 46 gig. I have, unfortunately, asymmetric Internet, with 54 Mbit down and 18 Mbit up, so uploading this thing would be extremely cumbersome. Would a bug report without a project replicating the bug still be useful?

    If I do submit a bug report, it may have to wait until the end of next week, because we're pushing toward a Dec 15 code milestone, and I can't take the time right now to re-enable the VC window and let it grind for an hour or two to generate logs.