[Windows] Issue with Nvidia NVS 810 and multiple displays

Hello!

I am seeing an issue with an Nvidia 810 GPU when using multiple displays. That card is made out of 2 separate GPUs and both are getting maxed out with the first output and have further slowdowns from each additional display.

I need to be able to output to 6 (+1) different screens, and it seems a bug in Unity is preventing me from doing so. I believe each screen is waiting from the previous one, causing some kind of feedback loop of

PostLateUpdate.PresentAfterDraw

and slowing down the game to a crawl.

Here are some info from the various debugging step I tried:

• Scene contents do not matter. An empty scene and a complex one have the same results.

• PostLateUpdate.PresentAfterDraw/Graphic.PresentAndSync/Device.Present

  represents 99.5% of rendering time and seemingly is the issue.

Hardware:

• GPU is up to date, tried multiple versions of the drivers
• Tried using different monitors and cables, no change.
• Slowness is directly impacted by the number of screens plugged in
• Issue does not happen on other GPUs I got to try.
• There is another graphics card plugged in to handle a 7th monitor, that handles the main rendering. I do not know how to change that for testing.

Software

• I tried enabling/disabling V-sync in both Unity and the Nvidia control Panel. No change.
  
• I tried both in editor and compiled build, development build and not. No change.
  
• Rendering multiple displays in the same output is fine, but as soon as the "Game" window is dragged to anotehr display, the slowdowns begin (main display is on another GPU, and does not have the same issue as the NVS 810)
• Disabling cameras on the additional screens impacts slowness as well.
• No difference between Forward and Deferred rendering paths.
• This is with the latest version of Unity on Windows 10.

I can run additional tests if needed. This is a very important issue and sadly I do not have control over hardware, so another GPU is not an option. I would really appreciate some help to figure this out.

Thanks!

 

Hi Tautvydas-Zilys, thanks for the quick reply!

Are they regular monitors or some kind of special displays?

Special displays I guess. They are in fact splitters that are emulating 1080p displays. Note that the issue does not exist in other applications not built in unity, and the displays are set up to request 30FPS.

What does the GPU usage look like in Unity Profiler or task manager?

100% in all of them, and same in the Nvidia profiler. The Unity GPU profiler show 99.5% of rendering time dedicated to

Device.Present

and clocks in at about 100ms per frame for an empty scene.

Are the monitors plugged in directly into the GPU?

The splitters are plugged in directly in the GPU using Nvidia-provided mini display port to HDMI adapters.

How are you making it draw on multiple displays?

Multiple cameras and a script that enables additional displays.


Thanks!

 

Hi Tautvydas,

Sorry about the late answer, I had some issues accessing the computer. Here are the answers to your questions:

How many monitors do you reach before the GPU usage gets to 100%? What's the frame rate at that point?

First monitor spikes the first GPU to 100 instantly. If I move the output to a screen plugged in to the second GPU (it's a single card with 2 gpus that have individual outputs), that one spikes instead. Unity reports about 30 FPS.


Does the same thing happen if you set background clear to solid color instead of skybox and use an empty scene?

Yes, I am already using solid color. Same thing happens for skybox, and even don't clear.

Here's the player log:

Code (CSharp):

1. GfxDevice: creating device client; threaded=1
2. Direct3D:
3.     Version:  Direct3D 11.0 [level 11.1]
4.     Renderer: NVIDIA Quadro P4000 (ID=0x1bb1)
5.     Vendor:  
6.     VRAM:     8089 MB
7.     Driver:   25.21.14.1881

Thank you once again for your help; I'll have access to the computer for the next couple hours so I'll be able to try thing or give you more info if needed.

 

Yes, this is the other graphics card I mentioned. The NVS810 is there to add outputs to the Quadro, which (theorically) handles rendering. That's what Nvidia suggested me to do.

 

The NVS. The Quadro barely any use. Since the NVS has two GPUs in one card, only the GPU that I output to using Unity get maxed out.

This happens when using the MultiDisplay options and also when just dragging a Game window to that screen in the editor.

 

Here's what the task manager looks like, with one game window in one of the NVS screens. The Quadro also has a game window and the editor and is fine.



What happens if you plug the displays into the Quadro (I believe it still has at least 3 outputs, right?) - does the lag go away?

Unfortunately I cannot test that as I don't have access to the machine's plugs. However, from the testing I was able to do before I'm fairly confident that the lag would go away. That being said I do need 7 outputs and cannot change the current configuration of the machine.

Thanks!

 

Interesting! Here's the graph with GPU 1 (the Quadro):



Here is with GPU 2:



There is an interesting drop in usage on GPU 2! Not sure what that means.

I tested with another 3d application, the same issue does not happen. GPU usage for the NVS raises only by a small amount on the card that's being used, as expected.

I also sent you the report privately. Thanks!

 

Hey there! I have tested without team viewer and got the same issue. I just happen to not have access to the device right now but the results are consistent with when I was testing hands on.

I have been testing both in Unity and outside, the sample I sent you was running the app standalone, but maybe something went wrong? It should not say Unity.exe as it was not running at the time. I'll try getting another fresh recording and look into Nsight or Pix.

 

Hi Tmpsrm and Tautvydas-Zilys.

I'm running into very similar issues. I have 6 monitors plugged in to a NVS810, and an empty scene gives me 99% GPU usage. Except that in my case I have also memory almost reaching 100%.
Were you guys able to solve this? Please help!

 

Yes Tautvydas, my app is using the GPU. NVS 810 actually has 2 stacked GPUs, so, maybe, as you say, the frame is rendered in one GPU and copied to the other one, causing some performance issues. I will try to use 4 monitors (only 1 GPU) and then I'll get back to you. Thanks for your super quick response.

 

Actually the most critical problem is the uneven framerate. The GPU profiler shows that RenderTexture.SetActive is the source of the problem, Even in and empty scene with no post-processing effects. Note that I am using HDPR.

 

Hey @sergiobd, were you able to find a solution? I'm gonna get back onto this project soon and I still haven't resolved it myself...

Thanks!

 

Hey all,
Not sure if anyone is still working on it half a year later but I've been trying to find more information about it but there is hardly any of it.

We've been trying to get 6 projectors (1920x1200) running for a project and figured out some stuff but we have questions as we're facing a project with 9x 4K projectors (I know wicked).

We tested with a system containing a RTX 2070 and a 1050Ti for additional outputs (Nvidia has max 4 displays per card) and we noticed as well with unity Display.displays[x].Activate(); as soon as you activate a screen on the other display the performance drops dramatically and the 1050TI was at 100%
So we got a new system with dual RTX 2070 super, thinking that 2 powerful card would do it without an issue but behold... same performance drop. (I almost quit my job over this).
So I tested a few scenarios
- 3 screens on 1 GPU on a basically empty project: ~1200 FPS
- 2 screens on 1 GPU and 1 screen on GPU2 on a basically empty project: ~100FPS
- 3 screens on 1 GPU running the deadwood demo: ~40FPS
- 2 screens on 1 GPU and 1 screen on GPU2 running the deadwood demo: ~40FPS

So it seems it wasn't simply GPU power as a heavy load did not suffer. So with GPU-Z tool running I saw a very particular thing. Once you activate a screen on the other GPU the Link-interface, a.k.a PCI-e bandwith was at a 100%
Our motherboard was a X570 without SLI support meaning a 16x and 4x slot. I brought my personal rig with dual 8x slots and behold instead of ~100FPS I got ~170FPS... not great but better. For some reason unity likes pumping data back and forth but not utilizing the other GPU.

Now one thing we didn't have the hardware for was a 16x + 16x slot or SLI (as we didn't had a NVlink) which would turn 2 GPU's into one (sort off) and maybe not have the same issue with PCI-e bandwith?

But we did try something else. Instead of fullscreen windows, we went with one borderless fullscreen, Aligned all 6 screen in a nice 3x2 grid (doing this in the Nvidia control panel is best) and then start the application via a .bat script at 5760x2400 resolution (all 6 projectors) and behold ~1000FPS and no PCI-e bandwith limitation. We even returned one of the 2070 supers and used our spare 1050TI and still it has the ~1000FPS

So what we've learned
- don't use Display.displays[x].Activate(); over multiple GPU's
- Unity only uses one GPU to render
What I'd still like to test
- Does SLI help with the two points mentioned, any difference between DX12 and Vulkan?
- Would a dual 5700 XT with both 16x @ PCI-e 4th gen make a difference?

So our next challenge is 9x 4K projectors. It will need at least 3 GPU's just for output but that massive resolution will be a bottleneck, especially if we're limited to one GPU having to do all the work.
Anyone have some idea's?

---------------------------
As a follow up test I activated DSR at 4x which means each projector is 3840x2400 and I'm running into issues again:
Having 3 projectors (11520 x 2400) on 1 GPU, empty project, gives about 700FPS
Dividing that onto 2 GPU's and it start to stutter, slipping back to ~25FPS about one a second

If I go on all 6 projectors with but only a canvas of 11520 x 2500, so just a 100 pixels on the 3 projectors on the 2nd GPU, and it's down to ~22FPS. PCI-e bandwith is about 85-95%.

 

But I find it very interesting that border less spanning is way less taxing than separate activated displays
So I tried 2020.2.13 alpha today and I also tried DX12 and Vulkan instead of DX11 (using Unity 2019.3.0f)

So same test as before with DSR on DX12:
Having 3 projectors (11520 x 2400) on 1 GPU, empty project, gives about the same 700+FPS
Dividing that onto 2 GPU's and it got better, it sometimes dropped more to 90 to 170 but way better that 25

going on all 6 projectors with but only a canvas of 11520 x 2500, so just a 100 pixels on the 3 projectors also got better with staying around 170FPS and dipping to about 80 every now and then

Going over to Vulcan with the 11520 x 2500 test and... ~650FPS solid...
Doing the entire 11520 x 4800 and still pushing 330FPS (hot day so reached thermal limits)
Keep in mind it's basically a blank canvas but it does show that it's not suffering from a lack of PCI-e bandwidth.

Now that I know that Vulkan does something magical that helps, maybe this also works in Unity 2019. And would the separate activated displays under Vulkan create a bottleneck again, have to test it I guess.
We're continuing to test, with real 4K screens instead of Nvidia DSR tricks, and a test with actual content. Maybe even add a 3rd graphics card for more pixels to push?