[Metal]can I edit Unity's "screen backbuffer"->color&depthBuffer->Load&StoreAction?

Hello, I am building rendering logic for an iOS game(target Metal+OpenGLES) using 2018.1.5f1's legacy render pipeline(not SRP, just old forward rendering path, the one that exist for many years), I need to make the game's rendering make use of iPhone's tiled-base rendering architecture(PowerVR / Apple GPU), which every RenderTexture's rendering should
-1.Load nothing from system memory to on chip memory (Load bathwidth = 0)
-2.Render everything for this RenderTexture inside on chip memory
-3.Store only resolved color from on chip memory to system memory (Store bandwidth = minimum), and discard depth&stencil&MSAA color buffer, so depth&stencil&MSAA color buffer don't even exist in system memory at all.
doing all of these, can reduce memory bandwidth, which reduce power consumption & heat.
*The memory bandwidth saving method, according to this talk(44:35 - 45:50):
https://developer.apple.com/videos/play/wwdc2015/610/

Before explaining what the problem I am facing is, the game's rendering logic are as follow,
//----------------------------------------------------------------------------------------------------------------
part (1):
first render all my RenderTextures one by one.(raw scene, bloom, motion vector......)
Every RenderTexture will call GL.Clear() before the first drawcall, to prevent Load. And RenderTexture.DiscardContent(false,true) right after their last drawcall & before next render target switch, to prevent Storing depth&stencil.

Code (csharp):

1.  
2. //1a: Switch renderTarget to myRenderTexture
3. RenderTexture.active = myRenderTexture;
4. someCamera.targetTexture = myRenderTexture;
5.  
6. //1b: this can hint Unity to generate Metal code that avoid LoadAction = Load,
7. //it is useful,LoadAction is Don'tCare/Clear, which means LoadBandwidth = 0, good!
8. //*camera with clear to solid color can do the same
9. GL.Clear(true,true, targetClearColor);
10.  
11. //1c: render everything for this RenderTexture
12. someCamera.Render();
13.  
14. //1d: then discard everything, other than color
15. myRenderTexture.DiscardContent(false,true);
16.  
17.  

*Why adding GL.Clear() & RenderTexture.DiscardContent()? I add them because of this:
https://community.arm.com/graphics/...rmance-2-how-to-correctly-handle-framebuffers

part (2):
now every RenderTexure has finished their render, it is time to render a full screen composite pass onto Unity's "screen backbuffer" ,
which use all my RenderTextures in part(1) as input.

Code (csharp):

1.  
2. //2a: Switch renderTarget to Unity's "screen backbuffer"
3. RenderTexture.active = null;
4.  
5. //2b: hope this can hint Unity to generate Metal code that avoid LoadAction = Load,
6. //but end up not useful,LoadAction is still Load, not Don'tCare/Clear
7. GL.Clear(true,true,new Color(0,0,0,0));
8.  
9. //2c: render my full screen composite pass to Unity's "screen backbuffer"
10. Graphics.DrawMeshNow(myFullScreenPlaneMesh,.......);
11.  
12. //2d: then render UI finally
13. UICamera.Render();
14.  
15. //for this frame,all my rendering code finished here
16.  

//----------------------------------------------------------------------------------------------------------------
In part (1), so far so good. In XCode metal frame debugger, I can see each RenderTexture now became a separate encoder,
which everyone of them:

-Load nothing from system memory to onchip memory
(LoadAction = Don'tCare/Clear) & (LoadBandwidth = 0MB)

-Store only resolved color from onchip memory to system memory
(StoreAction = Store for color attachment, StoreAction = Don'tCare for depth/stencil attachment) & (StoreBandwidth = minimum MB)

they all looks like this(just example image)

which is doing perfectly, exactly what I want!
//------------------------------------------------
The problem begins in part (2), which is the whole Unity's "screen backbuffer" pass, the one that I don't have explicit control.
In XCode metal frame debugger, I can confirm in part(2), unity triggered atleast
->color Load
->color+depth+stencil Store

this Unity's "screen backbuffer" pass (which exactly equals part(2)) look like this(just example image)

*as you can see, Load bandWidth is NOT 0!! And Store bandWidth is extremely high also

The problem is, I really don't need to Load anything from system memory back to on chip memory in part(2), I want this Unity's "screen backbuffer" pass(encoder) to act like all my previous encoders in part(1)(they all load nothing, LoadAction = Don'tCare/Clear .And store minimum color, StoreAction = store only resolved color)
//------------------------------------------------
so what I tried to do, is add the following code between 2(a) & 2(b)

Code (csharp):

1.  
2. RenderTexture.active = null; //switch to Unity's "screen backbuffer" RenderTexture
3. RenderBuffer colorBuffer = Graphics.activeColorBuffer; //I want to edit load/store action of this buffer = Unity's "screen backbuffer"->colorRenderBuffer
4. RenderBuffer depthBuffer = Graphics.activeDepthBuffer; //I want to edit load/store action of this buffer = Unity's "screen backbuffer"->depthRenderBuffer
5.  
6. //don't load anything
7. colorBuffer.m_LoadAction = RenderBufferLoadAction.DontCare;
8. depthBuffer.m_LoadAction = RenderBufferLoadAction.DontCare;
9.  
10. //store only resolved color,discard depth&stencil&MSAA color buffer
11. colorBuffer.m_StoreAction = RenderBufferLoadAction.Store;
12. depthBuffer.m_StoreAction = RenderBufferLoadAction.DontCare;
13.  

and hoping I can somehow make Unity create a correct encoder without all the unneeded LoadAction & StoreAction.
But I just can't access them(m_StoreAction,m_LoadAction) at all, RenderBuffer do not allow me to edit Load/Store actions, they are not public fields.
//------------------------------------------------
All I want to do is to reduce memory bandwidth, which reduce power consumption & heat a lot.
But I am stuck here, already knowing what the problem is, but have no idea how to solve it, so I need your help.

Question: Is there a way to edit Unity's "screen backbuffer"->color&depth buffer->Load&Store Action?

which the goal is to make Unity's "screen backbuffer" can do the following
-color&depth's LoadAction = Don'tCare
-color's StoreAction = Store
-depth's StoreAction = Don'tCare

thanks!

 

Thanks for the reply!
I tried your suggestion, but still I can't reduce frame's total LoadBandwidth to 0 MB.
//--------------------------------------------------------------------
I post the code here to better explain what I tried:

Code (CSharp):

1. //Do single frame's render logic, from (A) to (E)
2. //(A) to (C) is doing good(LoadBandwidth is 0), only (D) is the problem part
3. using UnityEngine;
4. using UnityEngine.Rendering;
5.  
6. [RequireComponent(typeof(Camera))]
7. public class MasterRenderLoop: MonoBehaviour
8. {
9.     void OnPreRender()
10.     {
11.         //(A):call cameras to render to lots of small RenderTextures
12.         //which will be used by (B)
13.         //-shadow map RenderTexture
14.         //-planar reflection RenderTexture
15.         //-depth RenderTexture
16.     }
17.  
18.     //(B): do the regular 3D scene rendering to a RenderTexture using RenderTextures result from (A)
19.  
20.     void OnPostRender()
21.     {
22.         //(C): render other RenderTextures for (D)'s postprocess usage, which use (B)'s result as input
23.         //-bloom RenderTexture (regular threshold then blur method)
24.  
25.         //-----------------------------------------------------------------------
26.         //*before this line, LoadBandwidth is still 0, which is good.
27.         //-----------------------------------------------------------------------
28.  
29.         //(D): finally switch rendering to Unity's screen backbuffer
30.         SwitchToRenderingOnUnityScreenBufferRT();
31.  
32.         //and draw the full screen composite drawcall
33.         Graphics.DrawMeshNow(fullScreenMesh, Matrix4x4.identity);
34.         //-----------------------------------------------------------------------
35.         //(E): also render the UI camera at last, still onto Unity's screen backbuffer
36.         //UICamera.Render();
37.  
38.         //[finish all my rendering code in this frame]
39.     }
40.     void SwitchToRenderingOnUnityScreenBufferRT()
41.     {
42.         //--------------------------------------------------------------------------
43.         //without this line,
44.         //I can't use Graphics.activeColorBuffer & Graphics.activeDepthBuffer to get Unity's screen back buffer's content
45.         RenderTexture.active = null;
46.         //--------------------------------------------------------------------------
47.         //we want to render to Unity's screen backbuffer, just like RenderTexture.active = null;
48.         //so extract Unity's screen backbuffer's color&depth buffer to be used by Graphics.SetRenderTarget()
49.         RenderBuffer unityScreenBackBufferColorBuffer = Graphics.activeColorBuffer;
50.         RenderBuffer unityScreenBackBufferDepthBuffer = Graphics.activeDepthBuffer;
51.  
52.  
53.         RenderTargetSetup rtSetup = new RenderTargetSetup();
54.         rtSetup.color = new RenderBuffer[] { unityScreenBackBufferColorBuffer };
55.         rtSetup.colorLoad = new RenderBufferLoadAction[] { RenderBufferLoadAction.DontCare };
56.         rtSetup.colorStore = new RenderBufferStoreAction[] { RenderBufferStoreAction.Store };
57.         rtSetup.cubemapFace = CubemapFace.Unknown;
58.         rtSetup.depth = unityScreenBackBufferDepthBuffer;
59.         rtSetup.depthLoad = RenderBufferLoadAction.DontCare;
60.         rtSetup.depthSlice = 1;
61.         rtSetup.depthStore = RenderBufferStoreAction.DontCare;
62.         rtSetup.mipLevel = 0;
63.         Graphics.SetRenderTarget(rtSetup);
64.     }
65. }
66.  

//--------------------------------------------------------------
Your suggestion did change the behavior of (D) & (E) in XCode frame debugger

from (using RenderTexture.active = null; to do renderTargetSwitch):
-single [Load/Store] encoder (it draws postprocess full screen mesh, then draws UI together in the same encoder)

to (using Graphics.SetRenderTarget() to do renderTargetSwitch):
-one extra [Clear/Store] encoder(it only draws postprocess full screen mesh)
-and then one [Load the same attachment we just store/Store] encoder(it only draws UI)

while the target is:
-single [Don'tCare/Store] encoder (it draws postprocess full screen mesh, then draws UI together in the same encoder)

net change = add 1 more encoder, LoadBandwidth reduced(still not 0),but StoreBandwidth increased

not sure if I do something wrong, happy to see this net change, but I am stuck again, I want to know how to reduce frame's total LoadBandwidth to 0 (all my RenderTextures are doing great, just the Unity's screenbackbuffer is doing unneeded Load/Store)

 

so it is actually safe to cache the renderbuffer @ start once,and use them for all later frames?
thats good news!
I will try that and not calling RenderTexture.active again in the render loop.
//------------------------------------------------------------------------------------------------------------
I will tell you the result later. thank you for the help, your information are really useful!

 

According to your info, I just switch from "RenderTexture.active = null" to Graphics.SetRenderTarget(rtSetup), it works perfectly in Editor(not yet check the LoadBandwidth in XCode).
//---------------------------------------------------------------------------------------------------------------------------------------------------

Since it is unsafe to cache Unity's screenbackbuffer's colorbuffer&depthbuff @ start,
How should I correctly get the lastest "Unity's screen backbuffer->colorbuffer&depthbuffer" per frame, without calling "RenderTexture.active = null" per frame(which may trigger unneeded renderTargetSwitch)?

I only know this "lines 42-50" method(but this need to call RenderTexture.active = null to update Graphics.activeColorBuffer & activeDepthBuffer per frame)

1. RenderTexture.active = null;
2. RenderBuffer unityScreenBackBufferColorBuffer = Graphics.activeColorBuffer;
3. RenderBuffer unityScreenBackBufferDepthBuffer = Graphics.activeDepthBuffer;

//---------------------------------------------------------------------------------------------------------------------------
I tried to get the lastest "Unity's screen backbuffer->colorbuffer&depthbuffer" in OnPreCull(), this will be before any of my Off screen RenderTexture's rendering, so I can assume the RenderTexture.active is null at this moment. (this method works in Editor without problem, RenderTexture.active is always == null in OnPreCull(), before my offscreen render target start rendering)

 

I wrote the minimum code that can always reproduce the problem I am facing,
If you open a new scene and attach this script to camera, then the problem can be reproduced.
(can check LoadStoreBandwidth in XCode frame debugger if you build IOS)

the goal is to write render code that can do simple postprocess, with minimum Load/Store Bandwidth usage by Unity's screen backbuffer.

//--------------------------------------------
if anyone know the solution to this, please help me!

Code (CSharp):

1. //this is the minimum code that reproduce the problem for
2. //https://forum.unity.com/threads/metal-can-i-edit-unitys-screen-backbuffer-color-depthbuffer-load-storeaction.554449/
3.  
4. //How to use this code:
5. //1.open a new scene
6. //2.attach this script to camera
7. //3.save scene
8. //4.build scene to XCode
9. //5.run this scene in XCode on real device(need A8 or up deivce with Metal support natively)(e.g. iPhone6 / iPhone6+)
10. //6.capture a frame using XCode frame debugger(metal frame debugger)
11. //-----------------------------------------------------------------------------
12. //what to check in XCode frame debugger:
13. //a.check "Frame statistic", which shows the total Load&Store Bandwidth of this frame
14. //b.check also each "Encoder statistic", which show each encoder's Load&Store Bandwidth, number of encoder varys from 1 to 3(if you click different setting when game is running)
15. //c.check also each Encoder's "RenderPassDescriptor", which shows the most important target setting of this test code, the LoadAction & StoreAction
16. //-----------------------------------------------------------------------------
17. //What is the final goal:
18. //I want to write code that can,
19.  
20. //(RenderPass1). render 3D scene to a RenderTexture, which has the following setting in this encoder's renderPassDescriptor
21. //-------[LoadAction/StoreAction]
22.  
23. //color  [Clear/MultiSampleResolve]
24. //depth  [Clear/DontCare]
25. //stencil[Clear/DontCare]
26.  
27. //(RenderPass2). then just blit the RenderTexture from (RenderPass1) to Unity's screen back buffer, which has the following setting in this encoder's renderPassDescriptor
28. //color  [DontCare/Store]
29. //depth  [Clear/DontCare]
30. //stencil[Clear/DontCare]
31.  
32. //This code can now generate (RenderPass1) correctly, but fail to generate correct (RenderPass2)
33. //-----------------------------------------------------------------------------
34. //Why writting this code:
35. //in RenderPass(2), no matter what code I tried, I can't make Unity output the minimum Load/Store setting
36.  
37. //The best I can do in (RenderPass2) is,
38. //color  [Clear/Store] ....clear is not needed, because we will Graphics.Blit() the full screen anyway
39. //depth  [Clear/Store] ....no need the store
40. //stencil[Clear/Store] ....no need the store
41.  
42. //which can be reproduced this in the following 2 cases,
43. //-enableRenderLogic == false (just 1 Encoder generated)
44. //-enableRenderLogic == true, useAleksandrkNewMethod == false, shouldClearUnityScreenbackBufferBeforeDraw = true;(2 Encoder generated)
45.  
46. //and in all other case, Unity will generate even worst setting
47. //color  [Load/Store]
48. //depth  [Load/Store]
49. //stencil[Load/Store]
50. //----------------------------------------------------
51. using UnityEngine;
52. using UnityEngine.Assertions;
53.  
54. public class RenderLogicTest : MonoBehaviour
55. {
56.     bool enableRenderLogic = true;//if set to false, turn off every rendering logic
57.     bool useAleksandrkNewMethod = false;//if set to true, use aleksandrk's new Graphics.SetRenderTarget method for unity's screen backbuffer, instead of RenderTexture.active = null
58.     bool shouldClearUnityScreenbackBufferBeforeDraw = true;//if set to true, call GL.Clear() right before drawing composite full screen quad
59.  
60.     RenderTexture myRenderTexture;
61.     RenderBuffer unityScreenBackBuffer_ColorBuffer, unityScreenBackBuffer_DepthBuffer;
62.  
63.     private void OnPreRender()
64.     {
65.         if (!enableRenderLogic)
66.             return;
67.         //=========================================================================================
68.         Assert.IsTrue(RenderTexture.active == null);
69.         //record RenderBuffer per frame, for later use in OnPostRender()
70.         unityScreenBackBuffer_ColorBuffer = Graphics.activeColorBuffer;
71.         unityScreenBackBuffer_DepthBuffer = Graphics.activeDepthBuffer;
72.  
73.         //render 3D scene to a 720p RenderTexture first
74.         myRenderTexture = RenderTexture.GetTemporary(1280,720,16+8, getSafeRGB565(), RenderTextureReadWrite.Default,4,RenderTextureMemoryless.MSAA,VRTextureUsage.None,false);
75.  
76.         //------------------------------------------------------------------------------------
77.         //render target switch start (from Unity's screen backbuffer to mainRT)
78.         GetComponent<Camera>().SetTargetBuffers(myRenderTexture.colorBuffer, myRenderTexture.depthBuffer);
79.    
80.         RenderTargetSetup rtSetup = new RenderTargetSetup();
81.         rtSetup.color = new RenderBuffer[] { myRenderTexture.colorBuffer };
82.         rtSetup.colorLoad = new UnityEngine.Rendering.RenderBufferLoadAction[] { UnityEngine.Rendering.RenderBufferLoadAction.DontCare };
83.         rtSetup.colorStore = new UnityEngine.Rendering.RenderBufferStoreAction[] { UnityEngine.Rendering.RenderBufferStoreAction.Store };
84.         rtSetup.cubemapFace = CubemapFace.Unknown;
85.         rtSetup.depth = myRenderTexture.depthBuffer;
86.         rtSetup.depthLoad = UnityEngine.Rendering.RenderBufferLoadAction.DontCare;
87.         rtSetup.depthSlice = 1;
88.         rtSetup.depthStore = UnityEngine.Rendering.RenderBufferStoreAction.DontCare;
89.         rtSetup.mipLevel = 0;
90.         Graphics.SetRenderTarget(rtSetup);
91.         //render target switch end
92.         //------------------------------------------------------------------------------------
93.         GetComponent<Camera>().clearFlags = CameraClearFlags.SolidColor; //clear before draw, to hint LoadAction = Clear for myRenderTexture
94.     }
95.  
96.     private void OnPostRender()
97.     {
98.         if (!enableRenderLogic)
99.             return;
100.         //=========================================================================================
101.  
102.         ////call this when ZTest&ZWrite are not needed anymore asap, but keep color for MSAAresolve->later sampling
103.         //to hint depth&stencil's StoreAction = DontCare for myRenderTexture
104.         myRenderTexture.DiscardContents(false,true);
105.  
106.         //-----------------------------------------------------------------------------------------------------------------------------------------------------------------
107.         //render target switch start (switch from myRenderTexture to Unity's screen backbuffer)
108.  
109.         //use the RenderBuffer we recorded in OnPreRender(), this should point to latest Unity's screen backbuffer
110.         GetComponent<Camera>().SetTargetBuffers(unityScreenBackBuffer_ColorBuffer,unityScreenBackBuffer_DepthBuffer);
111.  
112.         if (useAleksandrkNewMethod)
113.         {
114.             RenderTargetSetup rtSetup = new RenderTargetSetup();
115.             rtSetup.color = new RenderBuffer[] { unityScreenBackBuffer_ColorBuffer };
116.             rtSetup.colorLoad = new UnityEngine.Rendering.RenderBufferLoadAction[] { UnityEngine.Rendering.RenderBufferLoadAction.DontCare };
117.             rtSetup.colorStore = new UnityEngine.Rendering.RenderBufferStoreAction[] { UnityEngine.Rendering.RenderBufferStoreAction.Store };
118.             rtSetup.cubemapFace = CubemapFace.Unknown;
119.             rtSetup.depth = unityScreenBackBuffer_DepthBuffer;
120.             rtSetup.depthLoad = UnityEngine.Rendering.RenderBufferLoadAction.DontCare;
121.             rtSetup.depthSlice = 1;
122.             rtSetup.depthStore = UnityEngine.Rendering.RenderBufferStoreAction.DontCare;
123.             rtSetup.mipLevel = 0;
124.             Graphics.SetRenderTarget(rtSetup);
125.         }
126.         else
127.         {
128.             RenderTexture.active = null;
129.         }
130.         //render target switch end
131.         //-----------------------------------------------------------------------------------------------------------------------------------------------------------------
132.         if (shouldClearUnityScreenbackBufferBeforeDraw)
133.             GL.Clear(true, true, new Color(0,0,0,0));
134.  
135.         //act as the post process composite pass, no material defined here, because it will not affect reproducing the LoadStoreBandwidth problem
136.         Graphics.Blit(myRenderTexture, RenderTexture.active);
137.    
138.         RenderTexture.ReleaseTemporary(myRenderTexture);
139.         //[all rendering code finished here]
140.     }
141.  
142.     RenderTextureFormat getSafeRGB565()
143.     {
144.         if (SystemInfo.SupportsRenderTextureFormat(RenderTextureFormat.RGB565))
145.             return RenderTextureFormat.RGB565;
146.  
147.         return RenderTextureFormat.Default;
148.     }
149.  
150.     //just for reproduce the problem in different setting, to test Load/Store Bandwidth easily with XCode frame debugger
151.     private void OnGUI()
152.     {
153.         if (GUI.Button(new Rect(100, 100, 600, 50), enableRenderLogic ? "now running renderlogic" : "now running nothing"))
154.         {
155.             enableRenderLogic = !enableRenderLogic;
156.         }
157.         if (!enableRenderLogic)
158.             return;
159.  
160.         //--------------------------------------------
161.         if (GUI.Button(new Rect(200, 200, 550, 30), useAleksandrkNewMethod ? "now running Graphics.setRT()" : "now running RT.active=null"))
162.         {
163.             useAleksandrkNewMethod = !useAleksandrkNewMethod;
164.         }
165.         if (GUI.Button(new Rect(200, 300, 550, 30), shouldClearUnityScreenbackBufferBeforeDraw ? "now calling GL.clear" : "now NOT calling GL.clear"))
166.         {
167.             shouldClearUnityScreenbackBufferBeforeDraw = !shouldClearUnityScreenbackBufferBeforeDraw;
168.         }
169.     }
170. }
171.  

 

The above code will generate 2 encoders by default, which can be reproduced by

set enableRenderLogic = true
set useAleksandrkNewMethod = false
set shouldClearUnityScreenbackBufferBeforeDraw = true

with the above default setting(3 booleans),
the game will generate 2 RenderPass.
(RenderPass1, myRenderTexture) = this image, which has the perfect minimum LoadStoreAction already, the problem is not here

//---------------------------------------------------------
and (RenderPass2, unity's screen backbuffer) = this image, which depth&stencil DONT need to store, the problem is in this RenderPass

//-----------------------------------------------------
the ideal setting of RenderPass2 should be:
Color = [DontCare/Store]
Depth = [Clear/DontCare]
Stencil = [Clear/DontCare]

but no matter what code I wrote, I can never generate that ideal setting for RenderPass2
//-------------------------------------------------------------------------
additionally, I tried to "replace RenderTexture.active by Graphics.SetRenderTarget", but I fail to use Graphics.SetRenderTarget properly for (RenderPass2), it always generate (RenderPass2) with really bad LoadStoreAction.

you can try this by setting useAleksandrkNewMethod to true when the game is running,
not sure how this should be used correctly, any example code are welcome

 

the problem should be similar to this active bug report:
https://issuetracker.unity3d.com/is...he-system-frame-buffer-could-not-be-discarded

there are no way to discard unity screen backbuffer's depth&stencil, atleast in metal.

 

Build the current game with the lastest XCode (2018-9-20, XCode 10), only unity's screen backbuffer is producing the XCode purple warning, every other offscreen rendertargets(~10 RT) created by user(me) don't have XCode purple warning.

above image produced by Unity 2018.1.5.1f, can be reproduced with iPhone6+,iPhone6S,iPhoneX,iPadAir2 (all my iOS devices)

 

Thankyou, I will try the latest unity and using the display class to get the “backbuffer”. If it still can’t solve the load store problem in Xcode build, I will create a complete bug report that contains separate scene with no buttons, just click play and ready for test(with text about what is expected & what is actually happening).

 

@Alexey - how can we achieve this for a memoryless MSAA back buffer?

I've used your command buffer code above and set the store action to Resolve and modified the MetalHelper.mm file to set MSAA texture to MTLStorageModeMemoryless and it's resolve texture to MTLStorageModeShared.

https://imgur.com/a/XlT5Cey

Still doesn't seem to work though. Store action is still store.

See 20 mins in on this video https://developer.apple.com/videos/play/wwdc2019/606/ for what I'm trying to do.

I could reduce bandwidth a lot if I could get this working.

Cheers

Edit: I'm asking this same question in this thread as it is more appropriate

 

I use a skybox. Should I attach this script and just leave Clear Skybox active on the camera? Does it make things faster on ion/android?

 

Any update here @Alexey on @johnfrog 's problem that doesn't require editing the iOS Unity source files?