Support for UI elements in RenderToCubemap

I know this is not presently possible, but I would like to see an option in the RenderToCubemap() function that would allow you to capture UI elements. I use this function for non reflection probe based work, and process cubemap data to create 360 equirectangular images.

I dont want to open a BUG ticket in Unity, so I thought maybe someone on the graphics team at Unity could see this.

Ideally, I would want to have the UI captured in the world forward camera direction for worldspace items.

Thanks

 

Could also submit a feedback request here
https://feedback.unity3d.com/

 

As the bug is still not tackled, I added a feedback request: https://feedback.unity3d.com/suggestions/support-for-world-space-ui-elements-in-rendertocubemap Feel free to vote for it!

 

I ran into the same problem trying to render a world-space UI to a cubemap in 2017.2. I ended up throwing together a script to manually draw all UI elements in the scene. I couldn't find any documentation on how the UI system does its rendering, so don't expect this to work if you're using non-standard UI components. It's also pretty slow and uses some hacks that might not work on other versions.

Code (csharp):

1.  
2. using System.Linq;
3. using UnityEngine;
4. using UnityEngine.Rendering;
5. using UnityEngine.UI;
6.  
7. /// <summary>
8. /// Component that manually renders world-space UI Graphics to a Camera. Intended for
9. /// use with Camera.RenderToCubemap which seems to ignore the built-in UI system
10. /// </summary>
11. [RequireComponent(typeof(Camera))]
12. public class ManualUIRenderer : MonoBehaviour
13. {
14.     private static readonly int MainTexProperty = Shader.PropertyToID("_MainTex");
15.     private static readonly int TextureSampleAddProperty = Shader.PropertyToID("_TextureSampleAdd");
16.     private static readonly int ColorProperty = Shader.PropertyToID("_Color");
17.  
18.     private Camera targetCamera;
19.     private CommandBuffer commandBuffer;
20.  
21.     private void Awake()
22.     {
23.         this.targetCamera = this.GetComponent<Camera>();
24.  
25.         this.commandBuffer = new CommandBuffer();
26.         this.commandBuffer.name = "Manual UI rendering";
27.         this.targetCamera.AddCommandBuffer(CameraEvent.AfterSkybox, commandBuffer);
28.     }
29.  
30.     private void OnDisable()
31.     {
32.         this.commandBuffer.Clear();
33.     }
34.  
35.     private void Update()
36.     {
37.         this.commandBuffer.Clear();
38.         AddUiDrawingCommands(this.targetCamera, this.commandBuffer);
39.     }
40.  
41.     private static void AddUiDrawingCommands(Camera cam, CommandBuffer buffer)
42.     {
43.         // Root canvases ordered by screen space depth
44.         var rootCanvases = FindObjectsOfType<Canvas>()
45.             .Where(canvas => canvas.isRootCanvas && canvas.renderMode == RenderMode.WorldSpace)
46.             .OrderByDescending(canvas => cam.WorldToScreenPoint(canvas.transform.position).z);
47.  
48.         foreach (var canvas in rootCanvases)
49.         {
50.             // Graphics after culling sorted by depth
51.             var graphics = canvas.GetComponentsInChildren<Graphic>()
52.                 .Where(graphic => TestCullingMask(graphic, cam.cullingMask))
53.                 .OrderBy(graphic => graphic.depth);
54.  
55.             foreach (Graphic graphic in graphics)
56.             {
57.                 AddGraphicDrawingCommands(graphic, buffer);
58.             }
59.         }
60.     }
61.  
62.     private static bool TestCullingMask(Graphic graphic, int cullingMask)
63.     {
64.         return ((1 << graphic.gameObject.layer) & cullingMask) != 0;
65.     }
66.  
67.     private static void AddGraphicDrawingCommands(Graphic graphic, CommandBuffer buffer)
68.     {
69.         // Probably not needed, but let's call it anyway
70.         graphic.Rebuild(CanvasUpdate.PreRender);
71.  
72.         // Determine effective alpha from CanvasGroups (probably not how Unity does this)
73.         float effectiveAlpha = graphic.GetComponentsInParent<CanvasGroup>()
74.             .Aggregate(1f, (alpha, group) => alpha * group.alpha);
75.  
76.         var material = new Material(graphic.materialForRendering); // material with IMaterialModifiers already applied
77.         material.SetTexture(MainTexProperty, graphic.mainTexture);
78.         material.SetColor(ColorProperty, graphic.color * new Color(1f, 1f, 1f, effectiveAlpha));
79.         if (graphic is Text)
80.         {
81.             // Not sure how/when Unity decides to set _TextureSampleAdd, but Text is
82.             // the only component I've encountered that needs it
83.             material.SetVector(TextureSampleAddProperty, new Vector3(1, 1, 1));
84.         }
85.  
86.         var mesh = new Mesh();
87.         // Call protected member Graphic.OnPopulateMesh through reflection to
88.         // populate the mesh. Probably really slow and skipping quite a few steps
89.         // in Unity's UI render pipeline, but it seems to work
90.         using (VertexHelper vh = new VertexHelper())
91.         {
92.             graphic.GetType().InvokeMember("OnPopulateMesh",
93.                 System.Reflection.BindingFlags.Instance |
94.                 System.Reflection.BindingFlags.InvokeMethod |
95.                 System.Reflection.BindingFlags.NonPublic,
96.                 null, graphic, new object[] { vh });
97.  
98.             vh.FillMesh(mesh);
99.         }
100.  
101.         buffer.DrawMesh(mesh, graphic.rectTransform.localToWorldMatrix, material);
102.     }
103. }
104.  

Attach the component to the camera you're using for RenderToCubemap and you should be good to go.

Edit: updated to work with CanvasGroup alpha

 

Not working anymore with Unity 2018.2 any temporary fixes?

 

I just did a quick test with 2018.2 and it's still working here. What kind of components are you using in your UI? I only needed the bare basics for my purposes (basically only

Text

and

Image

), so it's possible this code never worked with your UI, even before 2018.2. For example, I know TextMesh Pro components don't work right now, but I'm not using those in my project so I haven't looked into why that is.

 

I am using the same with Unity Recorder to record a 360 video which uses targetCamera.RenderToCubemap(m_Cubemap1, 63, Camera.MonoOrStereoscopicEye.Left); to capture the cubemap in Camera360Input.cs but I am unable to get UI in the output. I am only using basic UI Text and Image.

 

I'm not sure what it could be then. It's possible the images are being rendered but at the wrong scale/position; like I said it's a very hacky solution. I attached my working test project in case you want to investigate further.

 

Thanks got it working in a fresh project.

 

In order to get the text to appear using these scripts, I had to change

Code (CSharp):

1.  this.targetCamera.AddCommandBuffer(CameraEvent.AfterSkybox, commandBuffer);

to:

Code (CSharp):

1. this.targetCamera.AddCommandBuffer(CameraEvent.AfterEverything, commandBuffer);

After that, text appeared! Thanks so much!

 

Has anyone here reported a bug about this? If yes, could you please post the case number here?
Also tagging @phil_lira for URP support of this and @LeonhardP as this is a problem in current alpha/beta still.

 

Doesn't work with TextMeshPro UI

 

Thanks for the script... I tested it U2021.1.16f1 and works for normal UI text but not TMP Pro. TMP Pro readers a solid block. So this is perfect for rendering normal text.

Result:
The red rectangle is TMP Pro
The Top text above the Cube is normal UI Text

 

Would love it if Unity can fix this because it is something I intend to use.

Does any know if any of the commercial assets work properly with 360 renders?

 

If anyone stumbles across this looking for a fix for TextMeshProUGUI elements (Unsure about TextMeshPro mesh text) I was able to modify Mnstrspeed's script with the following to get it to render correctly:

var mesh = new Mesh();



// Call protected member Graphic.OnPopulateMesh through reflection to

// populate the mesh. Probably really slow and skipping quite a few steps

// in Unity's UI render pipeline, but it seems to work

using (VertexHelper vh = new VertexHelper())

{

    graphic.GetType().InvokeMember("OnPopulateMesh",

        System.Reflection.BindingFlags.Instance |

        System.Reflection.BindingFlags.InvokeMethod |

        System.Reflection.BindingFlags.NonPublic,

        null, graphic, new object[] { vh });



    vh.FillMesh(mesh);

}



if (graphic is TextMeshProUGUI)

{

    var textMeshProUGUI = graphic as TextMeshProUGUI;

    mesh = textMeshProUGUI.mesh;

    material = textMeshProUGUI.fontMaterial;

}



buffer.DrawMesh(mesh, graphic.rectTransform.localToWorldMatrix, material);

 

UI Buttons that are tinted black are showing up white... Perhaps the button tints aren't rendered?


Edit:

For anyone else running into this issue, I Modfied Mnstrspeed's script with jumpgate_lewis's modification and then further modified it to resolve the above, I did also have to make a second script to keep track of UnityEngine.UI.Selectables selection states, because their selection state is protected, and i didn't want to replace button with a class that inherits from button or selectable.

My SelectableStateTracker:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using UnityEngine.EventSystems;
5. using UnityEngine.UI;
6.  
7. /// <summary>
8. /// Unitys default buttons in the normal UI  has selectionstate as a protected variable, can't check it to work around tint not showin in video correctly. so alternatively i track it here.
9. /// </summary>
10. public class SelectableStateTracker : MonoBehaviour, IPointerDownHandler,IPointerEnterHandler,IPointerExitHandler,IPointerUpHandler,ISelectHandler,IDeselectHandler
11. {
12.  
13.     private bool hovered = false;
14.     private bool pressed = false;
15.     private bool selected = false;
16.     public Selectable selectable;
17.  
18.     private void Awake()
19.     {
20.         selectable = GetComponent<Selectable>();
21.  
22.     }
23.  
24.     public enum SelectionState {highlighted,normal,pressed,disabled,selected};
25.  
26.     public SelectionState CurrentSelectionState = SelectionState.normal;
27.  
28.     public void OnPointerDown(PointerEventData eventData)
29.     {
30.         pressed = true;
31.     }
32.  
33.     public void OnPointerEnter(PointerEventData eventData)
34.     {
35.         hovered = true;
36.    
37.     }
38.  
39.     public void OnPointerExit(PointerEventData eventData)
40.     {
41.         hovered = false;
42.    
43.     }
44.  
45.     public void OnPointerUp(PointerEventData eventData)
46.     {
47.         pressed = false;
48.  
49.     }
50.  
51.     public SelectionState evaluate()
52.     {
53.         if (!selectable.interactable)
54.         {
55.             return SelectionState.disabled;
56.         }
57.      
58.  
59.         return (hovered, pressed,selected) switch
60.         {
61.  
62.             (false, false,false) => SelectionState.normal,
63.             (true, false, false) => SelectionState.highlighted,
64.             (false, true, false) => SelectionState.pressed,
65.             (false, true, true) => SelectionState.pressed,
66.             (true, true, false) => SelectionState.pressed,
67.             (true, true, true) => SelectionState.pressed,
68.             _ => SelectionState.selected
69.         };
70.     }
71.  
72.     public void OnSelect(BaseEventData eventData)
73.     {
74.         selected = true;
75.     }
76.  
77.     public void OnDeselect(BaseEventData eventData)
78.     {
79.         selected = false;
80.     }
81. }
82.  

And The Modified version of the ManualUIRenderer

Code (CSharp):

1.  
2. using System.Linq;
3. using TMPro;
4. using UnityEngine;
5. using UnityEngine.EventSystems;
6. using UnityEngine.Rendering;
7. using UnityEngine.UI;
8.  
9. /// <summary>
10. /// Component that manually renders world-space UI Graphics to a Camera. Intended for
11. /// use with Camera.RenderToCubemap which seems to ignore the built-in UI system
12. /// </summary>
13. [RequireComponent(typeof(Camera))]
14. public class ManualUIRenderer : MonoBehaviour
15. {
16.     private static readonly int MainTexProperty = Shader.PropertyToID("_MainTex");
17.     private static readonly int TextureSampleAddProperty = Shader.PropertyToID("_TextureSampleAdd");
18.     private static readonly int ColorProperty = Shader.PropertyToID("_Color");
19.  
20.     private Camera targetCamera;
21.     private CommandBuffer commandBuffer;
22.  
23.    
24.  
25.     private void Awake()
26.     {
27.         this.targetCamera = this.GetComponent<Camera>();
28.  
29.         this.commandBuffer = new CommandBuffer();
30.         this.commandBuffer.name = "Manual UI rendering";
31.         this.targetCamera.AddCommandBuffer(CameraEvent.AfterSkybox, commandBuffer);
32.         //this.targetCamera.AddCommandBuffer(CameraEvent.AfterEverything, commandBuffer);
33.  
34.        
35.  
36.     }
37.  
38.     private void OnDisable()
39.     {
40.         this.commandBuffer.Clear();
41.     }
42.  
43.     private void Update()
44.     {
45.         this.commandBuffer.Clear();
46.         AddUiDrawingCommands(this.targetCamera, this.commandBuffer);
47.     }
48.  
49.     private static void AddUiDrawingCommands(Camera cam, CommandBuffer buffer)
50.     {
51.         // Root canvases ordered by screen space depth
52.         var rootCanvases = FindObjectsOfType<Canvas>()
53.             .Where(canvas => canvas.isRootCanvas && canvas.renderMode == RenderMode.WorldSpace)
54.             .OrderByDescending(canvas => cam.WorldToScreenPoint(canvas.transform.position).z);
55.  
56.         foreach (var canvas in rootCanvases)
57.         {
58.             // Graphics after culling sorted by depth
59.             var graphics = canvas.GetComponentsInChildren<Graphic>()
60.                 .Where(graphic => TestCullingMask(graphic, cam.cullingMask))
61.                 .OrderBy(graphic => graphic.depth);
62.  
63.             foreach (Graphic graphic in graphics)
64.             {
65.                 AddGraphicDrawingCommands(graphic, buffer);
66.             }
67.         }
68.     }
69.  
70.     private static bool TestCullingMask(Graphic graphic, int cullingMask)
71.     {
72.         return ((1 << graphic.gameObject.layer) & cullingMask) != 0;
73.     }
74.  
75.     private static void AddGraphicDrawingCommands(Graphic graphic, CommandBuffer buffer)
76.     {
77.         // Probably not needed, but let's call it anyway
78.         graphic.Rebuild(CanvasUpdate.PreRender);
79.  
80.         // Determine effective alpha from CanvasGroups (probably not how Unity does this)
81.         float effectiveAlpha = graphic.GetComponentsInParent<CanvasGroup>()
82.             .Aggregate(1f, (alpha, group) => alpha * group.alpha);
83.  
84.         var material = new Material(graphic.materialForRendering); // material with IMaterialModifiers already applied
85.         material.SetTexture(MainTexProperty, graphic.mainTexture);
86.         material.SetColor(ColorProperty, graphic.color * new Color(1f, 1f, 1f, effectiveAlpha));
87.  
88.         if(graphic.GetComponent<SelectableStateTracker>() != null)
89.         {
90.             SelectableStateTracker s = graphic.GetComponent<SelectableStateTracker>();
91.             if (s.selectable.transition == Selectable.Transition.ColorTint)
92.             {
93.                 SelectableStateTracker.SelectionState state = s.evaluate();
94.  
95.                 Color c = (state) switch
96.                 {
97.                     SelectableStateTracker.SelectionState.highlighted => s.selectable.colors.highlightedColor,
98.                     SelectableStateTracker.SelectionState.normal => s.selectable.colors.normalColor,
99.                     SelectableStateTracker.SelectionState.pressed => s.selectable.colors.pressedColor,
100.                     SelectableStateTracker.SelectionState.disabled => s.selectable.colors.disabledColor,
101.                     SelectableStateTracker.SelectionState.selected => s.selectable.colors.selectedColor,
102.                     _ => s.selectable.colors.normalColor
103.                 };
104.  
105.  
106.                 material.SetColor(ColorProperty,c * new Color(1f, 1f, 1f, effectiveAlpha));
107.             }
108.  
109.         }
110.  
111.         if (graphic is Text)
112.         {
113.             // Not sure how/when Unity decides to set _TextureSampleAdd, but Text is
114.             // the only component I've encountered that needs it
115.             material.SetVector(TextureSampleAddProperty, new Vector3(1, 1, 1));
116.         }
117.  
118.         var mesh = new Mesh();
119.  
120.  
121.  
122.         // Call protected member Graphic.OnPopulateMesh through reflection to
123.  
124.         // populate the mesh. Probably really slow and skipping quite a few steps
125.  
126.         // in Unity's UI render pipeline, but it seems to work
127.  
128.         using (VertexHelper vh = new VertexHelper())
129.  
130.         {
131.  
132.             graphic.GetType().InvokeMember("OnPopulateMesh",
133.  
134.                 System.Reflection.BindingFlags.Instance |
135.  
136.                 System.Reflection.BindingFlags.InvokeMethod |
137.  
138.                 System.Reflection.BindingFlags.NonPublic,
139.  
140.                 null, graphic, new object[] { vh });
141.  
142.  
143.  
144.             vh.FillMesh(mesh);
145.  
146.         }
147.  
148.  
149.  
150.         if (graphic is TextMeshProUGUI)
151.  
152.         {
153.  
154.             var textMeshProUGUI = graphic as TextMeshProUGUI;
155.  
156.             mesh = textMeshProUGUI.mesh;
157.  
158.             material = textMeshProUGUI.fontMaterial;
159.  
160.         }
161.  
162.  
163.  
164.         buffer.DrawMesh(mesh, graphic.rectTransform.localToWorldMatrix, material);
165.  
166.     }
167. }
168.