Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Unity UI Interaction with objects displayed on render texture

Discussion in 'UGUI & TextMesh Pro' started by warpom, Feb 12, 2018.

  1. warpom

    warpom

    Joined:
    Nov 21, 2017
    Posts:
    3
    I am creating a menu for VR that is made to look like it is being displayed on a TV in a room where you sit. When placing objects in front of the TV to make UI elements rotation will clip elements and the freedom of how elements are displayed on the TV is limited.

    Therefore I want to render the menu from a separate camera and use that separate cameras view as a render texture on the TV.

    How can I wire this up so that interaction on the virtual TV gets propagated thru the secondary camera such that the Unity UI events can be used safely.

    In other words, I want to proxy all interaction from the main camera thru the secondary camera with its render texture used as viewport.
     

    Attached Files:

    Last edited: Feb 12, 2018
    SertanC, LaurieAnnis and linojon like this.
  2. warpom

    warpom

    Joined:
    Nov 21, 2017
    Posts:
    3
    I have come up with an intermediate solution that I want to share even though it has some drawbacks which I think can be solved when I understand the InputModules and EventSystem better.

    Providing my current implementation to help people reading this understand the problem better.

    Code (CSharp):
    1. using UnityEngine;
    2. using UnityEngine.EventSystems;
    3.  
    4. public class TVScreen : MonoBehaviour, IPointerClickHandler
    5. {
    6.     // Camera that provides the image on a TV.
    7.     public Camera tvCamera;
    8.  
    9.     // Callback that gets the collider the TV camera hit thru the TV
    10.     public delegate void OnHit(Collider collider);
    11.  
    12.     public void Awake()
    13.     {
    14.         // Set the TV camera to render to the render texture of this object.
    15.         RenderTexture tvScreen = (RenderTexture)GetComponent<MeshRenderer>().material.mainTexture;
    16.         tvCamera.targetTexture = tvScreen;
    17.     }
    18.  
    19.     public void OnPointerClick(PointerEventData eventData)
    20.     {
    21.         // When a user clicks this GameObject, we send it down to our Proxy method.
    22.         ProxyPointerEvent(eventData, "OnMouseDown");
    23.     }
    24.  
    25.     private void ProxyPointerEvent(PointerEventData eventData, string messageDestination)
    26.     {
    27.         // Create a ray from the PointerEventData which is used to figure out texture coordinate
    28.         // of the render texture for the camera.
    29.         // These coordinates may be available from the PointerEvent immediately in some form,
    30.         // but I have not been able to find a direct substitute.
    31.         Ray eyeRay = eventData.pressEventCamera.ScreenPointToRay(eventData.position);
    32.  
    33.         // Send this ray to our local Raycast method, and provide a callback which is called with
    34.         // the collider that is hit "behind" the TV.
    35.         Raycast(eyeRay, collider => collider.gameObject.SendMessage(messageDestination));
    36.     }
    37.  
    38.     public void Raycast(Ray eyeRay, OnHit onHit)
    39.     {
    40.         RaycastHit eyeHit, tvHit;
    41.  
    42.         // Cast the incoming ray to be able to access the textureCoord that is ultimately provided
    43.         // by the camera providing the TV image.
    44.         Physics.Raycast(eyeRay, out eyeHit);
    45.  
    46.         // Create second ray that extends from the Camera providing the TV image
    47.         Ray tvRay = tvCamera.ViewportPointToRay(new Vector3(eyeHit.textureCoord.x, eyeHit.textureCoord.y, 0));
    48.         Debug.DrawRay(tvRay.origin, tvRay.direction * 10, Color.red, 3);
    49.  
    50.         // Cast the secondary ray from the TV image camera.
    51.         if (Physics.Raycast(tvRay, out tvHit))
    52.         {
    53.             Debug.Log(tvHit);
    54.             // Call callback if collider was hit.
    55.             onHit(tvHit.collider);
    56.         }
    57.     }
    58. }
    59.  
     
    SertanC and LaurieAnnis like this.
  3. warpom

    warpom

    Joined:
    Nov 21, 2017
    Posts:
    3
    Wanted to update this and add the current solution which is more clean and honors all built in PointerEvents.

    Code (CSharp):
    1. using System.Collections.Generic;
    2. using UnityEngine;
    3. using UnityEngine.UI;
    4. using UnityEngine.EventSystems;
    5.  
    6. public class VirtualScreen : GraphicRaycaster
    7. {
    8.     public Camera screenCamera;
    9.  
    10.     // Called by Unity when a Raycaster should raycast because it extends BaseRaycaster.
    11.     public override void Raycast(PointerEventData eventData, List<RaycastResult> resultAppendList)
    12.     {
    13.         Ray ray = eventCamera.ScreenPointToRay(eventData.position); // Mouse
    14.         RaycastHit hit;
    15.         if (Physics.Raycast(ray, out hit))
    16.         {
    17.             RaycastBeyondTV(hit, resultAppendList);
    18.         }
    19.     }
    20.  
    21.     private void RaycastBeyondTV(RaycastHit originHit, List<RaycastResult> resultAppendList)
    22.     {
    23.         // Figure out where the pointer would be in the second camera based on texture position or RenderTexture.
    24.         Vector3 virtualPos = new Vector3(originHit.textureCoord.x, originHit.textureCoord.y);
    25.         Ray ray = screenCamera.ViewportPointToRay(virtualPos);
    26.         Debug.DrawRay(ray.origin, ray.direction * 10, Color.red, 0.2f);
    27.  
    28.         RaycastHit hit;
    29.         if (Physics.Raycast(ray, out hit))
    30.         {
    31.             RaycastResult result = new RaycastResult
    32.             {
    33.                 gameObject = hit.collider.gameObject,
    34.                 module = this,
    35.                 distance = hit.distance,
    36.                 index = resultAppendList.Count,
    37.                 worldPosition = hit.point,
    38.                 worldNormal = hit.normal,
    39.             };
    40.             resultAppendList.Add(result);
    41.         }
    42.     }
    43. }
    44.  
     
  4. unity_0Rf9MTCy1g-3mg

    unity_0Rf9MTCy1g-3mg

    Joined:
    Apr 1, 2018
    Posts:
    1
    What object do you attach a GraphicRaycaster to?
    I attached it to the game object that held the texture and got a null reference looking for a canvas.
    So I stopped and thought and looked around, and noticed that the canvas had a GraphicsRaycaster on it. So I removed the default one and added this one. Now I get tons of null references eventCamera is now null. That's a part of the base Raycaster. Is there something I'm missing?
     
  5. Peeling

    Peeling

    Joined:
    Nov 10, 2013
    Posts:
    442
    Hey :) Happened across this just now while looking for a virtual screen interaction solution.

    The code above doesn't seem to work with Unity's new UI system (or I wasn't able to make it work) as the Physics.Raycast() call doesn't detect anything I put in the canvas that I'm displaying via the rendertexture.

    However, with a minor modification, this does seem to work with the new UI:

    Code (CSharp):
    1. using System.Collections.Generic;
    2. using UnityEngine;
    3. using UnityEngine.UI;
    4. using UnityEngine.EventSystems;
    5.  
    6. public class VirtualScreen : GraphicRaycaster
    7. {
    8.     public Camera screenCamera; // Reference to the camera responsible for rendering the virtual screen's rendertexture
    9.  
    10.     public GraphicRaycaster screenCaster; // Reference to the GraphicRaycaster of the canvas displayed on the virtual screen
    11.  
    12.     // Called by Unity when a Raycaster should raycast because it extends BaseRaycaster.
    13.     public override void Raycast(PointerEventData eventData, List<RaycastResult> resultAppendList)
    14.     {
    15.         Ray ray = eventCamera.ScreenPointToRay(eventData.position); // Mouse
    16.         RaycastHit hit;
    17.         if (Physics.Raycast(ray, out hit))
    18.         {
    19.             if (hit.collider.transform == transform)
    20.             {
    21.                 // Figure out where the pointer would be in the second camera based on texture position or RenderTexture.
    22.                 Vector3 virtualPos = new Vector3(hit.textureCoord.x, hit.textureCoord.y);
    23.                 virtualPos.x *= screenCamera.targetTexture.width;
    24.                 virtualPos.y *= screenCamera.targetTexture.height;
    25.  
    26.                 eventData.position = virtualPos;
    27.  
    28.                 screenCaster.Raycast(eventData, resultAppendList);
    29.             }
    30.         }
    31.     }
    32.  
    33. }
    34.  
    Using this, hooked up to the correct camera and GraphicRaycaster, I was able to mouse over and click on standard canvas-based buttons displayed via a render-texture on an in-world screen. Hope that's of some use :)

    EDIT: I subsequently discovered some quite important caveats!

    The first, I have added to the code above (you need to check the raycast has hit the screen!)

    The second is that you must disable the Graphic Raycaster on the canvas that's generating the screen content (the one referenced as 'screenCaster'). If you don't, then clicks off the virtual screen will be handled by that raycaster as though they were on its own canvas. For example: if there were a button in the top left of the virtual screen, you would be able to click it by clicking the top left of the game screen, as well as by clicking the top left of the virtual screen. So: disable the graphic raycaster on the proxy canvas.
     
    Last edited: Aug 1, 2022
  6. augustomoreirapedro06

    augustomoreirapedro06

    Joined:
    Mar 18, 2022
    Posts:
    4
    When I try this code I get an error because of the "Override". I'm a beginner coder and don't know how to solve it.
     
  7. Peeling

    Peeling

    Joined:
    Nov 10, 2013
    Posts:
    442
    I just copied and pasted it back into a new project of mine and it compiled fine. Make sure you copied everything exactly, including this line:

    Code (CSharp):
    1. public class VirtualScreen : GraphicRaycaster
    GraphicRaycaster is the class that contains the Raycast method the code overrides. If you've created a blank class in Unity and tried to paste the methods into it, your blank class will be extending MonoBehaviour rather than GraphicRaycaster.
     
  8. ofirudwork

    ofirudwork

    Joined:
    Nov 3, 2021
    Posts:
    17
    @Peeling
    Hi has someone solved it?
    The if statement is never true...

    Code (CSharp):
    1. Ray ray = eventCamera.ScreenPointToRay(eventData.position); // Mouse
    2.         RaycastHit hit;
    3.  
    4.         if (Physics.Raycast(ray, out hit))
    5.         {
    6.             // Figure out where the pointer would be in the second camera based on texture position or RenderTexture.
    7.             Vector3 virtualPos = new Vector3(hit.textureCoord.x, hit.textureCoord.y);
    8.             virtualPos.x *= screenCamera.targetTexture.width;
    9.             virtualPos.y *= screenCamera.targetTexture.height;
    10.  
    11.             eventData.position = virtualPos;
    12.  
    13.             screenCaster.Raycast(eventData, resultAppendList);
    14.         }
     
    Last edited: Jul 11, 2022
  9. Peeling

    Peeling

    Joined:
    Nov 10, 2013
    Posts:
    442
    That's not a fault with the code; it must be a problem with your scene.

    You need to set it up as follows: your TV/phone/whatever object needs to have the following on:
    upload_2022-7-11_17-57-19.png

    The 'ScreenCamera' and 'Screen Caster' need to be set up to point at the camera doing the RTT and the Graphic Raycaster on the canvas used to render its content.

    I've just checked again and it works perfectly.
     
    SertanC likes this.
  10. ofirudwork

    ofirudwork

    Joined:
    Nov 3, 2021
    Posts:
    17
    Thanks for replying to me!
    But it still doesn't work, I mean it detects a hit from raycasting but the UI doesn't interact. I'll try to explain what I have.

    I have one camera and a canvas that regards this camera (The canvas has a lot of UI like buttons, text fields, and more..).
    This camera is displayed by rendering texture with RawImage on another canvas.
    The issue is that when I try to use the UI in the RT - it doesn't detect any interactions...
    So I tried to use your solution like this:

    To the canvas that has the Raw Image I added the script "VirtualScreen" and attached to it:
    1. Screen Camera - the camera that has the render texture.
    2. Screen Caster- the graphic raycaster of the canvas of the camera that has the render texture.
    3. Now I added the MeshCollider and tried to change the canvas to WorldSpace but still doesn't work...


    Have you understood?

    Hope you will help me :)
     
    Last edited: Jul 12, 2022
  11. Peeling

    Peeling

    Joined:
    Nov 10, 2013
    Posts:
    442
    Share your project and I'll take a look, if you like. It sounds as though you have things on the wrong objects.

    The camera and canvas that are rendering to the texture don't need anything adding to them.

    The virtual screen (the object with the texture on it, the one you want to click on in-game) needs a mesh collider, a world-space camera, and the virtualscreen script.
     
    Last edited: Jul 12, 2022
    SertanC likes this.
  12. Peeling

    Peeling

    Joined:
    Nov 10, 2013
    Posts:
    442
    Or try this example project!
     

    Attached Files:

  13. Creiz

    Creiz

    Joined:
    Jun 6, 2017
    Posts:
    130
    Sorry to necro this post.

    Is there a way to use this with an actual uGUI RenderTexture?
     
  14. Peeling

    Peeling

    Joined:
    Nov 10, 2013
    Posts:
    442
    What do you mean exactly?
     
  15. Creiz

    Creiz

    Joined:
    Jun 6, 2017
    Posts:
    130
    My render texture is on a UI canvas instead of being in WorldSpace. It's part of the GUI. This only works if your RenderTexture is in worldspace inside the scene itself.
     
  16. Peeling

    Peeling

    Joined:
    Nov 10, 2013
    Posts:
    442
    Interesting - I'm wondering why the GUI you're trying to click on needs to be on a rendertexture rather than just being part of the UI
     
  17. Creiz

    Creiz

    Joined:
    Jun 6, 2017
    Posts:
    130
    It's the other way around. The scene needs to be on the RenderTexture. Think of it as playing on a TV or something.

    You remember those old DOS dungeon crawlers? I'm trying to do something like that. On the left side, you have a window with all your characters, hp and mp and so on, on the right side you have your stats, inventory, minimap, compass, etc. On the bottom you have the message logger, ie: "you entered the crypt of the abyss" or "You attacked the goblin for 32hp of damage", etc.

    And just on top of that, you have your "Game Screen", which is the scenes themselves. The whole 3D world and all that. I did it so I could position the game screen snuggly in between the other "views" and not have it go bonkies with different resolutions and whatnot.

    However, sometimes I need to raycast into the scenes to be able to pick up items, click on things for puzzles, hover on objects to highlight them, etc.
     
  18. Peeling

    Peeling

    Joined:
    Nov 10, 2013
    Posts:
    442
    Ah, now I understand. That's not really the same problem as this code is intended to solve.

    Personally, I would stop using a rendertexture and use a script to adjust the game camera's viewport to fit an invisible rect autosized to fit that area. That will then allow you to cast rays by detecting clicks on the rect and converting the mouse coordinates to a world ray using the game camera in the usual way. There are a few methods you can find online for getting the screen space coords of a rect.
     
  19. tophattwaffle

    tophattwaffle

    Joined:
    Feb 28, 2022
    Posts:
    1
    Wanted to express my gratitude to @warpom and @Peeling for this post and the setup info. I was using a canvas to do some UI stuff and this allowed me to move it to a "TV" screen in world without a lot of effort. Works perfectly.
     
  20. mj-3dqr

    mj-3dqr

    Joined:
    Mar 30, 2017
    Posts:
    14
    Thank you for the example. It is very helpful. However, I have the problem that I can't get sliders (i.e. dragging) to pass through cleanly.

    I have already setup new values for
    1. eventData.delta
    2. eventData.pressPosition
    or recalculated it.

    No success. Maybe someone has a hint.