Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question Can't interact with UI when using a virtual mouse

Discussion in 'Input System' started by joeyvanlierop, Oct 16, 2020.

  1. joeyvanlierop

    joeyvanlierop

    Joined:
    Sep 21, 2014
    Posts:
    27
    I have created a "virtual" mouse through the new input system using:
    Code (CSharp):
    1. Mouse RemoteMouse = InputSystem.AddDevice<Mouse>()
    I the generate a new mouse state using:
    Code (CSharp):
    1. pos = new Vector2Int(x, y);
    2. newMouseState = new MouseState
    3. {
    4.     position = pos,
    5.     delta = pos - m_prevMouseState.position,
    6.     buttons = m_prevMouseState.buttons
    7. };
    8. newMouseState.WithButton(button, true);
    And finally update its position using:
    Code (CSharp):
    1. InputSystem.QueueStateEvent(RemoteMouse, newMouseState);
    2. m_prevMouseState = newMouseState;
    This all works fine for the most part. I am able to utilize this mouse as I would a normal mouse (camera controllers work completely fine), however I am unable to interact with UI. When the virtual mouse is hovering over a button the button doesn't transition to a hovered state, and when I click on a button or an input field nothing happens. Another thing to note is that
    EventSystem.current.IsPointerOverGameObject()
    always returns
    null
    when targeting the "virtual" mouse, but when I switch to the native mouse, it returns the correct values.

    Is this a limitation in the new input system? If not, any help would be appreciated.

    (I know that I could probaby work around this issue by using
    Mouse.WarpCursorPosition()
    , however I am unable to do so for my use case)
     
  2. Rene-Damm

    Rene-Damm

    Joined:
    Sep 15, 2012
    Posts:
    1,779
  3. joeyvanlierop

    joeyvanlierop

    Joined:
    Sep 21, 2014
    Posts:
    27
    Yes, I have a
    InputSystemUIInputModule
    component in the scene:
    upload_2020-10-20_12-4-17.png
    Is there anything else I have to do to get it to work with the virtual mouse? I know that the module is functioning for my hardware/native mouse because if I disable the module, I cannot interact with any UI.
     
  4. Rene-Damm

    Rene-Damm

    Joined:
    Sep 15, 2012
    Posts:
    1,779
    First thing I'd check is the input debugger to see whether the UI bindings are picking up the virtual mouse correctly. The controls for it should appear under the respective actions.

    If that is working, next thing I'd probably check is whether the coordinates are indeed right. Mouse operates in player screen space, not canvas UI space.

    If all that is working, could be there's some interference from the "Pointer Behavior" setting (though there should be none). If switching to "All Pointers As Is" makes a difference, that'd likely be the case.
     
  5. Rene-Damm

    Rene-Damm

    Joined:
    Sep 15, 2012
    Posts:
    1,779
    BTW looking at the VirtualMouseInput component that comes with the Gamepad Mouse Cursor sample may help. It adds a virtual mouse driven by the gamepad and works with the UI.
     
  6. joeyvanlierop

    joeyvanlierop

    Joined:
    Sep 21, 2014
    Posts:
    27
    Thanks for all of the suggestions!

    Starting off, both the native mouse and the virtual mouse look nearly identical when inspecting each device.
    Here is the native mouse (Mouse):
    upload_2020-10-20_13-45-3.png
    Here is the virtual mouse (Mouse1):
    upload_2020-10-20_13-45-30.png
    The coordinates of the virtual mouse are perfect and I can see that mouse clicks are being registered as expected. The only difference appears to be that the the native mouse has a "Native" flag.

    The UI bindings seem to be fine as well:
    upload_2020-10-20_13-47-45.png
    One thing that might be worth noting is that we assign the virtual mouse (and the other virtual devices) to an InputUser:
    upload_2020-10-20_13-49-49.png
    However we do not associate any actions with the user. Could that be the root of the issue?

    Finally, changing the "Pointer Behavior" setting had no effect.
     
  7. joeyvanlierop

    joeyvanlierop

    Joined:
    Sep 21, 2014
    Posts:
    27
    @Rene-Damm I don't mean to bug you, but do you have any more thoughts on this? I am still unsure of the root cause.
     
  8. KoalityGame

    KoalityGame

    Joined:
    Jan 30, 2016
    Posts:
    21
    I have the same issue in the demo scene for Gamepad Mouse Cursor. The analog sticks move the cursor just fine, but there is no interaction when pressing any of the buttons.
     
  9. zephybite0

    zephybite0

    Joined:
    Apr 25, 2018
    Posts:
    29
    I have this same issue
     
  10. RaspberryPuppies

    RaspberryPuppies

    Joined:
    Jul 20, 2012
    Posts:
    4
    +1 except I noticed that it's not automatically adding the bindings for the virtual mouse's actions. The mouse is associated to a user but none of the actions.
     
  11. RaspberryPuppies

    RaspberryPuppies

    Joined:
    Jul 20, 2012
    Posts:
    4
    I figured out my issue. I'm going to post a short list of all the confusing steps I ran into that could break things.
    1. Make sure the gamepad has a virtual mouse option
      virtualMouse3.jpg virtualMouse4.jpg
    2. Don't use a special Virtual Mouse input action. This action used the "Use specific device: Virtual Mouse" setting. Having this breaks everything
      virtualMouse.jpg
    3. Instead simply enable normal mouse input for the new platforms
      virtualMouse2.jpg
    4. If you notice that adding a virtual mouse seems to click a random UI element when first enabled, disable the InputSystemUIInputModule component before connecting the virtual mouse and then enable it a few updates after adding the virtual mouse.
    5. Disable the (Multiplayer) Event System's "Send Navigation Events" checkbox. Otherwise the game will highlight seemingly random UI elements as you move the mouse because it will interpret the joystick motion as both mouse and menu navigation. This one had me pulling my hair out trying to fix the wrong bug until I figured it out.
     
    Last edited: Dec 21, 2021
    CleverAI likes this.
  12. Timdeuces

    Timdeuces

    Joined:
    Nov 17, 2014
    Posts:
    2
    I'm using Unity 2021.3.12f1. LTS.

    I've tried everything in the above post by RaspberryPuppies, and I'm still unable to get the virtual mouse to interact with UI elements.

    The only thing that works is to remove the PlayerInput component from the scene. (Not a solution)

    The PlayerInput is doing something that prevents the virtual mouse from interacting with the UI.

    It's as though any updates to the mouse's position aren't reflected in the UI navigation. I am allowed to click the virtual mouse, but its position seems locked at (0,0).
     
    Last edited: Oct 18, 2023
  13. Timdeuces

    Timdeuces

    Joined:
    Nov 17, 2014
    Posts:
    2
    I've come up with a solution that doesn't depend on PlayerInput. I'll share the code here.
    I'm sure it could be improved, but this should be a decent start for anyone in a similar situation.
    Simply place this script onto the GameObject you want to act as the cursor.
    Code (CSharp):
    1. using System.Collections;
    2. using UnityEngine;
    3. using UnityEngine.InputSystem;
    4. using UnityEngine.InputSystem.LowLevel;
    5. using UnityEngine.UI;
    6.  
    7. [RequireComponent(typeof(RectTransform))]
    8. public class AutoHideVirtualMouse : MonoBehaviour
    9. {
    10.     [SerializeField] private InputActionReference _stickAction;
    11.     [SerializeField] private InputActionReference _mousePointAction;
    12.     [SerializeField] private InputActionReference _clickAction;
    13.     [SerializeField] private Graphic _image;
    14.     [SerializeField] private RectTransform _canvasTransform;
    15.     [SerializeField] private float _moveSpeed = 500;
    16.  
    17.     private Mouse _virtualMouse;
    18.     private RectTransform _cursorTransform;
    19.  
    20.     private Vector2 _delta;
    21.     private void Awake()
    22.     {
    23.         _cursorTransform = GetComponent<RectTransform>();
    24.     }
    25.     private void OnEnable()
    26.     {
    27.         if (_virtualMouse == null)
    28.         {
    29.             _virtualMouse = InputSystem.AddDevice<Mouse>("VirtualMouse");
    30.         }
    31.  
    32.         SetCursorEnabled(false);
    33.  
    34.         _stickAction.action.Enable();
    35.         _mousePointAction.action.Enable();
    36.         _clickAction.action.Enable();
    37.  
    38.         _stickAction.action.performed += OnStickMove;
    39.         _mousePointAction.action.performed += OnMouseMove;
    40.         _clickAction.action.performed += OnClick;
    41.  
    42.         InputSystem.onAfterUpdate += OnAfterUpdate;
    43.     }
    44.  
    45.     private void OnDisable()
    46.     {
    47.         SetCursorEnabled(false);
    48.  
    49.         InputSystem.onAfterUpdate -= OnAfterUpdate;
    50.  
    51.         _stickAction.action.performed -= OnStickMove;
    52.         _mousePointAction.action.performed -= OnMouseMove;
    53.         _clickAction.action.performed -= OnClick;
    54.     }
    55.     private void SetCursorEnabled(bool enabled)
    56.     {
    57.         Cursor.visible = !enabled;
    58.         _image.enabled = enabled;
    59.  
    60.         if (_virtualMouse == null) return;
    61.         if (enabled)
    62.         {
    63.             if (!_virtualMouse.added) InputSystem.AddDevice(_virtualMouse);
    64.  
    65.             //move virtual mouse to screen center instead of having it start at (0,0)
    66.             InputState.Change(_virtualMouse.position, new Vector2(Screen.width / 2, Screen.height / 2));
    67.         }
    68.         else
    69.         {
    70.             InputSystem.RemoveDevice(_virtualMouse);
    71.         }
    72.     }
    73.     private void OnAfterUpdate()
    74.     {
    75.         if (!_virtualMouse.added) return;
    76.  
    77.         var newPosition = (_virtualMouse.position.ReadValue() + _delta * _moveSpeed * Time.deltaTime);
    78.         newPosition.x = Mathf.Clamp(newPosition.x, 0, Screen.width);
    79.         newPosition.y = Mathf.Clamp(newPosition.y, 0, Screen.height);
    80.  
    81.         //move virtual mouse screen position
    82.         InputState.Change(_virtualMouse.position, newPosition);
    83.         InputState.Change(_virtualMouse.delta, _delta * _moveSpeed * Time.deltaTime);
    84.  
    85.         //move cursor transform accordingly
    86.         RectTransformUtility.ScreenPointToWorldPointInRectangle(_canvasTransform, newPosition, null, out Vector3 localPoint);
    87.         _cursorTransform.position = localPoint;
    88.     }
    89.     private void OnStickMove(InputAction.CallbackContext ctx)
    90.     {
    91.         if (!_image.enabled)
    92.         {
    93.             SetCursorEnabled(true);
    94.         }
    95.         if (!_virtualMouse.added) return;
    96.         _delta = ctx.ReadValue<Vector2>();
    97.     }
    98.     private void OnClick(InputAction.CallbackContext ctx)
    99.     {
    100.         if (!_virtualMouse.added) return;
    101.         _virtualMouse.CopyState(out MouseState mouseState);
    102.         mouseState.WithButton(MouseButton.Left, ctx.ReadValueAsButton());
    103.         InputState.Change(_virtualMouse, mouseState);
    104.     }
    105.     private void OnMouseMove(InputAction.CallbackContext ctx)
    106.     {
    107.         if (_image.enabled && ctx.control.device.name != "VirtualMouse")
    108.         {
    109.             StartCoroutine(Co_EndOfFrame());
    110.         }
    111.         //using local coroutine because removing the device before end of frame causes errors
    112.         IEnumerator Co_EndOfFrame()
    113.         {
    114.             yield return new WaitForEndOfFrame();
    115.             SetCursorEnabled(false);
    116.         }
    117.     }
    118.  
    119. }
    120.  
     
    Last edited: Oct 18, 2023
    MafiaMoe likes this.
  14. dherault

    dherault

    Joined:
    Dec 19, 2019
    Posts:
    8
    I'm also using Unity 2022.3.10f1.

    I confirm that the virtual mouse is not interacting with the UI, regardless of how it is configured.

    Could someone from unity look into this?
     
  15. paragraph7

    paragraph7

    Joined:
    May 9, 2016
    Posts:
    9
    Was implementing this today and had the same issues as above - the only way I got it working was by separating the input actions into separate files, one "UI" .inputactions file that is fed into the EventSystem, and one for everything else game related. I tried switching between active action maps when toggling the virtual mouse, but that didnt help either.
     
    lucian_mandc likes this.
  16. xXFUNNYJOKEXx

    xXFUNNYJOKEXx

    Joined:
    Jul 30, 2020
    Posts:
    2
    I was searching around and I found a solution on another forum; you need to set your Cursor.lockstate = CursorLockMode.Confined not Locked. The Locked will tick the virtual mouse to thinking it is on the center of the screen. That seems to help my project so good luck for yours!
     
  17. lucian_mandc

    lucian_mandc

    Joined:
    Jun 9, 2022
    Posts:
    1
    Thank you for posting this. I tried this solution, and it was exactly what was needed to overcome the almost inexplicable 'virtual mouse cursor does not register UI events like mouse-over or click' problem.

    I had only stumbled across Player Input as a potential culprit by going through my scene and deactivating parent Game Objects one by one until the virtual mouse started behaving correctly. That prompt led me to this thread, and then to your reply.

    Creating a brand new Input Actions and having the Event System's UI Input Manager reference it looks to be the fix.

    A truly maddening bug because every tutorial I could find on virtual mouse implementation didn't have any other Game Objects in the scene that had the Player Input component, so there was no indication that was the culprit. Ah, game development.