InputSystem.QueueStateEvent(...) Does Not Simulate InputActions?

I'm trying to simulate input on my game server, received from a client.
Currently, it's for a left mouse (LMB) click, so I have the following code:

Code (CSharp):

1. using UnityEngine.InputSystem;
2. using UnityEngine.InputSystem.LowLevel;
3.  
4. //...
5.  
6. MouseState buttonPress = new MouseState();
7. buttonPress = buttonPress.WithButton(MouseButton.Left);
8. buttonPress.position = new Vector2(850, 80); //Just an example value of screen pixel coordinates
9.  
10. Mouse mouse = InputSystem.GetDevice<Mouse>();
11. InputSystem.QueueStateEvent<MouseState>(mouse, buttonPress);

Nothing was triggering and my InputActions were silent.
So I checked the following event, which IS successfully firing the next frame (when the input queue is gone through):

Code (CSharp):

1. InputSystem.onEvent += OnInputEvent;
2.  
3. //...
4.  
5. private void OnInputEvent(InputEventPtr ptr, InputDevice device) {
6.     Debug.Log("OnInputEvent! ptr = " + ptr + ", device = " + device + " (" + device.displayName + ")");
7. }

However, I have no idea what I'm looking at with InputEventPtr, and why I would ever want it (oof, that sounds harsh! Haha).

1. Where's the input I tried to simulate?
2. Why did none of my UI register a mouse click on my button, despite having an EventSystem + InputSystemUIInputModule pair?

My goal is to simulate input as if it truly happened, so my game server can repeat the input the client does, and then validate it, and send results back.

 

Based on @Rene-Damm 's post on simulating input (possibly outdated, since it was from Jan, 2019?),
I wrote the following code today in Unity 2020.1.15f1 with InputSystem 1.0.1:

Code (CSharp):

1. private void TestClickAtCurrentMousePosition() {
2.     Mouse mouse = InputSystem.GetDevice<Mouse>();
3.     using (StateEvent.From(mouse, out InputEventPtr eventPtr)) {
4.         ((ButtonControl) mouse["leftButton"]).WriteValueIntoEvent<float>(1, eventPtr);
5.         InputSystem.QueueEvent(eventPtr);
6.     }
7. }

I was attempting to use a fake left-click to simulate clicking a UI button.
However, this does not work.

I submit bug report (Case 1297339) with many more details.

 

Innnnnnnnnnnnnteresting, I found a piece of code (by testing a LOT and having no idea what would work haha) --

This does work!!

1. I took the InputSystemUIInputModule's actionsAsset and duplicated it in my project,
2. Re-assigned the InputSystemUIInputModule's actionsAsset to my custom duplicated asset
3. Added a binding to the UI/Click action that points to "Any Key [Keyboard]"
4. I positioned my mouse over the UI Button in the Game View while in playmode
5. I ran the following code that simulated pressing and releasing the I (not L) key on the keyboard, and it WORKED!

(The goal was to simulate a "pointer click" EventSystems event. This was successful in clicking my UI Button).

Code (CSharp):

1. Keyboard keyboard = InputSystem.GetDevice<Keyboard>();
2. KeyboardState stateA = new KeyboardState();
3. KeyboardState stateB = new KeyboardState();
4. stateA.Press(Key.I);
5. stateB.Release(Key.I);
6.  
7. InputSystem.QueueStateEvent(keyboard, stateA);
8. InputSystem.QueueStateEvent(keyboard, stateB);

The thing I learned is it required two events.
One event to press the key,
One event to release the key.

If you use the same event, it won't register properly that the key was pressed, then released.
I'll come back with some more findings.. perhaps this will help me with my mouse button clicking..

 

Okay I finally understand:
My problem is that something about the mouse clicking in particular is not working. (Keyboard stuff is working great from above)

Neither of these 2 approaches work in simulating a click:

Approach A: 2 Mouse States (Press & Release)

Code (CSharp):

1. MouseState stateA = new MouseState();
2. MouseState stateB = new MouseState();
3.  
4. stateA.position = fakeMousePosition;
5. stateA = stateA.WithButton(MouseButton.Left, true);
6. stateB.position = fakeMousePosition;
7. stateB = stateB.WithButton(MouseButton.Left, false);
8.  
9. InputSystem.QueueStateEvent(mouse, stateA);
10. InputSystem.QueueStateEvent(mouse, stateB);

Approach B: 5 Mouse States (Includes Mouse Movement)

Code (CSharp):

1. Mouse mouse = InputSystem.GetDevice<Mouse>();
2. MouseState[] states = new MouseState[5];
3.  
4. states[0] = states[0].WithButton(MouseButton.Left, false);
5. states[1] = states[1].WithButton(MouseButton.Left, false);
6. states[2] = states[2].WithButton(MouseButton.Left, true);
7. states[3] = states[3].WithButton(MouseButton.Left, false);
8. states[4] = states[4].WithButton(MouseButton.Left, false);
9.  
10. Vector2 currentPos = mouse.position.ReadValue();
11. states[0].position = currentPos;
12. states[1].position = fakeMousePosition;
13. states[2].position = fakeMousePosition;
14. states[3].position = fakeMousePosition;
15. states[4].position = currentPos;
16.  
17. for (int i = 0; i < states.Length; i++) {
18.     states[i].delta = (i == 0) ? Vector2.zero : states[i].position - states[i - 1].position;
19.     InputSystem.QueueStateEvent(mouse, states[i]);
20. }
21.  
22. InputSystem.QueueStateEvent(mouse, states[0]);
23. InputSystem.QueueStateEvent(mouse, states[1]);
24. InputSystem.QueueStateEvent(mouse, states[2]);
25. InputSystem.QueueStateEvent(mouse, states[3]);
26. InputSystem.QueueStateEvent(mouse, states[4]);

I think my best bet is to just intercept the "Click" action for my InputSystemUIInputModule,
Assign it to a key on my Server (which is where I need mouse clicks to be simulated),
And simulate the mouse click by assigning a key to it.
Otherwise, I'll just be upset and my game will be broken lol

Hopefully the input queue will remember the MouseStates around my KeyboardStates setting the mouse position.
Let's see if this works...

 

Okay a big problem is that none of my MouseState events (where I explicitly set their position...) are effecting my mouse position.
I'm reading mouse position from:

Code (CSharp):

1. InputSystem.GetDevice<Mouse>().position.ReadValue();

EDIT: I'm now reading mouse position from:

Code (CSharp):

1. [SerializeField] private InputActionReference pointAction;
2.  
3. pointAction.action.ReadValue<Vector2>();

And I've been changing the bindings to see if anything changes the position anymore..
I created a Mouse device through C# code and assigned it in the InputActionsAsset window as a binding to the Point action.. it just stays stuck at (0, 0) even when I change it on Update..

Code (CSharp):

1. private void Update() {
2.    InputSystem.QueueDeltaStateEvent(virtualMouse.position, new Vector2(Random.Range(300, 900), 384));
3. }

This is extremely problematic because I can click, but I cannot click at any position.
Only where the mouse cursor happens to be.

Why can't I set the mouse position?..

This is a huge blocker for me...

 

Did you find a solution to this issue?

 

Nope, I did not.
If I recall correctly, I was able to simulate keyboard input, but not mouse input.

 

Hm, seems to work now, at least when reading via

Code (CSharp):

1. InputSystem.GetDevice<Mouse>().position

Works with writing to the "real" mouse device by retrieving it via

Code (CSharp):

1. InputSystem.GetDevice<Mouse>()

as well as with a virtual mouse created by

Code (CSharp):

1. InputSystem.AddDevice<Mouse>("FakeMouse");

 

Try this code snippet to change the mouse position.
InputState.Change(Mouse.current.position, new Vector2(deltaX, deltaY));

https://answers.unity.com/questions/1738031/force-update-of-mouse-position-with-the-new-input-1.html

 

I came up with a very legit way to move the mouse cursor and another to force a mouse click event because Unity is so annoying in not letting you do this when using Cursor.lockState. Now you can set Cursor.lockState and/or Cursor.visibility in Start() and call either of the following below to force the mouse to behave. (Verified working in macOS Big Sur, Unity 2021.1.17)

Stupid easy way to force mouse cursor position to center of game window in editor only from code:

Code (CSharp):

1.    
2. using UnityEngine.InputSystem;
3. using UnityEngine.InputSystem.LowLevel;
4. ...
5. public static void ForceMousePositionToCenterOfGameWindow()
6.     {
7. #if UNITY_EDITOR
8.         // Force the mouse to be in the middle of the game screen
9.         var game = UnityEditor.EditorWindow.GetWindow(typeof(UnityEditor.EditorWindow).Assembly.GetType("UnityEditor.GameView"));
10.         Vector2 warpPosition = game.rootVisualElement.contentRect.center;  // never let it move
11.         Mouse.current.WarpCursorPosition(warpPosition);
12.         InputState.Change(Mouse.current.position, warpPosition);
13. #endif
14.     }
15.  

Stupid easy way to force click in game window in editor only from code:

Code (CSharp):

1.    
2. using UnityEngine.InputSystem;
3. using UnityEngine.InputSystem.LowLevel;
4. ...
5. public static void ForceClickMouseButtonInCenterOfGameWindow()
6.     {
7. #if UNITY_EDITOR
8.         var game = UnityEditor.EditorWindow.GetWindow(typeof(UnityEditor.EditorWindow).Assembly.GetType("UnityEditor.GameView"));
9.         Vector2 gameWindowCenter = game.rootVisualElement.contentRect.center;
10.  
11.         Event leftClickDown = new Event();
12.         leftClickDown.button = 0;
13.         leftClickDown.clickCount = 1;
14.         leftClickDown.type = EventType.MouseDown;
15.         leftClickDown.mousePosition = gameWindow;
16.  
17.         game.SendEvent(leftClickDown);
18. #endif
19.     }
20.  

 

Thank you very much for sharing your findings.

For my case, this worked, just in case anyone needs:

Code (CSharp):

1.  
2.         var Keyboard = InputSystem.GetDevice<Keyboard>();
3.         var KeyboardState = new KeyboardState();
4.         KeyboardState.Press(Key.Space);
5.         InputSystem.QueueStateEvent(Keyboard, KeyboardState);
6.         KeyboardState.Release(Key.Space);
7.         InputSystem.QueueStateEvent(Keyboard, KeyboardState);
8.  

 

I have a very similar problem. To me, I am able to register events, but all actions fired through QueueStateEvent come in in the wrong order, through which my character just keeps running, because the release of the key is registered before the press.

Crossposting this because I found a solution