Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Trying to Automate Windows Calculator from Unity crashes unity

Discussion in 'Windows' started by dansav, Sep 18, 2019.

  1. dansav

    dansav

    Joined:
    Sep 22, 2005
    Posts:
    510
    I'm trying to automate the windows calculator from unity

    I put these dll's into unity's plugin folder not sure if that's what you're supposed to do.
    from c: program files/referenced assemblies/Microsoft/ framework /3.0
    uiautomationclient.dll
    windowsbase.dll
    uiautomationtypes.dll
    uiautomationprovider.dll
    uiautomationclientsideproviders.dll
    uiautomationprovider.dll

    The code I'm running is

    Code (CSharp):
    1. using UnityEngine;
    2. using System.Windows.Automation;
    3. using System.Windows;
    4. using System.Diagnostics;
    5.  
    6. public class main : MonoBehaviour
    7. {
    8.  
    9.     private void Start()
    10.     {
    11.        Process calculatorProcess = Process.Start("Calc.exe");
    12.        ElementFromCursor();
    13.     }
    14.  
    15.  
    16.    private void ElementFromCursor()
    17.     {
    18.         // Convert mouse position from System.Drawing.Point to System.Windows.Point.
    19.         //System.Windows.Point point = new System.Windows.Point(Cursor.Position.X, Cursor.Position.Y);
    20.        Point point = new Point(1500,1500);
    21.         AutomationElement element = AutomationElement.FromPoint(point);
    22.         //UnityEngine.Debug.Log(element);
    23.         //return element;
    24.     }
    25. }
    The Error is:
    * Assertion: should not be reached at ..\mono\metadata\marshal.c:7146
     
  2. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,646
    Mono (which Unity uses to run C# code) doesn't really support COM interop well. You're running into one of many of its limitations. If you want, you can report a bug on it but it's unlikely to be fixed in a a timely manner as this is a very niche use case.

    You might have more luck if you build your project using IL2CPP scripting backend.
     
  3. dansav

    dansav

    Joined:
    Sep 22, 2005
    Posts:
    510
    Thanks for the tip. I wasn't expecting any help since it's such a niche case so I was glad to see a response. I tried using IL2CPP but same result. Editor and Build same result on mono and on IL2CPP. I don't get any errors in the editor or player log so there's nothing to research.
     
  4. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,646
    When you say same result on IL2CPP, what do you mean? That assertion is coming from inside Mono, so it cannot be hit by IL2CPP.
     
  5. dansav

    dansav

    Joined:
    Sep 22, 2005
    Posts:
    510
    Hi. I changed something in the player settings from mono to IL2CPP is that what you meant to do? By "the same result" I meant it crashed. I did not see the mono error or any error in the logs on the IL2CPP build.
     
  6. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,646
    It crashed the build? Interesting, I wonder what kind of crash that is. Any chance you could attach a debugger to crashing IL2CPP process?
     
  7. dansav

    dansav

    Joined:
    Sep 22, 2005
    Posts:
    510
    It made the build without an error. Upon running it opens the calculator, then closes the application. The editor does the same thing. It does not hang anything, but actually closes the entire editor or the standalone. I'm assuming this is caused by the AutomationElement.FromPoint call. If I take it out it does not close/crash.
     
  8. dansav

    dansav

    Joined:
    Sep 22, 2005
    Posts:
    510
    How do I attach the debugger? there are some player settings and some settings in the build panel?
     
  9. dansav

    dansav

    Joined:
    Sep 22, 2005
    Posts:
    510
    I did a developmental build, script debugging, wait for managed debugger? not sure what that is.
    In the player settings I put on full stack trace on everything.
    But no new lines in the log. The last log lines of the player crash are...
    Begin MonoManager ReloadAssembly
    - Completed reload, in 0.895 seconds
    <RI> Initializing input.
    XInput1_3.dll not found. Trying XInput9_1_0.dll instead...
    <RI> Input initialized.
    <RI> Initialized touch support.
    UnloadTime: 2.270817 ms
     
  10. dansav

    dansav

    Joined:
    Sep 22, 2005
    Posts:
    510
    Okay now I got something in the editor that said it couldn't do the build. For some reason the editor switched back from IL2CPP to mono after it crashed. So now I'm unsure as to whether I had really made an IL2CPP build before.

    IL2CPP error for method 'System.Int32 MS.Internal.Automation.UiaCoreApi::RawUiaGetRuntimeId(MS.Internal.Automation.SafeNodeHandle,System.Int32[]&)' in assembly 'C:\Users\dan\desktopscanner\Temp\StagingArea\Data\Managed\UIAutomationClient.dll'
    Additional information: SafeArray element type None is not supported.
     
  11. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,646
    Indeed, this looks like just COM interop hell - those DLLs call system COM APIs and neither Mono nor IL2CPP are able to handle that. I think your only option is to call those COM APIs from a native plugin if you're determined to make this work.
     
  12. dansav

    dansav

    Joined:
    Sep 22, 2005
    Posts:
    510
    So far after many years I haven't run into something that unity can't do until now, but it seems like for a good reason. UIAutomation seems like an unreliable mess the more I read about it.

    Thanks for helping me determine the root source of the problem which is helping me avoid this library.