Search Unity

Unity2018.3, Microsoft Media Foundation and IMediaCapture

Discussion in 'Windows' started by vthemereau_asobo, Jan 18, 2019.

  1. vthemereau_asobo

    vthemereau_asobo

    Joined:
    Feb 27, 2018
    Posts:
    3
    Hello folks,

    We have a project for HoloLens that does video capture and use Microsoft Media Foundation for capturing and encoding the video stream. The project was working fine in Unity 2017.4.2f1 until we switch to Unity 2018.3.1.

    The video capture is done inside a DLL written in C++/WRL and forked from https://github.com/Microsoft/MixedRealityCompanionKit/tree/master/MixedRemoteViewCompositor/Source

    The issue in 2018.3.1 is that IMediaCapture::StartRecordToCustomSinkAsync method is not calling the callback while in 2017.2.4, it is working fine. We have noted that in 2018.3.1, the callback is called if we resize the Editor window ... which is very weird.

    Code (CSharp):
    1.     auto startRecordAsync = Callback<IAsyncActionCompletedHandler>(
    2.         [this, spInitAction](_In_ IAsyncAction *asyncResult, _In_ AsyncStatus asyncStatus) -> HRESULT
    3.     {
    4.        // NEVER CALLED IN UNITY 2018.3.1 !
    5.  
    6.         HRESULT hr = S_OK;
    7.         if (asyncStatus != AsyncStatus::Completed)
    8.         {
    9.             hr = E_ABORT;
    10.             IFC(hr);
    11.         }
    12.  
    13.         TRACE_OBJ();
    14.  
    15.         IFC(_networkMediaSink->SendCaptureStarted());
    16.  
    17.         IFC(SetCaptureState(CaptureEngineState_Started));
    18.  
    19.         spInitAction->Completed(S_OK);
    20.     done:
    21.         if (FAILED(hr))
    22.         {
    23.             IFC(SetCaptureState(CaptureEngineState_StartFailed));
    24.         }
    25.  
    26.         return hr;
    27.     });
    28.  
    29.     ComPtr<IAsyncAction> spStartRecordOperation;
    30.     IFR(_mediaCapture->StartRecordToCustomSinkAsync(_mediaEncodingProfile.Get(), spMediaExtension.Get(), &spStartRecordOperation));

    We have already checked that the initialization of the IMediaCapture is done on the Unity Main Thread. Microsoft's documentation says it must be done on the UI thread. I guess the UI thread is the same as the Unity main thread in Unity Editor.

    The problem does not appear in WSA and Standalone builds.

    Any clue of what could be the problem? What are the differences between 2017.2.4 and 2018.3 in terms of thread management?

    Thank you for your help
     
  2. timke

    timke

    Unity Technologies

    Joined:
    Nov 30, 2017
    Posts:
    67
    Hi,

    This appears to be a bug/regression. Could you please file a bug report on it and post the case number to this thread?

    I don't know offhand why this doesn't work in the Editor but does work in WindowsStandalone (since they're both using Win32). Basically, the "UI Thread" is a UWP construct that doesn't really exist in Win32, and so yes the main thread and UI thread are one in the same. This hasn't changed between versions.

    Instead, it sounds like the messages that trigger the callback event aren't be processed by the Win32 message handler. Based on your description, it might be the message queue isn't being "pumped" until a UI message is posted, e.g. WM_SIZE for when the window is resize.

    Our team will investigate further.
     
  3. vthemereau_asobo

    vthemereau_asobo

    Joined:
    Feb 27, 2018
    Posts:
    3
  4. vthemereau_asobo

    vthemereau_asobo

    Joined:
    Feb 27, 2018
    Posts:
    3
    After investigation by Unity QA, the bug is not on Unity side but probably on Microsoft Media Foundation. Here is a workaround provided by Unity team:


    Code (CSharp):
    1.  
    2. // This class implements a work-around to a bug within ABI::Windows::Media::Capture::IMediaCapture WinRT interface, in which the
    3. // Completed event on the IAsyncAction returned by IMediaCapture::StartRecordToCustomSinkAsync isn't invoked by COM. While the
    4. // asynchronous operation itself does appear to complete properly, the IAsyncAction representing the operation isn't updated by COM,
    5. // unless we explicitly perform some sort of COM operation, which is precisely what this work-around does.
    6.  
    7. // We create a basic timer which will (on regular intervals) call CoCreateInstance for some COM interface, this will "poke" COM into
    8. // doing work, at which time it'll process any pending messages, including our StartRecordToCustomSinkAsync completed event.
    9.  
    10. // IMPORTANT: Since IMediaCapture is created and processed on the main (UI) thread, we must also invoke COM on the same thread for this
    11. // to work. Since Timer events are always called from the Threadpool, we create a Task with the main thread Scheduler so COM is called
    12. // from the main thread.
    13.  
    14. // Usage:
    15. //
    16. // Call WorkaroundForWinRTCallbackBug.StartPokingCOM() within the StartCapture method to start the timer
    17. // Once the StartCapture operation completes (successfully or not) call WorkaroundForWinRTCallbackBug.StopPokingCOM() to stop the timer.
    18. //
    19. // The methods are thread-safe but StartPokingCOM() MUST be called from the main thread so we can retrieve the task Scheduler for it.
    20. // Note: currently in Unity Coroutines are all executed on the main "game" thread.
    21.  
    22. public class WorkaroundForWinRTCallbackBug
    23. {
    24.     public static void StartPokingCOM()
    25.     {
    26.         lock (_locker)
    27.         {
    28.             StopPokingCOM();
    29.  
    30.             _scheduler = System.Threading.Tasks.TaskScheduler.FromCurrentSynchronizationContext();
    31.  
    32.             _invokeTimer = new System.Timers.Timer(60);
    33.             _invokeTimer.Elapsed += OnInvokeTimerElapsed;
    34.             _invokeTimer.AutoReset = true;
    35.             _invokeTimer.Start();
    36.         }
    37.     }
    38.  
    39.     public static void StopPokingCOM()
    40.     {
    41.         lock (_locker)
    42.         {
    43.             if (_invokeTimer != null)
    44.             {
    45.                 _invokeTimer.Stop();
    46.                 _invokeTimer.Dispose();
    47.                 _invokeTimer = null;
    48.             }
    49.  
    50.             _scheduler = null;
    51.         }
    52.     }
    53.  
    54.     private static System.Timers.Timer _invokeTimer;
    55.     private static System.Threading.Tasks.TaskScheduler _scheduler;
    56.     private static object _locker = new object();
    57.  
    58.     private static void OnInvokeTimerElapsed(object sender, System.Timers.ElapsedEventArgs args)
    59.     {
    60.         try
    61.         {
    62.             System.Threading.Tasks.Task.Factory.StartNew(new Action(() =>
    63.             {
    64.                 try
    65.                 {
    66.                     // Create instance of INetworkListManager COM interface
    67.                     // The choice to use the interface is mostly arbitrary, but since it's the one that was making
    68.                     // IMediaCapture work in 2017.4 we'll keep using it.
    69.                     // NOTE: This call will probably fail will a COM exception; we don't care since all we need is to invoke CoCreateInstance
    70.  
    71.                     var tp = Type.GetTypeFromCLSID(new Guid("DCB00000-570F-4A9B-8D69-199FDBA5723B"));
    72.                     var networkListMan = Activator.CreateInstance(tp);
    73.                 }
    74.                 catch { };
    75.  
    76.             }), System.Threading.CancellationToken.None, System.Threading.Tasks.TaskCreationOptions.None, _scheduler);
    77.         }
    78.         catch { }
    79.     }
    80. }