Bug Unity Editor crash on play mode exit

Hello, we struggled with some problem on unity play mode exit while trying to delete channels. Normal cleaning up (delete channels and continue playing) is ok.


1) vivox ver: com.unity.services.vivox-15.1.170000
2) Unity ver: 2020.3.26f1
3) Build target: Android

Crash log:

Stacktrace:



  at <unknown> <0xffffffff>

  at (wrapper managed-to-native) VivoxCoreInstancePINVOKE.vx_req_sessiongroup_remove_session_t_session_handle_set (System.Runtime.InteropServices.HandleRef,string) [0x0001b] in <5437eccd060948ef9868f0cf9dbf3609>:0

  at vx_req_sessiongroup_remove_session_t.set_session_handle (string) [0x00001] in E:\UnityProjects\drive-zone\Library\PackageCache\com.unity.services.vivox@c9c00ead863c-1649177403252\Runtime\VivoxUnity\swig\vx_req_sessiongroup_remove_session_t.cs:63

  at VivoxUnity.Private.ChannelSession.Disconnect (System.AsyncCallback) [0x0005d] in E:\UnityProjects\drive-zone\Library\PackageCache\com.unity.services.vivox@c9c00ead863c-1649177403252\Runtime\VivoxUnity\Private\ChannelSession.cs:397

  at VivoxUnity.Private.LoginSession.WaitDeleteChannelSession (VivoxUnity.ChannelId) [0x00001] in E:\UnityProjects\drive-zone\Library\PackageCache\com.unity.services.vivox@c9c00ead863c-1649177403252\Runtime\VivoxUnity\Private\LoginSession.cs:1242

  at VivoxUnity.Private.LoginSession.DeleteChannelSession (VivoxUnity.ChannelId) [0x00057] in E:\UnityProjects\drive-zone\Library\PackageCache\com.unity.services.vivox@c9c00ead863c-1649177403252\Runtime\VivoxUnity\Private\LoginSession.cs:564

  at Core.VoiceChat.VivoxVoiceServiceImpl.CleanUp () [0x000ba] in E:\UnityProjects\drive-zone\Assets\Scripts\Services\VoiceSystem\VivoxImplementation\VivoxVoiceServiceImpl.cs:469

  at Core.VoiceChat.VivoxVoiceServiceImpl.Dispose () [0x00001] in E:\UnityProjects\drive-zone\Assets\Scripts\Services\VoiceSystem\VivoxImplementation\VivoxVoiceServiceImpl.cs:589

  at BehaviourInject.Internal.SingleAutocomposeDependency.Dispose () [0x00015] in E:\UnityProjects\drive-zone\Assets\BInject\Scripts\BehaviourInject\IDependency.cs:80

  at BehaviourInject.Context.DisposeDependencies () [0x00007] in E:\UnityProjects\drive-zone\Assets\BInject\Scripts\BehaviourInject\Context.cs:520

  at BehaviourInject.Context.Destroy () [0x00023] in E:\UnityProjects\drive-zone\Assets\BInject\Scripts\BehaviourInject\Context.cs:495

  at Contexts.ApplicationContext.OnDestroy () [0x00001] in E:\UnityProjects\drive-zone\Assets\Scripts\Contexts\ApplicationContext.cs:119

  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) [0x00020] in <695d1cc93cca45069c528c15c9fdd749>:0

 

Up

 

Hi, I'd just like to chime in to say that we are seeing the same issue and I was happy to see this post and that it therefore isn't just us.

It happens 100% of the time for us when we leave Play mode (we've had to add a developer option to disable Vivox entirely as it's obviously impeding our workflow).

Our stacktrace is the following (I had to remove some parts for NDA reasons):

Code (CSharp):

1.  
2. =================================================================
3.     Managed Stacktrace:
4. =================================================================
5.       at <unknown> <0xffffffff>
6.       at VivoxCoreInstancePINVOKE:vx_req_sessiongroup_remove_session_t_session_handle_set <0x0011b>
7.       at vx_req_sessiongroup_remove_session_t:set_session_handle <0x000ca>
8.       at VivoxUnity.Private.ChannelSession:Disconnect <0x0056a>
9.       [SNIP]
10.       at PlayerVoice:OnDestroy <0x000aa>
11.       at System.Object:runtime_invoke_void__this__ <0x00187>
12. =================================================================
13. Received signal SIGSEGV
14. Obtained 30 stack frames
15. 0x000002448817149c (Mono JIT Code) (wrapper managed-to-native) VivoxCoreInstancePINVOKE:vx_req_sessiongroup_remove_session_t_session_handle_set (System.Runtime.InteropServices.HandleRef,string)
16. 0x00000244881712fb (Mono JIT Code) [vx_req_sessiongroup_remove_session_t.cs:63] vx_req_sessiongroup_remove_session_t:set_session_handle (string)
17. 0x000002448817056b (Mono JIT Code) [ChannelSession.cs:403] VivoxUnity.Private.ChannelSession:Disconnect (System.AsyncCallback)
18. [SNIP]
19. 0x000002448816f53b (Mono JIT Code) [PlayerVoice.cs:35] PlayerVoice:OnDestroy ()
20. 0x00000244118f4518 (Mono JIT Code) (wrapper runtime-invoke) object:runtime_invoke_void__this__ (object,intptr,intptr,intptr)
21. 0x00007fff058ae0a4 (mono-2.0-bdwgc) [mini-runtime.c:3445] mono_jit_runtime_invoke
22. 0x00007fff057ee744 (mono-2.0-bdwgc) [object.c:3066] do_runtime_invoke
23. 0x00007fff057ee8dc (mono-2.0-bdwgc) [object.c:3113] mono_runtime_invoke
24. 0x00007ff6a0fff454 (Unity) scripting_method_invoke
25. 0x00007ff6a0ffa074 (Unity) ScriptingInvocation::Invoke
26. 0x00007ff6a0ffa15e (Unity) ScriptingInvocation::InvokeChecked
27. 0x00007ff6a1073ff6 (Unity) SerializableManagedRef::CallMethod
28. 0x00007ff6a0fd8f56 (Unity) MonoBehaviour::WillDestroyComponent
29. 0x00007ff6a09a1d01 (Unity) GameObject::WillDestroyGameObject
30. 0x00007ff6a0cac731 (Unity) PreDestroyRecursive
31. 0x00007ff6a0caa757 (Unity) DestroyObjectHighLevel_Internal
32. 0x00007ff6a0caa394 (Unity) DestroyObjectHighLevel
33. 0x00007ff6a0cb4cde (Unity) DestroyWorldObjects
34. 0x00007ff6a1ee1da1 (Unity) EditorSceneManager::RestoreSceneBackups
35. 0x00007ff6a1bf3aed (Unity) PlayerLoopController::ExitPlayMode
36. 0x00007ff6a1c02b41 (Unity) PlayerLoopController::SetIsPlaying
37. 0x00007ff6a1c05f3d (Unity) Application::TickTimer
38. 0x00007ff6a204e10a (Unity) MainMessageLoop
39. 0x00007ff6a20529db (Unity) WinMain
40. 0x00007ff6a337427e (Unity) __scrt_common_main_seh
41. 0x00007fff907954e0 (KERNEL32) BaseThreadInitThunk
42. 0x00007fff9142485b (ntdll) RtlUserThreadStart
43.  

Again, this happens any time we hit exit out of play mode in the editor when we are connected to a channel.

Can you advise on what we can do to avoid this?

 

1) It is happening in edtior. It happens 100% of the time for us when we leave Play mode.

2)

Code (CSharp):

1. List<ChannelId> channelsToDelete = new List<ChannelId>();
2.  
3.             foreach (var channelId in _loginSession.ChannelSessions.Keys)
4.             {
5.                 channelsToDelete.Add(channelId);
6.             }
7.  
8.             for (int i = 0; i < channelsToDelete.Count; i++)
9.             {
10.                 var id = channelsToDelete[i];
11.  
12.                 var session = _loginSession.GetChannelSession(id);
13.  
14.                 if (session != null)
15.                 {
16.                     _loginSession.DeleteChannelSession(id);
17.                 }
18.             }

3) It is happening on Gameobject context holder destruction

 

PC Windows editor

 

Hello, just wanted to confirm that we're also experiencing same issue. Once we login to vivox, if we exit play mode without firstly

• logging out,
• cleaning up client and
• uninitializing vivox

unity editor crashes. Using unity 2020.3.27 in windows and vivox 15.1.180000-pre.1 (currently latest version)

Haven't tested yet on android or other platforms.

 

Hello. Any information for us?

 

Unfortunalty, the last version of Vivox still has this problem.

@dylan_unity3d, we have the same clean up code inside. The problem is that we are trying to disconnect connected channel on exit from playmode

 

Temporary solved this problem by adding async channel disconnection and saving application quit flag so if disconnection proccess detect if game is exiting than interrupt disconnection. I've left it only in editor

 

Same here. Vivox 15.1.180001-pre.2, Unity 2020.3.26f1 (also tried 2020.3.36f1 and 2021.3.5f1), PC Windows Editor

Code (CSharp):

1. Stacktrace:
2.  
3.   at <unknown> <0xffffffff>
4.   at (wrapper managed-to-native) VivoxCoreInstancePINVOKE.vx_req_sessiongroup_remove_session_t_session_handle_set (System.Runtime.InteropServices.HandleRef,string) [0x0001b] in <eb812ede3c3f41b6abd899e76eac7407>:0
5.   at vx_req_sessiongroup_remove_session_t.set_session_handle (string) [0x00001] in W:\work\unity\sb-win\quantum_unity\Library\PackageCache\com.unity.services.vivox@15.1.180001-pre.2\Runtime\VivoxUnity\swig\vx_req_sessiongroup_remove_session_t.cs:63
6.   at VivoxUnity.Private.ChannelSession.Disconnect (System.AsyncCallback) [0x0005d] in W:\work\unity\sb-win\quantum_unity\Library\PackageCache\com.unity.services.vivox@15.1.180001-pre.2\Runtime\VivoxUnity\Private\ChannelSession.cs:422
7.   at Game.code.control.vivox.VivoxAPI/Channel/<>c__DisplayClass10_0.<leave>b__0 (System.AsyncCallback) [0x00000] in W:\work\unity\sb-win\quantum_unity\generated-by-compiler\game\macros\Assets\Game\code\control\vivox\VivoxAPI.Channel.transformed.cs:148
8.   at Game.code.control.vivox.VivoxAPI.callbackToFuture (System.Action`1<System.AsyncCallback>,System.Action`1<System.IAsyncResult>) [0x0001a] in W:\work\unity\sb-win\quantum_unity\generated-by-compiler\game\macros\Assets\Game\code\control\vivox\VivoxAPI.transformed.cs:163
9.   at Game.code.control.vivox.VivoxAPI/Channel.leave (bool) [0x000c9] in W:\work\unity\sb-win\quantum_unity\generated-by-compiler\game\macros\Assets\Game\code\control\vivox\VivoxAPI.Channel.transformed.cs:148
10.   at Game.code.control.GameController/Init/<>c__DisplayClass135_15.<.ctor>g__leaveCurrentVivoxChannel|116 () [0x00060] in W:\work\unity\sb-win\quantum_unity\generated-by-compiler\game\macros\Assets\Game\code\control\GameController.Init.transformed.cs:807
11.   at FPCSharpUnity.core.reactive.Subscription.unsubscribe () [0x0000f] in W:\work\unity\sb-win\quantum_code\FPCSharpUnity.core\reactive\Subscription.cs:48
12.   at FPCSharpUnity.core.reactive.Subscription.Dispose () [0x00000] in W:\work\unity\sb-win\quantum_code\FPCSharpUnity.core\reactive\Subscription.cs:58
13.   at FPCSharpUnity.core.dispose.DisposableTracker.Dispose () [0x000a9] in W:\work\unity\sb-win\quantum_code\FPCSharpUnity.core\dispose\DisposableTracker.cs:120
14.   at Game.code.control.GameController/Init.dispose (bool) [0x00096] in W:\work\unity\sb-win\quantum_unity\generated-by-compiler\game\macros\Assets\Game\code\control\GameController.Init.transformed.cs:885
15.   at Game.code.control.GameController/Init.Dispose () [0x00001] in W:\work\unity\sb-win\quantum_unity\generated-by-compiler\game\macros\Assets\Game\code\control\GameController.Init.transformed.cs:872
16.   at Game.code.control.GameInitController.disposeLastSession (FPCSharpUnity.core.log.ILog) [0x0006f] in W:\work\unity\sb-win\quantum_unity\generated-by-compiler\game\macros\Assets\Game\code\control\GameInitController.transformed.cs:285
17.   at Game.code.control.GameControllerBasic.shutdownSimulation (string) [0x00132] in W:\work\unity\sb-win\quantum_unity\generated-by-compiler\game\macros\Assets\Game\code\control\GameControllerBasic.transformed.cs:218
18.   at Game.code.control.GameController/Init.dispose (bool) [0x0006d] in W:\work\unity\sb-win\quantum_unity\generated-by-compiler\game\macros\Assets\Game\code\control\GameController.Init.transformed.cs:882
19.   at Game.code.control.GameController/Init.Dispose () [0x00001] in W:\work\unity\sb-win\quantum_unity\generated-by-compiler\game\macros\Assets\Game\code\control\GameController.Init.transformed.cs:872
20.   at Game.code.control.GameController.OnDestroy () [0x00000] in W:\work\unity\sb-win\quantum_unity\generated-by-compiler\game\macros\Assets\Game\code\control\GameController.transformed.cs:129
21.   at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) [0x00020] in <695d1cc93cca45069c528c15c9fdd749>:0
22.  
23. =================================================================
24. Got a SIGSEGV while executing native code. This usually indicates
25. a fatal error in the mono runtime or one of the native libraries
26. used by your application.
27. =================================================================
28.  
29. Received signal SIGSEGV
30. Stack trace:
31. 0x00000252c756d7d6 (Mono JIT Code) (wrapper managed-to-native) VivoxCoreInstancePINVOKE:vx_req_sessiongroup_remove_session_t_session_handle_set (System.Runtime.InteropServices.HandleRef,string)
32. 0x00000252c756d64b (Mono JIT Code) [vx_req_sessiongroup_remove_session_t.cs:63] vx_req_sessiongroup_remove_session_t:set_session_handle (string)
33. 0x00000252c756c9bb (Mono JIT Code) [ChannelSession.cs:422] VivoxUnity.Private.ChannelSession:Disconnect (System.AsyncCallback)
34. 0x00000252c756c4f4 (Mono JIT Code) [VivoxAPI.Channel.transformed.cs:148] Game.code.control.vivox.VivoxAPI/Channel/<>c__DisplayClass10_0:<leave>b__0 (System.AsyncCallback)
35. 0x00000250dff59cf9 (Mono JIT Code) [VivoxAPI.transformed.cs:163] Game.code.control.vivox.VivoxAPI:callbackToFuture (System.Action`1<System.AsyncCallback>,System.Action`1<System.IAsyncResult>)
36. 0x00000252c756bfa3 (Mono JIT Code) [VivoxAPI.Channel.transformed.cs:148] Game.code.control.vivox.VivoxAPI/Channel:leave (bool)
37. 0x00000252c756aa63 (Mono JIT Code) [GameController.Init.transformed.cs:807] Game.code.control.GameController/Init/<>c__DisplayClass135_15:<.ctor>g__leaveCurrentVivoxChannel|116 ()
38. 0x00000250d662751a (Mono JIT Code) [Subscription.cs:48] FPCSharpUnity.core.reactive.Subscription:unsubscribe ()
39. 0x00000250e228d063 (Mono JIT Code) [Subscription.cs:58] FPCSharpUnity.core.reactive.Subscription:Dispose ()
40. 0x00000250c37d4d4c (Mono JIT Code) [DisposableTracker.cs:120] FPCSharpUnity.core.dispose.DisposableTracker:Dispose ()
41. 0x00000252c756554e (Mono JIT Code) [GameController.Init.transformed.cs:885] Game.code.control.GameController/Init:dispose (bool)
42. 0x00000252c7564f4b (Mono JIT Code) [GameController.Init.transformed.cs:872] Game.code.control.GameController/Init:Dispose ()
43. 0x00000251ab0981f9 (Mono JIT Code) [GameInitController.transformed.cs:285] Game.code.control.GameInitController:disposeLastSession (FPCSharpUnity.core.log.ILog)
44. 0x00000252c75669d3 (Mono JIT Code) [GameControllerBasic.transformed.cs:218] Game.code.control.GameControllerBasic:shutdownSimulation (string)
45. 0x00000252c756546b (Mono JIT Code) [GameController.Init.transformed.cs:882] Game.code.control.GameController/Init:dispose (bool)
46. 0x00000252c7564f4b (Mono JIT Code) [GameController.Init.transformed.cs:872] Game.code.control.GameController/Init:Dispose ()
47. 0x00000252c7564e73 (Mono JIT Code) [GameController.transformed.cs:129] Game.code.control.GameController:OnDestroy ()
48. 0x00000250c32bdba8 (Mono JIT Code) (wrapper runtime-invoke) object:runtime_invoke_void__this__ (object,intptr,intptr,intptr)
49. 0x00007ffdbdb4f1e0 (mono-2.0-bdwgc) [mini-runtime.c:2849] mono_jit_runtime_invoke
50. 0x00007ffdbdad2ac2 (mono-2.0-bdwgc) [object.c:2921] do_runtime_invoke
51. 0x00007ffdbdadbb1f (mono-2.0-bdwgc) [object.c:2968] mono_runtime_invoke
52. 0x00007ff75753cf74 (Unity) scripting_method_invoke
53. 0x00007ff757538251 (Unity) ScriptingInvocation::Invoke
54. 0x00007ff7575383ae (Unity) ScriptingInvocation::InvokeChecked
55. 0x00007ff757592b86 (Unity) SerializableManagedRef::CallMethod
56. 0x00007ff757515ad6 (Unity) MonoBehaviour::WillDestroyComponent
57. 0x00007ff756f6c421 (Unity) GameObject::WillDestroyGameObject
58. 0x00007ff7572498c1 (Unity) PreDestroyRecursive
59. 0x00007ff7572475af (Unity) DestroyObjectHighLevel_Internal
60. 0x00007ff7572471c4 (Unity) DestroyObjectHighLevel
61. 0x00007ff757251d4e (Unity) DestroyWorldObjects
62. 0x00007ff7581e99c8 (Unity) EditorSceneManager::RestoreSceneBackups
63. 0x00007ff757f0c6ed (Unity) PlayerLoopController::ExitPlayMode
64. 0x00007ff757f1b7c7 (Unity) PlayerLoopController::SetIsPlaying
65. 0x00007ff757f1e627 (Unity) Application::TickTimer
66. 0x00007ff758363d61 (Unity) MainMessageLoop
67. 0x00007ff758367db1 (Unity) WinMain
68. 0x00007ff7599f75e2 (Unity) __scrt_common_main_seh
69. 0x00007ffe61067034 (KERNEL32) BaseThreadInitThunk
70. 0x00007ffe63042651 (ntdll) RtlUserThreadStart

Code for the workaround:

PlayStateUtils.cs

Code (CSharp):

1. using ExhaustiveMatching;
2. using FPCSharpUnity.core.reactive;
3.  
4. namespace FPCSharpUnity.unity.Utilities;
5.  
6. #if UNITY_EDITOR
7. [UnityEditor.InitializeOnLoad]
8. #endif
9. public static class PlayStateUtils {
10.   /// <summary>
11.   /// Becomes `false` when we exit the play mode and also when we're about to exit play mode.
12.   /// <para/>
13.   /// Works in Editor (by using <see cref="UnityEditor.EditorApplication.playModeStateChanged"/>) and Player (in player
14.   /// this always returns true).
15.   /// </summary>
16.   public static readonly IRxVal<bool> isPlayingAndNotExitingPlayMode;
17.  
18.   static PlayStateUtils() {
19.     isPlayingAndNotExitingPlayMode =
20. #if UNITY_EDITOR
21.       createIsPlayingAndNotExitingPlayModeForEditor();
22. #else
23.       // If we're not in editor, then we're just always playing.
24.       RxVal.cached(true)
25. #endif
26.    
27. #if UNITY_EDITOR
28.     static IRxVal<bool> createIsPlayingAndNotExitingPlayModeForEditor() {
29.       var rx = RxRef.a(UnityEditor.EditorApplication.isPlaying);
30.       UnityEditor.EditorApplication.playModeStateChanged += change => rx.value = change switch {
31.         UnityEditor.PlayModeStateChange.EnteredEditMode => false,
32.         UnityEditor.PlayModeStateChange.ExitingEditMode => false,
33.         UnityEditor.PlayModeStateChange.EnteredPlayMode => true,
34.         UnityEditor.PlayModeStateChange.ExitingPlayMode => false,
35.         _ => throw ExhaustiveMatch.Failed(change)
36.       };
37.       return rx;
38.     }
39. #endif
40.     ;
41.   }
42. }

Leave method:

Code (CSharp):

1.     /// <summary>
2.     /// Leaves the Vivox channel and (optionally) deletes the channel from the channel list.
3.     /// </summary>
4.     public Future<Try<Unit>> leave(bool deleteFromChannelList = true) {
5.       var isPlayingAndNotExitingPlayMode = PlayStateUtils.isPlayingAndNotExitingPlayMode.value;
6.      
7.       log.mDebug(
8.         $"{Macros.classAndMethodNameShort}({deleteFromChannelList.echo()}): "
9.         + $"channel state = {backing.ChannelState}, audio state = {backing.AudioState}, "
10.         + $"text state = {backing.TextState}, Vivox session state = {api.session.State}, "
11.         + $"{isPlayingAndNotExitingPlayMode.echo()}"
12.       );
13.      
14.       // Vivox has a bug where it segfaults the whole Unity editor if you try to leave the channel when play mode is
15.       // being exited in the editor.
16.       //
17.       // This is a workaround for that :)
18.       //
19.       // https://forum.unity.com/threads/unity-editor-crash-on-play-mode-exit.1264904/
20.       if (isPlayingAndNotExitingPlayMode) {
21.         return callbackToFuture(cb => backing.Disconnect(cb), _ => {
22.           // Delete the channel entry from the channel list after we successfully leave.
23.           if (deleteFromChannelList) api.session.DeleteChannelSession(id);
24.         });
25.       }
26.       else {
27.         log.mDebug(
28.           "Not really leaving Vivox channel to work around a Vivox bug that happens in Unity editor when currently "
29.           + "exiting play mode."
30.         );
31.         return Future.successful(Try.value(Unit._));
32.       }
33.     }

 

Any news? We have a lot of crashes on android platfrom, guys

 

@dylan_unity3d, why are you ignoring us? We have lots of crashes on android platform at the same conditions. Even crashlog almost the same in core as in the edtior. And as you know, crashes and anrs are the wrost google thing that can be...

 

• To confirm are you now seeing these crashes in build and not just editor?

Yes, i've seen this crashes on different android devices also. Almost all devices. The reason alwas the same: disconnecting channel on app quiting.

Also in stacktrace we ve seen the same point as in editor:

VivoxCoreInstancePINVOKE.vx_req_sessiongroup_remove_session_t_session_handle_set (System.Runtime.InteropServices.HandleRef,string) [0x0001b] in <eb812ede3c3f41b6abd899e76eac7407>:0

• Were you seeing these crashes before you implemented your workaround?
  
  Only on new version. We have been using 5.14 previously and there were no such crashes
  
  
• Are you seeing this on every android device you test on?
  
  Almost
  
  
• What android devices have you tested this on? (Pixel 4, Samsung 7, etc)

Samsung s8, Samsing s20, Redmi Note 6 Pro, etc

• Are you still seeing this issue in editor? I know you stated you found a workaround above, is that fixing your issue in editor?

After i've implemeted my "fix" by setting exit flag before exit, it has gone. But after release i've got another crashes that i've described in another topic.

 

Any news?

 

Apologies for the double post but I wasn't able to find this thread before. I'm experiencing the same issue on the newest vivox version: https://forum.unity.com/threads/cra...in-ondisable-while-exiting-play-mode.1420751/

 

Unfortunatly vivox is so disappointing with so many crashes at all platforms( even with such high pricing. We are working on another solution that allow us avoid vivox(

 

Hello,
i had the same problem. I solved it by removing LoginSession.Logout(); from my onDestroy method.

Sdk Ver: 15.1.20000000-pre.1

 

Hey - in case it's still bugging anybody, I tracked this down to an issue with unity's handling of async functions in OnDestroy. Simply triggering an async null task that had nothing to do with Vivox called it. Not sure if Vivox is part of the problem, but simply not logging out solved it for us. I suspect if we were willing to syncronously logout it'd have solved the issue as well.

 

I will say though, even if Vivox is 'innocent' on this - not sure it should have taken a single programmer 1 day to track it down after months of back and forth with a team dedicated to a commercial voip product... feels like there's customer support room for improvement