Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Unite 2023 Registrations are now LIVE! We are thrilled to announce that Unite 2023 Registration is now live and open for all!
    Dismiss Notice
New Forum User Notice Unite 2023 Registrations are now LIVE!

Ubuntu Headless Build - TLS Handshake fails

Discussion in 'Linux' started by ChainsawesomeJS, Aug 24, 2018.

  1. ChainsawesomeJS

    ChainsawesomeJS

    Joined:
    May 15, 2015
    Posts:
    29
    Hey there!

    We've been using Ubuntu docker containers to host our game servers for a while now and have never had issues with having this server connect to another via Secure Websockets. We've upgraded from Unity 5.6 up to 2018.1 without issue.

    However, after upgrading to Unity 2018.2, we first noticed an issue calling HTTPS requests through the UnityWebRequest. We were getting error codes 0 with the description Unknown error. Weirdly enough, installing curl in my container ended up resolving this issue. Now I'm stuck with the error described earlier. We're using Websocket-sharp and never had any issue connecting to our server until we updated to 2018.2. The websocket connector still works in 2018.2 and windows but fails on Linux with a TLS handshake error when trying to connect.

    I was wondering if it could be another issue of a missing dependency or that the Linux player is somehow wrongly wrapping the TLS encryption process? I've tried with Ubuntu 16.04 and 18.04.

    Cheers!
     
    ChainsawesomeJS, Aug 24, 2018
    #1

  2. andreasreich

    andreasreich

    Unity Technologies

    Joined:
    Sep 24, 2017
    Posts:
    55
    Hi! Unity should come with everything bundled, I don't think you're missing anything.

    In 2018.2 we got implemented a new TLS backend which we can use in Mono (.Net4 only) and UnityWebRequest. One of the consequences of that is that we can now use platform specific certificate stores on every individual platforms where before we would have shipped a "hardcoded store". On Linux we're using the store in /etc/ssl/certs/ca-certificates.crt

    So since it works on Windows it does sound a bit like your certificate is trusted by one OS and not by the other. I have not encountered something like that yet but this is also the first report of a TLS handshake with the new system working on one desktop platform but not the other. So maybe something wrong on our side after all.
    AlsoI am a bit troubled that you get "Unknown error" with UnityWebRequest, this should be more specific. Can you please file a bugreport with a repro case? Thank you!
     
    andreasreich, Aug 24, 2018
    #2

  3. ChainsawesomeJS

    ChainsawesomeJS

    Joined:
    May 15, 2015
    Posts:
    29
    Hey Andreas!

    Thanks for the quick response! We already filed a bug for the Unknown Error issue. It seems like the error is no longer Unknown Error in 2018.2.5f1 though. But I still need to install curl to get HTTPS working.

    I ran a couple tests, like adding our certificate to the list of trusted certificates, accepting any certificates and whatnot but it still doesn't work. Weirdly enough, in a smaller project, I'm unable to reproduce the issue, so I'm beginning to wonder if I might be doing something wrong.

    I have the full stack trace of the issue happening. We're on Unity 2018.2.5f1 with the 4.x scripting runtime :

    Code (CSharp):
    1.  
    2. WebSocketSharp.WebSocketException: An error has occurred during a TLS handshake. ---> System.AggregateException: One or more errors occurred. ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> System.ArgumentException: GCHandle value belongs to a different domain
    3.  at System.Runtime.InteropServices.GCHandle.op_Explicit (System.IntPtr value) [0x00025] in <f2e6809acb14476a81f399aeb800f8f2>:0
    4.  at Mono.Unity.UnityTlsContext.WriteCallback (System.Void* userData, System.Byte* data, Mono.Unity.size_t bufferLen, Mono.Unity.UnityTls unitytls_errorstate* errorState) [0x00006] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    5.  at (wrapper managed-to-native) System.Object.wrapper_native_0x8ab3370(Mono.Unity.UnityTls/unitytls_tlsctx*,Mono.Unity.UnityTls/unitytls_errorstate*)
    6.  at Mono.Unity.UnityTlsContext.ProcessHandshake () [0x0001c] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    7.  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status) [0x0003e] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    8.  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus)
    9.  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    10.  at Mono.Net.Security.AsyncProtocolRequest <ProcessOperation>d__24.MoveNext () [0x000ff] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    11. --- End of stack trace from previous location where exception was thrown ---
    12.  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <f2e6809acb14476a81f399aeb800f8f2>:0
    13.  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <f2e6809acb14476a81f399aeb800f8f2>:0
    14.  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <f2e6809acb14476a81f399aeb800f8f2>:0
    15.  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <f2e6809acb14476a81f399aeb800f8f2>:0
    16.  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable ConfiguredTaskAwaiter.GetResult () [0x00000] in <f2e6809acb14476a81f399aeb800f8f2>:0
    17.  at Mono.Net.Security.AsyncProtocolRequest <StartOperation>d__23.MoveNext () [0x0008b] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    18.  --- End of inner exception stack trace ---
    19.  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <f2e6809acb14476a81f399aeb800f8f2>:0
    20.  at Mono.Net.Security.MobileAuthenticatedStream <ProcessAuthentication>d__47.MoveNext () [0x00254] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    21.  --- End of inner exception stack trace ---
    22.  at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskCanceledExceptions) [0x00011] in <f2e6809acb14476a81f399aeb800f8f2>:0
    23.  at System.Threading.Tasks.Task.Wait (System.Int32 millisecondsTimeout, System.Threading.CancellationToken cancellationToken) [0x00043] in <f2e6809acb14476a81f399aeb800f8f2>:0
    24.  at System.Threading.Tasks.Task.Wait () [0x00000] in <f2e6809acb14476a81f399aeb800f8f2>:0
    25.  at Mono.Net.Security.MobileAuthenticatedStream.AuthenticateAsClient (System.String targetHost, System.Security.Cryptography.X509Certificates.X509CertificateCollection clientCertificates, System.Security.Authentication.SslProtocols enabledSslProtocols, System.Boolean checkCertificateRevocation) [0x0000d] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    26.  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.AuthenticateAsClient(string,System.Security.Cryptography.X509Certificates.X509CertificateCollection,System.Security.Authentication.SslProtocols,bool)
    27.  at Mono.Net.Security.MobileAuthenticatedStream.AuthenticateAsClient (System.String targetHost) [0x0000d] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    28.  at System.Net.Security.SslStream.AuthenticateAsClient (System.String targetHost) [0x00006] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    29.  at WebSocketSharp.WebSocket.setClientStream () [0x000de] in <2f2e2c67bdfd47ac97ec567117e0069d>:0
    30.  --- End of inner exception stack trace ---
    31.  at WebSocketSharp.WebSocket.setClientStream () [0x00101] in <2f2e2c67bdfd47ac97ec567117e0069d>:0
    32. at WebSocketSharp.WebSocket.doHandshake () [0x00001] in <2f2e2c67bdfd47ac97ec567117e0069d>:0
    33. at WebSocketSharp.WebSocket.connect ()
    34.  
     
    ChainsawesomeJS, Aug 24, 2018
    #3

  4. andreasreich

    andreasreich

    Unity Technologies

    Joined:
    Sep 24, 2017
    Posts:
    55
    Oh apparently we're using the system's curl on Linux, not our own. Wasn't aware, the configuration for curl (or whether we use it at all) is different on every platform. I'll confirm that later to be sure though.

    The inner most exception hints that this is not a verification issue hat all:
    Code (csharp):
    1. System.ArgumentException: GCHandle value belongs to a different domain
    2. at System.Runtime.InteropServices.GCHandle.op_Explicit (System.IntPtr value) [0x00025] in <f2e6809acb14476a81f399aeb800f8f2>:0
    As far as I know this should only happen if a disposed SSLStream was accessed. Looking at the current implementation of WebSocket.SetClientStream this can't be the case since it creates a new SSLStream every time. The only interesting thing in there I can think of is that if there is an exception, there is no explicit Dispose call to the SSLStream, so it would be cleaned up whenever the GC feels like it, potentially leading to issues.
    Another possibility is that Mono disposes the SslContext the GCHandle points to due to some other error, but then we should see that error instead.
    It's the first time I have an issue with GCHandle here, so really hard to diagnose without a repro project :(

    Does this issue happen on the first connection attempt? Do you have multiple connects in parallel in this project?
     
    andreasreich, Aug 25, 2018
    #4

  5. andreasreich

    andreasreich

    Unity Technologies

    Joined:
    Sep 24, 2017
    Posts:
    55
    andreasreich, Aug 31, 2018
    #5

  6. guido_unity

    guido_unity

    Joined:
    May 22, 2018
    Posts:
    3
    @andreasreich We also got a problem with Linux Standalone builds for Unity 2018.2. We didn't see the exact same errors as above, but it looks a lot like the same problem. When we run the app all external (web) calls fail with this error :

    Code (CSharp):
    1. <color=Red>Webcallback failed!   ==>> </color>Request (GET) => (<URL>/HealthCheck)
    2. Headers:
    3. Accept: application/json
    4.  
    5. Exception: System.Net.WebException: Error: SecureChannelFailure (One or more errors occurred.) ---> System.AggregateException: One or more errors occurred. ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> System.ArgumentException: GCHandle value belongs to a different domain
    6.   at System.Runtime.InteropServices.GCHandle.op_Explicit (System.IntPtr value) [0x00025] in <f2e6809acb14476a81f399aeb800f8f2>:0
    7.   at Mono.Unity.UnityTlsContext.WriteCallback (System.Void* userData, System.Byte* data, Mono.Unity.size_t bufferLen, Mono.Unity.UnityTls+unitytls_errorstate* errorState) [0x00006] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    8.   at (wrapper managed-to-native) System.Object.wrapper_native_0x8b49270(Mono.Unity.UnityTls/unitytls_tlsctx*,Mono.Unity.UnityTls/unitytls_errorstate*)
    9.   at Mono.Unity.UnityTlsContext.ProcessHandshake () [0x0001c] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    10.   at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status) [0x0003e] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    11.   at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus)
    12.   at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    13.   at Mono.Net.Security.AsyncProtocolRequest+<ProcessOperation>d__24.MoveNext () [0x000ff] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    14. --- End of stack trace from previous location where exception was thrown ---
    15.   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <f2e6809acb14476a81f399aeb800f8f2>:0
    16.   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <f2e6809acb14476a81f399aeb800f8f2>:0
    17.   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <f2e6809acb14476a81f399aeb800f8f2>:0
    18.   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <f2e6809acb14476a81f399aeb800f8f2>:0
    19.   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult () [0x00000] in <f2e6809acb14476a81f399aeb800f8f2>:0
    20.   at Mono.Net.Security.AsyncProtocolRequest+<StartOperation>d__23.MoveNext () [0x0008b] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    21.    --- End of inner exception stack trace ---
    22.   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <f2e6809acb14476a81f399aeb800f8f2>:0
    23.   at Mono.Net.Security.MobileAuthenticatedStream+<ProcessAuthentication>d__47.MoveNext () [0x00254] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    24.    --- End of inner exception stack trace ---
    25.   at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskCanceledExceptions) [0x00011] in <f2e6809acb14476a81f399aeb800f8f2>:0
    26.   at System.Threading.Tasks.Task.Wait (System.Int32 millisecondsTimeout, System.Threading.CancellationToken cancellationToken) [0x00043] in <f2e6809acb14476a81f399aeb800f8f2>:0
    27.   at System.Threading.Tasks.Task.Wait () [0x00000] in <f2e6809acb14476a81f399aeb800f8f2>:0
    28.   at Mono.Net.Security.MobileAuthenticatedStream.AuthenticateAsClient (System.String targetHost, System.Security.Cryptography.X509Certificates.X509CertificateCollection clientCertificates, System.Security.Authentication.SslProtocols enabledSslProtocols, System.Boolean checkCertificateRevocation) [0x0000d] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    29.   at Mono.Net.Security.MonoTlsStream.CreateStream (System.Byte[] buffer) [0x0007b] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    30.   at System.Net.WebConnection.CreateStream (System.Net.HttpWebRequest request) [0x00073] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    31.    --- End of inner exception stack trace ---
    32.   at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00058] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    33.   at System.Net.HttpWebRequest.GetResponse () [0x0000e] in <60233c34ef3e4e5f892759dbfc5ae27d>:0
    34.   at CI.HttpClient.Core.HttpBase.HandleStringResponseRead (System.Action`1[T] responseCallback) [0x00000] in <2a0ce68d4a9d476bb4d55a791bdf1068>:0
    35.   at CI.HttpClient.Core.HttpGet.GetString (System.Action`1[T] responseCallback) [0x00007] in <2a0ce68d4a9d476bb4d55a791bdf1068>:0
    36. Exception Body:
    This was working in Unity 2017.4. We are working on updating the project from Unity 2017.4.4f1 to Unity 2018.2.6f1

    Do you know why this happens?
     
    guido_unity, Sep 19, 2018
    #6

  7. andreasreich

    andreasreich

    Unity Technologies

    Joined:
    Sep 24, 2017
    Posts:
    55
    Yes, clearly the same issue. It's clear to me why it pops up starting with 2018.2 (because of the new UnityTls backend and more importantly its mono sided binding), but I'm still in the dark on how and why.

    Does this happen only in the Linux standalone? Do you have several connections in parallel?
    Would it be possible for you to file a bug with a repro case for this? Thank you
     
    andreasreich, Sep 20, 2018
    #7

  8. guido_unity

    guido_unity

    Joined:
    May 22, 2018
    Posts:
    3
    I already submitted a bug report on this number : 1082650 . QA also already reproduced this on there end, and have contacted development to look into this error.

    Thank you so much for your help
     
    Last edited: Sep 23, 2018
    guido_unity, Sep 20, 2018
    #8
    andreasreich likes this.

  9. BurningToad

    BurningToad

    Joined:
    Aug 14, 2013
    Posts:
    56
    We are having what sounds like a similar issue, with our players reporting that certain distros of Linux are unable to correctly communicate with various HTTPS URLs that our game tries to reach. It doesn't affect all Linux versions, just some. I don't have detailed error logs yet about the issue however. The issue mentioned here - https://issuetracker.unity3d.com/is...-gchandle-value-belongs-to-a-different-domain says fixed in 2019.1, has this fix been backported to 2018.2? We are using 2018.2.14. Our 2018.1.7 builds do not have this issue.
     
    BurningToad, Nov 6, 2018
    #9

  10. andreasreich

    andreasreich

    Unity Technologies

    Joined:
    Sep 24, 2017
    Posts:
    55
    Thanks for the feedback. I forgot to update this thread. The issue should be should be fixed respectively starting with:
    2019.1.0a6, 2018.3.0b8, 2018.2.14f

    Officially we only support Ubuntu. Apart from other potential Unity issues I heard that on other distros it can happen that we either fail to access the system certificate store or that we don't find all the root certificates there that are needed.
     
    andreasreich, Nov 7, 2018
    #10

  11. andreasreich

    andreasreich

    Unity Technologies

    Joined:
    Sep 24, 2017
    Posts:
    55
    Do you have the GCHandle issue described in this thread in particular or is it a "regular" authentication issue?
     
    andreasreich, Nov 7, 2018
    #11

  12. BurningToad

    BurningToad

    Joined:
    Aug 14, 2013
    Posts:
    56
    The issue persists even with 2018.2.15, so it doesn't look like the same issue. Fedora is the main distro that has been tested. The UnityWebRequest class error message is just "Unknown error" with no real information.
     
    BurningToad, Nov 13, 2018
    #12

  13. bartofzo

    bartofzo

    Joined:
    Mar 16, 2017
    Posts:
    150
    Getting the same error on 2018.2.16f1 Unity Linux headless build.

    Unknown error... running on an amazon EC2 instance.
     
    bartofzo, Nov 20, 2018
    #13

  14. dzonileona

    dzonileona

    Joined:
    Dec 2, 2012
    Posts:
    9
    Also have the same issue,
    2018.2.16f1 Linux headless build,
    it is working on Windows, iOS and android platform
     
    dzonileona, Nov 21, 2018
    #14

  15. PJayB

    PJayB

    Unity Technologies

    Joined:
    Apr 24, 2017
    Posts:
    105
    Unity only officially supports Ubuntu Linux, so it is looking (and failing to find) the certificate store where it would expect it to be. You can work around on Fedora by creating a symbolic link:

    mkdir -p /etc/ssl/certs && ln -s /etc/pki/tls/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt
     
    PJayB, Nov 21, 2018
    #15
    bartofzo likes this.

  16. bartofzo

    bartofzo

    Joined:
    Mar 16, 2017
    Posts:
    150
    Thanks a lot for this! It worked.
     
    bartofzo, Nov 27, 2018
    #16

  17. TheLastVertex

    TheLastVertex

    Joined:
    Sep 30, 2015
    Posts:
    126

    This is all new to me(Http requests / linux) , could I get some clarification on what to do with this info?

    I'm currently experiencing Unity web requests (v 2018.2.11 f 1) replying unknown error with error code 0 when running on Amazon Linux. I'm assuming this is potentially the same certificate problem?

    Is it possible to resolve this issue by avoiding UnityWebRequest and using .Net Http support instead?
     
    TheLastVertex, Feb 16, 2019
    #17

  18. TheLastVertex

    TheLastVertex

    Joined:
    Sep 30, 2015
    Posts:
    126
    I switched from UnityWebRequest to using System.Net.WebClient to send the request instead.

    This results in a certificate error, even on windows.

    I was however able to override the ServerCertificateValidationCallback to get the request to work on both windows and linux. I'm assuming this is not a safe or valid solution though?

    What's the best way to send http requests on linux?
     
    TheLastVertex, Feb 17, 2019
    #18

  19. starmindfr

    starmindfr

    Joined:
    Nov 7, 2016
    Posts:
    38
    hello any update on this ?
    I am using CentOS 7 clients and https urls form a client - server game and on centos with unity 2019.1.0f2 i could not connect webrequests anymore. Was working fine on old unity 2018. Also i build only for 64bits.
     
    starmindfr, Jul 3, 2019
    #19

  20. bartofzo

    bartofzo

    Joined:
    Mar 16, 2017
    Posts:
    150
    Don't know if it will work for your situation but I managed to get it working on Amazon Linux by running this shell script on the server instance:

    Code (csharp):
    1.  
    2. sudo mkdir -p /etc/ssl/certs
    3. sudo ln -s /etc/pki/tls/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt
    4.  
     
    bartofzo, Jul 3, 2019
    #20

  21. qrikko

    qrikko

    Joined:
    Mar 2, 2014
    Posts:
    9
    While we're at it, I'm on OpenSuse Leap and my ca-bundle is placed in: /etc/ssl/
    so need to link it like this:

    Code (CSharp):
    1. sudo ln -s /etc/ssl/ca-bundle.pem /etc/ssl/certs/ca-certificates.crt
     
    qrikko, Oct 21, 2019
    #21

  22. bartofzo

    bartofzo

    Joined:
    Mar 16, 2017
    Posts:
    150
    Be warned! This solution stopped working for me with Unity 2019.2.16f1.

    Downgraded back to 2019.2.12 and it works again. Remind me to never update Unity again.
     
    bartofzo, Dec 24, 2019
    #22

  23. andreasreich

    andreasreich

    Unity Technologies

    Joined:
    Sep 24, 2017
    Posts:
    55
    That's rather odd that this would break between those patch versions. My best guess is that it's because we backported a security fix where we switched from using not up-to-date OpenSSL version to MbedTLS in our backend. Would assume though that it still reads certs from the same location.
    You should consider filing a bug if you have a repro example. Note however that we officially don't support Amazon Linux. For what's supported in detail check here.
     
    andreasreich, Jan 2, 2020
    #23