Unity 2017.1 - TLS 1.2 still not working with .NET 4.6

@andreasreich Case 1076788

Thanks!

 

I think I have stumbled across the same problem as @EduardMatveev. During my tests, the crash only occured on androids 4.4 and below. Androids 5 and above work as expected.

I tried searching for the case number to see what the current status was but was unable anything. Could anyone point me to it or update me on the current status?

Thanks!

 

@andreasreich Just downloaded 2018.2.8f1 with your fix and it continues to exhibit the problem of not detecting the other end of the SslStream has shut down. Read and ReadAsync should eventually return a 0 result on a shutdown. The test project I sent you should log "Client: Remote end has closed." followed by "Client: Done." but it still hangs.

In the code UnityTlsContext.cs, it makes me wonder if Parent.InternalRead is actually blocking, or saying it would block, or something, because I don't see anything wrong in the fix. It looks like the 0 would get returned, but the read never gets the 0 result, or any result in fact, which is curious.

[ EDIT: SEE BELOW - I wrapped the await ReadAsync() in a try block and found that it is throwing an exception on a graceful close. ]

 

Not sure if it affects anything but do you still want to raise UNITYTLS_USER_READ_FAILED on a gracefully closed connection?

I just tested against a Linux openssl s_server client also by setting the host/port on my "TLS Client" object, and the client doesn't get the graceful shutdown.

 

@andreasreich Actually it looks like ReadASync does throw an exception on a graceful close, and a different exception if data has been successfully written but the remote end has shut down, both of which it shouldn't do. The exception when a shutdown happens is "A call to SSPI failed, see inner exception.", where the exception appears to be an abort of an internal MobileAuthenticatedStream async MoveNext(). Could this be related to the raised UNITYTLS_USER_READ_FAILED error upon a graceful shutdown that I mentioned above? A gracefully closed connection is not an error state, and if any treatment of this as an error causes the underlying task to throw exceptions, it could be causing this exception as a result in an unexpected way. For the situation where a Write()/WriteAsync has been called but the remote end has shut down with that data in the out buffer, ReadAsync should still continue to read data and give a 0 result on the stream if the remote shutdown was graceful, and should still not throw an exception based on write state (if that is indeed happening). Only a further write should cause an error, or an erroneous read after the 0 result has already been received.

 

@anderasreich I've also determined that ShutdownAsync() does not shut down the SSL stream cleanly, which makes testing confusing, and I have tested this against OpenSSL with Unity acting as client and server. However the above issue continues against OpenSSL if I just open SslStream and shutdown the remote end of the connection. An exception is thrown, whether it be "A call to SSPI failed, see inner exception.", or the other mystery exception, "Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.", even though data has been received before that shutdown occurred and the shutdown was graceful.

All of the above happens on SslStream built either from a TcpClient or from a NetworkStream attached to a Socket.

If a Wait() is performed on async code on shutdown a ReadAsync it freezes the editor and I have encountered asynchronous tasks staying open and logging after stopping execution.

I think SslStream needs more extensive testing against other tools for reads and writes on shutdowns and graceful/non-graceful disconnections, and the SSL shutdown itself. Undoubtedly there is a lot of complexity to creating this layer. If I can assist with more project examples let me know.

My tests are all Windows so far.

 

Hi,
We have had following problem with the TLS 1.2. Report id 1084800.

Basicly a great amount of GC allocations are made when reading from the TLS stream as each read created 16500 byte allocation. The amount of GC allocations was actually so huge per second that our application crashed randomly with getthreadcontext fails (this was probably some unrelated bug with many thread starts, lots of GC collects and avast). The crash went away (say propability decreased a lot) by reducing read calls by buffering reads to larger chunks, but still at least as many read are used as packets come in. However it's still causing too much GC allocations to run things smoothly.

 

@JoshPeterson Are the fixes mentioned here supposed to address the UNITYTLS_X509VERIFY_FATAL_ERROR error related to using System.net.websockets? I'm running Unity 2018.2.9f1 and consistently have this error when trying to open a connection with a ClientWebSocket in UWP with IL2CPP.

I make a call to

System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12 | System.Net.SecurityProtocolType.Tls11 | System.Net.SecurityProtocolType.Tls;

before opening the connection, but that does not seem to do anything.

There is no problem when I run the same code in the Unity editor. This only throws exceptions in a UWP build (both desktop and hololens) using IL2CPP and the 4.x scripting runtime..

 

@andreasreich I thought at least a partial fix made it into 2018.2.8f1 from the release notes so that's why I was checking it out. I'm happy to see work on modern TLS, because of the huge benefit it will bring to Unity.

The 2018.2.8f1 version did change the behavior of my test against other tools, with certain tests giving better results than others. Some even recognize a proper shutdown and throw no exception, closing normally (again, all in Windows so far). I went as far as to build a client and server in C++ with OpenSSL to test Unity's behavior during a shutdown or a sudden socket disconnection, on both ends. I wasn't sure whether other tools I was using (including web servers) were even doing shutdowns, so that was helpful.

I'll continue to monitor this thread and am happy to assist.

 

We realized that there was a possible issue with the certificate store (since its a self-signed cert during testing). We now load the cert through Windows.Security UWP apis, but the UNITYTLS_X509VERIFY_FATAL_ERROR issue still exists when trying to open up the connection. We're a little crunched for time trying to get ready for a demo, but will try to submit a sample project when we can. I did find an existing issue that is very similar to my problem: https://issuetracker.unity3d.com/is...is-generated-when-connecting-to-mumble-server. Supposedly this is fixed, but the comments suggest otherwise.

 

@andreasreich I see somewhat better behavior on 2018.3.0b5 with respect to SslStream shutting down, but both "ends" of my test only seem to detect a remote shutdown if the underlying socket is closed. This disallows the ability to reuse the socket after the session, such as should be allowed with leaveInnerStreamOpen. It's left open, but it's not shut down. (You might want to add underlying transport reuse to your regression test.)

The beta is an improvement from 2018.2, and I realize this is still being developed, but I wanted to give some insight. I'll try hard to keep this short! Again I'm speaking Windows-only for my observations here.

I wrote a test using OpenSSL C library to test against Unity's ShutdownAsync() and it appears that no shutdown negotiation happens then or when the SslStream is disposed.

In my test I'm performing a shutdown in the form of SSL_shutdown(SSL *), which you may know must be done twice in OpenSSL for a complete session shutdown, once for shutting down the local end (returning 0), and again for determining that the remote end has acknowledged (returning 1 on success), presumably passing over any data remaining in the stream, and completing the negotiation. In the case when I/O is non-blocking, this may involve possibly more shutdown calls; one must expect SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE "errors" on these and perform read and write calls until SSL_shutdown() finally succeeds or fails.

When I run my test against Unity 2018.3.0b5 in Windows, SSL_shutdown() on the second call does not return in OpenSSL when Unity's side runs ShutdownAsync(), so it looks like the shutdown negotiation isn't happening. The SslStream is not shut down cleanly when disposed as it is in .NET. In fact Unity behaves like it's using OpenSSL and only calling SSL_shutdown() once.

From the source it looks like the issue would be within UnityTls.NativeInterface.unitytls_tlsctx_notify_close(...) so that is beyond my visibility and suggests this is within the native handler and may be platform- or library-specific. UnityContext.Shutdown(), upon calling this, assumes that the context is done and destroys all native objects, and it's done asynchronously, so the native handler appears to be expected to complete negotiation before returning.

Happy to put together an example project.

Thanks Andreas!

 

@andreasreich I can file a bug report for this. Also, let me know if you want me to start a separate topic thread for this discussion.

I'm not affected with our project yet, but I can see a scenario where we will want to reuse a socket. To be honest I'm just excited about Unity and want it to be the ultimate for networking and multiplayer! I like Unity's approach and I'm glad the implementation is within your control as I'm sure it has to be for multi-platform.

I never tested Mono's behavior itself, so maybe I'll take a look.

I did a not-so-deep dive on .NET's reference source for SslStream, and it may be worth noting that it looks like ShutdownAsync() just sends a TLS shutdown alert token payload (assuming the stream is not already shut down), and it is asynchronous simply because it has to do a write. It doesn't appear to wait for a response from the remote end. When the SslStream is disposed though, if the stream does not control the underlying transport (leaveInnerStreamOpen presumably), then it appears to do quite a lot more cleanup, and the behavior suggests it waits for the remote end to send its alert. So I think if you are following that behavior your ShutdownAsync() only needs to call SSL_shutdown() once, but the Dispose() needs to call it again (or if it's non-blocking, then I think you actually do repeated calls and send empty reads and writes as it recommends, which might be how you can avoid Mono blocking?). Checking SSL_get_shutdown() for the SSL_RECEIVED_SHUTDOWN bit may allow you to skip doing this if the remote end already shut down.

The problem is I don't think that ShutdownAsync() is sending anything. Are you sure it's calling SSL_shutdown() even once? Even Dispose() isn't sending anything. Note I'm setting leaveInnerStreamOpen to true here.

So the proper bug report would be that ShutdownAsync() is not sending a TLS shutdown alert, and disposing the SslStream is not waiting for the remote end to acknowledge for a graceful close when leaveInnerStreamOpen is set to true.

Let me know if that works and I'll file a bug report.

Thanks!

 

Is this fixed in 2018.3.0f2? I'm still seeing

TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR, verify result: UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED

in editor (Windows). Have added the csc.rsp. The code is just

Code (CSharp):

1.  
2. System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
3. HttpClient HttpClient = new HttpClient();
4.  HttpClient.BaseAddress = new Uri("https://www.mocky.io/");
5.  HttpResponseMessage response = await HttpClient.GetAsync("v2/5185415ba171ea3a00704eed");

 

I'm getting the same thing while trying to connect to our Let's Encrypt secured websockets server (so wss:// protocol, via the ClientWebSocket class) - Unity 2018.3.3f1:

TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR, verify result: UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED
Mono.Unity.Debug.CheckAndThrow (Mono.Unity.UnityTls+unitytls_errorstate errorState, Mono.Unity.UnityTls+unitytls_x509verify_result verifyResult, System.String context, Mono.Security.Interface.AlertDescription defaultAlert) (at <3845a180c26b4889bc2d47593a665814>:0)
Mono.Unity.UnityTlsContext.ProcessHandshake () (at <3845a180c26b4889bc2d47593a665814>:0)
Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status) (at <3845a180c26b4889bc2d47593a665814>:0)
(wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus)
Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) (at <3845a180c26b4889bc2d47593a665814>:0)
Mono.Net.Security.AsyncProtocolRequest+<ProcessOperation>d__24.MoveNext () (at <3845a180c26b4889bc2d47593a665814>:0)

 

I'm getting the same result when I try to load the certificate (with or without private key) before connecting.

_ws.Options.ClientCertificates = new X509Certificate2Collection(certActual);

 

Thanks, the CertificatePolicy workaround is what I needed as well. And yeah now upon re-checking the docs the ClientCertificates is a different thing.

 

I have the same error with .NET 4.x in Unity 2019.1: "TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR, verify result: UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED". It happens in Android and iOS builds. Under Editor it works fine.

Example code:

Code (CSharp):

1.  
2. var _host = "wss://example.com"
3.  
4. Uri uri = new Uri(_host);
5. var client = new TcpClient( uri.DnsSafeHost, 443 );
6. var stream = new SslStream( client.GetStream(), false,
7.                 (sender, certificate, chain, sslPolicyErrors) =>
8.                 {
9.                     return true; // If the server certificate is valid.
10.                 }
11.             );
12. stream.AuthenticateAsClient( uri.DnsSafeHost );
13.  

 

Ticket 1161854. It happens in Android and iOS builds under Unity 2019.1 with .NET 4.x, with .NET 3.5 it works fine for our wss url, but for "wss://echo.websocket.org" it always works.

 

Unfortunately, we are experiencing something similar even in 2019.1 and 2019.3(alpha). In our case, the same code that works on the editor, windows build and Android build does not work on iOS, where we get the following error:

System.Net.Http.HttpRequestException: An error occurred while sending the request ---> System.Net.WebException: Error: TrustFailure (One or more errors occurred.) ---> System.AggregateException: One or more errors occurred. ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> Mono.Security.Interface.TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR, verify result: UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED

  at Mono.Unity.Debug.CheckAndThrow 

 

Sorry Andreas, what do you mean with a new certificate? What kind of certificates would Unity allow? Our game is able to connect to this one on Windows and Android, but not on iOS:

Code (CSharp):

1. $ openssl s_client -connect <oururl>:443 </dev/null 2>/dev/null | openssl x509 -text | grep -A1 'Key Usage'
2.             X509v3 Key Usage: critical
3.                 Digital Signature, Key Encipherment
4.             X509v3 Extended Key Usage:
5.                 TLS Web Server Authentication, TLS Web Client Authentication

Maybe it's the 'critical' part the one that is wrong? google.com for instance only has 'extended key usage'

 

Thank you Andreas. However, according to LetsEncrypt, "As of the end of July 2018, the Let’s Encrypt root, ISRG Root X1, is directly trusted by Microsoft products. Our root is now trusted by all major root programs, including Microsoft, Google, Apple, Mozilla, Oracle, and Blackberry." Indeed, ISRG Root X1 appears in the list Apple's trusted certificates https://support.apple.com/es-es/HT208125

Is there something else we can check?

 

Unity 2019.1.8 relase notes include "IL2CPP: Fixed Unity TLS being incorrectly stripped with Medium and High Managed Stripping Levels. (1134343, 1154766)" I'll try with this version...

 

Well, we finaly were able to solve this by setting up a certificate validation callback that always returns true inside the constructor of the class that handles communications in our game... Thanks for your time!

Code (CSharp):

1. ServicePointManager.ServerCertificateValidationCallback = delegate {return true;};

 

Hi! I'm getting the same error:

TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR, verify result: UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED


My setup:

- Node.JS ws
- C# ClientWebSocket
- Nginx proxy for nodeJS, nginx configuration:

location / {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_pass http://127.0.0.1:3008;
}


Tried everything from this post but nothing works

 

Hi @andreasreich,

We have a problem initialising SSLStream in a WebGL build, essentially we are trying to do a InitTLS where the innerStream for the SSLStream has it's IO from an existing WebSocket. Unity 2019.1.12f1. Is this something you think is currently possible on the platform?

An Array out of Bounds exception is thrown.

Code (CSharp):

1. System.Runtime.InteropServices.Marshal:ptrToStructure(IntPtr, Type)
2. System.Runtime.InteropServices.Marshal:ptrToStructure(IntPtr)
3. Mono.Unity.UnityTls:get_NativeInterface()
4. Mono.Unity.UnityTls:get_IsSupported()
5. Mono.Net.Security.MonoTlsProviderFactory:InitializeProviderRegistration()
6. Mono.Net.Security.MonoTlsProviderFactory:InitializeInternal()
7. Mono.Net.Security.MonoTlsProviderFactory:GetProviderInternal()
8. Mono.Net.Security.MonoTlsProviderFactory:GetProvider()
9. Mono.Net.Security.NoReflectionHelper:GetProvider()
10. Mono.Security.Interface.MonoTlsProviderFactory:GetProvider()
11. System.Net.Security.SslStream:GetProvider()
12. System.Net.Security.SslStream:.ctor(Stream, Boolean)

 

Hi, we are facing the same issue reported by Bhearus last year, with the "InvalidOperationException: invalid nested call." message. I see that it is an expected error related to interleaved write calls, but could you guys clarify which scenarios can cause this? For example, simply calling BeginWrite twice before an EndWrite is called shouldn't be an issue, right?
I tried to get more information from the bug report number (#1057764) but I couldn't find it (I don't know exactly where I should look for this code).
By the way, we're using Unity 2018.4.6.
Thanks!

 

facing this error today while trying to call web api on localhost.
works fine when calling the same web api from remote server.
Postman has option to disable ssl certification so we can test on localhost. Why can't Unity have that option too?

 

Intermittently Experienced this error on PC,Androis build is fine

 

This is not work in Unity 2019.3.0f6 with HttpClient
What else I can do?

 

Thank you! Worked here on 2019.1.8

 

Unity 2019.3.2f1 and Android build:

Try to connect with ssl:

Code (CSharp):

1. IOException: Remote prematurely closed connection.
2. Mono.Net.Security.MobileAuthenticatedStream.InnerRead (System.Boolean sync, System.Int32 requestedSize, System.Threading.CancellationToken cancellationToken) [0x00000] in <00000000000000000000000000000000>:0
3. System.Threading.ContextCallback.Invoke (System.Object state) [0x00000] in <00000000000000000000000000000000>:0
4. System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0
5. System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () [0x00000] in <00000000000000000000000000000000>:0
6. System.Action.Invoke () [0x00000] in <00000000000000000000000000000000>:0
7. System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) [0x00000] i

ios build works correct.
How can I fix it?

 

We had this error running a game server on particular distributions of Linux. It appears that it looks for the root certificate under the path:

/etc/ssl/certs/ca-certificates.crt

Which, on RHEL distributions, should be:

/etc/ssl/certs/ca-bundle.crt

Creating a symbolic link solved the problem for us.

 

Similar curl here for MacOS build: Curl error 51: Cert verify failed: UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED

Making the call via Unitywebrequest.

I'm on Unity 2019.3.13. Tested on MacOS Mojave 10.14.6. Works fine on Editor PC, builds for UWP and Standalone. Fails on editor and build MacOS.

I confirmed azure host where unitywebrequest is going has a certificate.

I checked the project with Fiddler and built project only seems to handshake with host on tunnel, I can't see Multipartformdatasection fields exchanged.

I created a fresh test project and tested the same url unitywebrequest. Worked fine. Then run fiddler in parallel and got the same Curl error above.

This didn't work for us: ServicePointManager.ServerCertificateValidationCallback = delegate {return true;};

 

@andreasreich I see that this was fixed. But I have a repro on a more recent version - Unity 2019.3.2f1. As we see from our users, problem affects users with old androids. We have a repro on device: SM-T530 android 4.4.2
Android log attached, what happens there: UnityWebRequest to our server via https fails with UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED. Server certificate checked and is valid. We noticed this after an update from unity 2018.2.20f1 and finished moving all our requests from besthttp plugin to UnityWebRequest

 

As a test, we fixed UnityWebRequest with overriding certificate validate callback:

Code (CSharp):

1.  
2.     public class AllowAllCertificateHandler : CertificateHandler
3.     {
4.         protected override bool ValidateCertificate(byte[] certificateData)
5.         {
6.             Logger.LogWarning("AllowAllCertificateHandler: ValidateCertificate");
7.             return true;
8.         }
9.     }
10.  
11.         var form = new WWWForm();
12.         var www = UnityWebRequest.Post(ServerDescription.GameUri, form);
13.         www.SetRequestHeader(HEADER_SIGNATURE, hash);
14.         www.uploadHandler = new UploadHandlerRaw(payloadBytes);
15.         www.certificateHandler = new AllowAllCertificateHandler();