A Unity ID allows you to buy and/or subscribe to Unity products and services, shop in the Asset Store and participate
in the Unity community.
Discussion in 'Experimental Scripting Previews' started by Zocker1996, Aug 9, 2017.
@andreasreich Case 1076788
Thank you! Turns out this is a different issue than the one I fixed. So far it looks like an issue in Mono where reads on a closed NetworkStream block, but I'm not done investigating yet.
Shame on me for blaming Mono so quickly! Was a mistake on our side in the Mono integration of the Tls backend. Fix is on the way.
I think I have stumbled across the same problem as @EduardMatveev. During my tests, the crash only occured on androids 4.4 and below. Androids 5 and above work as expected.
I tried searching for the case number to see what the current status was but was unable anything. Could anyone point me to it or update me on the current status?
There was some delay on this one since the issue doesn't reproduce on all Android devices, but the Mobile team has it successfully isolated now and is working on a fix.
@andreasreich Just downloaded 2018.2.8f1 with your fix and it continues to exhibit the problem of not detecting the other end of the SslStream has shut down. Read and ReadAsync should eventually return a 0 result on a shutdown. The test project I sent you should log "Client: Remote end has closed." followed by "Client: Done." but it still hangs.
In the code UnityTlsContext.cs, it makes me wonder if Parent.InternalRead is actually blocking, or saying it would block, or something, because I don't see anything wrong in the fix. It looks like the 0 would get returned, but the read never gets the 0 result, or any result in fact, which is curious.
[ EDIT: SEE BELOW - I wrapped the await ReadAsync() in a try block and found that it is throwing an exception on a graceful close. ]
Not sure if it affects anything but do you still want to raise UNITYTLS_USER_READ_FAILED on a gracefully closed connection?
I just tested against a Linux openssl s_server client also by setting the host/port on my "TLS Client" object, and the client doesn't get the graceful shutdown.
@andreasreich Actually it looks like ReadASync does throw an exception on a graceful close, and a different exception if data has been successfully written but the remote end has shut down, both of which it shouldn't do. The exception when a shutdown happens is "A call to SSPI failed, see inner exception.", where the exception appears to be an abort of an internal MobileAuthenticatedStream async MoveNext(). Could this be related to the raised UNITYTLS_USER_READ_FAILED error upon a graceful shutdown that I mentioned above? A gracefully closed connection is not an error state, and if any treatment of this as an error causes the underlying task to throw exceptions, it could be causing this exception as a result in an unexpected way. For the situation where a Write()/WriteAsync has been called but the remote end has shut down with that data in the out buffer, ReadAsync should still continue to read data and give a 0 result on the stream if the remote shutdown was graceful, and should still not throw an exception based on write state (if that is indeed happening). Only a further write should cause an error, or an erroneous read after the 0 result has already been received.
@anderasreich I've also determined that ShutdownAsync() does not shut down the SSL stream cleanly, which makes testing confusing, and I have tested this against OpenSSL with Unity acting as client and server. However the above issue continues against OpenSSL if I just open SslStream and shutdown the remote end of the connection. An exception is thrown, whether it be "A call to SSPI failed, see inner exception.", or the other mystery exception, "Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.", even though data has been received before that shutdown occurred and the shutdown was graceful.
All of the above happens on SslStream built either from a TcpClient or from a NetworkStream attached to a Socket.
If a Wait() is performed on async code on shutdown a ReadAsync it freezes the editor and I have encountered asynchronous tasks staying open and logging after stopping execution.
I think SslStream needs more extensive testing against other tools for reads and writes on shutdowns and graceful/non-graceful disconnections, and the SSL shutdown itself. Undoubtedly there is a lot of complexity to creating this layer. If I can assist with more project examples let me know.
My tests are all Windows so far.
Sorry for the delayed reply, my EMail notifications didn't do their job.
The issue is fixed in 2018.3 (not sure if it is in the first beta already) and is currently backported 2018.2
I don't think the case is visibly public.
Sorry for all the extra effort you put in there, but the fix did not land in 2018.2 yet. In fact there was apparently some delay and the fix did not even land in 2018.3b1 yet . I'm chasing up when it lands and will double check that your proejct works as expected now - quite sure though that I tried with the fix already and it was fine.
Was there something in the release notes that indicated a fix?
Confirmed again that the fix ("will") work with your project. I saw now that I even added a regression test that resembles your project. The fix is in the queue of being merged to both the 2018.2 and the 2018.3 line. So chances are high it will be in the next release of both of those.
We have had following problem with the TLS 1.2. Report id 1084800.
Basicly a great amount of GC allocations are made when reading from the TLS stream as each read created 16500 byte allocation. The amount of GC allocations was actually so huge per second that our application crashed randomly with getthreadcontext fails (this was probably some unrelated bug with many thread starts, lots of GC collects and avast). The crash went away (say propability decreased a lot) by reducing read calls by buffering reads to larger chunks, but still at least as many read are used as packets come in. However it's still causing too much GC allocations to run things smoothly.
We have good news on this specific front. In the last few days we've tracked down what we think is the cause of this intermittent GetThreadContext crash (which is often exasperated by anti-virus tools). We're still banging on it in testing, but we're hoping to have a fix available in patch releases in the next few months.
@JoshPeterson Are the fixes mentioned here supposed to address the UNITYTLS_X509VERIFY_FATAL_ERROR error related to using System.net.websockets? I'm running Unity 2018.2.9f1 and consistently have this error when trying to open a connection with a ClientWebSocket in UWP with IL2CPP.
I make a call to System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12 | System.Net.SecurityProtocolType.Tls11 | System.Net.SecurityProtocolType.Tls; before opening the connection, but that does not seem to do anything.
There is no problem when I run the same code in the Unity editor. This only throws exceptions in a UWP build (both desktop and hololens) using IL2CPP and the 4.x scripting runtime..
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12 | System.Net.SecurityProtocolType.Tls11 | System.Net.SecurityProtocolType.Tls;
@delphinius81 The fixes Josh mentioned are not related to any verification error.
The fact that it works in the Unity editor but not in UWP builds is quite odd. I'm not sure if the certifcate store is different for UWP applications than for Windows Desktop. Internally we use the same method to query it for both UWP and "classic windows" but it might still yield a different set.
Can you open a bugticket with a repro project please? Thank you!
@andreasreich I thought at least a partial fix made it into 2018.2.8f1 from the release notes so that's why I was checking it out. I'm happy to see work on modern TLS, because of the huge benefit it will bring to Unity.
The 2018.2.8f1 version did change the behavior of my test against other tools, with certain tests giving better results than others. Some even recognize a proper shutdown and throw no exception, closing normally (again, all in Windows so far). I went as far as to build a client and server in C++ with OpenSSL to test Unity's behavior during a shutdown or a sudden socket disconnection, on both ends. I wasn't sure whether other tools I was using (including web servers) were even doing shutdowns, so that was helpful.
I'll continue to monitor this thread and am happy to assist.
We realized that there was a possible issue with the certificate store (since its a self-signed cert during testing). We now load the cert through Windows.Security UWP apis, but the UNITYTLS_X509VERIFY_FATAL_ERROR issue still exists when trying to open up the connection. We're a little crunched for time trying to get ready for a demo, but will try to submit a sample project when we can. I did find an existing issue that is very similar to my problem: https://issuetracker.unity3d.com/is...is-generated-when-connecting-to-mumble-server. Supposedly this is fixed, but the comments suggest otherwise.
@andreasreich I see somewhat better behavior on 2018.3.0b5 with respect to SslStream shutting down, but both "ends" of my test only seem to detect a remote shutdown if the underlying socket is closed. This disallows the ability to reuse the socket after the session, such as should be allowed with leaveInnerStreamOpen. It's left open, but it's not shut down. (You might want to add underlying transport reuse to your regression test.)
The beta is an improvement from 2018.2, and I realize this is still being developed, but I wanted to give some insight. I'll try hard to keep this short! Again I'm speaking Windows-only for my observations here.
I wrote a test using OpenSSL C library to test against Unity's ShutdownAsync() and it appears that no shutdown negotiation happens then or when the SslStream is disposed.
In my test I'm performing a shutdown in the form of SSL_shutdown(SSL *), which you may know must be done twice in OpenSSL for a complete session shutdown, once for shutting down the local end (returning 0), and again for determining that the remote end has acknowledged (returning 1 on success), presumably passing over any data remaining in the stream, and completing the negotiation. In the case when I/O is non-blocking, this may involve possibly more shutdown calls; one must expect SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE "errors" on these and perform read and write calls until SSL_shutdown() finally succeeds or fails.
When I run my test against Unity 2018.3.0b5 in Windows, SSL_shutdown() on the second call does not return in OpenSSL when Unity's side runs ShutdownAsync(), so it looks like the shutdown negotiation isn't happening. The SslStream is not shut down cleanly when disposed as it is in .NET. In fact Unity behaves like it's using OpenSSL and only calling SSL_shutdown() once.
From the source it looks like the issue would be within UnityTls.NativeInterface.unitytls_tlsctx_notify_close(...) so that is beyond my visibility and suggests this is within the native handler and may be platform- or library-specific. UnityContext.Shutdown(), upon calling this, assumes that the context is done and destroys all native objects, and it's done asynchronously, so the native handler appears to be expected to complete negotiation before returning.
Happy to put together an example project.
I'm quite surprised that the Beta behaves differently, I think I backported pretty much everything I did. Well maybe the patch release with some change is not out yet.
`UnityTls.NativeInterface.unitytls_tlsctx_notify_close` does indeed only a single (unidirectional) shutdown in our OpenSSL based implementation (which is used on all desktop platforms). The mono backend api doesn't look like it does any async operation here and we don't want to block for the shutdown.
Didn't really think of the scenario of reusing the socket to be honest, that is indeed quite problematic. Would you mind filing a separate bug for this? I can't guarantee a quick fix on that unless it turns out that it is critical. Will also need to check if blocking in the Mono backend at this point is an option or not.
@andreasreich I can file a bug report for this. Also, let me know if you want me to start a separate topic thread for this discussion.
I'm not affected with our project yet, but I can see a scenario where we will want to reuse a socket. To be honest I'm just excited about Unity and want it to be the ultimate for networking and multiplayer! I like Unity's approach and I'm glad the implementation is within your control as I'm sure it has to be for multi-platform.
I never tested Mono's behavior itself, so maybe I'll take a look.
I did a not-so-deep dive on .NET's reference source for SslStream, and it may be worth noting that it looks like ShutdownAsync() just sends a TLS shutdown alert token payload (assuming the stream is not already shut down), and it is asynchronous simply because it has to do a write. It doesn't appear to wait for a response from the remote end. When the SslStream is disposed though, if the stream does not control the underlying transport (leaveInnerStreamOpen presumably), then it appears to do quite a lot more cleanup, and the behavior suggests it waits for the remote end to send its alert. So I think if you are following that behavior your ShutdownAsync() only needs to call SSL_shutdown() once, but the Dispose() needs to call it again (or if it's non-blocking, then I think you actually do repeated calls and send empty reads and writes as it recommends, which might be how you can avoid Mono blocking?). Checking SSL_get_shutdown() for the SSL_RECEIVED_SHUTDOWN bit may allow you to skip doing this if the remote end already shut down.
The problem is I don't think that ShutdownAsync() is sending anything. Are you sure it's calling SSL_shutdown() even once? Even Dispose() isn't sending anything. Note I'm setting leaveInnerStreamOpen to true here.
So the proper bug report would be that ShutdownAsync() is not sending a TLS shutdown alert, and disposing the SslStream is not waiting for the remote end to acknowledge for a graceful close when leaveInnerStreamOpen is set to true.
Let me know if that works and I'll file a bug report.
Thanks for diving so deeply into this! Sounds like a perfect report to me
To be honest I don't know how the higher level APIs trickle down to the place where we hinged in our TLS binding. The stuff goes through so many layers inside Mono that it is often very hard to tell.
Is this fixed in 2018.3.0f2? I'm still seeing
TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR, verify result: UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED
in editor (Windows). Have added the csc.rsp. The code is just
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
HttpClient HttpClient = new HttpClient();
HttpClient.BaseAddress = new Uri("https://www.mocky.io/");
HttpResponseMessage response = await HttpClient.GetAsync("v2/5185415ba171ea3a00704eed");
I'm getting the same thing while trying to connect to our Let's Encrypt secured websockets server (so wss:// protocol, via the ClientWebSocket class) - Unity 2018.3.3f1:
TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR, verify result: UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED
Mono.Unity.Debug.CheckAndThrow (Mono.Unity.UnityTls+unitytls_errorstate errorState, Mono.Unity.UnityTls+unitytls_x509verify_result verifyResult, System.String context, Mono.Security.Interface.AlertDescription defaultAlert) (at <3845a180c26b4889bc2d47593a665814>:0)
Mono.Unity.UnityTlsContext.ProcessHandshake () (at <3845a180c26b4889bc2d47593a665814>:0)
Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status) (at <3845a180c26b4889bc2d47593a665814>:0)
(wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus)
Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) (at <3845a180c26b4889bc2d47593a665814>:0)
Mono.Net.Security.AsyncProtocolRequest+<ProcessOperation>d__24.MoveNext () (at <3845a180c26b4889bc2d47593a665814>:0)
I'm getting the same result when I try to load the certificate (with or without private key) before connecting.
_ws.Options.ClientCertificates = new X509Certificate2Collection(certActual);
`https://www.mocky.io` is misconfigured in that it only sends the leaf certificate but not any intermediaries. See: https://whatsmychaincert.com/?www.mocky.io
Due to caching of intermediate most browsers will still trust this server since they can establish a chain of trust using the cached (or otherwise retrieved) certificate.
Same applies to Microsoft's .Net/.NetCore implementation. However, other applications like curl will fail to connect to the server just as Unity does (note that the curl version preinstalled on recent Windows 10 versions will connect just fine, but a new one from https://curl.haxx.se/windows/ won't)
We may be able to solve this issue by doing a verification via the system specific TLS api instead of using OpenSSL/MbedTLS to validate against root certificates as we do today, however this solution would then not work cross-platform. So we don't want to implement it today, as it would hide the misconfigured server from the user on some but not all platforms.
To work around the issue at hand you can set your own verification callback via `System.Net.ServicePointManager.CertificatePolicy`. Sadly, today Mono doesn't implement neither `HttpClientHandler.ClientCertificates` nor `HttpClientHandler.ServerCertificateCustomValidationCallback`.
> _ws.Options.ClientCertificates = new X509Certificate2Collection(certActual);
I think you're misunderstanding what this option does. As I understand it this specifies the client certificates that are provided to the server in case it requests any, not the certificates used for validation. (similar to the clientCertificate parameter here to which this field seems to be passed on in Mono)
Thanks, the CertificatePolicy workaround is what I needed as well. And yeah now upon re-checking the docs the ClientCertificates is a different thing.
I have the same error with .NET 4.x in Unity 2019.1: "TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR, verify result: UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED". It happens in Android and iOS builds. Under Editor it works fine.
var _host = "wss://example.com"
Uri uri = new Uri(_host);
var client = new TcpClient( uri.DnsSafeHost, 443 );
var stream = new SslStream( client.GetStream(), false,
(sender, certificate, chain, sslPolicyErrors) =>
return true; // If the server certificate is valid.
stream.AuthenticateAsClient( uri.DnsSafeHost );
Odd, that certificate handler should bypass validation so that you never get `FLAG_NOT_TRUSTED`!
It could be that it is related to this issue on renegotiation https://forum.unity.com/threads/res...r-failed-to-receive-data.651616/#post-4383880 which was fixed in 2019.3 (queued for backporting to 2018.4, 2019.2 & 2019.1, but still hasn't arrived in any earlier Unity version ) but I'm not sure about this.
Any chance you could open a ticket with a repro project? (and send me the ticket number so I can bypass/speedup the usual bug processing )
Ticket 1161854. It happens in Android and iOS builds under Unity 2019.1 with .NET 4.x, with .NET 3.5 it works fine for our wss url, but for "wss://echo.websocket.org" it always works.