Search Unity

  1. Unity Asset Manager is now available in public beta. Try it out now and join the conversation here in the forums.
    Dismiss Notice
  2. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  3. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

Bug 2021.2.0b6 and System.Memory/ReadOnlySpan under .NET 4.8

Discussion in '2021.2 Beta' started by Baawk, Aug 5, 2021.

  1. Baawk

    Baawk

    Joined:
    Nov 15, 2017
    Posts:
    20
    Hello!

    Referencing #1355846, it seems that b6 has some nasty behaviour changes (likely due to the update to .NET 4.8). System.Memory seems to be suddenly merged into the mscorlib even for 4.8, which means types that previously had to be supplied using assemblies (System.Span/System.ReadOnlySpan from System.Memory.dll) are suddenly available/can no longer be imported by an assembly. This is a bit odd because System.Memory is not a part of the .NET Framework 4.8.

    In theory, this change could be nice, but sadly ReadOnlySpan seems to be broken. In particular, attempting to index it will throw an error:

    Code (CSharp):
    1.  
    2. System.ReadOnlySpan<int> span = new System.ReadOnlySpan<int>(new int[10]);
    3. UnityEngine.Debug.Log(span[5]);
    4.  
    will result in

    This is a bit frustrating, because it effectively means that ReadOnlySpan is no longer usable (and, by extension, a big chunk of whoever was using it before).

    Because it effectively bricks projects using ReadOnlySpan, I'll have to skip b6 and revert back to b5.
     
  2. TieSKey

    TieSKey

    Joined:
    Apr 14, 2011
    Posts:
    228
    ReadOnlySpan<T> seems to not be even defined in mscorlib as I'm getting a ton of:

    error CS0012: The type 'ReadOnlySpan<>' is defined in an assembly that is not referenced. You must add a reference to assembly 'System.Memory, Version=4.0.1.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.

    Whenever I try to remove System.Memory to solve the double declaration.


    Looking closely there are some:

    error CS0433: The type 'IAsyncEnumerator<T>' exists in both 'Microsoft.Bcl.AsyncInterfaces, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' and 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'

    How did that class even get there?!?! b6 mscorlib is cursed as hell.
     
  3. Baawk

    Baawk

    Joined:
    Nov 15, 2017
    Posts:
    20
    Hm, quite odd. For me, when I had System.Memory still there, it talked about a clash between mscorlib and System.Memory, and Visual Studio's F12 (after re-creating the csprojs) also led me to the mscorlib.

    It seems that with their upgrade to 4.8 and more specifically netstandard2.1, somehow classes of the latter got thrown into the former.
     
  4. SugoiDev

    SugoiDev

    Joined:
    Mar 27, 2013
    Posts:
    395
    The issue on my side is the same as @Baawk
    Will submit a bug report soon.
     
  5. TieSKey

    TieSKey

    Joined:
    Apr 14, 2011
    Posts:
    228
    Mmm part of my code is compiled as a dll in a separate project, those ReadOnlySpan errors are actually coming from there since that project references System.Memory.
     
  6. Baawk

    Baawk

    Joined:
    Nov 15, 2017
    Posts:
    20
    Nice to hear that it's reproducible for others as well! I'm not too familiar with Unity bug reporting, but as (perhaps badly stated) in the first post, I've already reported it as 1355846. Not sure if multiple reports will help or hinder progress.

    Ah, that's a good point actually. If you have third party dependencies which expect these types to be in System.Memory.dll, but they're actually now in mscorelib... How is that supposed to work? You could, probably, create an own System.Memory assembly that type forwards them to the mscorlib but... uh.

    I'm beginning to think that it was probably a mistake that these types came in for 4.8, and should probably get removed again.
     
    stonstad likes this.
  7. SugoiDev

    SugoiDev

    Joined:
    Mar 27, 2013
    Posts:
    395
    Oh yeah, I completely missed it!
    Well, now we have two developer reporting in!

    It seems so, since .net 4.8 does not support span at all, only 5.0+ (according to https://docs.microsoft.com/en-us/dotnet/api/system.span-1)
    But, since it's mono, maybe it does, partially?
     
  8. runner78

    runner78

    Joined:
    Mar 14, 2015
    Posts:
    804
    So far i know, mono implements .NET Standard 2.1 completely, including Span and Memory.
    It would also make no sense for the .NET 4.8 profile to leave out the Span and Memory if the runtime (Mono) supports it anyway.
     
  9. Baawk

    Baawk

    Joined:
    Nov 15, 2017
    Posts:
    20
    That entirely depends on how it is done, however. The current way (simply throwing it into mscorlib without providing type forwarding or similar) is just not working. Every dependency that relies on any type inside System.Memory (or other classes that got "merged" into mscorlib) will not work, therefore severely restricting the possible .NET libraries that one can use within Unity.

    On top of that, it might make sense to allow users to ship their own System.Memory or similar. There might be special cases where a certain version is required (e.g. some odd bugs/library dependencies), and the b6 behaviour is a (major) breaking change from that.
     
    stonstad and TieSKey like this.
  10. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    3,136
    Thanks for flagging this and for submitting reports! We'll look into it.
     
    SugoiDev and olavrv like this.
  11. olavrv

    olavrv

    Joined:
    May 26, 2015
    Posts:
    516
    Any news on this? Same problem in b7...
     
  12. Randhall

    Randhall

    Joined:
    Feb 21, 2013
    Posts:
    17
    This change makes 90% of my game framework unusable. Really hope this will get resolved before the 2021.2 goes out of Beta
     
  13. SugoiDev

    SugoiDev

    Joined:
    Mar 27, 2013
    Posts:
    395
    Yeah, it's chaotic for us too.
    A lot of stuff we use depends on System.Memory.
    I managed to switch to netstandard to test and got it working, but this is not a project where I want to just switch to standard like that.

    I think this might be the first time I'm fully blocked out from a beta.
     
  14. Kichang-Kim

    Kichang-Kim

    Joined:
    Oct 19, 2010
    Posts:
    1,021
    Same issue here. It also breaks so many libraries like System.IO.Pipelines depends on System.Memory.
     
  15. olavrv

    olavrv

    Joined:
    May 26, 2015
    Posts:
    516
    Is this a bug in Unity that will be fixed, or a change which developers will have to adapt to?
     
  16. TieSKey

    TieSKey

    Joined:
    Apr 14, 2011
    Posts:
    228
    Regardless of unity intention, this is definitely to be treated as a bug since it breaks a ton of 3rd party libraries including "official" .NET ones.
    Avoiding this is exactly half of the reason why Unity won't change to .NET5 yet (the other half being the lack of app domains).
     
    olavrv likes this.
  17. Baawk

    Baawk

    Joined:
    Nov 15, 2017
    Posts:
    20
    I suppose it could be doable to remove the types from the supplied mscorlib, but it would definitely be much nicer and less stressful if Unity could fix this obviously gone-bad migration.

    In the issue system, my bug is still open. The lack of communication from Unity regarding this topic is quite disappointing, seeing as it's really severely crippling .NET 4.8 has backend, which still has some advantages over netstandard or is flat out required for some products.
     
  18. SugoiDev

    SugoiDev

    Joined:
    Mar 27, 2013
    Posts:
    395
    I didn't get a reply either. Unity is in dire need of improving their communication with developers.

    Let's hope it's not something that's harder to fix than it looks.
     
  19. Alex-Thibodeau

    Alex-Thibodeau

    Unity Technologies

    Joined:
    Apr 18, 2013
    Posts:
    42
    I am investigating this now as far as I know this should work.
     
    Roc916, TieSKey and SugoiDev like this.
  20. olavrv

    olavrv

    Joined:
    May 26, 2015
    Posts:
    516
    One of the assets we are using (Dissonance) has these errors in both 2021.0b6 & 0b7 - so perhaps you can test with this?
     
  21. Nyarlathothep

    Nyarlathothep

    Joined:
    Jun 13, 2013
    Posts:
    399
    I'm one of the developers of Dissonance, if I can help debug this issue feel free to contact me at martin@placeholder-software.co.uk

    To be honest right now I'm not sure why this issue would affect Dissonance - as far as I know we don't use Span/ReadOnlySpan anywhere (we use the older ArraySegment struct instead).
     
    olavrv likes this.
  22. kristianstoyl

    kristianstoyl

    Joined:
    Aug 10, 2018
    Posts:
    5
    You're correct. If I create a new project in 2021.2.0b7, download and import the latest Dissonance, I get some compile errors, but it looks like only the syntax has changed (Specifically CopyTo no longer returns an ArraySegment).
     
    Nyarlathothep likes this.
  23. castor76

    castor76

    Joined:
    Dec 5, 2011
    Posts:
    2,521
    I am also suddenly getting all this error :

    Assets\Plugins\Ulid\Ulid.cs(188,62): error CS0433: The type 'ReadOnlySpan<T>' exists in both 'System.Memory, Version=4.0.1.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' and 'netstandard, Version=2.1.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'
     
  24. castor76

    castor76

    Joined:
    Dec 5, 2011
    Posts:
    2,521
    Is this bug currently available at the Issue Tracker? So we can vote for it?
     
  25. Nyarlathothep

    Nyarlathothep

    Joined:
    Jun 13, 2013
    Posts:
    399
    I just imported this into 2021.2 and I see what has happened. We defined our own
    CopyTo
    extension method on
    ArraySegment
    - now there's a built in
    CopyTo
    method which takes the same arguments but has a different return type. Fortunately this particular case should be easy to fix (I can just rename the extension) and will probably not affect other assets (not many people will have defined a
    CopyTo
    extension method that just happens to clash).
     
    Last edited: Aug 17, 2021
  26. runner78

    runner78

    Joined:
    Mar 14, 2015
    Posts:
    804
    Don't copy System.Memory.dll manually to an .NET standard 2.1 Unity project.
     
  27. castor76

    castor76

    Joined:
    Dec 5, 2011
    Posts:
    2,521
    Oh yeah, I see it... so it is now part of .net standard 2.1?
     
  28. runner78

    runner78

    Joined:
    Mar 14, 2015
    Posts:
    804
    Yes
    Edit: However, there still the bug, that Unity cannot find System.Memory, even if it is part of .NET Standard 2.1.
     
    Last edited: Aug 18, 2021
  29. TieSKey

    TieSKey

    Joined:
    Apr 14, 2011
    Posts:
    228
    Beta8 just hit the hub but I don't see this issue in either known issues nor fixes.
     
    arkenstn likes this.
  30. Baawk

    Baawk

    Joined:
    Nov 15, 2017
    Posts:
    20
    This entire/thread revolves around .NET 4.8 as scripting backend however, not netstandard 2.1. There's no fix for this problem on .NET 4.8. For 4.8, these types should not exist, and if Unity shipped System.Memory themselves (which perhaps could be slightly problematic), they shouldn't do it in mscorlib (or unless they're also shipping a forwarding assembly to make sure external assemblies work fine).

    My bug report is still open, so I suppose it hasn't been fixed yet.
     
    kevdome3000 likes this.
  31. SugoiDev

    SugoiDev

    Joined:
    Mar 27, 2013
    Posts:
    395
    My report is still open too, with no reply.
    It's a bit concerning not seeing this issue in the "known issues".

    I managed to hack my project to work on netstandard for the time being, just so I can continue participating in the beta testing, but this is a pretty serious issue and we'll have a lot of trouble if it's not fixed before release.
     
  32. runner78

    runner78

    Joined:
    Mar 14, 2015
    Posts:
    804
    Mono has an Forwarder for System.Memory. And so far i know, Unity use the mscorlib from mono, not an custom fork.
     
    stonstad likes this.
  33. Baawk

    Baawk

    Joined:
    Nov 15, 2017
    Posts:
    20
    You can check the b7 installation directory. There's no System.Memory assembly that could do the forwarding (unless they're doing this in some other, non-standard fashion) - plus the error messages that come up also point towards this not being the case for Unity.

    Point being: Just removing the System.Memory.dll does not help; ReadOnlySpan is still broken, and external dependencies aren't working because they can't find System.Memory.dll. It might be possible to provide an own System.Memory.dll with those forwards, but frankly, that shouldn't be our job, and it still would not resolve the problem that at least some of the implementations that Unity provides don't work.
     
  34. runner78

    runner78

    Joined:
    Mar 14, 2015
    Posts:
    804
  35. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    Hey everyone, I just wanted to give a brief update on this issue - I'm actively working on it now. I'm not sure what the proper resolution is, but we will work it out.

    I appreciate the feedback on this thread. Please feel free to continue to post about any related issues so that we can make sure we get this all cleaned up. Thanks!
     
  36. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    3,136
  37. runner78

    runner78

    Joined:
    Mar 14, 2015
    Posts:
    804
    The error in the tracker seems to be more of a problem with the wrong C# version. The indexer is an readonly ref return, an only work with C# >= 7.2.
    The other problem many have here is that third party libraries with references to System.Memory don't work. If you add the dll manually then the members like ReadOnlySpan<T> collide with the existing ones. And if you remove the DLL, you get the message System.Memory cannot be found.
     
  38. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    We've corrected the original issue reported here internally (https://issuetracker.unity3d.com/is...s-not-supported-and-throws-an-error-when-used). I'll describe a bit about the cause of this problem, then address some more general issues in a separate post.

    First, the specific bug related to

    Code (CSharp):
    1. error CS0570: 'ReadOnlySpan<T>.this[int].get' is not supported by the language
    occurs because the reference assemblies provided to the C# compiler for the .NET Framework Api Compatibility Level option were incorrect. Specifically, the indexed on ReadOnlySpan was incorrect. The IL code for it should look like this:

    Code (CSharp):
    1. .method public hidebysig specialname
    2.     instance !T& modreq(System.Runtime.InteropServices.InAttribute) get_Item (
    3.         int32 index
    4.     ) cil managed
    5. {
    6.     .param [0]
    7.     .custom instance void System.Runtime.CompilerServices.IsReadOnlyAttribute::.ctor() = (
    8.         01 00 00 00
    9.     )
    10.     // Method begins at RVA 0x2050
    11.     // Code size 2 (0x2)
    12.     .maxstack 8
    13.  
    14.     // throw null;
    15.     IL_0000: ldnull
    16.     // (no C# code)
    17.     IL_0001: throw
    18. } // end of method ReadOnlySpan`1::get_Item
    Note the IsReadOnlyAttribute on the param[0] (which is the return type of the property getter method). Our profile was missing this attribute. You can check that yourself in a tool like ILSpy by inspecting the mscorlib.dll reference assembly located at Data\UnityReferenceAssemblies\unity-4.8-api\mscorlib.dll in the Unity Editor installation.

    I have a fix in process now to correct this, and allow code using ReadOnlySpan to compile properly. I'll respond here when I know which beta version will correct this issue.
     
    SugoiDev likes this.
  39. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    Regarding the more general issues, starting in b6, the Api Compatibility Level ".NET Framework" is more than just .NET Framework 4.8. As I alluded to above, Unity is shipping a custom reference profile that includes the union of APIs from .NET Framework 4.8 and .NET Standard 2.1.

    We do have documentation about this change on the way, but I've just checked the published Unity Manual for 2021.2, and it looks like the documentation changes are not there yet. I'll try to describe our reasoning here:

    Microsoft supports .NET Standard 2.0 in the .NET Framework 4.8, but chose not to support .NET Standard 2.1 in the .NET Framework ecosystem. Since .NET Standard 2.1 brings some really useful APIs (like Span and ReadOnlySpan), we want to support it across the entire Unity ecosystem. That includes via packages shipped by Unity and in the UnityEngine and UnityEditor APIs. We're not ready to compile the UnityEngine.dll assembly against .NET Standard 2.1 though, as that would break many existing pre-compiled DLLs. Instead, Unity created this hybrid .NET Framework 4.8 + .NET Standard 2.1 reference profile.

    This all works because Mono supports all of the .NET Standard 2.1 APIs in its mscorlib-based base class library implementation. As @runner78 said in this forum thread, all of the code exists, so we wanted to expose it to all users.

    There are three place where this can be difficult though (all of which will be mentioned in the upgrade guide for 2021.2).
    1. .NET Standard 2.1 includes new APIs, so they might conflict with existing C# code, leading to ambiguous references. For example, .NET Standard 2.1 introduces System.Range. If your code has a Range type already, the C# compiler in b6 and later might not know which one you want to use, and you will need to adjust using statements or fully qualify the names to indicate which to use.
    2. As @Nyarlathothep mentioned, your code might have extension method that are now implemented on the types directly. You can choose to rename your extension methods or to use the BCL-implemented methods.
    3. As originally reported in this thread, the use of System.Memory.dll from Nuget to get Span with .NET Framework will now cause problem, because the APIs in System.Memory.dll are now implemented in the BCL Unity ships and in the reference profile Unity uses. It should be possible to work around this issue by removing the System.Memory.dll file and using the provided APIs.
    After adjusting for these issues, we think the end result will be better. Project should have access to more .NET APIs with less extra work (e.g. pulling in assemblies from Nuget) which should work correctly no matter the Api Compatibility Level or Scripting Backend.

    Our long term goal is to move Unity to a .NET Core-based ecosystem - this transition to .NET Standard 2.1 is a stepping stone on that path.

    As usual in software engineering, everything is a trade-off. Here we are trying to provide the best future with the least pain now. We're open to suggestions though - we want to get this right during the beta phase. So please let me know your feedback. Thanks!
     
    fherbst, bitinn, TieSKey and 6 others like this.
  40. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    3,536
    What are the benefits of waiting to switch to .Net Core instead of already doing the switch now?
     
  41. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    We're not ready to switch to a .NET Core-based ecosystem now. I've outlined the process we need to go through at a high level in this forum thread: https://forum.unity.com/threads/unity-future-net-development-status.1092205/

    Since the Mono Unity is using in b6 and later supports .NET Standard 2.1 APIs, we wanted to at least expose them to users, as that is something we can do much sooner than .NET Core support.
     
    tbg10101_, JesOb and Lars-Steenhoff like this.
  42. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    3,536
  43. Kichang-Kim

    Kichang-Kim

    Joined:
    Oct 19, 2010
    Posts:
    1,021
    @JoshPeterson Hi, thanks for detailed explanation. But about 3, removing System.Memory.dll simply throws missing reference error for many libraries from nuget. Do you mean that these libraries are unavailable until Unity completely move to .NET Core ecosystem?
     
  44. castor76

    castor76

    Joined:
    Dec 5, 2011
    Posts:
    2,521
    I tried this with Unity 2021.2.0b9 today, and got the same bunch of errors :

    Assets\Plugins\Ulid\Ulid.cs(188,62): error CS0433: The type 'ReadOnlySpan<T>' exists in both 'System.Memory, Version=4.0.1.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' and 'netstandard, Version=2.1.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'

    I don't have any manual dll copies of System.Memory, but upon tracking it down a bit more reveals that:
    There is one such as:
    UnityProject\Library\PackageCache\com.unity.searcher@4.8.0\Editor\Lucene\System.Memory.dll

    Which belongs to the Searcher package from package manager.

    Wondering if Searcher isn't compatible? What gives?
     
  45. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    3,136
    Thanks for bringing this up, I've forwarded it to the team.
     
  46. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    The fix for this issue will be available in 2021.2.0b11 and 2022.1.0a7. Thanks!
     
  47. Roc916

    Roc916

    Joined:
    Jun 5, 2021
    Posts:
    2
    The issue unsolved in 2022.1.0a7.1073.
    upload_2021-9-8_15-36-24.png
     
  48. TieSKey

    TieSKey

    Joined:
    Apr 14, 2011
    Posts:
    228
    If the fix made it to alpha7, u have to remove your own dlls from plugins folder and it should work

    Oh nvm, seems like a package is still including it?
     
    Last edited: Sep 8, 2021
  49. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    Sorry for the delay in response - I missed this question in the forum thread.

    These libraries should work. All of the APIs from System.Memory.dll should be available with Unity in version 2021.2b6 and later.

    Can you provide details about the specific error message and nuget package that triggers it?
     
  50. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    It looks like the protobuf.net package is including System.Memory.dll itself, which likely was necessary before. But now Unity should provide it. Is there a version of protobuf.net that is built to work with .NET Standard 2.1? I suspect that version would work.