Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

The current state of link.xml in packages

Discussion in 'Package Manager' started by at_, Oct 28, 2020.

  1. at_

    at_

    Joined:
    Oct 29, 2018
    Posts:
    24
    Hi,

    I've been trying to get packages to properly propagate their link.xml to the project using them, but I haven't figured out a good workflow for this. Native thirdparty dlls (source outside of our control) in packages is a common occurrence in our private package registry so using the PreserveAttribute isn't an option.

    Right now our developers have to copy the link.xml files manually from each package into the project, while ensuring that none of the link.xml files accidentally overwrite each other. While a minor annoyance, this adds a potential point of failure to the workflow.

    In addition, copying these link.xml files from the package into the project will inevitably create GUID collisions unless the meta files are regenerated. This is pretty hard to miss thanks to Unity tossing the console full of warnings but it's just another pain point in the workflow.

    I noticed some earlier threads on this acknowledging that it's an issue, but haven't heard anything since. In the long term I see this preventing a lot of assets from ever being transferred over to packages. Is there improvement coming up on the roadmap?
     
    Favo-Yang and bdovaz like this.
  2. at_

    at_

    Joined:
    Oct 29, 2018
    Posts:
    24
    Now that it's been almost two weeks since I made this thread, I think it's safe to assume this isn't a very pressing issue for the development team.
    We'll keep looking into other options.
     
  3. bdovaz

    bdovaz

    Joined:
    Dec 10, 2011
    Posts:
    973
    The truth is that it is quite unfortunate that in 3 years which is the life that UPM has, it is halfway:
    - It does not allow link.xml files.
    - It does not allow csc.rsp files.
    - It does not allow you to mark a package as deprecated.
    - No Nuget integration. This is something I have been demanding for years and more since UPM exists because there can be several assets depending on the same Nuget package in different versions and there is no dependency and conflict management
    - Integration with the non-existent Asset Store. I mean as an Asset Store developer to be able to publish it as a package and for users to benefit from all that it entails such as being able to update it, remove it or change versions simply by editing the manifest.json file and not flooding the /Assets folder.
     
    DerrickBarra likes this.
  4. UnityMaru

    UnityMaru

    Community Engagement Manager Unity Technologies

    Joined:
    Mar 16, 2016
    Posts:
    1,227
    Hey there, so sorry for the delay in reply. I have put your question forward to the team shortly after your first post and waiting for feedback so I'll be sure to come back once I get some to share :)
     
  5. mike-voorhees

    mike-voorhees

    Unity Technologies

    Joined:
    Aug 9, 2016
    Posts:
    43
    Thank you for voicing your concerns about link.xml files in packages. It is helpful to hear why users feel they need support for it.

    Yes, we do not currently respect link.xml files in packages. I will elaborate on why that is later. First let me explain two ways to get link.xml files in packages working that would probably be easier (or at least more robust) than what you are doing.

    a) You can embed the link.xml file into your package assembly. If you do this, the embedded file name must match the assembly name.

    b) I haven't tried this approach personally, but I don't see why it wouldn't work. You can implement https://docs.unity3d.com/2020.1/Doc...rProcessor.GenerateAdditionalLinkXmlFile.html in your package and then return your package's link.xml file path when the callback is called.

    I recognize both of these are obscure and add needless complication.

    Let me switch back to why we don't respect link.xml files in packages. There are a number of reasons which, combine, have led to us not doing anything about it to date.

    1) link.xml files are painfully awful to use. It's a bad experience. Maintaining type names in them, much less generic method names. It is a fundamentally poor experience.

    2) There is 1 legitimate use case for them, fixing stripping problems in prebuilt assemblies or assemblies you don't have/want to change the source for.

    3) Today the owner of the project mostly has control over what is preserved. If we allow link.xml files in packages interesting questions come up. If some package preserved the entire `mscorlib` assembly how would the owner of the project ever realize that was happening?

    4) Other questions come up such as would you want every package's link.xml file respected for every build? With the embedded approach above (a) link.xml files are not processed unless a given assembly is needed for the build. With link.xml files on disk we loose that knowledge. Instead the link.xml would force whatever it preserves to be included in every build no matter what. All that is to say, it's not as straight forward as searching for files in a few more places.

    5) About a year ago, Microsoft started to invest heavily in managed code stripping. This is great for the .NET ecosystem in the long run. However, in the short term it has meant that we (Unity) have taken a back seat and shifted to following their lead. Only adding features when we felt our users needs were not going to be met by the plans for the larger .NET ecosystem.

    6) The default behavior of the `Low` managed stripping level is very hard to understand. We want to do something about it, but haven't for it's own set of reasons (mainly increased iteration time). The solution would likely be to expand the set of assemblies that are not stripped. This would likely reduce the number of users that would feel the need for a link.xml file in a package.

    7) It's relatively uncommon to need to use a link.xml file in a package from what I've been able to gather. And most of the link.xml files I've seen in a package preserve the entire assembly which goes back to (6). And rightfully so, after about 5-10 lines in a link.xml you realize (1) above and just preserve the entire assembly which then ties into (5)/

    8) And lastly, there are workarounds (a) and (b) above.

    We revisit the subject of link.xml files in packages a few times a year. We may allow it in the future. Or we may continue to put our time into other managed code stripping improvements such as (6) above. Recently our efforts have been on https://forum.unity.com/threads/sup...g-annotation-attributes.1000246/#post-6528926, keeping up with Microsoft https://github.com/mono/linker , and other things you don't see under the hood such as improved detection of reflection.

    That may not be the answer you want to hear. And maybe we made the wrong choice in doing nothing. But that's the rational for why we've done what we've done (nothing).

    Thanks,
    -Mike
     
  6. codestage

    codestage

    Joined:
    Jul 27, 2012
    Posts:
    1,843
    Hey @mike-voorhees,

    Thank you for the detailed explanation, perfectly makes sense for me and b) is working fine in my case, but just to let you know about one more use case of link.xml in packages, here is my story short:

    I make Asset Store plugin which uses some Unity components (such as MeshCollider) from code, and those components gets stripped with IL2CPP's "Strip Engine Code" feature if they are not used in the project, so I had to implement link.xml preventing those components stripping in my asset which often is used in form of UPM package by customers.
     
  7. bobbaluba

    bobbaluba

    Joined:
    Feb 27, 2013
    Posts:
    81
    I had a go at the GenerateAdditionalLinkXmlFile / IUnityLinkerProcessor approach, this is what I ended up with:

    Code (csharp):
    1.  
    2. #if UNITY_EDITOR
    3. using System.IO;
    4. using UnityEditor;
    5. using UnityEditor.Build;
    6. using UnityEditor.Build.Reporting;
    7. using UnityEditor.UnityLinker;
    8. namespace MyUpmPackage
    9. {
    10.     public class LinkXmlInstaller : IUnityLinkerProcessor
    11.     {
    12.         int IOrderedCallback.callbackOrder => 0;
    13.         string IUnityLinkerProcessor.GenerateAdditionalLinkXmlFile(BuildReport report, UnityLinkerBuildPipelineData data)
    14.         {
    15.             // This is pretty ugly, but it was the only thing I could think of in order to reliably get the path to link.xml
    16.             const string linkXmlGuid = "9bbe747a148f00540828ab8521feb770"; // copied from link.xml.meta
    17.             var assetPath = AssetDatabase.GUIDToAssetPath(linkXmlGuid);
    18.             // assets paths are relative to the unity project root, but they don't correspond to actual folders for
    19.             // Packages that are embedded. I.e. it won't work if a package is installed as a git submodule
    20.             // So resolve it to an absolute path:
    21.             return Path.GetFullPath(assetPath);
    22.         }
    23.         void IUnityLinkerProcessor.OnBeforeRun(BuildReport report, UnityLinkerBuildPipelineData data)
    24.         {
    25.         }
    26.         void IUnityLinkerProcessor.OnAfterRun(BuildReport report, UnityLinkerBuildPipelineData data)
    27.         {
    28.         }
    29.     }
    30. }
    31. #endif
    32.  
    It works, but it ain't pretty.
     
    daleth90 and CodeLane_Wouter like this.
  8. benjaminbigrun

    benjaminbigrun

    Joined:
    Mar 16, 2020
    Posts:
    1
    We ended up making a script that automatically searches all packages and copies link.xml files into the project. Then you can put this function in the IUnityLinkerProcessor:OnBeforeRun or just in a more-usual preprocess build step:


    Code (CSharp):
    1. private static void CopyLinks()
    2. {
    3.     var req = UnityEditor.PackageManager.Client.List(true);
    4.     while (!req.IsCompleted)
    5.     {
    6.         continue;
    7.     }
    8.  
    9.     var linkPaths = new List<(string, string)>();
    10.     var collection = req.Result;
    11.     foreach (var p in collection)
    12.     {
    13.         CollectLinks(linkPaths, p.resolvedPath, p.resolvedPath);
    14.     }
    15.  
    16.     if (linkPaths.Count > 0)
    17.     {
    18.         foreach (var (absPath, relPath) in linkPaths)
    19.         {
    20.             var destPath = $"{Application.dataPath}/PackageLinks/{relPath}";
    21.            
    22.             Debug.Log($"Copying {absPath} to {destPath}...");
    23.  
    24.             Directory.CreateDirectory(Path.GetDirectoryName(destPath)!);
    25.             File.Copy(absPath, destPath);
    26.         }
    27.     }
    28.     else
    29.     {
    30.         Debug.Log("No package links found.");
    31.     }
    32. }
    33.  
    34. private static void CollectLinks(List<(string, string)> linkPaths, string rootPath, string path)
    35. {
    36.     // check files
    37.     var files = Directory.GetFiles(path, "link.xml");
    38.     foreach (var file in files)
    39.     {
    40.         linkPaths.Add((file, file.Replace(rootPath, "")));
    41.     }
    42.  
    43.     // check directories
    44.     var directories = Directory.GetDirectories(path);
    45.     foreach (var dir in directories)
    46.     {
    47.         CollectLinks(linkPaths, rootPath, dir);
    48.     }
    49. }
     
    DerrickBarra and codestage like this.
  9. SimonDarksideJ

    SimonDarksideJ

    Joined:
    Jul 3, 2012
    Posts:
    1,677
    @mike-voorhees Great write up, but can you expand on the point:

    > a) You can embed the link.xml file into your package assembly. If you do this, the embedded file name must match the assembly name.

    I've tried adding a link.xml file with the name of my assembly (assembly.xml) in to my UPM package, yet it produced no different results, the code was stripped.

    Is this method documented, or better yet, has an example somewhere?
     
  10. bobbaluba

    bobbaluba

    Joined:
    Feb 27, 2013
    Posts:
    81
    So it seems IUnityLinkerProcessor was deprecated in 2021.1: https://unity3d.com/unity/whats-new/2021.1.0

    What's the recommended replacement that supports source distribution (no .dlls) of packages and doesn't tamper with files in the main project?

    EDIT: Sorry, it's just the OnBeforeRun and OnAfterRun methods that have been deprecated. And those we don't actually need. IUnityLinkerProcessor.GenerateAdditionalLinkXmlFile(BuildReport report, UnityLinkerBuildPipelineData data) is still there :)

    Sorry for the noise. I guess I can just remove those two behind an ifdef on a unity version define
     
    Last edited: Sep 29, 2021
  11. codestage

    codestage

    Joined:
    Jul 27, 2012
    Posts:
    1,843
    Only two IUnityLinkerProcessor APIs were deprecated but GenerateAdditionalLinkXmlFile still in place.
     
  12. huulong

    huulong

    Joined:
    Jul 1, 2013
    Posts:
    205
    I started relying a lot on VisualScripting (ex Bolt) and custom Units are the first things to get stripped and break the game when I enable stripping, as they are not referenced in code, yet referenced by Script Graphs.

    Low stripping means custom Units defined inside Project Assemblies are preserved, while those defined in external Package Assemblies are not. If you own said package, you can just add [assembly: Preserve]. But again, the issue lies with external packages. I started using https://github.com/RealityStop/Bolt.Addons.Community, which recently transitioned from the old Bolt to Unity's native VisualScripting, and was wondering how to solve this issue.

    It turns out they made their own solutions, https://github.com/RealityStop/Unity.LinkMerge, which is similar to @benjaminbigrun 's solution, except it does actual XML merging.

    The file containing the copy/merge operation:
    https://github.com/RealityStop/Unity.LinkMerge/blob/master/Editor/BuildHelper.cs

    Personally, I like the idea of letting the user pick which link.xml to use/embed in their project. Maybe we could define a dedicated asset or Project Setting where we drag and drop a list of xml (or even assembly references that we want to preserve?), and let the engine do the rest of the job. As if the script calling GenerateAdditionalLinkXmlFile (like the one from @bobbaluba) was predefined in the engine, and would just pick our asset / Project Setting info to know with XML to merge / Assembly to fully preserve.

    In the meantime, I guess we can parameterize the list of XML files to pass to GenerateAdditionalLinkXmlFile in some custom ScriptableObject... In any case though, letting the developer using the plugin customize stripping/preservation requires to inform them of what they can/should do. Unity on their side can provide a simple interface, while the developers of plugins relying on reflection, etc. can lead their user to the right steps.

    For users who "just want it to work", being able to set a link.xml as enabled by default, and having a power user disable it in the Asset / Project Setting XML list could also be the way.