Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Dismiss Notice

Question How to make custom packages from GCP Artifactory show up to browse them

Discussion in 'Package Manager' started by MegaMileyStudios, Jun 12, 2023.

  1. MegaMileyStudios

    MegaMileyStudios

    Joined:
    Aug 19, 2017
    Posts:
    72
    I just got my GCP Artifact Registry setup as an NPM service and I'm able to both deploy and install Unity packages now (transfering everything over from JFrog as my free account will be sunset soon).

    The only thing I'm missing right now is the ability to see all the packages I have under my registry and install them directly. If I add them via their package id it works as expected but just like my assets or Unity's own packages I'd like to be able to see all my packages in the package manager before they are installed

    Here's what it looks like in a test project with just one of them installed (note, this is the GameConsole project hence why it says Custom here, the custom registry is setup in the same way as the main project though) Screenshot 2023-06-12 at 09.49.07.png
    And what it looks like with the main project with both packages installed in the main project (the GCP in the name is just because I am transfering between JFrog and GCP atm)

    Screenshot 2023-06-12 at 09.49.23.png The packages are fully available but as you can see for the top one it doesn't show the RemoteConfigHelpers package in the list as an available package, how could I get this to show up?

    I'm authenticated through a GCP service account that has the Artifact Registry Reader role so I would imagine it to show all the packages through the artifactregistry.npmpackages.list permission but they don't seem to show up. The repository is setup as a regular NPM repository on a single region.

    How can I get my packages to show up within having to be manually added first?

    Oh I just tried to refresh and got this error:

    Access to 'REDACTED/megamileystudios/internal-unity-packages/-/all' was denied


    After some more investigating, it seems that /-/all has been deprecated back in 2017, https://blog.npmjs.org/post/157615772423/deprecating-the-all-registry-endpoint and /-/v1/search should now be used instead, odd that Unity is still calling such an old endpoint
     
    Last edited: Jun 12, 2023
  2. maximeb_unity

    maximeb_unity

    Unity Technologies

    Joined:
    Mar 20, 2018
    Posts:
    481
    Hi @MegaMileyStudios,

    The Unity Package Manager uses the "/all" endpoint as a fallback for older npm registries that have not implemented /v1/search in an attempt to be compatible with as many of them. It uses the /v1/search endpoint first, using each configured scope as a search query (so, yes, limiting the number of scopes to limit the number of search queries might be ideal) and combining the results after filtering the results to only those scopes (hence guaranteeing the package space partitioning the scoped registry feature promises).

    If you're seeing a 403 Forbidden error from the /all endpoint, that means the /v1/search endpoint likely returned 404 and UPM used the /all fallback; if /v1/search failed with any other error or returned data successfully, /all does not get invoked. This could mean that the GCP registry does not support/provide the /v1/search endpoint, or it's not enabled in your current setup. You can find more information by turning up logging to Debug level, in which case you'll see failed HTTP requests logged in the upm.log file. I hope this helps!
     
  3. maximeb_unity

    maximeb_unity

    Unity Technologies

    Joined:
    Mar 20, 2018
    Posts:
    481
    Or, maybe a more likely scenario, it is enabled, but GCP returns 404 if it doesn't get valid credentials mapped to the proper permissions (instead of returning 403), which would mislead UPM into thinking the endpoint is not supported, when in fact GCP would just not allow access nor "leak" information about its existence as a security measure.
     
  4. MegaMileyStudios

    MegaMileyStudios

    Joined:
    Aug 19, 2017
    Posts:
    72
    Hey there, yeah the issue with the /-/all being called was due to my credentials expiring (GCP auth tokens have a 1 hour expiration to them), I fixed this by creating a custom PackageManager extension and refreshing the token every 50 minutes when I'm accessing the Package Manager (I'm using the package selected event and a EditorPrefs integer with the last refresh time to keep track of that).

    The /v1/search is working as expected most likely but I don't see any packages that I haven't installed yet which is what I do want to see in my package manager. Is there something I can do to enable showing all not yet installed packages?
     
  5. maximeb_unity

    maximeb_unity

    Unity Technologies

    Joined:
    Mar 20, 2018
    Posts:
    481
    Do you happen to see "My Registries" in the Package Manager Window? If the search endpoint works as expected and returns the packages in the search results, then those packages should be visible under "My Registries". In addition to those you have installed in your project showing up under "In Project", of course. Is there anything in the upm.log file that could provide a hint of what's going on?
     
  6. MegaMileyStudios

    MegaMileyStudios

    Joined:
    Aug 19, 2017
    Posts:
    72
    I do see the My Registries in the PackMan window (also part of the screenshots I send at the start of this post), and when I look at my upm.log file I see some errors when my auth token has expired but once I authenticate again (this is in a project wihtout the code to re-authenticate automatically) those all go away.

    There doesn't seem to be any error/issue in terms of listing the packages, it just doesn't seem to list them all including the ones that aren't installed at the moment

    The excat log entries after I did a refresh look like this:

     
  7. maximeb_unity

    maximeb_unity

    Unity Technologies

    Joined:
    Mar 20, 2018
    Posts:
    481
    Have you tried using the command-line to verify that your registry is actually returning anything with the search? You can do that by logging in with npm, first by logging in (using
    npm login
    , I think you would use your auth token as a password) and then by doing
    npm search --registry <your-registry> <your scope>
    . E.g.
    npm search --registry https://gcp.example.com/path/to/registry com.my-org
    .
     
  8. MegaMileyStudios

    MegaMileyStudios

    Joined:
    Aug 19, 2017
    Posts:
    72
  9. maximeb_unity

    maximeb_unity

    Unity Technologies

    Joined:
    Mar 20, 2018
    Posts:
    481
    I checked your SO thread and thought I'd point out something: in your example URL, if
    https://europe-west1-npm.pkg.dev/PROJECT_NAME/REPOSITORY_NAME
    is the registry URL, then the search endpoint should be
    https://europe-west1-npm.pkg.dev/PROJECT_NAME/REPOSITORY_NAME/-/v1/search?text=BASE_SCOPE
    (missing
    /-
    before
    /v1/search
    in your test URL). Otherwise, you're essentially asking the registry to return an subresource "search" of the "v1" package ;)
     
  10. MegaMileyStudios

    MegaMileyStudios

    Joined:
    Aug 19, 2017
    Posts:
    72
    When I add that `/-` to the URL I just end up with a 404 HTML page rather than a regular API response. Also when I remove the REPOSITORY_NAME and just use the `-` there it says that it can't find the repository `-` rather than it not finding the requested entity within it. I've tried different variations as well but none of them seem to work.

    The `gcloud artifacts files list` command does work and shows all the packages and their versions that are available but it just seems that the NPM api isn't working.
     
  11. maximeb_unity

    maximeb_unity

    Unity Technologies

    Joined:
    Mar 20, 2018
    Posts:
    481
    It's possible that their repository software does not implement /-/v1/search (as evidenced by the 404 response) and actively does not support /-/all (as evidenced by the Forbidden error, which would be an underlying 403). Unfortunately, this seems to point to a limitation of the registry software that's being used.

    Google's documentation seems to point to this as being some in-house solution that doesn't fully implement the npm protocol, but only the metadata retrieval endpoint ("npm view", "yarn info"), which, incidentally, is all that's needed to install them. The page makes no mention of npm search, but instead seems to rely on their gcloud tool to list packages (as you found). Unfortunately that isn't compatible with the Unity Package Manager.
     
  12. MegaMileyStudios

    MegaMileyStudios

    Joined:
    Aug 19, 2017
    Posts:
    72
    Aha, that's unfortunate. As a workaround then, is there any way for me to provide extra information to the Package Manager so that I can show all the packages through running the `gcloud artifacts list` command locally? Or would I be better off creating a custom UI for this through a sort of "core" package? (I'll likely need to create such a package anyways given the authentication window of 1 hour before needing to re-authenticate)
     
  13. maximeb_unity

    maximeb_unity

    Unity Technologies

    Joined:
    Mar 20, 2018
    Posts:
    481
    Unfortunately the Package Manager does not provide such capabilities at this time. But since you need to manually support your custom authentication requirements, like you said, you could do this yourself and use the
    UnityEditor.PackageManager.Client
    Scripting API. For instance, you can fetch the information that you want to display for a package by using the
    Client.Search
    method with that package's name.

    If you would like, you could submit a feature request for something like this (e.g. configuring a custom list of packages for a scoped registry that does not support the npm search API) on this page, by clicking the "Submit a new idea" tile under the Roadmap's Package Management tab.
     
    MegaMileyStudios likes this.
  14. MegaMileyStudios

    MegaMileyStudios

    Joined:
    Aug 19, 2017
    Posts:
    72
    Just made the feature request, guess I'll be creating a custom simple UI for this for now