Search Unity

scoped registry packages not displayed in 2019.3.0f6

Discussion in 'Package Manager' started by guillaume_unity135, Jan 29, 2020.

  1. guillaume_unity135

    guillaume_unity135

    Joined:
    Dec 3, 2019
    Posts:
    5
    Hi, I just updated Unity to v2019.3.0f6 and the packages from my company's private registry are not displayed in UPM UI anymore.

    - We are using Verdaccio
    - it is working correctly on Unity 2019.2
    - on our local registry I can see a 404 error for the route "/-/v1/search" when Unity updates the UPM package list

    I know that Unity had support for "/all" only in versions previous to 2019.3. It seems that 2019.3 implements the new npm search route. Is the "/all" route support discontinued for the new Unity version ?
     
  2. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    756
    HI @guillaume_unity135,

    You are right, the Package Manager now first try to fetch packages through the "/-/v1/search". But, if it fails, we resort to the "/-/all" route to fetch all packages. You don't see a "/all" request just after the 404?

    Regards,

    Pascal
     
  3. guillaume_unity135

    guillaume_unity135

    Joined:
    Dec 3, 2019
    Posts:
    5
    Thank you for your fast answer. I just checked and you are right. I compared the requests made by Unity 2019.2 and Unity 2019.3.

    2019.3 : The "/-/v1/search" fails. It falls back on "/-/all". The server replies with the list of packages and a 200 status code as intended. However the packages aren't displayed in Unity and the Unity's npm cache stays empty.

    2019.2 : The request to "/-/all" works the same way but then Unity seems to make requests individually to every package that matches the scope. The npm cache is filled and the packages are shown in UPM.
     
  4. guillaume_unity135

    guillaume_unity135

    Joined:
    Dec 3, 2019
    Posts:
    5
    Hi Pascal,

    Do you have any news ? We are studying the possibility to upgrade our production pipeline to Unity 2019.3 at my company and this issue is quite blocking for us.
     
  5. amarcolina

    amarcolina

    Joined:
    Jun 19, 2014
    Posts:
    65
    I am encountering this issue as well, and I do not see the "/all" request after the "/search" requests 404.
     
  6. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    756
    Sorry. I missed your post. This may be a bug. Would you mind providing a project manifest exposing the bug? If you could create a bug report we the required information it would be awesome.

    Regards,

    Pascal
     
  7. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    756
    Can you create a bug report with a sample scoped registry configuration exposing the issue?

    Regards,

    Pascal
     
  8. kevinq_vr

    kevinq_vr

    Joined:
    Feb 5, 2020
    Posts:
    1
  9. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    756
    Thank you!
     
  10. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    802
    We're seeing the same, this is a regression in 2019.3. After adding a scoped registry Unity needs at least one, sometimes multiple, restarts until the packages are shown. Sometimes they're only shown after manually adding one of the packages in that registry to the manifest.

    This is a regression to 2019.3.0f3 or even 0f5 - again a workflow regression introduced in a release version...
     
  11. guillaume_unity135

    guillaume_unity135

    Joined:
    Dec 3, 2019
    Posts:
    5
    Hi Pascal, thank you for you answer. Unfortunately the manifest I use won't be of any help to reproduce the bug as my company's Verdaccio server can only be accessed from a local network for privacy purposes. Although I made a test with OpenUPM packages added as scopedRegistries that has this bug too, I will make a bug report with it.
     
  12. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    756
    Thank you so much! This will be really helpful!
     
  13. nilsdr

    nilsdr

    Joined:
    Oct 24, 2017
    Posts:
    374
    I encounter the same bug, for what its worth. 2019.3.0f6, I can see the /all endpoint is succesfully queried but packages don't show in the UI (adding them manually works fine)
     
  14. Wolfram

    Wolfram

    Joined:
    Feb 16, 2010
    Posts:
    261
    We're experiencing the same bug as well. 2019.2 works, 2019.3.0f6 does not.
    When accessing our registry server in 2019.3, the following error appears in %localappdata%\Unity\Editor\upm.log :

    Code (CSharp):
    1. [2020-02-12T14:27:13.756Z][ERROR]
    2.  
    3. {
    4.   "message": "The url [http://192.168.0.8:4873/-/all] did not return a JSON response.",
    5.   "stack": "Error: The url [http://192.168.0.8:4873/-/all] did not return a JSON response.\n    at Object.getJson (C:\\snapshot\\upm\\node_modules\\@upm\\core\\lib\\private\\utilities\\request.js:0:0)\n    at <anonymous>\n    at process._tickDomainCallback (internal/process/next_tick.js:228:7)"
    6. }
    We're running Verdaccio 4.4.0, and the verdaccio log says:
    Code (CSharp):
    1.  http <-- 404, user: null(192.168.0.142), req: 'GET /-/v1/search?text=com.nmy&from=0&size=250', error: File not found
    2. http <-- 404, user: null(192.168.0.142), req: 'GET /-/v1/search?text=com.nmy&from=0&size=250', error: File not found
    3. http <-- 200, user: null(192.168.0.142), req: 'GET /-/all', bytes: 0/2940
    4. http <-- 200, user: null(192.168.0.142), req: 'GET /-/all', bytes: 0/2940
    5.  
    , so nothing unexpected.

    Accessing http://192.168.0.8:4873/-/all directly via browser or curl DOES return a valid JSON file, containing all packages of the registry.

    Further info: 2019.2 does generate the file %localappdata%\Unity\cache\npm\192.168.0.8_4873\192.168.0.8_4873\-\all\.cache.json , which IS the downloaded JSON file.
    2019.3 does NOT generate this file,but it DOES evaluate it, if it was generated by 2019.2 (=> so this is a viable workaround to still be able to use scoped registries with 2019.3, except the necessity to have 2019.2 update this file each time the list of packages on the registry server changes...
     
    Last edited: Feb 12, 2020
  15. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    756
    Our QA was able to reproduce the issue. A developer is trying to find the cause at this moment. We should have a resolution to this problem soon. Really sorry about this!

    Regards,

    Pascal
     
  16. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    756
    This is also really helpful. Thank you!
     
  17. Wolfram

    Wolfram

    Joined:
    Feb 16, 2010
    Posts:
    261
    Thanks for the update! Did you also see my edited post (paragraph "Further info:" at the end of my post)?
     
  18. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    756
    Glad to see you found a temporary workaround. I'll give you more details on the issue. Maybe it will help you figure out another workaround. The gist of the issue is that we added HTTP response validation that was a bit too strict. UPM would reject the response as not being valid JSON even if the body was (ex: Missing response content-type header or different than `application/json`). I believe Verdaccio doesn't return the `content-type` in the response headers and makes UPM simply reject the request :(.

    We were already aware of this issue (but with a different symptom) and made a fix that landed in 2020.1.0a17. If you have a chance to try with this release and see if the issue is still there for you, that would be really appreciated!

    Thank you!
     
  19. samuelb_unity

    samuelb_unity

    Unity Technologies

    Joined:
    Nov 13, 2017
    Posts:
    151
    As @okcompute_unity says, we believe this was the Content-Typer header check we implemented. If you like, you could run
    curl -I http://192.168.0.8:4873/-/all
    and verify the
    Content-Type
    value or lack thereof. If it's not
    application/json
    then we've found the culprit :)
     
    Wolfram likes this.
  20. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    756
    I confirm we were not able to reproduce the issue in 2020.1.a17. So we have in our hands. I'll push to get the fix in the next 19.3 release.
     
    Wolfram likes this.
  21. Wolfram

    Wolfram

    Joined:
    Feb 16, 2010
    Posts:
    261
    The problem indeed appears to be fixed in 2020.1.a17 (I did not test versions earlier than a17 though).
    However, the auto-refresh when modifying manifest.json does not appear to work correctly in that version (and possibly others, haven't tested this in 2019.2):
    • open/create project with 2020.1.a17
    • open Package Manager
    • edit manifest.json externally and add a scopedRegistry
    • switching back to Unity, the PM detects a modified manifest.json and refreshes "something" - but no packages from the scopedRegistry appear
    • even after closing and re-opening the PM, or manually refreshing, the packages are still not listed
    • only after completely closing Unity and re-opening it, and then the PM, the packages from the scopedRegistry are listed
     
  22. Wolfram

    Wolfram

    Joined:
    Feb 16, 2010
    Posts:
    261
    Yep, the returned result does not contain a Content-Type entry.
    Thanks! Waiting for 2019.3.2 then... ;-)
     
    fherbst likes this.
  23. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    756
    This was a regression in 2020.1 that was fixed in 2020.1.a18 :mad:
     
    Wolfram, fherbst and samuelb_unity like this.
  24. guillaume_unity135

    guillaume_unity135

    Joined:
    Dec 3, 2019
    Posts:
    5
    Hi, sorry I didn't have time to make a bug report for the issue. I'm glad you found a fix, thank you ! I just tried to curl our private registry and I can confirm that Content-Type field is missing in the response Header.
     
  25. Wolfram

    Wolfram

    Joined:
    Feb 16, 2010
    Posts:
    261
    fherbst and okcompute_unity like this.
  26. Chadobado

    Chadobado

    Joined:
    Oct 19, 2016
    Posts:
    26
    Anyone wanting a quick-fix can simply tell verdaccio to return the correct header.

    <your node_modules folder>/verdaccio/build/api/endpoint/api/search.js

    add at line 17:

    res.set('Content-Type','application/json; charset=utf-8');

    Save and restart verdaccio and it will return the proper Content-Type for the endpoint.
     
    Wolfram likes this.
  27. FodderMK

    FodderMK

    Joined:
    Jan 21, 2013
    Posts:
    5
    I submitted a fix to Verdaccio which should be in the next release.
     
    fherbst, amarcolina and supita_unity like this.