Search Unity

Feedback PackMan can reduce the unnecessary scope search request for pagination

Discussion in 'Package Manager' started by Favo-Yang, Aug 4, 2022.

  1. Favo-Yang

    Favo-Yang

    Joined:
    Apr 4, 2011
    Posts:
    464
    I have noticed that PackMan always fire two http requests to search a scoped registry for a given scope. The second request changed the "from" parameter from 0 to first_response.size+1. But the second request seems unnecessary.

    /-/v1/search?text=com.littlebigfun.action-sender&from=0&size=250
    /-/v1/search?text=com.littlebigfun.action-sender&from=1&size=250

    The first request will return a list with only one object (total=1). Since the request asked for size=250 and the server only returns 1 object. Packman can assume no need for a following pagination request.

    Code (JavaScript):
    1. {
    2.   "objects": [
    3.     {
    4.       "package": {
    5.         "name": "com.littlebigfun.action-sender",
    6.         "displayName": "Unity Action Sender",
    7.         "description": "A type-safe replacement to SendMessage.",
    8.         "dist-tags": {
    9.           "latest": "1.0.0"
    10.         },
    11.         "maintainers": [
    12.           {
    13.             "name": "openupm",
    14.             "email": "openupm"
    15.           }
    16.         ],
    17.         "author": {
    18.           "name": "Favo Yang",
    19.           "url": "https://github.com/favoyang"
    20.         },
    21.         "repository": {
    22.           "type": "git",
    23.           "url": "git+https://github.com/favoyang/unity-action-sender.git"
    24.         },
    25.         "readmeFilename": "README.md",
    26.         "homepage": "https://github.com/favoyang/unity-action-sender#readme",
    27.         "bugs": {
    28.           "url": "https://github.com/favoyang/unity-action-sender/issues"
    29.         },
    30.         "license": "MIT",
    31.         "time": {
    32.           "modified": "2021-05-03T08:16:14.249Z"
    33.         },
    34.         "versions": {
    35.           "1.0.0": "latest"
    36.         }
    37.       },
    38.       "flags": {},
    39.       "local": true,
    40.       "score": {
    41.         "final": 1,
    42.         "detail": {
    43.           "quality": 1,
    44.           "popularity": 1,
    45.           "maintenance": 0
    46.         }
    47.       },
    48.       "searchScore": 100000
    49.     }
    50.   ],
    51.   "total": 1,
    52.   "time": "Thu, 04 Aug 2022 13:42:34 GMT"
    53. }
    But Pacman still fires the second request and get an empty list.

    Code (JavaScript):
    1. {
    2.   "objects": [],
    3.   "total": 0,
    4.   "time": "Thu, 04 Aug 2022 13:43:10 GMT"
    5. }
    Packman only needs to fire the following request if response.total == request.size.
     
    Favo-Yang, Aug 4, 2022
    #1

  2. maximeb_unity

    maximeb_unity

    Unity Technologies

    Joined:
    Mar 20, 2018
    Posts:
    556
    Hi @Favo-Yang,

    Nice analysis and write-up! I remember this logic of "going one page too far" was implemented as a workaround for a bug in Bintray or Artifactory a few years ago (I forgot which one), which we used while developing the Package Manager and internally at Unity; the bug caused the registry search results to be unreliable, returning packages already listed in previous pages, so we had to implement some algorithm that guarantees it eventually finishes. What we did was continue as long as we didn't get an empty page, or one that included only results already seen on older pages. (Yeah, that was a weird bug!)

    The workaround might no longer be necessary but as we're targeting as wide interop as possible, there's a risk removing this workaround might introduce a regression for some users who might be using unpatched registries. Thanks for your feedback though, we'll definitely take your suggestion under consideration.
     
    maximeb_unity, Sep 1, 2022
    #2
    Favo-Yang likes this.