Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

Now Available: Private Buckets

Discussion in 'Unity Cloud Content Delivery' started by ChristinaGuo, Sep 21, 2021.

  1. ChristinaGuo

    ChristinaGuo

    Unity Technologies

    Joined:
    Feb 20, 2020
    Posts:
    48
    We've just launched private buckets!

    Private buckets protect read access to buckets with an access token, so that only those users with that access token can retrieve content from that bucket.
    • You now have the option to make a bucket private upon creation.
    • You can edit buckets in a new Settings page in the CCD dashboard.
    • You can view, create, edit or delete bucket access tokens in the new Settings page.
    • You can delete buckets on individual bucket pages.
    Screen Shot 2021-09-21 at 3.57.04 PM.png
     
  2. ferretnt

    ferretnt

    Joined:
    Apr 10, 2012
    Posts:
    401
    This sounds very promising.

    We'd like to try this out in order to have secured https downloads of addressable at runtime.

    However, it's still not clear from searches or reading docs exactly what we're meant to do to do this? TransformInternalId? Copying AssetBundleProvider.cs and and modifying CreateWebRequest()? Something else in recent addressables? Deeper forum searches reveals fear about whether Content Catalogs or json need separate treatment from AssetBundleProvider.cs.

    Our use case is that we we already have a large project using CCD with Addressables, and we'd like to move the content to a new bucket, change our RemoteLoadPath, and somehow append the header?

    Is there any documentation on a minimal way to append the auth header to all requests to a private bucket? Iterating on auth or addressables load failures isn't quick or fun.

    Thanks.

    [Edit: It sounds like in 1.19 WebRequestOverride is the way to go and that this (and appending auth headers for any other service) is finally painless? Is that now the recommendation?]
     
    Last edited: Sep 22, 2021
  3. ryan_ngo

    ryan_ngo

    Unity Technologies

    Joined:
    Feb 2, 2021
    Posts:
    35
    Hi @ferretnt! I've confirmed with the Addressables team that what you would need to do is utilize the `WebRequestOverride` within Addressables 1.19+ to add that authentication header. Here is the documentation for reference. It appears like you would just need to set this delegate and it should utilize the custom UnityWebRequest rather than the default. I hope this helps! Please let us know if there's more that we can help with.
     
    adamgolden likes this.
  4. ferretnt

    ferretnt

    Joined:
    Apr 10, 2012
    Posts:
    401
    After spending a full day debugging this, including reading bundles directly using UnityWebRequest and the API, using HttpRequest, and playing with PostMan, this still doesn't work. Specifically our initial download of a 1.58Gb initial bundle (it contains a lot of 16k-sq textures...) gets this error, which it does *not* get when the same code is run on our non-authenticated bucket containing the same content:

    Curl error 18: transfer closed with 200924547 bytes remaining to read.

    Now suspiciously, the initial bundle is 1585044867 bytes, meaning transfer is closing after precisely 1320Mb, which seems like a very round number...

    Any thoughts?
     
  5. ferretnt

    ferretnt

    Joined:
    Apr 10, 2012
    Posts:
    401
    OK, so after careful evaluation, some of our problems are caused by the apparent 1,320Mb limit introduced by private buckets.

    I can run the following code (which does not use addressables - it just uses UWR) twice - once with a URL of a bundle in the private bucket, and once with ONLY the bucket ID changed so that it points to our public bucket. The private bucket always closes the connection after transferring precisely 1,320Mb. The public bucket succeeds. Both buckets contain the same file. (Identical hash, uploaded separately. We can't prove it because private buckets don't support promotion.)

    I hoped this was server-side, but I can also confirm that if I use System.Net.HttpClient to download we get the full 1.48Gb file. So ??confused??.

    static string authenticate(string username, string password)
    {
    string auth = username + ":" + password;
    auth = System.Convert.ToBase64String(System.Text.Encoding.GetEncoding("ISO-8859-1").GetBytes(auth));
    return auth;
    }

    IEnumerator Fetch()
    {
    using (var www = UnityWebRequestAssetBundle.GetAssetBundle(url))
    {
    string authorization = "Basic " + authenticate("", token);
    www.SetRequestHeader("Authorization", authorization);
    www.SendWebRequest();
    while (!www.isDone)
    {
    Debug.Log($"Bytes downloaded: {www.downloadedBytes}");
    yield return new WaitForSeconds(0.5f);
    }
    Debug.Log($"Error {www.error}");
    }
    }
     
  6. ferretnt

    ferretnt

    Joined:
    Apr 10, 2012
    Posts:
    401
    It also looks like in some cases, addressables is generating URLs for dependent asset loads that do not hit the API, but go to <very long url which it would take too long to sanitize for posting> starting with

    cloudcontent.unity3dusercontent.com

    Rather than going through the v1 CCD API.

    We get an initial WebRequestOverride, which looks like this...

    https://CLIENT_REMOVED.client-api.u...est/entry_by_path/content/?path=/PATH_REMOVED

    And append an auth header to it.

    The failure that comes back is with a much longer URL, that note doesn't include the v1 API but hits the bucket directly with some sort of token. I've tried to sanitize this whilst keeping it meaningful.

    RemoteProviderException : Unable to load asset bundle from : https://CLIENT_REMOVED.cloudcontent.../REMOVED?__token__=exp=SNIP~acl=/buckets/SNIP with auth Basic SNIP_ENCODE_AUTH_TOKEN

    (Note that we have modified the log in addressables source to confirm that URL has the header auth appended to it, even though it is not identical to a URI our WebRequestCallback received, implying that something in Addressables changed the URI after WebRequestCallback?)

    And those are the ones that seem to fail auth (?? because they don't hit the API that reads the header and then the protected bucket rejects them?? - this was the cause of my earlier confusion about auth headers on UWR seemingly not working or because the __token__ in the query string conflicts with the auth header?)

    This has been an epically frustrating waste of a day that I thought would be 3 lines of drop-in code and clicking a button to promote our current data into a private bucket.

    [Edit: Sorry that this turned into a bug report on a feature announcement.]
     
    Last edited: Sep 23, 2021
  7. dannyd

    dannyd

    Unity Technologies

    Joined:
    Jun 3, 2014
    Posts:
    779
    Hey @ferretnt, sorry for the slow follow up here. We had several people investigating this on different fronts last week and I forgot to respond before heading out on Friday. I also want to apologize that this has been a frustrating experience for you. Hopefully, we can work through this with you and get things sorted out.

    1. We've been unsuccessful in reproducing the large file download issue. I was able to successfully download a 2 GB bundle using both curl/postman and in the editor using the example code you provided (tried both UnityWebRequestAssetBundle.GetAssetBundle and UnityWebRequest.Get). Just to confirm, are you testing this in editor or at runtime on a specific device?
    2. Regarding the second issue with Addressables/WebRequestOverride, we were able to confirm this issue on our end and have since made a configuration change on our end that should address this particular issue. Just to help fill in some details here, the way our Client API works is when a request comes in for content (i.e. on client-api.unity3dusercontent.com), the API responds with a redirect to the actual content location on the CDN (i.e. cloudcontent.unity3dusercontent.com). In the case of private content, the Client API request expects an authorization header to be present and responds with a redirect that is signed (which is why it includes that __token__ parameter in the redirect). There was a problem on the CDN side with our configuration when both the __token__ and an Authorization header were present, so we've updated the configuration to completely ignore the Authorization header on the CDN side. In general, UWR should not have appended to the Authorization header after redirect, which we will investigate separately, but this should fix the issue for you now.
     
    ferretnt likes this.
  8. villaman

    villaman

    Joined:
    Apr 21, 2021
    Posts:
    68
    Hello,
    Is it possible to make these bucket tokens expire, say after 1 hour? This would allow them to be distributed to end users
    so they can access the bucket. However, if they were to somehow figure out this token, even if they shared it, it would only be "useful" for an hour (in this example).
     
  9. kaanyalti

    kaanyalti

    Unity Technologies

    Joined:
    Dec 14, 2020
    Posts:
    8
    Hello @villaman , this is currently not supported, thank you for suggesting this feature, we are going to discuss it
     
  10. villaman

    villaman

    Joined:
    Apr 21, 2021
    Posts:
    68
    thanks..
     
  11. ChristinaGuo

    ChristinaGuo

    Unity Technologies

    Joined:
    Feb 20, 2020
    Posts:
    48
    Hi @villaman - we're currently investigating this ask. Are you currently able to script a solution for this using our APIs?
     
  12. villaman

    villaman

    Joined:
    Apr 21, 2021
    Posts:
    68
    I assume you’re asking whether I have the time to script with some new api’s for this functionality?

    if that’s the question, I think I’ll be at that part in my project in maybe 45 days or so.

    however, if you let me examine the api’s I can give you some feedback (without fully scripting) as I have a sense about how they should work.

    thanks
     
  13. dannyd

    dannyd

    Unity Technologies

    Joined:
    Jun 3, 2014
    Posts:
    779
    @villaman Christina was suggesting that this could already be achieved through our current APIs using the CreateBucketAccessToken and DeleteBucketAccessToken functionality. You'd have to manage the "expiration" yourself and explicitly call delete on the tokens you create after a set time period.

    Out of curiosity - can you expand a bit more on the use case? What are you trying to achieve with expiring tokens like this?
     
  14. villaman

    villaman

    Joined:
    Apr 21, 2021
    Posts:
    68
    ok, I'll take a look at these API's.

    Regarding the use case, imagine that the game has two types of assets. Those assets common to all users and those that are specific to a user.

    The assets that would be common to all users would be things like levels that have been designed by the developer of the game and the private bucket mostly works well for that. (more on this at the end).

    Now, the assets that are specific to a user would be assets that the user themselves have created and uploaded. One reason for users to upload such assets would be to synch those assets across (say) Windows, iOS etc.

    Now there are two ways to make those user-specific assets only accessible to the appropriate user. The first way would be to proxy all requests through a proxy server which in turn accesses the private bucket. This works, but requires a proxy server that one would have to stream lots of large assets through. This would cause one to lose the benefits of a CDN.

    The second approach is that the user connects to the central server, but instead of streaming assets through this central server, the server instead returns a signed time-limited URL to the user client, which then makes the request to the CDN.

    This way one gets the advantage of a CDN and also avoid having to develop a server infrastructure which can stream multi-gb requests through it.

    In summary this facilitates a redirect workflow rather than a proxy workflow.

    Regarding the CDN advantage, imagine a request originated in Asia, but one's proxy server is in the US. In this case the client will connect to the US server which in turn will likely get the data from a US CDN endpoint, which sort of defeats the purpose of CDN end-points "near" a user.

    So...
    With a proxy flow one has

    Client In Asia -> US Central Server -> US-based CDN end-point

    With a redirect flow one has

    Client In Asia -> US Central Server -> Redirect with signed URL
    Client In Asia -> Asia-based CDN end-point

    Even for the first class of assets (those common to all users), a redirect workflow is more appealing than a proxy workflow. In theory, one could distribute the same private bucket key to all users. But this is unappealing from a security standpoint since one must assume that eventually this key will get compromised. So, even here a redirect flow where a user gets signed URL's back from the central server is a better approach.

    Hope this makes sense.
     
  15. Yato-Hataraku

    Yato-Hataraku

    Joined:
    Feb 4, 2017
    Posts:
    1
    Hi,
    When do you think we will be able to get the same functionality as this one -> Securing Azure CDN assets with token authentification. Which is literally what the post above describes.

    Thx.
     
  16. neal-wang

    neal-wang

    Unity Technologies

    Joined:
    Feb 23, 2021
    Posts:
    6
    Hi @NicolasDexx, Thank you for the valuable suggestion! This isn't currently on our roadmap, and not being worked on. We will further discuss it.