Now Available: Private Buckets

This sounds very promising.

We'd like to try this out in order to have secured https downloads of addressable at runtime.

However, it's still not clear from searches or reading docs exactly what we're meant to do to do this? TransformInternalId? Copying AssetBundleProvider.cs and and modifying CreateWebRequest()? Something else in recent addressables? Deeper forum searches reveals fear about whether Content Catalogs or json need separate treatment from AssetBundleProvider.cs.

Our use case is that we we already have a large project using CCD with Addressables, and we'd like to move the content to a new bucket, change our RemoteLoadPath, and somehow append the header?

Is there any documentation on a minimal way to append the auth header to all requests to a private bucket? Iterating on auth or addressables load failures isn't quick or fun.

Thanks.

[Edit: It sounds like in 1.19 WebRequestOverride is the way to go and that this (and appending auth headers for any other service) is finally painless? Is that now the recommendation?]

 

After spending a full day debugging this, including reading bundles directly using UnityWebRequest and the API, using HttpRequest, and playing with PostMan, this still doesn't work. Specifically our initial download of a 1.58Gb initial bundle (it contains a lot of 16k-sq textures...) gets this error, which it does *not* get when the same code is run on our non-authenticated bucket containing the same content:

Curl error 18: transfer closed with 200924547 bytes remaining to read.

Now suspiciously, the initial bundle is 1585044867 bytes, meaning transfer is closing after precisely 1320Mb, which seems like a very round number...

Any thoughts?

 

OK, so after careful evaluation, some of our problems are caused by the apparent 1,320Mb limit introduced by private buckets.

I can run the following code (which does not use addressables - it just uses UWR) twice - once with a URL of a bundle in the private bucket, and once with ONLY the bucket ID changed so that it points to our public bucket. The private bucket always closes the connection after transferring precisely 1,320Mb. The public bucket succeeds. Both buckets contain the same file. (Identical hash, uploaded separately. We can't prove it because private buckets don't support promotion.)

I hoped this was server-side, but I can also confirm that if I use System.Net.HttpClient to download we get the full 1.48Gb file. So ??confused??.

static string authenticate(string username, string password)
{
string auth = username + ":" + password;
auth = System.Convert.ToBase64String(System.Text.Encoding.GetEncoding("ISO-8859-1").GetBytes(auth));
return auth;
}

IEnumerator Fetch()
{
using (var www = UnityWebRequestAssetBundle.GetAssetBundle(url))
{
string authorization = "Basic " + authenticate("", token);
www.SetRequestHeader("Authorization", authorization);
www.SendWebRequest();
while (!www.isDone)
{
Debug.Log($"Bytes downloaded: {www.downloadedBytes}");
yield return new WaitForSeconds(0.5f);
}
Debug.Log($"Error {www.error}");
}
}

 

It also looks like in some cases, addressables is generating URLs for dependent asset loads that do not hit the API, but go to <very long url which it would take too long to sanitize for posting> starting with

cloudcontent.unity3dusercontent.com

Rather than going through the v1 CCD API.

We get an initial WebRequestOverride, which looks like this...

https://CLIENT_REMOVED.client-api.u...est/entry_by_path/content/?path=/PATH_REMOVED

And append an auth header to it.

The failure that comes back is with a much longer URL, that note doesn't include the v1 API but hits the bucket directly with some sort of token. I've tried to sanitize this whilst keeping it meaningful.

RemoteProviderException : Unable to load asset bundle from : https://CLIENT_REMOVED.cloudcontent.../REMOVED?__token__=exp=SNIP~acl=/buckets/SNIP with auth Basic SNIP_ENCODE_AUTH_TOKEN

(Note that we have modified the log in addressables source to confirm that URL has the header auth appended to it, even though it is not identical to a URI our WebRequestCallback received, implying that something in Addressables changed the URI after WebRequestCallback?)

And those are the ones that seem to fail auth (?? because they don't hit the API that reads the header and then the protected bucket rejects them?? - this was the cause of my earlier confusion about auth headers on UWR seemingly not working or because the __token__ in the query string conflicts with the auth header?)

This has been an epically frustrating waste of a day that I thought would be 3 lines of drop-in code and clicking a button to promote our current data into a private bucket.

[Edit: Sorry that this turned into a bug report on a feature announcement.]

 

Hello,
Is it possible to make these bucket tokens expire, say after 1 hour? This would allow them to be distributed to end users
so they can access the bucket. However, if they were to somehow figure out this token, even if they shared it, it would only be "useful" for an hour (in this example).

 

thanks..

 

I assume you’re asking whether I have the time to script with some new api’s for this functionality?

if that’s the question, I think I’ll be at that part in my project in maybe 45 days or so.

however, if you let me examine the api’s I can give you some feedback (without fully scripting) as I have a sense about how they should work.

thanks

 

ok, I'll take a look at these API's.

Regarding the use case, imagine that the game has two types of assets. Those assets common to all users and those that are specific to a user.

The assets that would be common to all users would be things like levels that have been designed by the developer of the game and the private bucket mostly works well for that. (more on this at the end).

Now, the assets that are specific to a user would be assets that the user themselves have created and uploaded. One reason for users to upload such assets would be to synch those assets across (say) Windows, iOS etc.

Now there are two ways to make those user-specific assets only accessible to the appropriate user. The first way would be to proxy all requests through a proxy server which in turn accesses the private bucket. This works, but requires a proxy server that one would have to stream lots of large assets through. This would cause one to lose the benefits of a CDN.

The second approach is that the user connects to the central server, but instead of streaming assets through this central server, the server instead returns a signed time-limited URL to the user client, which then makes the request to the CDN.

This way one gets the advantage of a CDN and also avoid having to develop a server infrastructure which can stream multi-gb requests through it.

In summary this facilitates a redirect workflow rather than a proxy workflow.

Regarding the CDN advantage, imagine a request originated in Asia, but one's proxy server is in the US. In this case the client will connect to the US server which in turn will likely get the data from a US CDN endpoint, which sort of defeats the purpose of CDN end-points "near" a user.

So...
With a proxy flow one has

Client In Asia -> US Central Server -> US-based CDN end-point

With a redirect flow one has

Client In Asia -> US Central Server -> Redirect with signed URL
Client In Asia -> Asia-based CDN end-point

Even for the first class of assets (those common to all users), a redirect workflow is more appealing than a proxy workflow. In theory, one could distribute the same private bucket key to all users. But this is unappealing from a security standpoint since one must assume that eventually this key will get compromised. So, even here a redirect flow where a user gets signed URL's back from the central server is a better approach.

Hope this makes sense.

 

Hi,
When do you think we will be able to get the same functionality as this one -> Securing Azure CDN assets with token authentification. Which is literally what the post above describes.

Thx.