Loading content with Signed Urls

Our current setup is along the lines of this:

1. Request list of all asset bundles for a particular build from our server
2. List is returned with signed urls (google cloud storage)

Desired Outcome:
Load the content catalog, hash file, and asset bundles using the new signed urls.

Issue: How do I override the Content Catalog and Hash loading to use the new signed url?

I believe I can create my own AssetBundleProvider and simply replace the urls with the new signed ones when it tries to download. But can I create my own ContentCatalogProvider?

The reason for this is we don't want people to learn the url scemes and try to snoop prying for access to files that aren't ready for public yet. Signed urls allow per-file authentication process to avoid this.

Thanks,
Quinn

 

Why not use remote catalog, then convert all urls in the json file to signed urls, then upload to your RemoteLoadPath. So you won’t need to *hack* the system, though I guess you will find a way eventually.

 

yes, you can do both of those things, and we are working on an example (or perhaps built in provider) to do that.

As to @Favo-Yang's suggestion, that could work too (editing the catalog at build time to include the correct paths). This obviously won't work if your URLs are dynamic.

 

Thanks @Favo-Yang and @unity_bill.

The signed urls are dynamically generated and expire after 15 minutes. So there's no way to just compile them into the build sadly. I am currently using a Remote Catalog.

I saw there's an Addressables.LoadCatalogAsync(string path) which I've tried to send it the signed-url path to the catalog on the server but was having issues still... Maybe having a remote catalog isn't necessary though. The way I'm storing bundles is in a per-build specific folder (to keep backwards compatibility for testing previous builds). So maybe I just have a local catalog and append the signed urls when downloading the bundles?

 

To actually get the bundles, yeah, I'd recommend a custom provider that appends the signed url as needed.
To get the remote catalog, you may need to derive from

JsonAssetProvider

and override the Start() method.

A remote catalog is only necessary if you want to be able to update content without publishing a new build. If every time you change something, you ship a new full build, then the remote catalog is not needed.

 

An update on this, I successfully am using runtime generated signed urls to load asset bundles. Turns out was actually fairly simple. Here's how I'm achieving this:

Step 1: Download list of all the signed urls
For us this is easy, we're dumping all the bundles into 1 folder, so our server engineers setup a separate API call for me to get all signed urls for a folder in a Google Bucket. (game specific, aka not a unity call here)

Step 2: Create your own AssetBundleProvider
I simply found the Addressables default

AssetBundleProvider

and

AssetBundleResource

, copied them into my own ProjectAssetBundleProvider and made the following changes:

Code (CSharp):

1. UnityWebRequest CreateWebRequest(IResourceLocation loc)
2. {
3.     // This is your Step 1 which contains all of your signed urls
4.     _assetBundleUrlResolver = AssetBundleUrlResolver.Instance;
5.    // Get the signed url for this particular asset
6.    // The default loc.InternalId here would be the original unsigned url (http://www.storage.google.com/yourbucket/blah.bundle)
7.     string signedUrl = _assetBundleUrlResolver.GetSignedUrl(loc.InternalId);
8.     if (m_Options == null)
9.         return UnityWebRequestAssetBundle.GetAssetBundle(signedUrl);
10.  
11.     // Note the rest of this is untouched from original CreateWebRequest function.
12.     var webRequest = !string.IsNullOrEmpty(m_Options.Hash) ?
13.         UnityWebRequestAssetBundle.GetAssetBundle(signedUrl, Hash128.Parse(m_Options.Hash), m_Options.Crc) :
14.         UnityWebRequestAssetBundle.GetAssetBundle(signedUrl, m_Options.Crc);
15.  
16.     if (m_Options.Timeout > 0)
17.         webRequest.timeout = m_Options.Timeout;
18.     if (m_Options.RedirectLimit > 0)
19.         webRequest.redirectLimit = m_Options.RedirectLimit;
20. #if !UNITY_2019_3_OR_NEWER
21.     webRequest.chunkedTransfer = m_Options.ChunkedTransfer;
22. #endif
23.     if (m_ProvideHandle.ResourceManager.CertificateHandlerInstance != null)
24.     {
25.         webRequest.certificateHandler = m_ProvideHandle.ResourceManager.CertificateHandlerInstance;
26.         webRequest.disposeCertificateHandlerOnDispose = false;
27.     }
28.     return webRequest;
29. }

The AssetBundleUrlResolver.GetSignedUrl function is simple, just doing a simple string.Contains search of the signed urls using the original url.

Code (CSharp):

1. public string GetSignedUrl(string originalUrl)
2. {
3.     // SignedUrls is simply list of string urls that were signed by the server.
4.     foreach (var url in SignedUrls)
5.     {
6.         if (url.Contains(originalUrl))
7.             return url;
8.     }
9.  
10.     return null;
11. }

Step 3: Set your bundle provider in Addressables window to be your own

I hope this helps someone!

EDIT: Btw, I'm also still using a remote catalog. Turns out it runs through my provider to get the remote catalog so it's checking the signed url version of the catalog.

 

@unity_bill This approach works above, but one concern here is if there are any updates to how Addressables BundleProviders work, i'll have to copy/paste this code back in with each update to be sure we're using the latest code. Which I assume is a problem for anyone writing custom providers for things.

 

That's not as bad an issue as you think. We have custom stuff in several packages, Addressables, TextMeshPro, LWRP, etc. If you are using custom stuff you should:

• Move the package from Library/PackageCache/ to Packages/
• Rename the package not to have a version suffix.
• Commit it to your repo without your changes.
• Commit your changes.

To update the package:

• Update your working directory back to the revision you committed the vanilla package in.
• Delete the package.
• Update using the package manager.
• Move the package from Library/PackageCache/ to Packages/
• Rename the package not to have a version suffix.
• Commit.
• Merge your changes in.

We use Mercurial for this, where I have a branch for import of each package. It works a treat and you can easily inspect what changes are made to the packages by the team this way. If you have a custom provider, it's very easy to check the diff of the original one and make the same mods on yours.

 

We host our assets files on a Playfab CDN and to access them we must first call GetContentDownloadUrl API (a HTTP request to get a signed URL for accessing that file) and then make the GET request to that signed URL to actually get the file. But The URL expires in x time, so to make sure everything is still valid, we must call GetContentDownloadUrl API before every attempt to download a file.
We managed to implement our own AssetBundleProvider and set it to the AssetBundleProviderType in our remote group asset, as @CodeBombQuinn suggested. The problem is we don't have the list of all the signed URLs and we must get each one before making each get resource request. Does anyone knows how inject another HTTP request before the one that gets the asset using a custom AssetBundleProvider? Or how to try another approach?

 

One way of doing that would be modifying AssetBundleProvider::BeginOperation() to detect whether you are going to need to sign the Url and chain the operations if so, setting the creation and sending of the AssetBundle web request to be send by the completion handler of your GetContentDownloadUrl() operation. You need to be careful with caching, you don't want to end up doing this even if the file is available locally.

 

The only thing on our roadmap to do to the AssetBundleProvider eventually is "add signed url support". So with any luck, the first change we throw at you that conflicts with your work will also make it unneeded.

But as @AlkisFortuneFish has pointed out, it is a manageable problem.

 

Hey any update on this thread its been year but i havent found anything helpful to change my url path after lazy init

 

@unity_bill

Just curious. I see you've posted a lot in threads about this issue which makes private usage of addressables kind of useless. Has there ever been a demo made for this or any addressable updates to account for a custom provider or signed urls?

 

just checking to see if there is any example to reference for this - setting out to implement a solution soon

 

This worked for me! Was able to remove my custom Addressables code completely!

My Implementation (hope it helps someone):

Code (CSharp):

1. Addressables.ResourceManager.InternalIdTransformFunc += MyTransformAddressableIdFunc;

Code (CSharp):

1. public string MyTransformAddressableIdFunc(IResourceLocation location)
2. {
3.     // If we're trying to load an asset from online, swap the URL for signed version.
4.     if(location.InternalId.StartsWith("http") &&
5.         !location.InternalId.Contains("localhost")) // Don't swap if we're running Editor Hosting
6.     {
7.         // This function is all on you to determine what to do with this.
8.         return GetSignedUrl(location.InternalId);
9.     }
10.  
11.     // Return default location if not a server url
12.     return location.InternalId;
13. }

My GetSignedUrl is simply swapping out that URL for my signed version, notice I'm ignoring Editor Hosting swaps since those don't need Signed Urls.

And here's official Documentation on that func: https://docs.unity3d.com/Packages/com.unity.addressables@1.14/manual/TransformInternalId.html

 

Absolutely useless if you need to make some 3rd party API request (i.e. PlayFab CDN) in order to acquire the bundle URL.
Moreover there are NO working solutions out there as of right now. This must be a joke.

 

there has to be some way using the transform func. It works pretty well.. I am using it extensively in both setting my remoteload path and signed url.

My remote location link completely changes depending on which section they want to look download so its not as easy as just adding a signed url to the end. Since we have 30 different procedures in the app we didnt want to download them all at once because they can be big. This gives them the option to only download content they wish to view. Because of this the different catalogs are stored in different directories on private Azure Blob Storage. The TransformInternalId func actually works great for this if i set the content link right before downloading the addressabels so it applies both the remote location url and adds the private blobs signed policy to the url to allow access to the private content. So the Remote Location url is completely dynamic for me in this way. And looks similar to this in the Addressables Profile for Remote Load Path:

{AddressableURL.remoteLoadPathUrl}/[BuildTarget]/{AddressableURL.remoteLoadPathVersion}
AddressableURL.remoteLoadPathUrl =
https://azurestorageaccountname.blob.core.windows.net/release/MyApp/MyProcedureName/
[BuildTarget] = Android
AddressableURL.remoteLoadPathVersion = 1.0
catalog_2021.xx.xx.xx.xx.xx.json
and then ?my_signed_url_to_add_to_the_end

Final Transformed RemoteLoadPath:
https://azurestorageaccountname.blob.core.windows.net/release/MyApp/MyProcedureName/Android/1.0/catalog_2021.xx.xx.xx.xx.xx.json?my_signed_url_to_add_to_the_end

The biggest hurdle was new builds with new addressable content without affecting previous builds addressables. So I used the app build version which is in the url to download the new correct content for that build and found in the the addressable profile the Cache Clear Behaviour which allows me to Clear When New Version Loaded. (Although in atleast in addressables version1.17.17 its called Clear When When New Version Loaded)

So far everything I am doing between these has addressables, versioning and url signing for private blobs working like a charm.

 

Hi guys
@pillakirsten if you are still around I have a question:
InternalIdTransformFunc seems not to work with LoadContentCatalogAsync.
It calls it several times resulting in double encrypion or just calls it plain without changing the cataloge.
I have several catalogues that I load at runtime one by one
Is suspect this works for bundles.
Is there some special way to change URL for loading a catalogue?
Tnx