Search Unity

Loading content with Signed Urls

Discussion in 'Addressables' started by CodeBombQuinn, Jul 31, 2019.

  1. CodeBombQuinn

    CodeBombQuinn

    Joined:
    Apr 17, 2018
    Posts:
    7
    Our current setup is along the lines of this:
    1. Request list of all asset bundles for a particular build from our server
    2. List is returned with signed urls (google cloud storage)
    Desired Outcome:
    Load the content catalog, hash file, and asset bundles using the new signed urls.

    Issue: How do I override the Content Catalog and Hash loading to use the new signed url?

    I believe I can create my own AssetBundleProvider and simply replace the urls with the new signed ones when it tries to download. But can I create my own ContentCatalogProvider?

    The reason for this is we don't want people to learn the url scemes and try to snoop prying for access to files that aren't ready for public yet. Signed urls allow per-file authentication process to avoid this.

    Thanks,
    Quinn
     
  2. Favo-Yang

    Favo-Yang

    Joined:
    Apr 4, 2011
    Posts:
    268
    Why not use remote catalog, then convert all urls in the json file to signed urls, then upload to your RemoteLoadPath. So you won’t need to *hack* the system, though I guess you will find a way eventually.
     
  3. unity_bill

    unity_bill

    Unity Technologies

    Joined:
    Apr 11, 2017
    Posts:
    913
    yes, you can do both of those things, and we are working on an example (or perhaps built in provider) to do that.

    As to @Favo-Yang's suggestion, that could work too (editing the catalog at build time to include the correct paths). This obviously won't work if your URLs are dynamic.
     
  4. CodeBombQuinn

    CodeBombQuinn

    Joined:
    Apr 17, 2018
    Posts:
    7
    Thanks @Favo-Yang and @unity_bill.

    The signed urls are dynamically generated and expire after 15 minutes. So there's no way to just compile them into the build sadly. I am currently using a Remote Catalog.

    I saw there's an Addressables.LoadCatalogAsync(string path) which I've tried to send it the signed-url path to the catalog on the server but was having issues still... Maybe having a remote catalog isn't necessary though. The way I'm storing bundles is in a per-build specific folder (to keep backwards compatibility for testing previous builds). So maybe I just have a local catalog and append the signed urls when downloading the bundles?
     
  5. unity_bill

    unity_bill

    Unity Technologies

    Joined:
    Apr 11, 2017
    Posts:
    913

    To actually get the bundles, yeah, I'd recommend a custom provider that appends the signed url as needed.
    To get the remote catalog, you may need to derive from
    JsonAssetProvider
    and override the Start() method.

    A remote catalog is only necessary if you want to be able to update content without publishing a new build. If every time you change something, you ship a new full build, then the remote catalog is not needed.
     
  6. CodeBombQuinn

    CodeBombQuinn

    Joined:
    Apr 17, 2018
    Posts:
    7
    An update on this, I successfully am using runtime generated signed urls to load asset bundles. Turns out was actually fairly simple. Here's how I'm achieving this:

    Step 1: Download list of all the signed urls
    For us this is easy, we're dumping all the bundles into 1 folder, so our server engineers setup a separate API call for me to get all signed urls for a folder in a Google Bucket. (game specific, aka not a unity call here)

    Step 2: Create your own AssetBundleProvider
    I simply found the Addressables default
    AssetBundleProvider
    and
    AssetBundleResource
    , copied them into my own ProjectAssetBundleProvider and made the following changes:

    Code (CSharp):
    1. UnityWebRequest CreateWebRequest(IResourceLocation loc)
    2. {
    3.     // This is your Step 1 which contains all of your signed urls
    4.     _assetBundleUrlResolver = AssetBundleUrlResolver.Instance;
    5.    // Get the signed url for this particular asset
    6.    // The default loc.InternalId here would be the original unsigned url (http://www.storage.google.com/yourbucket/blah.bundle)
    7.     string signedUrl = _assetBundleUrlResolver.GetSignedUrl(loc.InternalId);
    8.     if (m_Options == null)
    9.         return UnityWebRequestAssetBundle.GetAssetBundle(signedUrl);
    10.  
    11.     // Note the rest of this is untouched from original CreateWebRequest function.
    12.     var webRequest = !string.IsNullOrEmpty(m_Options.Hash) ?
    13.         UnityWebRequestAssetBundle.GetAssetBundle(signedUrl, Hash128.Parse(m_Options.Hash), m_Options.Crc) :
    14.         UnityWebRequestAssetBundle.GetAssetBundle(signedUrl, m_Options.Crc);
    15.  
    16.     if (m_Options.Timeout > 0)
    17.         webRequest.timeout = m_Options.Timeout;
    18.     if (m_Options.RedirectLimit > 0)
    19.         webRequest.redirectLimit = m_Options.RedirectLimit;
    20. #if !UNITY_2019_3_OR_NEWER
    21.     webRequest.chunkedTransfer = m_Options.ChunkedTransfer;
    22. #endif
    23.     if (m_ProvideHandle.ResourceManager.CertificateHandlerInstance != null)
    24.     {
    25.         webRequest.certificateHandler = m_ProvideHandle.ResourceManager.CertificateHandlerInstance;
    26.         webRequest.disposeCertificateHandlerOnDispose = false;
    27.     }
    28.     return webRequest;
    29. }
    The AssetBundleUrlResolver.GetSignedUrl function is simple, just doing a simple string.Contains search of the signed urls using the original url.

    Code (CSharp):
    1. public string GetSignedUrl(string originalUrl)
    2. {
    3.     // SignedUrls is simply list of string urls that were signed by the server.
    4.     foreach (var url in SignedUrls)
    5.     {
    6.         if (url.Contains(originalUrl))
    7.             return url;
    8.     }
    9.  
    10.     return null;
    11. }
    Step 3: Set your bundle provider in Addressables window to be your own

    I hope this helps someone!

    EDIT: Btw, I'm also still using a remote catalog. Turns out it runs through my provider to get the remote catalog so it's checking the signed url version of the catalog.
     
    unity_bill and Favo-Yang like this.
  7. CodeBombQuinn

    CodeBombQuinn

    Joined:
    Apr 17, 2018
    Posts:
    7
    @unity_bill This approach works above, but one concern here is if there are any updates to how Addressables BundleProviders work, i'll have to copy/paste this code back in with each update to be sure we're using the latest code. Which I assume is a problem for anyone writing custom providers for things.
     
  8. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    640
    That's not as bad an issue as you think. We have custom stuff in several packages, Addressables, TextMeshPro, LWRP, etc. If you are using custom stuff you should:
    • Move the package from Library/PackageCache/ to Packages/
    • Rename the package not to have a version suffix.
    • Commit it to your repo without your changes.
    • Commit your changes.
    To update the package:
    • Update your working directory back to the revision you committed the vanilla package in.
    • Delete the package.
    • Update using the package manager.
    • Move the package from Library/PackageCache/ to Packages/
    • Rename the package not to have a version suffix.
    • Commit.
    • Merge your changes in.
    We use Mercurial for this, where I have a branch for import of each package. It works a treat and you can easily inspect what changes are made to the packages by the team this way. If you have a custom provider, it's very easy to check the diff of the original one and make the same mods on yours.
     
    unity_bill likes this.
  9. Petruamber

    Petruamber

    Joined:
    Mar 13, 2019
    Posts:
    1
    We host our assets files on a Playfab CDN and to access them we must first call GetContentDownloadUrl API (a HTTP request to get a signed URL for accessing that file) and then make the GET request to that signed URL to actually get the file. But The URL expires in x time, so to make sure everything is still valid, we must call GetContentDownloadUrl API before every attempt to download a file.
    We managed to implement our own AssetBundleProvider and set it to the AssetBundleProviderType in our remote group asset, as @CodeBombQuinn suggested. The problem is we don't have the list of all the signed URLs and we must get each one before making each get resource request. Does anyone knows how inject another HTTP request before the one that gets the asset using a custom AssetBundleProvider? Or how to try another approach?
     
  10. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    640
    One way of doing that would be modifying AssetBundleProvider::BeginOperation() to detect whether you are going to need to sign the Url and chain the operations if so, setting the creation and sending of the AssetBundle web request to be send by the completion handler of your GetContentDownloadUrl() operation. You need to be careful with caching, you don't want to end up doing this even if the file is available locally.
     
  11. unity_bill

    unity_bill

    Unity Technologies

    Joined:
    Apr 11, 2017
    Posts:
    913
    The only thing on our roadmap to do to the AssetBundleProvider eventually is "add signed url support". So with any luck, the first change we throw at you that conflicts with your work will also make it unneeded.

    But as @AlkisFortuneFish has pointed out, it is a manageable problem.