Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question download addressables from aws S3 bucket (not public)

Discussion in 'Addressables' started by elfasito, Jun 22, 2021.

  1. elfasito

    elfasito

    Joined:
    Jul 4, 2017
    Posts:
    51
    Hello people, I need to download files from s3 private bucket using addressables system.
    I see a lot of guides about S3 and addressables working, but all set the bucket as public, so Anybody can download, thats dont is what I need.
    I want to download a addressable scene from a bucket in S3 only for my cognito authenticated users.
    I already have configurated my aws backend (I think).

    for example from unity I can execute this function:
    Code (CSharp):
    1. public async void ExecuteLambda()
    2.     {
    3.         Debug.Log("ExecuteLambda actually S3 test");
    4.  
    5.         S3Client = new AmazonS3Client(_authenticationManager.GetCredentials(), AuthenticationManager.Region);
    6.  
    7.         ListObjectsRequest request = new ListObjectsRequest
    8.         {
    9.             BucketName = _s3BucketName,
    10.             MaxKeys = 10
    11.         };
    12.  
    13.         ListObjectsResponse response = await S3Client.ListObjectsAsync(request);
    14.         Debug.Log("Response code: " + response.HttpStatusCode);
    15.  
    16.         foreach (S3Object entry in response.S3Objects)
    17.         {
    18.             Debug.Log("key = " + entry.Key + " size = " + entry.Size);
    19.         }
    20.     }
    and that list the objects what I have in my private bucket. (only a authenticated user can see the bucket).

    but if I try to download a file/scene with addressables system I get:
    UnityWebRequest result : ProtocolError : HTTP/1.1 403 Forbidden
    ResponseCode : 403, Method : GET


    this is my function for download a bundle: (not working)
    Code (CSharp):
    1. public async void ExecuteLambda2()
    2.     {
    3.         Debug.Log("ExecuteLambda actually S3 test");
    4.  
    5.         S3Client = new AmazonS3Client(_authenticationManager.GetCredentials(), AuthenticationManager.Region);
    6.  
    7.         ListObjectsRequest request = new ListObjectsRequest
    8.         {
    9.             BucketName = _s3BucketName,
    10.             MaxKeys = 10
    11.         };
    12.  
    13.         ListObjectsResponse response = await S3Client.ListObjectsAsync(request);
    14.         Debug.Log("Response code: " + response.HttpStatusCode);
    15.  
    16.         var locations = await Addressables.LoadResourceLocationsAsync(SceneToLoad).Task;
    17.  
    18.         foreach (var location in locations)
    19.             await Addressables.InstantiateAsync(location).Task;
    20.     }
    can someone give me a hint about what im doing wrong?.

    PD: if I set the s3 bucket as public access, I can download the bundles, so I think I need to call addressables system using my aws credentials (no clue about how)
     
  2. LE-Peter

    LE-Peter

    Joined:
    Sep 15, 2020
    Posts:
    44
    Bumping this thread because we want to do the same thing.

    How do we use addressables and S3 buckets set to private?
     
  3. TeachingComJeff

    TeachingComJeff

    Joined:
    Mar 27, 2020
    Posts:
    1
    Any answers to this? We need the same issue resolved.
     
  4. TreyK-47

    TreyK-47

    Unity Technologies

    Joined:
    Oct 22, 2019
    Posts:
    1,816
    I'll flag with the team for some guidance!
     
    AlonLivne and KingKRoecks like this.
  5. nilsdr

    nilsdr

    Joined:
    Oct 24, 2017
    Posts:
    374
    Kasi2302 likes this.
  6. unity_krista

    unity_krista

    Unity Technologies

    Joined:
    Apr 28, 2020
    Posts:
    40
    @nilsdr is correct that used to be the only way to do what you're trying to do.

    However starting in Addressables 1.19.0, you can assign a function to the Addressables object's WebRequestOverride property to individually modify the UnityWebRequest from which is used to download files, such as an AssetBundle or catalog json file.

    You must assign the function before the relevant operation starts, otherwise the default UnityWebRequest is used.
    The ResourceManager calls your WebRequestOverride function before UnityWebRequest.SendWebRequest is called, passing the UnityWebRequest for the download to your function.

    I believe something like this should help you with AWS S3 authenticating requests:
    Code (CSharp):
    1. using UnityEngine;
    2. using UnityEngine.Networking;
    3. using UnityEngine.AddressableAssets;
    4.  
    5. public class WebRequestOverride : MonoBehaviour
    6. {
    7.     private string AWSauthorizationToken;
    8.  
    9.     //Register to override WebRequests Addressables creates
    10.     //The UnityWebRequests will default to the standard behaviour
    11.     private void Start()
    12.     {
    13.         Addressables.WebRequestOverride = EditWebRequestURL;
    14.     }
    15.  
    16.     //Override the url of the WebRequest
    17.     private void EditWebRequestURL(UnityWebRequest request)
    18.     {
    19.         request.SetRequestHeader("Authorization", AWSauthorizationToken);
    20.     }
    21. }
    For getting your authentication token to use, I think this page on AWS documentation will help
     
  7. GameSmith2020

    GameSmith2020

    Joined:
    Jul 14, 2018
    Posts:
    3
    Hey, I know this is a bit of a necro, but I wanted to add my solution to this, in case anyone else (Like I was) came here looking for a solution.
    While the solution posted by Krista was super helpful and my starting point, I had a lot of difficulty figuring out the Header syntax and authorization.. stuff (It's not my area of expertise)

    I found a much simpler method.

    Using Cognito to set up a role for your bucket (see any of the many simple tutorials for setting up roles), you can then use the SDK to get a presigned key for any item in your bucket, which I thought was useless, as you are only getting a URL to go off, but if you delete the general bucket url from the request.url (as from above) you get the item key, which you can then feed into aws sdk to get the presigned key, and overwrite the request.url to allow it to get whatever it needs to.

    Code (CSharp):
    1.  
    2. using Amazon.CognitoIdentity;
    3. using Amazon;
    4. using Amazon.S3;
    5. using Amazon.S3.Model;
    6.  
    7. public class WebRequestOverride : MonoBehaviour
    8. {
    9.  
    10.     string bucketName = "bucket";
    11.     string amazonUrlEnd = ".<your region endpoint>.amazonaws.com/";
    12.  
    13.     string completeUrl => "https://" + bucketName + amazonUrlEnd;
    14.  
    15.     //Register to override WebRequests Addressables creates
    16.     //The UnityWebRequests will default to the standard behaviour
    17.     private void Start()
    18.     {
    19.         Addressables.WebRequestOverride = EditWebRequestURL;
    20.  
    21.         // Initialize the Amazon Cognito credentials provider
    22.         CognitoAWSCredentials credentials = new CognitoAWSCredentials(
    23.             "your credentials here", // Identity pool ID
    24.             RegionEndpoint.yourRegion// Region
    25.         );
    26.  
    27.  
    28.         // need to use both the credentials and the region endpoint to connect
    29.         client = new AmazonS3Client(credentials, RegionEndpoint.yourRegion);
    30.     }
    31.  
    32.     string GetItemFromURL(string url)
    33.     {
    34.         return url.Substring(completeUrl.Length);
    35.     }
    36.  
    37.     private void EditWebRequestURL(UnityWebRequest request)
    38.     {
    39.         string itemKey = GetItemFromURL(request.url);
    40.  
    41.         GetPreSignedUrlRequest preSignedRequest = new GetPreSignedUrlRequest()
    42.         {
    43.             BucketName = bucketName,
    44.             Key = itemKey,
    45.             Expires = DateTime.UtcNow.AddHours(1)
    46.         };
    47.  
    48.         string url = client.GetPreSignedURL(preSignedRequest);
    49.  
    50.         request.url = url;
    51.     }
    52. }
    53.  
    don't forget to replace the region endpoints, bucket and identity information, but this worked for me, and I haven't seen this posted anywhere before (also beats trying to figure out header authorization shenanigans)

    Hope this helps :D
     
    Last edited: Feb 25, 2022
    Mk_dd, Blockehh, TomPorter1 and 3 others like this.
  8. pihels

    pihels

    Joined:
    May 13, 2018
    Posts:
    8
    Has anyone managed to figure out how to set the authorization header yet? I am also having difficulties figuring out how to compute the token and any help would be appreciated.
     
  9. nilsdr

    nilsdr

    Joined:
    Oct 24, 2017
    Posts:
    374
    Kasi2302 likes this.
  10. pihels

    pihels

    Joined:
    May 13, 2018
    Posts:
    8
    Thanks for the fast reply!

    Currently I am getting 403 Forbidden error (detailed message is "AWS authentication requires a valid Date or x-amz-date header"). Here is my test code which shows that the authorization header is set to:
    Authorization: AWS4-HMAC-SHA256 Credential=[AccessKey]/20220308/eu-north-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=80864ebee43afd7a2e97fe43b9af487181cc4fc53f12c520a3f9ea749a528010

    Does this look like it is in the correct format? Based on https://docs.aws.amazon.com/AmazonS...tion.html#ConstructingTheAuthenticationHeader I thought it would look something like
    AWS [AccessKey]:qgk2+6Sv9/oM7G3qLEjTH1a1l1g= .

    Code (CSharp):
    1.    
    2. private void EditWebRequestURL (UnityWebRequest request) {
    3.         var signer = new AWS4SignerForAuthorizationHeader {
    4.             EndpointUri = request.uri,
    5.             HttpMethod = "GET",
    6.             Service = "s3",
    7.             Region = "eu-north-1"
    8.         };
    9.  
    10.         var authorization = signer.ComputeSignature (headers,
    11.             "", // no query parameters
    12.             AWS4SignerBase.EMPTY_BODY_SHA256,
    13.             accessKey,
    14.             secretKey);
    15.         Debug.Log ("EditWebRequestURL Authorization token  " + authorization);
    16.         request.SetRequestHeader (AWS4SignerBase.X_Amz_Content_SHA256, AWS4SignerBase.EMPTY_BODY_SHA256);
    17.         request.SetRequestHeader ("content-type", "text/plain");
    18.         request.SetRequestHeader ("Authorization", authorization);
    19.     }
    20.  
     
    Kasi2302 likes this.
  11. nilsdr

    nilsdr

    Joined:
    Oct 24, 2017
    Posts:
    374

    Here's our implementation of signing for get requests. I'm pretty sure the content-type header has to be passed ad input to the signer aswel, thats the main difference.

    Code (CSharp):
    1.  
    2. var headers = new Dictionary<string, string>
    3.                         {
    4.                             {AWS4SignerBase.X_Amz_Content_SHA256, AWS4SignerBase.EMPTY_BODY_SHA256},
    5.                             {"content-type", "text/plain"}
    6.                         };
    7.  
    8.                         var signer = new AWS4SignerForAuthorizationHeader
    9.                         {
    10.                             EndpointUri = request.uri,
    11.                             HttpMethod = request.method,
    12.                             Service = "s3",
    13.                             Region = "eu-west-1"
    14.                         };
    15.  
    16.                         var authorization = signer.ComputeSignature(headers,
    17.                                                                     request.uri.Query,   // no query parameters
    18.                                                                     AWS4SignerBase.EMPTY_BODY_SHA256,
    19.                                                                     identity.m_AccessKey,
    20.                                                                     identity.m_Secret);
    21.  
    22.                         // place the computed signature into a formatted 'Authorization' header
    23.                         // and call S3
    24.                         headers.Add("Authorization", authorization);
    25.  
    26. foreach (KeyValuePair<string, string> header in headers)
    27.                         {
    28.                             if (header.Key != "Host")
    29.                             {
    30.                                 try
    31.                                 {
    32.                                     request.SetRequestHeader(header.Key, header.Value);
    33.                                 }
    34.                                 catch (Exception ex)
    35.                                 {
    36.                                     Debug.LogWarning(ex.Message);
    37.                                 }
    38.                             }
    39.                         }
    40.  
    41.  
     
    Kasi2302 likes this.
  12. pihels

    pihels

    Joined:
    May 13, 2018
    Posts:
    8
    Ah my bad, I was doing that also, but accidentally removed the line from the cleaned up code I pasted here.

    Anyway looks the problem in my script was that I was adding the headers manually in the end and didn't add the X-Amz-Date (this was added to the headers dictionary in ComputeSignature). After I added the headers from the dictionary like in your code it works. Thanks the help, you saved me a lot of headache :).
     
    Kasi2302 and nilsdr like this.
  13. Kasi2302

    Kasi2302

    Joined:
    Sep 30, 2020
    Posts:
    25
    Thank you very much, it works great. :)

    But now the next step, for me and maybe others.... o_O

    The access to S3 with authenticated user data, a cognito user.

    I have for authentication, the following data available:

    Code (CSharp):
    1. UserSessionCache userSessionCache = new UserSessionCache(
    2.             authFlowResponse.AuthenticationResult.IdToken,
    3.             authFlowResponse.AuthenticationResult.AccessToken,
    4.             authFlowResponse.AuthenticationResult.RefreshToken,
    5.             userid);
    Can someone please please help me with this problem. :(

    I need a code example to modify the header accordingly to load the addressable assets from the S3 bucket with the new authentication method.

    Do you still need the additional resources like AWS4SignerBase.cs, etc. ???
     
  14. epl-matt

    epl-matt

    Joined:
    Sep 9, 2021
    Posts:
    13
    A year later and neither method works. I have tried the auth headers and the presigned url and always get unable to load url. Anyone ever get this work?
     
  15. WavyRancheros

    WavyRancheros

    Joined:
    Feb 5, 2013
    Posts:
    176
    I want to bump this as well. I need this.

    Supporting Addressables on a private AWS S3 bucket should be standard. Why is Unity making this so hard for everyone?
     
  16. epl-matt

    epl-matt

    Joined:
    Sep 9, 2021
    Posts:
    13
    Because they hav their own system and want people to buy into is my guess. But that solution doesn't work for us.
     
  17. nilsdr

    nilsdr

    Joined:
    Oct 24, 2017
    Posts:
    374
    Why on earth would unity publish Amazon integrations. They allow for you to change requests before dispatching them, which is all you need to authenticate with any storage provider you want.
     
  18. Kiran_Jodhani

    Kiran_Jodhani

    Joined:
    Mar 8, 2022
    Posts:
    3
    Has anyone managed to make it working. I am not able to download content from private bucket
    I am using Version 1.19.19
    I tried with public bucket and it works well but making private and adding Addressables.WebRequestOverride is not working.

    For testing purpose I create presigned url manually and opening it in browser works this mean there is no problem with link but something in Adddressable.