Search Unity

Question Unity CCD Management API Content Patch Endpoint not working

Discussion in 'Unity Cloud Content Delivery' started by halvaren_nugadog, Mar 29, 2022.

  1. halvaren_nugadog

    halvaren_nugadog

    Joined:
    Feb 16, 2022
    Posts:
    31
    Hi! I've started working with Cloud Content Delivery service last week and I'm trying to manage posting and getting content of my buckets through the management API. My workflow is the next:

    - I post some assets (not Addressables, just individual assets) using a Unity Editor tool - so, I post some entries, one for each asset, and then update its content with the corresponding asset.
    - I get that assets in an external Blazor app.

    The assets in question are basically images and json files, so they don't give me much problems working with them. The problem I'm facing is with the PATCH entry content endpoint from the Management API. When I access it from my Unity tool it just sends me a 204 Status Code, which is ok, but nothing updates it in the bucket. It also happens the same using the Swagger UI of the API documentation and using Postman.

    I can't post code because of a NDA.

    I have another problem with the "get assets" step of the workflow, but I will explain it in other thread.
     
    Last edited: Mar 29, 2022
    halvaren_nugadog, Mar 29, 2022
    #1

  2. ryan_ngo

    ryan_ngo

    Unity Technologies

    Joined:
    Feb 2, 2021
    Posts:
    35
    Hi @halvaren_nugadog! So one thing I'm seeing off the bat is that you're using `MultipartFormData` while setting the content type to "application/octet-stream". We do utilize TUS which allows us to do resumable uploads. If you intend to use multipart/form-data, I believe you may need to change the content type. Another thing I noticed is that the `upload-offset` and `upload-length` headers are not assigned here. That could be another issue. When using the TUS protocol, you'll need a few headers along with the content-type "application/offest+octet-stream". You can view the documentation at tus.io.

    Could you also clarify what you mean by "nothing updates it in the bucket? From the dashboard, are you not able to see the updated file or grab the content?

    Please let us know if you have any other difficulties!
     
    ryan_ngo, Mar 29, 2022
    #2

  3. halvaren_nugadog

    halvaren_nugadog

    Joined:
    Feb 16, 2022
    Posts:
    31
    When I post a new entry and not patch its content, I can see the next in the dashboard: upload_2022-3-29_15-14-55.png
    With that "Added (Incomplete)" as its state.

    Then, when I try to patch the content, it remains with this state because nothing seems to be updated inside the entry.

    I will also make the changes you suggest to the code, but it is still confusing how the behaviour is the same if I patch the content through the API documentation Swagger UI and using Postman.

    Thank you for your help!
     
    halvaren_nugadog, Mar 29, 2022
    #3

  4. ryan_ngo

    ryan_ngo

    Unity Technologies

    Joined:
    Feb 2, 2021
    Posts:
    35
    Does the same thing happen if you try uploading through the dashboard?
     
    ryan_ngo, Mar 29, 2022
    #4

  5. halvaren_nugadog

    halvaren_nugadog

    Joined:
    Feb 16, 2022
    Posts:
    31
    No, when I upload manually through the dashboard, everything works correctly. But I need to do it using the API, considering my workflow.
     
    halvaren_nugadog, Mar 29, 2022
    #5

  6. halvaren_nugadog

    halvaren_nugadog

    Joined:
    Feb 16, 2022
    Posts:
    31
    Hi @ryan_ngo I made some progress with this issue but it's still unsolved: I changed the content type to multipart/form-data and added the headers you tell me to add, upload-length and upload-offset.

    Also, I've changed the way I was posting new entries: I was creating entries with a size of 1MB by default, assuming it wouldn't be a problem if the content didn't occupy all the capacity, but I considered changing that using the file length, and also, when I just want to update the content of an already existing entry, I make a PUT request to update that value.

    Finally, I realized that I was uploading the entries giving them a random content hash value, when this hash value is the one associated with the file I upload. So now, entries are posted or put with the proper hash value.

    So now the behaviour is the next:
    • If I post a new entry and upload content through API endpoints, the entry is created with the correct size and hash, but its state in dashboard is Added (incomplete), except if the file I tried to added to that entry is already in other entry (they share the same hash) - then, the state is Added and the file can be downloaded through the dashboard and, I assume, through a GetContent API endpoint.
    • If I put an existing entry and upload content through API endpoints, the entry is updated with the correct size and hash, but with content occurs just the same than the previous case.
    • If I put an existing entry but I don't upload any content, changing the entry size and content hash to a different one that doesn't correspond to any other content in the bucket, the result is still the same: Added (incomplete) state in the updated entry.
    • If I do the same as the previous case but the new content hash corresponds to one of the contents of the bucket, the state of the entry is Added and the content can be downloaded.
    What I assuming it's happening here could be two things:
    • Patch endpoint doesn't work correctly
    • I'm not calling patch endpoint correctly. Probably I didn't add the "upload" headers to the correct object. I add my current patch request code
    upload_2022-3-31_16-8-23.png

    The commented lines are different combinations of adding headers I tried - some of them just result on uploading nothing and some of them make that the request doesn't arrive to the endpoint and I don't even get a response from the endpoint.
     
    halvaren_nugadog, Mar 31, 2022
    #6

  7. ryan_ngo

    ryan_ngo

    Unity Technologies

    Joined:
    Feb 2, 2021
    Posts:
    35
    Hi @halvaren_nugadog, So I did some experimenting and did a quick mock implementation which allowed me to upload to our server. Instead of using multipart form content I opted to utilize the TUS protocol to enable resumable downloads. Please take a look at this and see if this could help you with your implementation.


    Code (CSharp):
    1. using System;
    2. using System.IO;
    3. using System.Net;
    4. using System.Net.Http;
    5. using System.Text;
    6. using System.Threading.Tasks;
    7.  
    8. namespace CcdUploader
    9. {
    10.     class MainClass
    11.     {
    12.         const string BUCKET_ID = "BUCKET ID HERE";
    13.         const int NUM_MB = 3;
    14.         static int CHUNK_SIZE = (int)Math.Ceiling(NUM_MB * 1024.0 * 1024.0);
    15.  
    16.  
    17.         public static async Task Main(string[] args)
    18.         {
    19.             FileStream fs = File.OpenRead("/PATH/TO/testfile.txt");
    20.             string entryId = "ENTRY ID HERE";
    21.             string path = "testfile.txt";
    22.  
    23.             Console.WriteLine("Uploading File");
    24.             await PatchEntryContent(fs, entryId, path);
    25.             Console.WriteLine("Finished Uploading file");
    26.             return;
    27.  
    28.         }
    29.  
    30.         private static async Task PatchEntryContent(FileStream fs, string entryId, string path)
    31.         {
    32.             var offset = 0;
    33.             CcdClient ccdClient = new CcdClient();
    34.             while (offset < fs.Length)
    35.             {
    36.                 fs.Seek(offset, SeekOrigin.Begin);
    37.                 int chunk = CHUNK_SIZE;
    38.                 if (offset + CHUNK_SIZE > fs.Length)
    39.                 {
    40.                     chunk = (int)fs.Length - offset;
    41.                 }
    42.                 byte[] buffer = new byte[chunk];
    43.                 var bytesRead = fs.Read(buffer, 0, chunk);
    44.                 var request = new HttpRequestMessage();
    45.                 request.Method = new HttpMethod("PATCH");
    46.  
    47.                 request.Headers.Add("Tus-Resumable", "1.0.0");
    48.                 request.Headers.Add("Upload-Offset", $"{offset}");
    49.                 request.Headers.Add("Upload-Length", $"{bytesRead}");
    50.  
    51.                 request.RequestUri = new Uri($"https://content-api.cloud.unity3d.com/api/v1/buckets/{BUCKET_ID}/entries/{entryId}/content/");
    52.  
    53.                 request.Content = new ByteArrayContent(buffer);
    54.                 request.Content.Headers.Add("Content-Type", "application/offset+octet-stream");
    55.  
    56.                 try
    57.                 {
    58.                     var response = await ccdClient.SendAsync(request);
    59.                     if (response.StatusCode == HttpStatusCode.NoContent)
    60.                     {
    61.                         offset += bytesRead;
    62.                     }
    63.                     else
    64.                     {
    65.                         throw new Exception("Failed to upload");
    66.                     }
    67.                 }
    68.                 catch (IOException ex)
    69.                 {
    70.                     throw ex;
    71.                 }
    72.  
    73.             }
    74.         }
    75.     }
    76.  
    77.     class CcdClient : HttpClient
    78.     {
    79.         const string API_KEY = "";
    80.         const string AUTH_HEADER = "Authorization";
    81.  
    82.         public CcdClient()
    83.         {
    84.             this.DefaultRequestHeaders.Add(AUTH_HEADER, $"Basic {encodeAPIKey()}");
    85.         }
    86.  
    87.         private string encodeAPIKey()
    88.         {
    89.             return Convert.ToBase64String(Encoding.UTF8.GetBytes($":{API_KEY}"));
    90.         }
    91.     }
    92. }
    93.  
    After some quick testing, I was able to do chunked uploads as well as smaller uploads. I hope this helps! Please let us know if you have any questions.
     
    ryan_ngo, Mar 31, 2022
    #7

  8. halvaren_nugadog

    halvaren_nugadog

    Joined:
    Feb 16, 2022
    Posts:
    31
    Thank you so much @ryan_ngo it's finally working! Sorry if I bother you in any moment.

    Just, as feedback, I don't know why the Patch endpoint doesn't work in the Swagger UI of the Management API docs - it receives a NoContent result but the content doesn't update correctly. Probably it has to do with the fact that I'm just patching the content and not putting the entry to addapt the content hash and size to the new content, but it is confusing that there's no any warning about that.

    Thank you so much again!
     
    halvaren_nugadog, Apr 1, 2022
    #8

  9. ryan_ngo

    ryan_ngo

    Unity Technologies

    Joined:
    Feb 2, 2021
    Posts:
    35
    No problem! Glad I could help! I'll be sure to look into the Swagger docs to see if I notice anything peculiar!
     
    ryan_ngo, Apr 1, 2022
    #9

  10. halvaren_nugadog

    halvaren_nugadog

    Joined:
    Feb 16, 2022
    Posts:
    31
    Hi @ryan_ngo I made some testing of the solution and I have a question about resumable uploading: is there a size limit for an upload chunk? I mean, if I try upload a "huge" file, will the API stop the upload somehow?

    This is the behaviour I found: If I tried to upload some "large" content (forcing to make a resumable upload) to an entry which has already this content (f.ex. I'm uploading an image of me to an entry where I already upload the same page, with the same content hash), the execution arrives to the line where the request is executed (line 58 in your code) with the first chunk of the file and the response never arrives. It doesn't throw any exception and the code after that line will never be executed.
    The operations I'm doing to reupload this content is PutEntry (which does nothing because the path, the content length and the content hash are the same) and PatchContent. If I delete the entry and, then, I make a PostEntry operation and PatchContent operation, the result is the same. The only way to make possible this upload is by modifying the content I want to upload, forcing the content hash to change.
    So, to sum up, when uploading a "huge" file totally new for the bucket, everything works perfectly, but when try to reupload the exact same file, the request stops and never ends.

    I use quote marks when talking about the size of the file because, testing this behaviour, I found that this size hasn't to be very large to suppose a problem. In fact, I found that the problematic sizes begin around the 800KB of file. The solution I found then it's uploading the files in chunks of 500KB, so I don't have any problem if there's a size limit for the uploads, but I find a little confusing that I have not read anything about that in the docs. And also, it is strange that nothing of this happens if it's the first time you upload that content, no matter what it is the size of it.

    Hope this helps as feedback!
     
    halvaren_nugadog, Apr 4, 2022
    #10

  11. StefanNovak96

    StefanNovak96

    Joined:
    May 4, 2022
    Posts:
    3
    Hey Ryan, thank you for this code snippet! It was really helpful to me.

    I'm having a strange problem where I'm getting a 406 Not Acceptable using this code, and my own code that I adapted from this. I can see in the Bucket that at first I have an Imcomplete Upload after creating the entry, then using that entryId I upload the file and I can see that it's uploaded and CCD has no problem with it. I can create a release, and view the file online. But the response is always a 406.

    Could you help me understand this problem?

    Edit: Using the Management API I've found the problem to be a 'size mismatch'. Must be something to do with my files, I'll be updating when I find the solution!

    Edit 2: Yes this is very strange for me. Using the code provided pretty much exactly, I continue to get 406s. I use the same file, just naming them different, and some will upload okay to the CCD Bucket (though still with a 406), and some will remain Incomplete for a few minutes, then OK a few minutes later. Very strange.

    Edit 3: Getting further problems with larger file uploads, 416 response, Requested Range Not Satisfiable. Why is there no documentation on this? What are the satisfiable ranges?

     
    Last edited: May 23, 2022
    StefanNovak96, May 19, 2022
    #11

  12. ryan_ngo

    ryan_ngo

    Unity Technologies

    Joined:
    Feb 2, 2021
    Posts:
    35
    Apologies for the delayed response! On 416, this indicates that when uploading the file the content range that you specified was probably out of the range that was expected with the entry. Could you elaborate on the size of this larger file. My guess would be that something happened with uploading the different chunks of the file.
     
    ryan_ngo, Jun 7, 2022
    #12