Search Unity

Resolved TextAsset Addressable encoding changes once loaded

Discussion in 'Addressables' started by _watcher_, Mar 13, 2020.

  1. _watcher_

    _watcher_

    Joined:
    Nov 7, 2014
    Posts:
    261
    Hi!

    Let me explain my problem, because its the strangest thing:

    I have a TextAsset file marked as addressable. Its encoding is ANSI, its a binary file created by protobuf-net.

    After it is asynchronously loaded by Addressables:
    Code (CSharp):
    1. var op = worldDefault.LoadAssetAsync<TextAsset>();
    2. await op.Task;
    3. byte[] fileContents = op.Result.bytes;
    and when i then look at the extracted bytearray, i can clearly see, that the encoding has changed to UTF-8.

    My question is, how can i tell addressables to preserve the ANSI encoding in the .bytes extracted output?

    My research indicates, that the encoding change is performed when you convert a string to an array of bytes. Id assume this is done 'under the hood' through op.Result.text, but am not sure. I can't use op.Result.text either, because this is a binary file with special characters, neither do i see a way to set the encoding when asking for op.Result.text.

    Anyway, the bottom line is: I am getting an UTF-8 encoded output from ANSI encoded input, when using Addressables. The output needs to preserve its encoding and be ANSI-encoded.

    Ideas?
     
    Last edited: Mar 13, 2020
  2. Fatcat-Games

    Fatcat-Games

    Joined:
    Nov 9, 2016
    Posts:
    34
    Does this help at all?

    Code (CSharp):
    1.  
    2. TextAsset result = op.Result as TextAsset;
    3. byte[] fileContents = result.bytes;
    4. string fileText = result.text;
    5.  
     
  3. _watcher_

    _watcher_

    Joined:
    Nov 7, 2014
    Posts:
    261
    Same result, but thanks for response.
     
  4. _watcher_

    _watcher_

    Joined:
    Nov 7, 2014
    Posts:
    261
    I've been unable to reproduce with simple class structure (serialized into a file, referenced as addressable, loaded as TextAsset) - simple file structure with an int wrapped in a class works fine. What fails is my complex structure with nested classes with byte[] arrays, Lists, and other types. Is there a way to access the Addressables source code? I might need to look at how the TextAsset wraps the text file and how the encoding into bytes is handled inside the Addressables source code.

    Or use Resources and reference the text files directly, avoiding Addressables altogether (this works as expected).

    Ive been profiling the byte[] output, and what i get from FileStream->Read is exactly like what is on the drive, and what i get from Addressables's TextAsset (AssetReference) is the same bytes, with the exception of some of the bytes being exploded into 3 bytes instead (different value), which then of course, is impossible to deserialize. I'd assume changed encoding, because if i simply save that byte[] output from Addressables into file, i get a UTF-8 encoded text file (the file marked as addressable is ANSI encoded).
     
    Last edited: Mar 18, 2020
  5. _watcher_

    _watcher_

    Joined:
    Nov 7, 2014
    Posts:
    261
    Solved now. You need to make sure Unity treats the file (Addressable TextAsset) as binary. This is not as straightforward as it may seem however, some use-cases completely void it.

    Some hints:
    > "Resources' folder does a special treatment on imported files. It 'assumes' their encoding from extension. Beware changing the extension from outside of Unity!!!
    > AssetBundles are stored in the above folder, and Addressables is a wrapper around AssetBundles.
     
  6. Serge144

    Serge144

    Joined:
    Oct 2, 2018
    Posts:
    65
    Hey _watcher_ , how do you load a TextAsset (binary file) with Addressable Assets? I want it to load synchronously because I need it to show right away when the scene is started. If I load it asynchorously then there's a chance that it may take a few time to show an essential part of the level. Im also using a binary formatter to deserialize it to a c# class.

    thank you in advance.
     
  7. lyflike

    lyflike

    Joined:
    Dec 23, 2018
    Posts:
    12
    Same issue. I am trying to access .dat file from the s3 bucket by addressable.
    upload_2021-3-15_17-45-26.png
    Please help if anyone tried before.
     
  8. Serge144

    Serge144

    Joined:
    Oct 2, 2018
    Posts:
    65