Bug Pulling my hair out - Unable to upgrade to Unity 2020 [RESOLVED]

Hello all,

I'm trying to upgrade my project from unity 2019 to unity 2020. After upgrading, my project works great:
- In the editor
- On my mac
- On all x6 of my android test devices
- On all x5 android devices belonging to each of my family members

But when I publish the Unity 2020 version it to Google Play, it becomes a buggy mess for the majority of my users:


Many of the models are appearing white (simple Lambert surface shader, attached). All player models are invisible, or have messed up scale

Normally I'd have no trouble de-bugging and fixing this situation, but there are no errors thrown, and I can't get my hands on a device where it's bugged out. It works perfectly on every device I can lay my hands on. None of my players have issues with the Unity 2019 version. Just Unity 2020. I am using Addressables, Unity Ads, Blender files, and Unity IAP

I am completely stumped. Any suggestions?

[Edit]
This bug is affecting ALL devices - not just old ones. I'm getting reports from users with brand new 2022 devices.

[Edit2]
Could this bug be related to using multiple UV maps (4 different UV maps) on each object? Perhaps the Play Store is compressing/corrupting the app bundle in a weird way? But only in certain regions of the world...? Because when I download it from Google Play, there is no problem.

[Edit3]
Player Settings and Quality Settings:

 

Never saw anything like this myself, but assuming you're using source control - have you tried diff'ing your Project Settings file? Maybe some new Android-specific settings were added or their defaults changed?

Also worth maybe looking into which devices it's broken on - and comparing to the ones where it's fine - and checking their technical specs (are they using different hardware, software, etc.)

 

I have scanned through the project settings - nothing seems to have changed. I asked a player to send me his device specs, and they're pretty much the exact same as my test devices (CPU/GPU/OpenGL/etc). So confusing.

Deleting Library/GI cache/etc had no effect. I wonder if it's an Addressables caching issue?

 

Could be, though then I would expect it to be magenta instead of white, but still very much possible!

Have you asked a user to maybe completely uninstall the game and then reinstall the latest version? Presumably if they do that and it's still wrong that it can't be caching.

Also just making sure when you test it on your own devices you grab it off Google Play, right? Not push a local apk - just to be extra sure you're testing the same version as your users.

 

can you share google play link? curious to see how it looks.

 

that looks like the phone doesn't fully support ES3.1

 

I downloaded the Google Play version. Still seems to work perfectly on my devices... which is strange because some users having problems have the exact same device as me =(

The user is hesitant to re-install - they're a long time player who is extremely high level, and they don't want to lose their progress

https://play.google.com/store/apps/details?id=com.abstractsoft.hybridanimals

The public release is unity 2019. If you send me your email, I can add you to 'internal testing' so you can download the glitched out unity 2020 version. Anyone else who wants to test, send me your email too =)

Hmm. How could I test this? I tried launching the unity editor with the command "-force-gles31" etc, to no avail. The user has OpenGLES 3.0 on their device. Users with the exact same device as me are having issues, which is the most confusing part. Is my custom shader (attached) is causing problems? The shader multiplies 4 textures together, each with unique UVs. I added the shaders to 'always include shaders' list, but nothing seems to change. Even tried following this advice to get addressables to always include the shaders - no change.

 

This does seem weird... this "PaintAdvanced" shader works fine for me even on an ancient version of Unity. It looks sort of like a multi-source channel splatmap shader, if I read it correctly, nothing too crazy.

What happens if you clone the problematic shader(s) (change their names!), make fresh materials and then change the assets in question to use the clones?

My thinking is if something is borked with the GUIDs that connect the shader to the material, the source textures to the material, the material to the model, etc... cloning it and putting in a new test pattern object that uses the new copies might show that it just works.

 

I'll give that a shot. However, it seems there's more than just shader errors going on - if you look closely at the first two images, the main player object that is supposed to be in the center of the screen, is also invisible. The main player object doesn't use any custom shaders - just a default "standard" shader with a custom texture. It almost seems like something is going wrong with the Unity Engine code, causing a cascade of errors.

If you would like, I'd be happy to add your email to the Internal Testing list so you can try the 'buggy' Unity2020 version

[Edit] I suppose I should also mention - objects that use the PaintAdvanced shader also have multiple materials assigned.

[Edit2] At this point, I'm almost so desperate as to completely recreate the project from scratch in Unity 2020

 

what device we are talking about. some devices may have same name but different hardware

 

This also makes me think you might be using Blender3D models here, imported straight from .blend files, and the Blender import scripts from Unity changed between these two versions, which might break linkages in your prefabs.

But you say you are absolutely NOT seeing the problem yourself?

 

Here are the hardware specs of one user having the problem, who was generous to share:
- Device: Samsung Galaxy J4 Core (SM-J410F)
- Processor: Qualcomm Snapdragon 455
- CPU: Cortex-A53
- GPU: Qualcomm Adreno 308
- Vulkan support: Not found
- OpenGL: ES 3.0 V@269.0 AU@08.01.00.312.045 (GIT@lf99a9f7fh1f)
- OS: Android 8.1.0 (API 27)
- Architecture: armv7l
- Instruction Set: armeabi-v7a armeabi

As for the device with the same name that I was testing on (Moto G6 Play), I suspect a lot of the players having problem are using the "Brazil" edition of the phone, not the "USA" edition. Looks like it has an Octa-core CPU instead of a Quad-core CPU, among other differences.

Other blender files in the screenshot (the tree models, rock models, etc) look like they're working fine. I'm definitely not seeing the problem myself. Some devices I have tested it on:
- x2 Different moto G6 phones
- x1 Old Samsung tablet
- x2 Old Huawei phones
- x1 New Google pixel
- x1 New Samsung Galaxy
- x1 Old macbook air
- x1 New macbook air

 

Okay, if it was a Blender thing, it would break every use of that model.

The relevant change to the import script was that Camera and Light began to be imported (whereas before they were ignored). Importing Camera and Light changes the hierarchy of the imported FBX. If you had no Camera or Light in a Blender file, it would not change. If you did, then it would change and break 100% of everything for that model.

So... never mind, don't think it's Blender imports.

But you could also try fork the model and export it as an FBX and see if the problem affects that...

 

Not sure if the player settings or quality settings have anything to do with it?

 

I see that the device you have posted is ES3.0, not ES3.1

I suspect one of these two options:

The device graphic hardware doesn't support something included in the game, some shader instruction or something

unity implementation for ES3.0 may have some bugs and issues.

hard to tell what could be.

maybe just enable ES3.1 and cut off older devices, or keep using unity 2019.

think about the GPUs unit as a box of legos with different pieces included. Theoretically a hardware manufacture should conform to ES3.0 standards and include all the hardware modules required by this standard. But if the manufacture wants to cut cost maybe he took some older chips without full support and lied in the specs that their phone support ES3.0

Lets say ES3.0 require 10 modules to be complied to the standard. but the chip just had 9 of them. Well the phone manufacture could say, we get this chip for this phone batch, and we emulated the 10th missing module. But they may have done a poor job of doing this emulation or unity code conflicts with this hack

You should take a look at google information of world wide devices, in the end maybe it makes more sense to just drop ES3.0 support and be done. Instead of loosing weeks of dev time figuring stuff you most likely can't fix anyway either because of hardware issues or unity own bugs in the code

 

Thanks for the in-depth response. You've convinced me to drop support for OpenGL ES 3.0. I might just drop support for Android 8.1 and lower as well

It's a shame though... a large portion of my users are in countries where they use 5+ year old devices. Many of the most devout fans are going to lose access to the game. Also, I'll have to stomach a drop in installs for a few years, since it may take a while for my top countries to acquire up to date devices

 

if a large user base have phones that will be cut out better stick with 2019.

 

So... I forced it so that only the newest devices could run it (OpenGL ES3.2, android 31). And it made no difference - the Unity 2020 version is still visually a mess when downloaded from Google Play. But not when I download it from Google Play myself ... for some reason.

Could this have anything to do with the fact that my models use 4 different UV maps (One for details, one for pattern, and 2 for various symbols)? Also, my objects often have 2-5 materials applied.

Perhaps Google Play is compressing/corrupting the app bundle incorrectly, but only in certain regions of the world (ie. South America)? Perhaps Google's clouds servers are encountering an error due to a non-english character in the file paths on certain cloud servers?

 

I tried converting my entire app to URP - still completely unusable. I need to upgrade to Unity 2020 by November 1st since Google requires us to update the Billing Library, and target API 31. Any ideas anyone???

 

Looking back at your original post, here's a few more random thoughts:

- make sure you rebuild 100% of the Addressables; those may not be compatible anymore

- make sure the Blender export to FBX to import in Unity isn't changing on you. You say you have four UV maps... crosscheck to make sure all data in those UV maps are coming over properly.

- try experimentally refactoring one of your problem setups so that it uses two objects, placed at the same location, each with only two UV maps, if that is possible at all.

Are you localizing the paths to these resources?! Ideally never use anything but A-Z and 0-9 in all paths and URLs, or you are courting disaster with some silly little subsystem or webserver somewhere chonking on it.

 

@Kurt-Dekker Thanks for the suggestions!

I think I may have solved it. I checked unity "Cloud Diagnostics" (which I didn't know existed). Looks like 99% of users are encountering crashes/errors upon Resources.Load() being called. Weird.

Just spit-balling some theories as to why this is happening:
- Google Play is compressing the Resources folder in a weird way? Perhaps something is going wrong due to a non-ASCII character in the path of one of their South American / Russian cloud computer servers when processing the .aab files. Having worked with Google's plugins, I know that their code is plagued with issues like this.
- Addressables is conflicting with the Resources folder now, for some reason

Anyways...I will be switching away from Resources folder entirely, and I'll post an update to see if it worked.

 

Are you using Resources.Load<T>() correctly in ALL places?

Always use

Resources.Load<T>()

, never use

Resources.Load() as T

https://forum.unity.com/threads/resources-load-not-working.1137463/#post-7306567

And the same goes for

Resources.LoadAll<T>()

Generally you want to avoid writing code that is simply waiting to mysteriously fail simply by the future appearance of an identically-named file of a different type.

Just so you know, I have NEVER seen Resources.Load<T>() fail, not even once in a decade of using Unity. Every time I thought "Aha! I got it failing!" it turns out it was some other failure on my part.

Everybody pooh-poohs Resources.Load<T>() but I find that it has an extremely critical advantage over everything else they suggest I use:

Resources.Load<T>() actually works

 

Interesting! I was indeed using "Resources.Load() as T", even sometimes "(T)Resources.Load()", but I had no identically named assets (ie. Flame.prefab / Flame.png). Everything was unique (ie. Flame_prefab.prefab / Flame_image.png). Unless it was picking up the meta files accidentally or something (I am using PlasticSCM). Maybe Unity changed how Resource Assets type casting works behind the scenes, between Unity 2019 & 2020+

The super strange part is : it worked on all my test devices, and even when downloaded from Google Play on a fresh phone. That's why I believe it has to do with geo-location, or a previous version caching issue with addressables

 

This is also possible, and could be something as subtle as casing... for instance, if you have

UIStuff/Button.png

it might work to import it as "uistuff/button" and suddenly it stopped, or varies on platform to platform. I always strive to match the casing PERFECTLY, including all the paths to it.

Copy/paste when putting these in is your friend. Or better yet, collect all the things into a larger ScriptableObject containers, so you can load those containers and know every object explicitly beyond that.

 

AND... I figured it out


float.Parse()

In Unity 2019, "en-USA" was the default CultureInfo for float.Parse(), and it used to understand 'period' characters as decimals points. In Unity 2020+, float.Parse() apparently uses the local CultureInfo. So in Brazil etc, it was incorrectly expecting commas as the decimal points. Not sure why, but no errors are thrown

I spent over a month trying to fix this.... =(

 

exactly my unity experience, lost two years with this exact stuff, do some work rapidly then lose weeks figuring out some stupid bug. without wanting I've become a small expert in how unity should work but it doesn't because this or that bug. and then I never finished one game. i should had created an youtube channel teaching unity to others like all those yotuber experts that have no game in their portfolio.

good find btw.