AssetBundle missing asset data (and Unity reports that it built successfully)?

We just came across a strange error that we had never seen before, but is very concerning in its implications. We have a bundle which was built without the asset data included. The manifest that was returned had the correct CRC, but the bundle fails to load due to not being able to be decompressed. Fortunately, we performed the build on 2 separate machines and have the log files, bundles, and CRC values for comparison.
Here's the log file for the good bundle with the 2 surrounding bundles for context:

And here's the corresponding log file for the bad bundle:

The only difference in the 2 is the compressed size of the edisonstringlights bundle. When I inspect the 2 bundles in a binary viewer, it looks like there are 2 size values in the header info that are correct for each bundle, but the rest of the bundles match aside from the last 10 bytes of the bad bundle not matching the corresponding bytes of the good bundle. We write the CRC values into a manifest file, and the CRC is the same from each build. Which indicates the Unity generates the CRC values of bundles on the asset data in memory and not from the actual bundle file.
When I try to load the bundles from file, the good bundle loads fine, but the bad bundle fails with a cannot decompress error.
From all appearances, it simply looks like Unity did not write the asset data to the bad bundle but reported that everything was fine. Has anybody else seen this issue? I feel like we need to have a test program iterate through our bundles and make sure that each one can be loaded before we deploy now.

 

So, it turns out we had the same issue with another build earlier in the week with a different bundle, and this time it happened on both Windows and Android bundles. Unfortunately, this particular bundle only had 1 asset as well, so we still don't know if there may be bundles with several assets that are just missing a subset.
In any case, I'm adding a post-build step to our build script that will crawl our CDN and try to load/unload every bundle to make sure they are valid. Once we have this in place, we'll at least have some idea how often this is happening, and ideally will be able to get a repro case for a bug report.

 

I added a post-build step in our build script to load/unload each bundle, and we found that several bundles failed over the weekend. This happened on Mac and Windows build machines and the bundles were for Mac, Windows, iOS, and Android.

We have seen bundles with multiple assets fail now, so we know that it's an all-or-nothing issue. So far, all bad bundles contain textures, so we're starting to wonder if there's a memory issue and a texture is failing to load and/or compress, and Unity doesn't handle it properly. Since this happens during a single BuildPipeline.BuildAssetBundles() call, I don't know how to check memory usage during the build. Perhaps some external tool that can log memory usage and we can see if there is a spike or maximum near the time that the bad bundles are written.

At least we can make sure our builds are valid before deploying now, but we still don't know why this is happening. The only thing we can do at this point is rebuild and hope for the best. Whee!

I realized that I forgot to mention that we are using Unity 2018.2.15f1, in case that matters. We build our bundles for Windows/Android on a Windows 10 machine, and Mac/iOS on a Mac of some sort.

 

Well, I found the issue in the tracker: https://issuetracker.unity3d.com/is...ces-dot-resources-or-dot-ress-will-be-0-bytes
Looks like we have to workaround until we upgrade to 2019.2, whenever that will be. Le sigh.