[NEW RELEASE] AutoBuilder - a multi-platform builder, and editor-friendly asset bundle system

Website: https://reachablegames.com/unity-assets-autobuilder/
PDF Docs: https://reachablegames.com/AutoBuilder.pdf

Hi game developers!

After spending a substantial amount of time creating a useful build system (command line build-all, build-target, etc) and making all the hidden code settings mouse-clickable, I added a basic asset bundle building system. Then it grew. Now you can just click a couple buttons and have a player and asset bundles based on a folder structure.

It has:

• Automatic build versioning (manual or auto-increment or UTC time ticks)
• Asset Bundles produced with hashes, so bundles that don't change don't need to be downloaded again
• A custom bundle Manifest per version that your built player loads to know what bundles to download
• A fully working loader scene with a smooth progress bar, with everything async loading behind it
• Automatic bundle cache management, so old data is deleted and only new bundles d/l when needed
• Command-line build handler for individual platforms or all platforms
• Individual build settings for bundles and player options stored per platform, easy to check into version control
• Allows partial asset name matches so you can actually rearrange where files are in different bundles without having to change any of your code or data.

What's unique about it?
Bundle builds are great, of course, because the loading times are much faster than using single-threaded loading in the Editor. But building bundles can take a LONG time on a big project. So, I built a differential bundle system that lets you download (or build) a set of bundles by version, then click one button and AutoBuilder generates an asset bundle with all the new assets packaged in it, which it uses automatically when you run in the Editor. No coding required. It's seriously amazing for iterative workflow, because in a big project, the load times can be much longer than the time it takes to make a small override bundle.

So... I think it's pretty amazing. Obviously, or I wouldn't have bothered packaging it all up for you guys. I'm using it in my own projects, and expect there to be small changes and fixes as time goes on.

Any comments or questions? I'm happy to answer them.

 

Super excited to announce that AutoBuilder hit the asset store yesterday! Anybody looking to use out Asset Bundles, or wants a complete multi-platform command line driven (or menu driven) setup, with automatic caching on startup... check it out!

Even more useful is a way to work in the editor using asset bundles all the time, with a quick-to-create override bundle.

Hope you guys like it!

 

I just pushed a small fix to AutoBuilder that has corrected paths for the package so everything lives underneath the project folder instead of having a few things outside it.

Turns out that several people are buying multiple assets from us, and some of the folder names were the same under the company folder. So I cleaned up the structure a bit. While I was in there, I slightly improved the documentation and fixed a minor bug in the code.

I was just starting a new prototype the other day and pulled Split/Scenes and AutoBuilder into it. The coolest thing was, in about ten minutes, I had a root scene with several child layers with different stuff on each that I could use in-editor, then dragged all the scenes to the Bundles folder, configured the AutoBuilder profile, built the prototype. No kidding, in ten minutes, I had an executable that didn't have any direct linkage to the game (the root scene is loaded by name), asset bundles asynchronously downloaded automatically from a CDN, the root scene async loaded and when it launched, Split/Scenes automatically async loaded all the child scenes and started them. I didn't have to declare the scenes in the build other than the loading scene that comes with AutoBuilder.

It was pretty epic! As a framework for building asset-bundle-based games, this combo is definitely a huge win.

 

I've just come across this, it seems amazing and I am very interested. My game is massive, and it's all in one scene. Is it possible to use this to make asset bundles easily without having to tag all the different assets Ive used? Can I feed it my scene to do the work? I would really love to get my asset bundle workflow sorted once and for all and this seems to be the ticket.. Are there options to automatically limit the size of the bundles by splitting them up etc? My scene is 300meg and I believe the assets in use are about 100gig so it's a pretty challenging project to work with.. would love to know if you think this asset can help me out!

edit: PS. my assets are spread all over in MANY MANY messy dirs, will this be a problem? I mean it could take me until next xmas to manually specify them, which is why im looking for automation.

 

Hi!

Sounds like a pretty daunting task. Before promising you the moon, I would ask what you expect asset bundles will do for your massive project? Do you need incremental loading as you move across space, or the ability to change content after shipping without updating the whole thing?

The AutoBuilder asset makes organization really simple. Just specify the root folder and each subfolder becomes a bundle. That takes care of your organizational effort, for the most part. The issue will be if you have cyclical dependencies somewhere, it will fail (and tell you what the problem assets are). Assets can depend on things in other bundles, but one must load before the other, and if A depends on B, B loads first, then A. If B depends on something in A, that object will not load properly. So AutoBuilder tells you, at which point you have to move some files around (or factor out shared files into their own bundle).

What you do with bundles once they are created is up to you. So roaming and loading CAN be done, but that isn't something I can do for you. Loading is a single line of code though, so basically trivial to script if you can figure out how to organize bundles spatially. AutoBuilder does come with a loader scene that simply loads everything for you as an example.

The build process itself is pretty smart about not building unnecessarily, and uses a slightly different manifest file layout so that you can have different versions of your game hosted side by side without duplicating bundles that haven't changed. And you can run bundles entirely locally without hosting anything just to kick the tires on it and test your workflow. The fancy feature nobody else has is an override bundle that generates only the deltas from a specific build version into one bundle which let's you keep testing without having to do a full rebuild until you're ready.

If you have specific questions, I can answer them, though.

Best,
JH

 

Hey, I may have over estimated how much memory will be in use. I load the scene entirely into the memory and it runs just fine in the editor, so I think it uses about 16 gig of RAM on high quality (I think my build would hopefully be less than 20 gig). I am targeting PS5/High End PC ultimately so I don't mind about the RAM, ie I dont need to load stuff in and out of memory (yet).. I turn on and off parts of the scene as needed but everything is pretyt much in memory as soon as the scene finishes loading.

The main reason I need asset bundles (in order of importance) is:

A) I have exceeded the hardcoded 4 gig texture limit allowed in a scene. I don't notice in editor but if I build out my game all the graphics are corrupted. This is a known unity issue that can apparently be completely avoided by using asset bundles.

B) Updating the game without having to get it all again, as you said. There will no doubt be many changes to all kinds of assets, I guess it would be nice to keep asset bundles lower than say 1 to 5 gig so new ones can be grabbed relatively easily.

C) Speed, I am hoping the time for the game to play may speed up, as right now it takes about 5 minutes to start after I tap play.

I guess my main question is:

- how much manual work will be needed on this *massive* and very messy project. The root folder I would need to specify would simply be assets/ . All my content is illogically spread across many subdirs etc over the last 8 years. There is A LOT (200gig) in there that I don't need in my build. I need some way of using the scene file to work out which things to include in my bundles automatically, it would be impossible by hand I would say (since theres no way to just tag objects in the hierarchy).

 

Well, I have good news and bad news. The good is that bundles are fast to load, cache well, and generally solve the post launch updating problem really well, after all the work that I went through to make AutoBuilder do a good job caching, I mean.

The bad news...code does not go into bundles, ever, so you would have to make a proper patch for that. Also, AutoBuilder does a lot of things, but it does not strip assets for you. Bundles are literally everything you tell them to contain, which is up to you to pare down.

Even as easy as AutoBuilder is to generate bundles, it is definitely a manual process you have to do. In the case of explicit references, dragging whole folders into bundle folders, unity will not break because it finds assets via guid, as I'm sure you know. That's why I built AutoBuilder such that organizing files on disk IS the bundle structure.

But if you're looking for something to remove unneeded assets, look into some of the build report tools out there that can tell you what is not referenced anywhere, based on what doesn't go into the build. I think one was called AssetH Pro? I haven't used it, but it sounds like something you might look into.

Once you get down to a definitive set of assets you actually want to make into asset bundles, that's when you want AutoBuilder.

Hope that helps!

 

Hi JH,

I am starting considering if I should use Addressables or AssetBundles. At this point I have too little knowledge on that to decide, but your AutoBuilder seems to be close to what I need, so I'd like to learn more on that. I have a similar problem as radiantboy, but I also need to be able to build quickly different versions of the same app, but with different data and object configurations (positions, textures etc.). I will have many app templates and will build those apps after providing small customizations based on customer requirements. My current approach is that I have many scenes which use reusable prefabs. Then, once I get customer's data and requirements, I will change the app and build it.

Initially, I will need a solution that will speed up loading the first scene and will load the main one later. For example, the first scene would load with some low resolution textures and then either a bundle would load from a local drive (ready customer app) or from internet (demo app with demo data).

Without asset bundles Unity itself recognizes what is used by the scenes and includes only needed assets in the build. Correct me if I am wrong, but your AutoBuilder will not recognize automatically what is actually used, will it? It will just create bundles based on folders even if some assets are not used by the current build. If you agree, then it will cause similar problem as Resources folder that I stopped using for the same reason - the build always includes assets in that folder. If you agree, would it be difficult to develop this kind of auto detection?

Finally, how would you compare your AutoBuilder system with Addressables? Does it have some advantages? Does it change anything to the diagram on Complexity/Development time included in the below document?
https://docs.google.com/document/d/1hPLNLdrF0qAvjEJTpKf-cuO_d4FCV0H2cqBeP1Zo6mA/

Cheers,

Daniel

 

Hi Daniel,

On vacation right now so I don't have a laptop in front of me to answer more thoroughly, but I will try to hit the high points quickly.

What it sounds like you want is something to generate bundles based on a set of root assets and their dependencies, which is not what AutoBuilder does. Unity builds do that naturally, as long as you add a scene to the build and out the prefabs into that scene. I recommend you could easily get what you want by having the last scene in your build have all your resources in it and just never launch that scene. Then your built client is fully self contained and has all the data and only the data you need.

Could I write something to flag unused assets? Yeah. It might be a bit more of a manual process to decide if you wanted to delete them or move them away from the bundle folders or something like that. I don't know at this point how it would work. But walking dependency trees is easy. The problem is aasets can be loaded by full pathname, (or in AutoBuilder by partial path), and there is no way I can determine if that functionality is going to be used, so I have to assume everything in a folder is necessary, unless you tell me otherwise by calling some menu operation indicating how to remove unrelated assets.

As for addressables, I can't really say. It seems like it is similar to how I have implemented partial path loading, where the bundles you mount can be reorganized without changing the code that loads assets. But I haven't spent enough time looking at the way addressables load or are directed to cdns. Someone somewhere has to create a manifest of where files live and where to fetch those addressables, so the process is going to be similar to building bundles. Maybe they are similar to if you were to make each asset and it's dependencies a single bundle? Maybe it's smart enough to fold each subdependency together if there is a texture in multiple places? I can't say.

Will AutoBuilder do what you want out of the box? Some of the things you want, yes. Some no. I hope this helps.

JH

 

Hello fellow devs!

I'm excited to announce that a new version of AutoBuilder hit the store today.

What's new?

• A couple of minor issues fixed with the override bundle generator during tight iterations.
• Added support for all the new build features Unity added up to 2019.3, and the new platforms too.
• Added a toggle to enable/disable logging so you can see exactly what is going into each bundle.
• Added 3 flavors of simple filename based filtering: Exact Path, Contains, and EndsWith. You can do just about anything (short of regular expressions) with this. Filtered files also show up in red in the logs, so you can catch accidentally filtered files easily.

I hope you all enjoy the update and look forward to suggestions, bug reports, and positive reviews!

Best regards,
JH

 

Quick question, does AutoBuilder work with the MRTK package installed. Developing industrial app for HoloLens 2. I don't want to spend forever working on this project. Thanks.

 

I don't see why not? All the C# code for AutoBuilder is completely namespaced, so there won't be any conflicts with other code. You can move the folder and it's tolerant of that.

Have you bought AutoBuilder and are running into a problem with it? If so, shoot me an email at support@reachablegames.com with details.

 

My company has multiple teams and partners along with external suppliers all of which provide somewhat convoluted materials in information. It goes through a scrubbing process and into databases. This app will be delivering a fairly diverse visualizations process for the IoT data. I need to load scenes (hopefully), and at a minimum objects. From the description your tool should help me accomplish this. I will be purchasing it today. If you are interested, I will let you know how it goes. Thanks for the reply and the honest assessment. BTW my handle is my linkedin name.

 

In our government services, how would we procure and license AutoBuilder to a government customer on the PO?

 

Yes, indeed, AutoBuilder will allow you to load a scene from a remote-loaded asset bundle and anything stored inside it. Technically, Unity performs this feat, AutoBuilder just makes it really easy to generate the bundles. One thing to note, the executable must already have all the C# code compiled into it before trying to load any bundle that has code bindings, otherwise it won't work properly. That's a limitation of how asset bundles work, not something I can fix.

Unfortunately, I don't know anything about how purchase orders are fulfilled by the Unity Asset Store. A quick search didn't turn anything up. If you don't find what you need after contacting the store (support@unity3d.com), reach out to me directly and we can discuss the best way forward for you.

 

Purchased, downloaded/imported. Followed initial setup instructions. With Unity 2019.4.10f1 I had to build the editor configuration for AutoBuilder to work.

 

I will test this for a while and see if it is something we will be able to deliver to our government customers. Some as you might imagine are not able to establish Internet connections or to purchase from an AssetStore. I really hope this will work for us.

 

Can you explain further, C# scripts must all exist in the app at compile time and be delivered as part of the packaged app? That seems to limit the utility fairly significantly.

 

Are you by chance a consultant with available hours?

 

For anyone using this tool with current Unity releases, 2019.4.x, if you have any problem, before doing any troubleshooting follow the instructions to use Unity main menu: Tools/ReachableGames/AutoBuilder/Build [Current] for Editor. So far this has worked for me to get things working as expected. I am at Unity 2019.4.10f1 LTS.

 

EDIT: Issue does NOT occur with 2020.1.6f1


When I import the asset I get many

InvalidOperationException: Sequence contains no matching element

System.Linq.Enumerable.First[TSource] (System.Collections.Generic.IEnumerable`1[T] source, System.Func`2[T,TResult] predicate) (at <351e49e2a5bf4fd6beabb458ce2255f3>:0)

ReachableGames.AutoBuilder.WebViewHook..cctor () (at Assets/ReachableGames/AutoBuilder/Editor/WebViewHook/WebViewHook.cs:38)

Rethrow as TypeInitializationException: The type initializer for 'ReachableGames.AutoBuilder.WebViewHook' threw an exception.

ReachableGames.AutoBuilder.AssetsPopup.OnGUI () (at Assets/ReachableGames/AutoBuilder/Editor/ABAssetsPopup.cs:116)

UnityEditor.HostView.OldOnGUI () (at <b17f35b08b864a3ca09a7032b437596e>:0)

UnityEngine.UIElements.IMGUIContainer.DoOnGUI (UnityEngine.Event evt, UnityEngine.Matrix4x4 parentTransform, UnityEngine.Rect clippingRect, System.Boolean isComputingLayout, UnityEngine.Rect layoutSize, System.Action onGUIHandler, System.Boolean canAffectFocus) (at <cdec8475f3524a2780f0322b5e8bd8da>:0)

UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&)

When I press on an build [current] for editor I get (besides other stuff):

[win32] Asset bundle build failed for C:/Users/Klaus/privat/myProjects/Unity/AutoBilder Test/Assets/ReachableGames/AutoBuilder/Bundles/

UnityEngine.Debug:LogError(Object)

ReachableGames.AutoBuilder.ABBundleBuilder:DoBuildBundles(String, String, String, BuildAssetBundleOptions, BuildTarget, String, Boolean, String[], String[], String[]) (at Assets/ReachableGames/AutoBuilder/Editor/ABBundleBuilder.cs:75)

ReachableGames.AutoBuilder.ABMenuItems:DoBuildCurrentPlatformForEditor() (at Assets/ReachableGames/AutoBuilder/Editor/ABMenuItems.cs:62)

I am totally new to your asset, this is simply what I get from a clean new project in 2020.1.01f. I am not even sure this is the correct forum...

 

Ah, the WebViewHook stuff should be completely disabled on anything Unity 2019.4+. They removed some of the hidden classes that I was using, so I wrapped all that in #if to prevent them being used. I'll investigate and see if that somehow isn't compiling out properly. It may have been their versioning #if's weren't setup properly in 2020.1.01f... not certain about that.

 

I should have mentioned this already--Version 1.7 is out and has a big new feature!

Now, there's a checkbox on the configuration for each platform called Self-Contained. When you turn this on, it allows that platform's build to happen in such a way that the asset bundles are built before (rather than after) the player, and are moved into the /StreamingAssets/ folder. That build of the player is instructed to read all its bundles from within its own folders, without talking to the internet. This should help folks who need to release a fully self-contained app, and just want to use asset bundles for organizing data or coming up with a custom streaming system.

I also pulled out the WebHook code, since it has outlived its usefulness. A significant number of console logs have been improved to help you understand what is going wrong during the initial setup, too, which is where most people have trouble. It should be a lot easier to understand and get working!

Happy building!

 

Hallo Jason, many thanks for the previous support by email!

(1) When I build for Android I get the error "No ABBuild object found in build." What is a build info and how do I add it to a bundle?

(2) What is the best practice for loading only some bundles? I want to use Autobuild to download parts of the game once they are purchased.

Many thanks!

 

Hi Klaus!

That's strange. ABBuild is automatically generated during the build process. If you put a breakpoint at ABAutoBuilder.cs (line 128), that's where it gets created during the build. If that doesn't get you sorted, shoot me an email and we can work out what's happening.

As for loading some number of bundles, it's not how AutoBuilder works. To make it easy for most people to get going, it builds all bundles at build-time, and at run-time it loads all the bundles it finds in the manifest. I would recommend getting that working first. Getting separate asset bundle sets out of AutoBuilder isn't impossible, but it will require adding some method for deciding what bundle goes in what build, and being able to kick off different builds (probably a prefix on the bundle folder, ie "main-*" being for the game build, "pack1-*" for your first bundle pack, etc.) It doesn't matter what the bundle folders are called, in the end anyway.

So, making some changes to the build process to produce wholly different sets of builds, plus changing the runtime to know to download the extra packs at the right time is relatively easy once you're able to host the different asset packs... just modify the loading scene to accept the pack's manifest, which lists the bundles to download.

It's a solid day of coding, if you know your way around C#, but AutoBuilder gives you all the pieces. Some assembly required.

JH

 

Hello JH,

First off, let me say your asset looks incredibly useful and well made. Congrats!

This may be an edge-case or not entirely what the extension was designed for, but I would like to ask if your asset would be useful to solve my problem.

I have a long running (4+ years) large (1 gb and growing) mobile game that uses the legacy APK + OBB delivery system for Android. We are looking for a relatively painless way to shift from this to the new Play Asset Delivery method. However, this first requires us to make Asset Bundles, which will be almost impossible time and resource-wise for us to do using Unity's manual method.

We aren't really looking to change the way we deliver on iOS, this is primarily just to get us away from all the permission problems and bugs related to the apk + OBB method.

Can you let me know if your asset would be useful in this specific case?

Regards,

Josh

 

Hi Josh,

To be honest, I had never heard of this new system until you mentioned it. For Android, though, it looks like a big win to let the OS handle the downloads even without requiring the user to open the app (pretty impressive!). From what it appears, Google has provided a way in Unity to take a single asset bundle and wrap it up in a 'Play Asset' and upload it to their CDN. This can be automated, which would seem like the logical thing to do in your case, seeing as how you'd probably want to have the user download all the bundles automatically, and have them generated automatically at build time and uploaded to Google.

Here's the Unity-specific details as it pertains to Android: https://developer.android.com/guide/app-bundle/asset-delivery/build-unity

As for AutoBuilder, yes, you could trivially use AutoBuilder to generate your .apk and all the asset bundles that would (normally) be uploaded to a CDN of your choosing. In your case, you'd want to take something like the block of code mentioned in the link above, and wedge it into the build process, or slightly afterwards in a build script of some sort, and convert each asset bundle to an install-time Android Play Asset. Since they would automatically be installed, you would simply skip using our bundle downloading scene and go about your business. Simple.

Hope that helps!
JH

 

Hey JH,

Thanks for the answer. Yes using Google's CDN will save us a lot of headaches in the end.

The absolute best situation for us would be to be able to take our current build format apk + OBB and instead keep the apk the same and take what was the OBB and just make that whole chunk one asset bundle. Is this possible with your asset? Or maybe that is not what it was designed for.

Sorry to be thick and take up so much of your time with this.

Regards,
Josh

 

To be completely honest, I have no idea. I wish I could tell you AutoBuilder will solve all the issues you face, but all I can tell you is what it was written to do. It builds executables and asset bundles and knows how to load them before progressing. If you find a use beyond that, that's great. I expect you can do anything with AutoBuilder that you can do with any other build system, since it supports all the available options, but I have not personally built an obb. So I can't attest to it. There's a lot of options I haven't tried every possible combination.

JH

 

Understood JH,

Thank you for your answer. I will give the Documentation another through read and make try to intuit if what we want to accomplish is possible with your asset.

I think there is a relatively large market for assets that will simplify the asset bundle creation process specifically to fit in with the new google play system (which will be a requirement for new games going forward). On the forums there are many threads about the trials and tribulations of the dreaded APK + OBB system with people begging for solutions, and Unity doesn't seem super motivated to provide a good tool built in to the editor yet, so there may be a window of opportunity there. Especially if it is already possible with your tool and you just have to rewrite a few lines in the description/change a few tags.

Either way, I with you luck with your plugin and your future development.

 

Interesting. I have been getting a few questions about OBB's lately. I didn't realize that was the new requirement from Google. If you do try out the asset and need help getting that to work, let me know and I can make some adjustments, but my guess is building bundles of any sort, through any method, is step one. AutoBuilder does that part very well, at least

Thanks,
JH

 

Hey, just a heads up, I tried to find a support e-mail on the website but I didn't see one so I apologize for posting this here. I updated to the latest version of Unity (2020.2.1f1) and latest version of AutoBuilder (1.7) and I am getting a number of these errors:

Assets\Externals\ReachableGames\AutoBuilder\Runtime\ABLoadingStateManifest.cs(101,28): warning CS0618: 'UnityWebRequest.isHttpError' is obsolete: 'UnityWebRequest.isHttpError is deprecated. Use (UnityWebRequest.result == UnityWebRequest.Result.ProtocolError) instead.'

 

Oh, thanks for the note. Sounds like I need to make a couple of small fixes to keep the compiler happy. Technically, the code should still work okay for now (it's a warning, not an error), but I'll get an update on the asset store as soon as I can to address it.

For reference, support@reachablegames.com is our support email address, and you're right--it's all over the Asset Store pages and our documentation, but I neglected to put it on our website. I'll fix that too!

Thanks for the reports. And happy new year!

 

@elitegoliath13 Both the website and AutoBuilder package (and my other 5 assets) have been updated and submitted to the store. It often will take a day or two for them to sort through all the package submissions, but you should get an alert when they go live. Then you can update and the warning will be gone!

Thanks again.