We've started work on a new memory profiler

Oh snap. I actually googled recently looking for some kind of memory profiler. This will be great.

 

Is there going to be a snapshot diffing option ?

 

Always great to have some low level API's! Any details on that API?

 

Very exciting! I hope we see this become stable & il2cpp becoming active for more platforms so we can also use it for android etc.

 

Any hope of this making it back into 4.6?

A lot of mobile developers are still in 4.x, with pretty good reasons for staying away from 5.x for now.

And the current profiler gives some unexpected/inconsistent stats when trying to investigate memory use on iOS, especially when compared with the XCode memory use figures.

 

What are the mobile issues in 5.x preventing your migration? that would be the most important thing for Unity afaik.

 

I believe you can already use IL2CPP for Android in 5.2. At least in the betas you could, haven't yet switched to 5.2.

 

The loss of Beast, really.

Even if the new system was completely bug free (it's not - lightprobes have been broken for months - apparently fixed now, but the fix hasn't shipped yet), there's also the issue of having to relight and rebake a fairly large and lightmap-heavy game to make it work with the new system - and there's still a question of how lighting quality (and overall game performance) would compare with 4.x.

 

Hi Lucas, thanks for the write-up and making this tool available, even in an early state. You mentioned that it's available in Unity5.3a4. Does that mean it's also available in 5.3.0b1? If not, where can we find a4?

 

The new memory profile is revealing some interesting things.

We make use a of a few singletons in our game - singletons that manage some parts of our UI (NGUI). For example, a ScoreboardUI script/singleton that exists in some game scene. In initial menu scene, the game UI atlas has not yet been loaded. When I load the game scene, the game atlas gets loaded, the ScoreboardUI script sets the singleton reference to it self, and the ScoreboardUI script updates some UI Labels to show the scores for the game. When I leave the game scene to go back to the menu scene, everything is destroyed, but the Memory Profiler reveals that the game UI Atlas has not been unloaded because the ScoreboardUI's static Instance field is referencing it (Indirectly - the Scoreboard UI references some UILabels which in turn reference the atlas but the shortest path to root is ScoreboardUI). The strange thing is, none of these objects exists anymore. Everything is already destroyed. The gameobject with the ScoreboardUI script is destroyed and the entire UI was destroyed too.

Turns out I have to explicitly set the static ScoreboardUI.instance value to 'null' in the ScoreboardUI's OnDestroy method. When I do this, Unity releases the atlas correctly.

 

Thanks for the reply Lucas. We're trying to solve one of these issues right now. The memory profile just says one of our objects (an object that is DontDestroyOnLoad, not a singleton) is referencing the atlas material from the previous scene (And, in turn, that material is referencing a huge atlas). The strange thing is, this script has no material references (even though the memory profiler says it is referencing the material directly). I have no choice but to go through the script and just explicitly set anything that could be causing an issue to null when the scene changes.

At least with this new memory profile I can build the game and take a memory sample to check if I fixed the problem.

This is all making me realize just how, uhh, 'dangerous' DontDestroyOnLoad and static fields can be if you are not super careful.

 

@Lucas Meijer sure thing, tomorrow when I get in to work. I'll PM you the script but post here too in case others want to read your follow up as well.

 

I am very happy to see this - I can tell you, if we had this in 4.x (I know it's not coming to 4.x) I'd make great use out of it, as I'm currently trying to track down what's holding rogue textures in memory after their objects (and scenes) got destroyed. A very useful tool this new memory profiler will be!

 

Hi @Lucas Meijer could I get a link to this documentation please? I haven't been able to find anything myself. I've tried using MemorySnapshot.RequestNewSnapshot and comparing the instance ids of objects to detect the similarities but there are so many objects in the snapshot I suspect it's also reading objects in Editor prefabs that aren't loaded into the scene. That being a UnityEditor function I can't use it in a build either.

Thanks for your help

 

Yet another feature while a looong list of existing features do not work as advertised...? Thanks a lot....

 

A random selection of things that really do not work as expected or advertised:

http://forum.unity3d.com/threads/speedtree.364906/#post-2362216
http://forum.unity3d.com/threads/substance-support-not-working-as-expected.256431/#post-2349933
http://forum.unity3d.com/threads/shadow-fadeoff-approaching-far-clip-plane.356893/#post-2311360

I could go on. How much time do you have?
Now, please do FIX the things that we report before you add more new features that work so-so.

 

I'm pretty sure the unity developer didn't want a smattering of other bugs with other areas of the engine, but rather a bug report or detailed description of the problems encountered with the memory profiler.

 

I usually log bugs for every issue I encounter. Why would I not, and why do you seem to imply I do not? What kind of strange defence is that?
I will have you know that out of the multitude of bugs I report I seldom get feedback and nor have the bugs been fixed in a timely manner. The substance problem is a year old or more. How do you think that affects the willingness to log bugs in the long term. I still log them, but when they are seldom fixed, I am starting to wonder why bother...?

I respect that you do not want OT posts in this topic, and I apologise for it. But is extremely frustrating to get a feature announcement when there are so many things that need fixing before you head on to the next shiny feature. You might want to forward that to your project manager as I am sure I am not alone. There is even a thread in the beta forum urging you to slow down and fix what is broken.

 

Yeah, well. That goes to show how bug-fixing works, since it still did not work last time I tried. (Which was much later than the bug report date). I probably missed the closure/resolution message (if I even got one). But the sad part is that we actually ditched using substances since it was so unreliable in Unity.
So, a feature that you marketed very heavily a while back was dropped by a paying customer since it did and does not work well enough for production. Same goes with Speed Tree. I know a lot of devs who have reverted back to the old trees or other home-brew options. Because the advertised SpeedTree support is just a quick hack, and not usable in the present state.

How do you think that makes us feel? Paying for supposedly professional versions of the software and all too often learning that, oh no, the feature that looked as if it would elegantly solve a certain need does not, since it really does not work well enough or is buggy. Or both. Then you see new feature announcements while threads about problems and your bug reports go unanswered and not acted upon.

Think about it.

 

@Lucas Meijer: THANK YOU! Seriously, memory management is one of the things that often gets put to the side as prototypes get promoted to projects and production schedules ask for features.

And as someone targeting WebGL for a significant chunk of business I'm really looking forward to using this.

 

@Lucas Meijer: Thank you so much. Even in it's current janky state, this has already helped me fix several extremely obscure problems in our game.

Did you know that Mono 2.0's standard lib SortedList has a bug where it retains a reference to the last item in the list even after you remove everything from the list? I do now! (Check out RemoveAt line 462) Our code was completely correct by inspection, the ref chain was extremely obscure and the old memory profiler just reported "Referenced by some static. Not gonna tell ya which one lol." Finding that leak would have been impossible before.

 

Anything new on the subject? I can't see the memory profiler anywhere on the roadmap

 

Ey, such a great tool!

We are using it right now looking for a 100 MB memory leak we have in each menu->scene-> menu loop. The tool is very useful, but, is there any way to compare between snapshots? Because visually is very difficult to find any different.

Thank you

 

What's the current status of this?

I'm using it to profile a windows standalone build made with Unity 5.5.0p1, and the information I'm seeing doesn't appear to be accurate.

According to the snapshot, there's a 2.7MB texture (splash-image) that's always in memory. The profiler states that it's referenced by a an object called 'CharacterButton', but that object should never reference that particular image.

The gameobject that should reference this image (SplashPage) is created at the start of the game and then destroyed shortly after.

Am I reading this wrong? Or is the info simply not correct?

 

Is there any way to use this on a PS4/X1/PC project?

Building for windows store has it's own troubles converting and taking a sample on PS4 crashes the PS4 right away (even in our main menu). Anyone else have success on X1/PS4?

 

Update: We made a windows 10 store build and profiled using that. The memory tool helped us find a leak that we would've never found otherwise (leaking 100mb every scene load).
Thanks a lot for the tool, it's reaaly promising. Can't wait for il2cpp to be usable in normal standalone builds!

 

Hello,

Is this project stopped? I say that because I am using it and I have seen this video

In it, @Lucas-Meijer uses a very interesting feature, Reference Chain, that I can not find in the bitbucket project. Moreover, I have seen interesting features in branches, like spreadsheet, but not merged.

So I do not know if this project is abandoned.

I hope not, because it is a great tool, but it feels like not completed.

Regards

 

This is exactly the kind of tool we need. Too bad it only works for IL2CPP platforms. Is there anything similar for mono backend?

 

Hello,
I am using 5.5.0f3. I assume this tool should still work?

I cannot get any data out of it. I get a couple of asserts, a lot, maybe thousands of warnings and end with with an index out of range error.

Asserts:

Code (CSharp):

1. Assertion failed on expression: 'kSnapshotRuntimeInfoMagicBytes == magicBytes'

Sample Warnings:

Code (CSharp):

1. Exception parsing object header. Skipping. System.ArgumentException: Unexpected pointersize: 0
2.   at MemoryProfilerWindow.BytesAndOffset.ReadPointer () [0x0005d] in /Projects/Unity-Technologies-memoryprofiler-40911eb35d1d/Assets/Editor/BytesAndOffset.cs:20
3.   at MemoryProfilerWindow.Crawler.ParseObjectHeader (MemoryProfilerWindow.StartIndices startIndices, UnityEditor.MemoryProfiler.MemorySection[] heap, UInt64 originalHeapAddress, System.UInt64& typeInfoAddress, System.Int32& indexOfObject, System.Boolean& wasAlreadyCrawled, System.Collections.Generic.List`1 outManagedObjects) [0x00011] in /Projects/Unity-Technologies-memoryprofiler-40911eb35d1d/Assets/Editor/Crawler.cs:231
4.   at MemoryProfilerWindow.Crawler.CrawlPointer (UnityEditor.MemoryProfiler.PackedMemorySnapshot packedMemorySnapshot, MemoryProfilerWindow.StartIndices startIndices, UInt64 pointer, Int32 indexOfFrom, System.Collections.Generic.List`1 out_connections, System.Collections.Generic.List`1 out_managedObjects) [0x00037] in /Projects/Unity-Technologies-memoryprofiler-40911eb35d1d/Assets/Editor/Crawler.cs:176
5. UnityEngine.Debug:LogWarningFormat(String, Object[])
6. MemoryProfilerWindow.Crawler:CrawlPointer(PackedMemorySnapshot, StartIndices, UInt64, Int32, List`1, List`1) (at Assets/Editor/Crawler.cs:180)
7. MemoryProfilerWindow.Crawler:Crawl(PackedMemorySnapshot) (at Assets/Editor/Crawler.cs:31)
8. MemoryProfilerWindow.MemoryProfilerWindow:IncomingSnapshot(PackedMemorySnapshot) (at Assets/Editor/MemoryProfilerWindow.cs:165)
9. UnityEditor.MemoryProfiler.MemorySnapshot:DispatchSnapshot(PackedMemorySnapshot)

Code (CSharp):

1. Skipping field Empty on type System.String
2. UnityEngine.Debug:LogWarningFormat(String, Object[])
3. MemoryProfilerWindow.Crawler:CrawlRawObjectData(PackedMemorySnapshot, StartIndices, BytesAndOffset, TypeDescription, Boolean, Int32, List`1, List`1) (at Assets/Editor/Crawler.cs:153)
4. MemoryProfilerWindow.Crawler:Crawl(PackedMemorySnapshot) (at Assets/Editor/Crawler.cs:36)
5. MemoryProfilerWindow.MemoryProfilerWindow:IncomingSnapshot(PackedMemorySnapshot) (at Assets/Editor/MemoryProfilerWindow.cs:165)
6. UnityEditor.MemoryProfiler.MemorySnapshot:DispatchSnapshot(PackedMemorySnapshot)

Final Error:

Code (CSharp):

1. IndexOutOfRangeException: Array index is out of range.
2. MemoryProfilerWindow.CrawlDataUnpacker.Unpack (MemoryProfilerWindow.PackedCrawlerData packedCrawlerData) (at Assets/Editor/CrawledDataUnpacker.cs:35)
3. MemoryProfilerWindow.MemoryProfilerWindow.Unpack () (at Assets/Editor/MemoryProfilerWindow.cs:154)
4. MemoryProfilerWindow.MemoryProfilerWindow.IncomingSnapshot (UnityEditor.MemoryProfiler.PackedMemorySnapshot snapshot) (at Assets/Editor/MemoryProfilerWindow.cs:166)
5. UnityEditor.MemoryProfiler.MemorySnapshot.DispatchSnapshot (UnityEditor.MemoryProfiler.PackedMemorySnapshot snapshot) (at /Users/builduser/buildslave/unity/build/Editor/Mono/Profiler/MemorySnapshot.cs:24)

Is there something simple I am doing wrong? I'd cant package this project and send a bug. I will try it out on a smaller, sample/empty project for comparison.

Thanks!

 

Hey @Lucas-Meijer, @Tautvydas-Zilys, can we get a word from Unity in this thread? It seems people are having a few issues with the tool and it's unclear if Unity is still supporting it.

 

It's still working for us on PS4 (Unity 5.5.2)

We couldn't have solved our memory issues without it, I do hope Unity realizes this should be a built-in feature and get some higher priority.

 

Tried this with Unity 5.3.4p6 and 5.4.0f3, but I get an error when opening the bitbucket project: "The type or namespace name 'Profiling' does not exist in the namespace 'UnityEngine'." throughout PackedMemorySnapshotUtility.cs. What version am I supposed to be using for this to work? Or, how would I fix that error?

 

Hi,

I'm running Unity 5.5.1f1 and try to use MemoryProfiler but failed to see allocations like huge arrays of ints or classes. Looking through sources I've found that typeDescriptions field of PackedMemorySnapshot does not contain anything, it is empty. As far as I know the only way to find that the object is an array is to use TypeDescription.isArray property, but there is nothing in typeDescriptions.

Could it be the reason why I can not see int arrays in MemoryProfiler?

 

Finally had a chance to use this tool and it worked nicely (in Unity 5.5.2p4). Was quite easy to find assets that required optimizing.

One odd thing though: One object was listed as having a reference to a certain texture, but it (and its children in the hierarchy) only referenced fonts. Either there actually is a reference somehow or I'm just reading it wrong.

 

Downloaded 5.6.0f1 and faced the same issue. TypeDescriptions is empty. I remember that I was able to see arrays of any objects in new memory profiler before (in 5.3.6f1) and it was really helpful. But now I allocate an 25 MB int array and see nothing about it in memory profiler.

So, I suppose that now memory profiler shows to me just Assets and objects that use UnityEngine.Object as base type. That's really disapponting.

 

So, I've finally found what I did wrong. After installing new Unity I forgot to set IL2CPP backend. But after building on Android device MemoryProfiler failed to get a snapshot at all (I can see stats in regular Profiler from device). I used Unity 5.6.0f1 and Meizu MX3. Seems like there is still no Android support.

Is there a way to participate in New Memory Profiler intiative actively?

 

Hey,

I'm trying this on iPhone 6 with Unity 5.5.0f1 and everything connects as per the video. After I click Snapshot the profiler pauses a bit and then continues - but I don't see anything else displayed in the Profiler window i.e. nothing gets displayed - no grid no anything - even if I leave it for 20 minutes or so.

Does anyone have any ideas as to why this could be and what I may be able to do to fix it?

thanks

Nalin

 

The info the profiler gave me was so wacko I don't think you're missing out on much.

 

Why couldn't you provide a package for it? Tried to add it to my project, but it doesn't work if I put content to custom directory in Editor folder. Looks like it should be only in Assets folder... Very uncomfortable