Search Unity

  1. Unity Asset Manager is now available in public beta. Try it out now and join the conversation here in the forums.
    Dismiss Notice
  2. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  3. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Feature Request [Memory Profiler] Export to csv

Discussion in 'Profiler Previews' started by NuclearC00kie, Jul 14, 2020.

  1. NuclearC00kie

    NuclearC00kie

    Joined:
    Apr 11, 2019
    Posts:
    32
    Having an export to CSV option available for table view would be a HUGE lifesaver.
    We could use the power of spreadsheet applications to really leverage our analysis.

    Thank you,
    Pieter
     
  2. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,424
    Hi,

    and thank you for the Feedback, a csv export. As mentioned in your other thread, we're preparing to completely overhaul the table views and underlying database, which are both currently in the unfortunate proto-duction state to be way too involved to maintain, extend and improve upon. And yes, that goes so far as to making it tricky to even add a csv export that would reliable be printing out all the right info.

    We are aware that there are a multitude of reasons why one might want to export this data to a csv and have had several requests of that nature already. We'll try to provide this as soon as it is reasonably possible to slot it into the production. However, we're also always very interested what prompts these requests because it tends to hint at something the Memory Profiler package is currently not delivering on, but could, any maybe even reasonably should provide out of the box (or as something that can be easily extended upon by its users in some form or another). And yes, that includes CI tooling.

    So, could you maybe describe what kind of analysis you're expecting to be running over this and/or what kind of Problems you are looking to solve with this, so that we can maybe even deliver that out of the box to you at some point? :)
     
  3. NuclearC00kie

    NuclearC00kie

    Joined:
    Apr 11, 2019
    Posts:
    32
    Hi @MartinTilo,

    The usecases I have in mind are mostly:

    * Removing data that I'm currently not interested in (removing entire columns or rows)
    * Quick sorting / grouping / counting / searching data
    * Plotting data
    * Pivot tables for data analysis
    * ...

    Some of these points could be implemented in the memory profiler, but I don't think the memory profiler is ever going to (or even should) beat software that is specialized in manipulating a grid of data. Excel is going to be faster to work with and it's 1 less tool to learn if you want somebody else to analyze your data in a straightforward way.

    Thank you!
     
  4. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,424
    Thanks for the reply and going a bit more into the details here. This is good info and I agree that maybe the Memory Profiler doesn't need to fully replace Excel or Google Sheets for all the range of data investigation one might be able to come up with. We're instead trying to cover all specific questions or problems related to memory that any user might come across and then make sure that the UI can provide some answers to or insights into that.
    Quite honestly, we just need to trim some columns as well that provide marginal or no value. We'll be looking into reducing this to a more concise form with more details maybe as optional columns.

    I'd be more interested in what kind of rows you might find that you want to filter out. Is there any specific problem you are trying to tackle that doesn't require them or where they just add noise?
    Is it what you described in the other thread with this:

    Once everything is just NativeArrays, performance should become less of an issue but is there some other aspect of quickness there? Is it more about the workflow of how you have to add these kind of filters in the UI, or how you have to re-add them?

    I guess this can mean a number of things but is there any workflow/ problem investigation for which you could foresee certain graphs to be useful?

    I can understand that some of this might be "I'll know it when I see it" and a bit more exploration and experimenting needed to know what you'd want out of this. I guess once we do have csv support I'll also be asking for anyone willing to share what they did with the data to send it our way as feedback or suggestions. Still would be good to have any obvious shortcomings of the current solution mapped out before we rework it :)
     
  5. NuclearC00kie

    NuclearC00kie

    Joined:
    Apr 11, 2019
    Posts:
    32
    Indeed it's partially filtering rows based on what I'm not interested in but the filters already do an OK job at that (except what was explained in the thread you previously quoted from), but it really depends on the usecase.

    The last few weeks I'm trying to understand which objects have been changed between 2 sessions (old build vs new build).
    In that case, IMO the best way to deduce which objects have changed, is by looking at the name of the objects and their size. (sort size high to low, compare names between old and new, figure out which assets are gone or new.)
    That means I don't really need the Name, Index, Value, Address, Unique String columns.
    But I might need them in other research areas maybe?

    Quick here refers to the performance of the tool, not the workflow. I'm personally quite ok with the way the filters are implemented in terms of UI, but in general the tool often feels sluggish when manipulating data or filters. Glad to hear that's going to be improved with NativeArrays :)

    As an example:
    if we can export to CSV, the captures could be taken by an automated build system which then generates small reports, this way you can track memory usage for specific places in game over the development of your game, and show that in a line graph.

    Since you already show a memory map view, I think most usages for plotting would come from plotting values from multiple snapshots easily.

    Having CSV support in general would allow the user a bit more freedom in how they work with the data and if something in the memory profiler is a bit bugged, or not as straightforward as we'd like, this would allow the user to work in a familiar tool and avoids having to wait for an update to the package, which can take weeks or months depending on the urgency of the request / bug.

    Thank you for taking the time to get user feedback!
     
  6. sebj

    sebj

    Joined:
    Dec 4, 2013
    Posts:
    70
    Resurrecting because this is the first result in Google for exporting Memory Profiler to CSV.

    A workaround is to expose the internals of the MemoryProfiler package to custom Editor windows in the project, then get a reference to the MemoryProfilerWindow, and from here extract the data from the loaded snapshot.

    See the following snippet that exports the type names and sizes for all managed objects.

    To create a local copy of the package, copy it from the Library PackageCache into the Packages folder of the project (not into Assets). There is no need to change the manifest; just copy the folder and the Editor will update itself.

    To use IntelliSense and the Debugger, you also need to make sure the Editor will generate .csproj files for the local package. This is set in "Preferences -> External Tools -> Generate .csproj files for:".
    (I have Embedded packages, Local packages, Registry packages and Packages from unknown sources set).

    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4. using UnityEditor;
    5. using System;
    6. using System.Linq;
    7. using Unity.MemoryProfiler.Editor;
    8. using Unity.MemoryProfiler.Editor.UI;
    9. using Unity.MemoryProfiler.Editor.EnumerationUtilities;
    10. using Unity.MemoryProfiler.Editor.Format;
    11. using System.IO;
    12.  
    13. namespace MySample
    14. {
    15.     /// <summary>
    16.     /// This Window contains additional methods for exploring snapshots loaded by the MemoryProfilerWindow.
    17.     /// </summary>
    18.     public class MemoryProfilerHelperWindow : EditorWindow
    19.     {
    20.         [MenuItem("MySample/Memory Profiler Helper")]
    21.         static void Init()
    22.         {
    23.             MemoryProfilerHelperWindow window = (MemoryProfilerHelperWindow)EditorWindow.GetWindow(typeof(MemoryProfilerHelperWindow));
    24.             window.Show();
    25.         }
    26.  
    27.         public class QnDCSV
    28.         {
    29.             private TextWriter writer;
    30.  
    31.             public QnDCSV(string filename)
    32.             {
    33.                 writer = new StreamWriter(filename);
    34.             }
    35.  
    36.             public void Next(string token)
    37.             {
    38.                 writer.Write(token);
    39.                 writer.Write(",");
    40.             }
    41.  
    42.             public void NextLine()
    43.             {
    44.                 writer.Write("\n");
    45.             }
    46.  
    47.             public void Close()
    48.             {
    49.                 writer.Dispose();
    50.             }
    51.         }
    52.  
    53.         void OnGUI()
    54.         {
    55.             if (GUILayout.Button("Save"))
    56.             {
    57.                 // Get the opened window Memory Profiler Window.
    58.                 // We know the type of this from the UI Toolkit Debugger.
    59.                 // To access it we need to copy the Package locally, and add the
    60.                 // attribute:
    61.                 //     [assembly: InternalsVisibleTo("Assembly-CSharp-Editor")]
    62.                 // to the namespace (MemoryProfilerWindow.cs line 44).
    63.  
    64.                 MemoryProfilerWindow window = EditorWindow.GetWindow<MemoryProfilerWindow>();
    65.  
    66.                 // Get the current Mode (UIState::BaseMode) from the UI, which
    67.                 // will give us access to the tables cached when the Mode is
    68.                 // first created.
    69.  
    70.                 var table = (ObjectAllManagedTable)window.UIState.FirstMode.GetSchema().GetTableByName("AllManagedObjects");
    71.  
    72.                 // Get the Columns we are interested in...
    73.                 // Look at the ObjectListTable.cs ~860 for the column names (or use Search in Project)
    74.  
    75.                 List<Unity.MemoryProfiler.Editor.Database.Column> columns = new List<Unity.MemoryProfiler.Editor.Database.Column>();
    76.                 columns.Add(table.GetColumnByName("Type"));
    77.                 columns.Add(table.GetColumnByName("OwnedSize"));
    78.                 // columns.Add(table.GetColumnByName("NativeSize")); // Not available in the AllManagedObjects, but in some others.
    79.  
    80.                 QnDCSV csv = new QnDCSV(@"D:\3drepo\ISSUE_434\types.csv");
    81.  
    82.                 // Check all the columns have the same count.
    83.  
    84.                 var expected = table.GetRowCount();
    85.                 foreach (var column in columns)
    86.                 {
    87.                     Debug.Assert(column.GetRowCount() == expected);
    88.                 }
    89.  
    90.                 // Finally write the CSV
    91.  
    92.                 for (int i = 0; i < table.GetRowCount(); i++)
    93.                 {
    94.                     foreach (var column in columns)
    95.                     {
    96.                         // The formatter is used to convert the type to a string only. The result is the 'raw' value in string format.
    97.                         // (i.e. the value will always be in bytes for a size column)
    98.                         csv.Next(
    99.                             column.GetRowValueString(i, Unity.MemoryProfiler.Editor.Database.DefaultDataFormatter.Instance));
    100.                     }
    101.  
    102.                     csv.NextLine();
    103.                 }
    104.                 csv.Close();
    105.             }
    106.         }
    107.     }
    108. }
     
    phobos2077 and idyakonov like this.
  7. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,424
    Yes, that would be one way of doing it but will require rework if you want to eventually use version 1.0 of the package, where ObjectAllManagedTable and friends have been removed from the codebase.
     
  8. phobos2077

    phobos2077

    Joined:
    Feb 10, 2018
    Posts:
    350
    Any update on this? Seems like really needed feature missing. The current functionality for filtering objects is not sufficient. We need to be able to filter and sort by any field, use regular expressions, etc.
     
    idyakonov likes this.
  9. ruudvangaal

    ruudvangaal

    Joined:
    May 15, 2017
    Posts:
    27
    CSV export could be interesting. I'm stuck on 2021.3 (it's not easy to transfer all created content to later versions just like that). When comparing 2 snapshots, I see a 'new in B (110)' and 'deleted in B (101)' for example. Just knowing which 9 don't match would be so helpful. Right now I can't even export or copy/paste the text to other software to match the 101 deletes vs the 110 news.
     
  10. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,424
    No updates on this.

    @ruudvangaal: you don't need to update your project or it's contents to use the 1.0 or 1.1 versions of the Memory Profiler. Just installing a 2022.2+ version of the Unity Editor, creating an empty project with that version, installing the memory Profiler in that empty project and importing the snapshots into it allows you to use the new UI on data from older Unity versions.

    For your case, I'd recommend that you try version 1.1.0-pre.1 (which might require that you modify the manifest to specify that version). That version might make it more obvious what the 9 extra new objects are.
     
    ruudvangaal likes this.
  11. ruudvangaal

    ruudvangaal

    Joined:
    May 15, 2017
    Posts:
    27
    @MartinTilo Thanks for the tip, I'll give the new Memory Profiler a try!
     
    MartinTilo likes this.