Search Unity

[REBRANDED] Unity Asset Tracker (Formerly Unity Package Asset Index) - Editor Utility

Discussion in 'Assets and Asset Store' started by RKeown3D, Jan 14, 2016.

  1. ibyte

    ibyte

    Joined:
    Aug 14, 2009
    Posts:
    1,047
  2. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    Project Status is still in Development, and is slated to be released prior to v1.0.0.0. Right now, I'm working on Asset Favorites and History, and improving upon the Asset Dependency Checking for Models (FBX, DAE, OBJ, etc..)

    The ultimate goal for Project Status will be to track what packages/assets you've imported, so the system can tell you if any packages/assets can be updated. I'm shooting for v0.9.9.5 for the first true iteration of Project Management.

    I'm also looking at creating an asset archive manager, that will allow you to keep x-number of previous versions of Packages, and this is slated for a release in November.

    I'm still working towards 2 releases per month, during the Beta.

    Send me an email with your contact info, and include your order id, and I'll add you to the distribution list of beta users if you want.

    Rob
     
  3. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    Can you zip up your log files, and send them to me via an email. The folder path can be found in the UPAI settings window.

    Go to UPAI/System/Settings, and click on the Folders tab.
     
  4. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I'm working on a patch for the latest couple of issues that have been reported over the last few days. I'm planning on getting 0.9.9.2 rolled up, along with fixes for the latest issues. One of the issues includes a namespace issue, regarding the Newtonsoft.Json library that a few other Publishers include with their packages. The version that I included is not the version that is sold on the Asset Store specif for serializing Unity objects as Json. Instead, it's included as a reference library specific for back-end functionality for managing the index.

    I will alter the namepsace so that it doesn't conflict with any other versions of the Newtonsoft.Json.Net library that you may need for supporting other Assets/Packages.

    My goal is to have this submitted to the asset store by this upcoming Thursday/Friday.

    Please continue to email me any issues you run into, and I will address them each.
     
  5. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I'm posting this response on the Community thread, instead of on the AssetStore page, where it's more appropriate.

    A user recently posted a review asking about why certain things didn't appear to be working, and I'd like to address that here.

    - Compiler errors if you have Playmaker, PlaymakerPUN and Photon Unity Networking, these can be rectified but you need time and knowledge.

    This is being addressed in v0.9.9.2, as the Newtonsoft.Json assembly is going to be replaced with RKGamesDev.Systems.Serialization.Json, which is the original .net 2.0 Netwonsoft.Json.dll, with it's namepsace altered from Netwonsoft.Json => RKGamesDev.Systems.Serialization.Json. This will allow the assembly to reamin in the project as Editor Only, without causing any conflicts with other packages that include their required version Newtonsoft.Json.dll.

    A temporary fix, is to delete the /Plugins/Newtonsoft.Json.Net folder only....

    - No image available by many assets, some in fbx are there, some not and that also by prefabs, models and so on.

    The reason for missing images is that the system extracts the generated asset preview image file from the package file itself. This is only included if the Publisher SELECTS the Asset as one to include as a featured asset from within the the Unity Asset Store Tools, when uploading a new version of a package.

    I am looking to work on a way to try to generate a preview image, but this involves having to actually import the asset into Unity, and calling the function to generate the preview image, and then serialize it to an external .PNG file.


    - Import single asset only working sometimes, well actually practically not at all.

    The user is correct with his first part of the statement. I am looking into what may have caused the Asset Import to function a few times, and then stop working. This appears to have become broken in the v0.9.9.0 release, and should hopefully be fixed with v0.9.9.2

    - Dependencies NO, by lots of assets. I would expect them to be there.

    Dependency data is actually not included in the Package file, but instead, within the .Asset file, itself. I had to write a parsing algortihm to locate the dependency references from within the binary data contained in the .Asset file that is generated when the publisher originally imported the asset to include.

    Currenlty, 3D model files contain references only to material names and texture paths. and aren't actually converted to .Asset files, but some of the data is included within the {3DModelFile}.{Extension}.meta, after the file is imported. Again, my application is parsing all of the data from the package itelf, and not importing the package into Unity. After working with trying to import the package into anpother instance of unity and then reading all of the data was extremely slow and tedious. I know it's slow now, but that was the reason for parsing the data directly from the Package file itself.


    This is the main reason why no one else has tried putting an application together like this, on the Asset Store before.

    - Interface needs attention

    This is constantly being worked on....

    - Needs a pause function when scanning in case one needs to gain full control of the computer again and later carry on.

    This is expected to go into the application sometime after v0.9.9.5

    - Needs an option when searching to show all assets that have a thumbnail, I don’t want to see assets that have no picture.

    I'm looking at deserializing the internal default icon data from within the unity editor iteslf, so some form of picture is generated. This is already working on PhysicMaterial, Physic2DMaterial, AudioClip, and a few others. See the reason for why the image file is missing, in the - No image available by many assets... section, above.

    - Needs control information, how many assets are going to be scanned and information about how far through the tree the process currently is.

    Currently, the Unity Editor only allows for one progress bar to be displayed; however, I've written logic into my application that tracks all of the control information that you mentioned, and updates the progress bar, appropriately, as events are fired off on the back-end. I am working on a custom editor window to address this.

    - Needs right click on a thumbnail to go directly to asset store description.
    - Needs right mouse button actions


    Both f these will be added in either v0.9.9.2, or in v0.9.9.3

    Please remember, that this is still a Beta, and will continue to be so, for at least the next few months. There is a lot of complexity to what my application is doing. I hope I have addressed as many questions and concerns.

    The main difference between Grab Your Assets and my application, is that GYA is only showing you Package Information, retrieved from the Package file Header itself. Mine is showing this as well, and including as much of the package contents, as can be deduced, currently.

    GYA is a truly wonderful application, and I hope that both can be used together, and I am trying to incorporate some functionality integration, between both applications, to make Game Development a lot easier from the Package/Asset Management side of things.

    Please keep the issues coming, and also especially any ideas you may have for the future of this application. I am very open to new ideas, and a lot of whats there now, came from many of the users who got in, during the ALPHA, starting in January of this year.!!!
     
    Last edited: Oct 15, 2016
  6. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    So with all of the feedback coming in, the biggest issue currently, is the Package Scan process, and the time it takes to complete on systems with large quantities of Packages, as well as the number of Assets included in those packages. After reviewing the process, the biggest time consumption is the deserialization of the Package File to the file system.

    This appears to be consuming about 85% of the processing due to the massive IO operations that are going on.

    The post-deserialization processing is taking the least amount of time, and is where the heart of the application really lies.

    I'm looking at updating the current Indexing process in one of 3 ways:

    1) Real-time processing (Internal Unity Editor), with Progress bars, updates and notifications.
    2) Background processing (Internal Unity Editor), with no notifications not updates, but with the ability to monitor the processing via a separate Editor Window
    3) Command Line processing (External Unity Editor).

    This means that I will be delaying v0.9.9.2 for a few days, and longer if necessary, which throws out attempting to get an update out during the Level 11 event for October.

    I will also be extending the Beta period, so a January 1, 2017 go-live will also be delayed.

    I am continuing to review all feedback that's come in, and am working on numerous other process improvements, etc, so please continue to send me errors, questions, comments, etc to robert.keown@rk-softwaredev.net.

    I hope to post an update with a new timeline in the next coming days.

    Thanks again for your patience.

    Rob
     
  7. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I have updated the link for support forums to be this page, until I've verified that the issues with Email verification have been resolved and confirmed as fixed on my website.
     
  8. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I am pushing back all of the dev for v0.9.9.2 to 0.9.9.3, and releasing an emergency fix to address the namespace issue with Newtonsoft.Json.Net. This will be published as v0.9.9.2, and will include a script to automatically cleanup old UPAI files no longer needed. I'm hoping to get this submitted this evening, and hopefully approved within the next couple of days.
     
  9. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I have submitted v0.9.9.2 to the asset store as of this evening, and it is currently pending review. It addresses the Newtonsoft.Json.dll/Namespace conflict. The asset script to remove old UPAI Assets is included as well, and this has been tested with PUN and Playmaker.

    If anyone would like an advance copy, send me your invoice number in an email, and once confirmed, i will forward on the pre-submission version. The only difference between the submitted version and the pre-submission version is that the pre-submission doesn't include the Asset Preview nor Package Icon files, generated using the Asset Store Tools.
     
  10. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I am currently finishing up development on v0.9.9.3, and a major change is being put into place. I am changing the analysis method by which the system analyzes all of the Package data. Currently, it extracts the entire package to disk, before analyzing all of its contents; however, going forward, the majority of the analysis will now be done while the package is loaded into memory. This will be a significant performance increase, as it's cutting out about 80% of the Scan time.

    I'm also updating the dependency analysis for 3d Model formats that Unity currently supports, and hope to have this included as well.

    I'm hoping to have it submitted by the end of this month.

    I'm also still working on the ability for users to add custom tags and notes, regarding Packages and Package assets, without modifying the package contents, as well.
     
  11. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I've made some progress with the in-memory scanner, but need to go back and re-work some of the sub-processes, since I'm no longer reading all asset files, only those that can contain dependency data.

    I apologize for the delays, but i just got over the flu, even though i had a flu shot, go figure...

    I'm shooting for December 15th for the next release.
     
  12. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I am really excited to announce that I am officially starting the port to Mac, for the unity Editor in Mac. I'm hoping to have it in the next release (Same Package will support multiple OS's). Please hold off on purchasing if you are a Mac user, because I'm just getting setup on my Mac for development, and need to run through some tests to make sure that it will in fact port over, without issue.

    I'll announce it's official release, prior to submitting the package for review.

    I'm also going to be porting this to Linux as well.

    My goal is to get these versions out, during the beta, so that you'll be able to participate in the pricing for Beta, itself.

    I'll post more once I'm closer to knowing the port process for the different OS's
     
  13. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    Just wanted to let you know that i haven't forgotten about you, as I am starting the process of porting the assemblies to Mac.

    I just posted an update at the bottom of this thread.

    Rob
     
  14. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    406
    Hi, just upgraded to Unity 5.5.0f3 and ScriptInspector 3 is throwing errors

    I have to then delete UPAI so that everything works again except your package.

    Here is the error

    Assets/Plugins/Editor/ScriptInspector3/Scripts/FGTypeSystem.cs(800,67): error CS0433: The imported type `System.Runtime.CompilerServices.ExtensionAttribute' is defined multiple times


    It would be nice to have both working in the same project, but ScriptInspector is more important.

    Thanks in advance.
     
  15. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    406
    Just got this E-Mail from the author of ScriptInspector


    @eagleeyez cool! :) It might be worth informing the author, asset authors should never declare symbols which are already defined in .Net or Unity assemblies...
     
  16. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I'm looking at this this afternoon. Without the ExtensionAttribute being included in the assembly, itself, untiy would throw exceptions regarding not being able to call extension methods inside the editor. I know that this is already defined in the System.Core assembly for .Net, which is in fact being referenced from the UPAI Assemblies.
     
  17. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    406
    Woe, you guys know your stuff. For me it’s all a little double Dutch.

    So don’t shoot the messenger.

    Keep up your good work as I am looking forward to this working well as it is a brilliant idea and concept, but I can imagine how hard it must be.
     
  18. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    406
    Wow, great developer. This issue was solved in hours and not days or weeks

    He sent me the updated package immediately and it worked. I have noticed, that this developer is committed to his product like glue.

    That means he sticks to his promise.
     
  19. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    @Eageleyez, thanks for finding the issue, as well as testing the updated package. I had to go into the code and mark the ExtensionAttribute as internal sealed and not public sealed, so that it won't conflict with any other assets either referencing it via script in the unity editor, or any other assemblies that had to include this, due to Visual Studio Compilation errors.

    I have just submitted v0.9.9.2a to the Asset Store, for review, and hope that it will be approved soon!!!

    I've also gone into the upcoming revision, v0.9.9.3, and rolled forward these changes as well.

    :)
     
  20. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    406
    You are great!!!
    thanks
     
  21. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    If you are experiencing the above issue, please contact me, via email, and I will be more than happy to get you an advanced copy of the package.

    Please be sure to include your Unity Purchase Order number, for verification purposes.
     
  22. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I'm going to take a break from development of the UPAI over the next couple of weeks, and pick back up the first week of January.

    I'll have more details regarding v0.9.9.3 that week.

    Have a very happy holidays and a happy new year!!!!

    Look forward to resuming in January 2017!!!
     
  23. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    Enjoyed a nice little break, and am back to working on the UPAI next release.
     
  24. marked_one

    marked_one

    Joined:
    Jun 21, 2015
    Posts:
    17
    Hello, I'm trying to use UPAI, but getting the following errors while it indexes the assets:

    Some assets are processed normally, some are giving me such errors.
     
  25. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I'm looking at this now.
     
  26. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    Do you know what package was being scanned at the time? If possible, could you send me the last log file from the C:\Users\marked\AppData\Roaming\Unity\UPAI-5.x\Logs folder? Send it to me via email at robert.keown@rk-softwaredev.net, preferably in a zip file, as it may be large.

    For some reason, System.IO seems to think that either the source or the destination is a folder, and not a file path, which is odd.

    I am reworking the file naming convention also, so that it's not long, as it is now. This is being addressed in the next release, which is hopefully going to drop on the 20th or so.

    EDIT: updated to correct email address
     
    Last edited: Jan 13, 2017
  27. marked_one

    marked_one

    Joined:
    Jun 21, 2015
    Posts:
    17
    Sent you the e-mail with logs attached.

    There were different assets giving similar errors. For example, The Courtyard was one: https://www.assetstore.unity3d.com/en/#!/content/49377
     
  28. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    Just wanted to let you know that i sent you an email, this evening, with my findings.
     
  29. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I am excited to announce that as of this evening, I have re-written the Asset Scanning engine, which will improve on performance and speed. I've also updated the logic for detecting asset dependencies, including 3d model formats supported by Unity!!!.

    I'm going to wrap up testing here within the next few days, and have a new version of the engine and ui submitted to the Asset Store for review and approval by this Friday, January 27th, 2017.

    This is the first major BETA update for 2017!!!
     
    AGregori likes this.
  30. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    Just an update. I wasn't able to get things finished up this week, as I had hoped. I've got the scan engine re-written, and am going back and updating the progress tracking logic and UI.

    I may have to postpone adding in the addition of managing labels and notes, until 0.9.9.4, so I can get the updated scan engine out in the next few days.

    I'll keep posting updates.
     
  31. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    In order to reduce disk space usage, I've added in functionality that will reduce all Asset Preview Icons having a height or width greater than 128, down to a properly scaled 128 pixel height/width.

    Again, this will only affect the Asset Preview Icons that have length or width greater than 128 pixels. All others will remain unchanged.
     
  32. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    While testing the new Scanning Engine this evening, I found an issue that will affect a certain group of users. The users that will be affected by the issue I found (including myself...), has to do with the Asset Store-5.x directory being setup as a junction to another location on the system. Most people do this, so that they don't run out of space on their C: drive, due to all of the packages that they've purchased/downloaded.

    Because accessing the file, without .Net being involved, because the file itself contains a ReparsePoint attribute, the OS instinctively decides to add the file to the System Page Files Standby (Reserved) memory, which overtime, can become quite large in size.

    Because the process is scanning virtually every package file, this causes the system to eventually run out of memory. The problem is that unity only supports .Net v3.5 (CLR 2.0), but MemoryMappedFile management wasn't added until .Net 4.x, which isn't currently supported by Unity at the moment. I've researched the issue and have found one backport to .net 2.0, but it relies on Local Operating system files from Windows.

    I'm currently modifying the process as follows:

    1) Create a new Assembly, which will be set to Windows only, in the Unity Editor.
    • Because the system is in compiled DLL form, I ca't utilize Unity's Pre-Compiler options, for C#
    2) Determine if the unity package file is being accessed via a Reparse Point Attribute check (Done)
    • If not, no need to go any further...
    3) Detect if the OS is running under Windows
    • If not, no need to go any further (exception of checking for similar Mac/Linux issues as well)
    4) Dynamically load up the additional .Net Windows Only assembly, and use the Win32API calls for MemeoryMappedFiles, and Flush the View from Memory, and got on to the next file...

    I'm hoping to have this re-coded and testing wrapped up by February 15th. i know I missed the Jan 15th deadline, but a lot of changes have been made to the scanning engine, and a lot of code has to be re-factored to utilize the updated functions.

    I've also decided that the UPAI, from now on, will run in the background, in a separate thread, with below normal thread priority, so as to not interfere with your use of Unity for Game Development. I'm thinking of leaving in the dedicated scan, as an option, if the user wants to just let it run, by itself.

    Also, i may have to limit the size of the unity packages that can be indexed, due to processing time required. One of the issues I ran into was with the Unity Technologies Adam Project files over 2.0 GB in size. Currently the system is handling well with files sizes up to 1.5 GB.

    also, the scan times have been greatly improved, and the process is running as multi-threaded, which means multiple Packages/Assets are being scanned simultaneously. This next version will also have the first iteration of parsing all currently Supported 3D Model formats, currently supported by Unity, out of the box. this doesn't include Cinem4D, 3DS Max, nor Maya files, currently, and aren't planned for any future release, as of yet.
     
    yash_sg likes this.
  33. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    Akshara, thanks for the feedback. The good news is that I did say that I MAY have to limit the scanning of large files,a nd it's not out of the picture yet. I'm evaluating whether its better to scan large package files by extracting them, then scanning at the file system level, or continue to try and scan them in memory.

    Its still a work in process, and is still in BETA.

    Please don't worry about the feedback, I under the frustration, trust me, been there. I'm continuing to do the best I can, with the limitations of the .Net 2.0 framework, at this time. The problem is that the OS is actually controlling the aspect of MemoryMappedFiles, due to file junction reparse points. If Unity supported .Net v4.x, the MemoryMappedFilew wouldn't be an issue. I have found a possible solution, but it's time to implement

    As for the Scanning of large packages, it's coming down to time to parse all of the contents of the Assets, due to large numbers of them, as well as the size of individual assets, as well.

    Hang in there, and keep the feedback coming.

    Please don't feel bad. I'm the one who feels bad for being late with the deliverables.

    ;)
     
  34. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    Ok, so after working with Microsoft, it doesn't look as if I'll be able to programatically do anything about the standby memory, built into the system, for now. It turns out that they changes the Algorithm that decides which files get mapped in memory, by default, and there are no plans to change it. So standby memory is an issue going forward, at least for Windows 8 and 10 users.

    for the time being, there is a tool that users can run, for now, called Rammap.exe, which allows you to clear the Standby lists.

    They did say that the files in standby aren't active, but shouldn't be affecting memory being freed up, but they are seeing an increase in reports that this may not be the case, and that the OS may not be releasing or zeroing the memory fast enough.

    I'm going to leave the engine alone, for now, regarding this aspect, and move on to improving scan times for larger packages.

    I'll keep you posted as this progresses.
     
    yash_sg likes this.
  35. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    Also, i will be leaving up to the user in regards to how they implement the scanning. I'll have both On-Demand and Background scanning options.
     
  36. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    As of this evening, I'm still continuing to work on re-integrating the changes to the scan engine into the Unity Editor UI.

    I apologize for the delays, but please note, that i am working overtime to get the latest version out, with all of the improvements and updates.
     
  37. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    After testing these last few days, i have a few bugs I want to knockout, prior to submitting the updated package to the Asset Store.

    I hope to have it all wrapped up by this weekend, and have it submitted to the Asset Store Admins for review by mid-next week.

    Thanks again for all your patience.
     
  38. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I'm now going back into the project and updating it for both Mac and Linux compatibility. this will be the first release that will support all 3 OS's!!! I've finished setting up my dev environments on both Mac and Linux.

    I want to give the other OS users a chance to get in on the BETA at the lower price, prior to launching v1.0.0.0.

    This will add just a few more days to the development release.
     
  39. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    Still working on the UI, and am also starting integration testing on both Mac and Linux. My plans are to send this out to the registered users, first, and then get feedback, prior to posting to the Asset Store. this way we can get some initial testing from a small group, prior to launching it for all who currently own it.

    Also, I will need 1 volunteer user for both Mac and Linux, so it can be tested by others on those OS's.

    Any users wanting to Alpha test on Mac or Linux, please respond to this thread, and I will provide a copy of the Package. Please, only serious users who are willing to test the software and report back issues, fixes, etc...
     
  40. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    So, I hate to do this, but I may have to delay the asset due to a PC issue. During testing over the last few weeks, I've been getting inconsistent results, with no visible explanation. After working with a few other developers and Microsoft Tech support, it looks like either my processor and/or my motherboard is failing. the problem is I don't have the funds to update my current system.

    I'm currently running on (sad to say) an old AMD Phenom II X4 945 (3.0 GHz -> OC'd to 3.3GHz) on a FoxxConn AMG-88X motherboard. Both are several years old (2011), and due to our financial situation, I don't have the means to be able to replace the parts :-(

    I'm NOT going to deprecate the asset, and am hoping to come up with a solution, as soon as possible. If anyone has any questions, comments, concerns, please send me an email listed on my asset page.

    Rob
     
  41. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    So, it doesn't look like I'm going to be down as long as I thought. We narrowed it down to the processor, and I was able to find a Phenom II X4 965, and it arrives on Friday afternoon. I should be back up and running by Saturday morning, and resuming development

    Rob
     
  42. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    As of this evening, i am back up and running with a newer Phenom II X4 965. We did confirm the old X4 945 in fact failed. The good news is I'm back on track, and am running through some more diagnostics just to make sure there are no other issues, and will post an update, over the weekend, about when the next (way long over-due update) will be submitted to Unity for approval.

    Thanks for your patience and understanding!!!

    Rob
     
  43. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    Spent the weekend refactoring code to get me back to where I was, before all of the issues started happening on my PC, due to a failed/failing processor.

    Also, updating the Asset Dependency Scanning logic for handling Unity Assets in YAML format. Binary format, and Unity Script's are being analyzed for Resource.Load, calls to AssetDatabase.Load, as well as other Shader functionality to determine which scripts are attempting to utilize assets from the Asset Package.

    It looks like I've wrapped the logic to now properly scan all default model file formats that Unity Supports out of the box, without additional software being required. this is detecting both Material and Texture references from all of the different file formats (Binary, ASCII, and XML)

    All of this is going in the next release. I'm still re-working the timeframe for the next update, and hope to have more information for everyone shortly. I'm also working on a few backlogged feature requests for filtering and searching Packages and individual Assets included in those packages by Asset Types (Material, Shader, Texture, etc...) By Category, and also by Publisher. I'm also adding in the ability to manage the sizes of the cached Package/Individual Assets Icon files, in order to save on disk space.

    I've also now made the application multi-threaded, and will limit the number of processes to the number of logical cores on a system.
     
  44. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    Also forgot to mention that SpeedTree Dependency detection is also working, and that was a tough one to figure out. This is also going live in the next update as well.

    Also, all though I did have some major set backs and delays, I am still shooting for this update to be the first version that supports Windows, Mac and Linux, and all three versions will be available in the same package. If you are planning on using this for Linux and/or Mac, please hold off on purchasing unitl I've actually confirmed that it will launch with support for both additional OS's.

    Rob
     
  45. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I'm also working on getting SSL set up on www.rk-softwaredev.net. I have SSL setup on my webservices site, but am in the works of adding a second static IP and purcahsing an SSL cert for the www site.
     
  46. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    Okay, just a quick status update. Here is how the scanning logic is being established, which you will have full control over:

    Foreground Scanning (Current Implementation)
    Progress provided via progress bars...

    Multi-threading options:
    • Enable/Disable
    • Processor Thread Limit : Selectable via user interface
    • Process Thread Priority - Above Normal

    3 Scan Types:
    • Quick - Just overview of Package Data only, images, and meta data extracted from package headers.
    • Full - Similar to Quick, but also extracts Generic Asset Information included in package, files sizes, asset types, preview images (if available), and label data set by the publisher.
    • Deep - Full + Asset Dependency Detection ( this is where the system is processor/memory intensive)
    Background Scanning (Additional Implementation)
    Progress provided via status window...

    Multi-threading options:
    • Enable/Disable
    • Processor Thread Limit - Max is 1/2 Logical Processor Count
    • Process Thread Priority - Below Normal
    For Deep scanning of assets only (Dependency checking)....
    Due to Unity currently supporting the CLR 2.0 (.Net 2.0/3.5) I've added in a limit of a maximum of 2GB of parallel processing, as this is a memory limitation of that Frameworks ability. So if you have multi-threading enabled, and you have 4 logical processors, it will attempt to process 4 packages, simultaneously, only if their combined sizes is less than or equal to 2GB.

    This does NOT mean that i am limiting the size of packages, just the processing of packages, simultaneously.

    The Assets Scanning themselves, will still be multi-threaded, with the same memory limitations imposed on the packages. Up to 2GB of Assets will be scanned simultaneously.

    Packages greater than or equal to 2GB in size will be processed one at a time.

    I'm still evaluating the best method of Deep Scanning unity packages larger than 1.5 GB.

    Index Size
    I'm also shrinking the size of the index and removing storing some unnecessary data. It should be a significant drop in the index sizes.

    Icon File Sizes
    You will also have the ability to control the max sizes of the Package/Asset Images that are cached, in order to save on disk space.

    I've been working all weekend to finish integration testing, and updating the UI.

    Friday, April 15th, is my new target date for submitting the v0.9.9.3 package to the Asset Store for approval.

    Any questions, comments or concerns, please contact me directly at robert.keown@rk-softwaredev.net

    Rob
     
  47. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I'm also working on adding SSL to my web-site, and hope to have it up and running by mid-week. I just finished converting the site's IP over to a static one, with the hosting provider, so I can add a certificate tot he site.

    Rob
     
  48. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    If you are using the UPAI and have upgraded to Unity 5.6, this may break some of the web functionality under UPAI -> Help menu items. The local help file should still work, but any calls out to my web services may fail due to some of the changes made from 5.5.x to 5.6.x.

    The indexing/search functionality should continue to work.

    Currently, I am focusing on getting the latest update out for compatibility to 5.0.x to 5.5.x.

    Once 0.9.9.3 has been launched, i will then work on the comparability issues for 5.6.x and above, and then release an alternate package, specifically targeting that version of unity.

    Rob
     
  49. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    I'm not making this evenings deadline to have the latest version ready to be sent to the Asset Store Publishers. I have to go back and re-write some of the Index Management Functions, due to the major changes implemented on the Back-end.

    the Scan Engine is finished, and does work in detecting dependencies on all Unity Asset Types, that support Dependencies. i also added in a bug fix for detecting Assets that are generally stored in binary form with some of the newer versions of Unity, but may also be stared in YAML form. I added in YAML parsing of those types of Assets, in order to retrieve their Asset Dependencies.

    I've also confirmed that all dependency scanning is working against all 3D model formats, that Unity supports out of the box. This doesn't include 3DS Max (.max), CD4, Maya (.mb), LightWave (.lwo) and other formats where an external editor is required to do format conversion into Unity3D.

    Also, I have finished testing on the SpeedTreeProceduralModel reader, and it is also extracting dependencies for Materials, Models/Meshes, and Textures.

    Once I've gone back and re-written the Index Manager Functions, I'll update the UI.

    Before i send the latest version to the Asset Store, I will want to send this out to the Beta Testers, in order to get some feedback, prior to submitting to the Asset Store. If you'd like to participate in this set of testing, please reply this this email.

    My new goal is for next Friday, April 21st, 2017, to get this out to the beta testers, and then once all feedback comes in, 2-3 weeks, we'll submit to the Asset Store for approval.
     
  50. RKeown3D

    RKeown3D

    Joined:
    Oct 14, 2014
    Posts:
    188
    So, I believe I found a solution to scanning very large package files. It involves splitting up the package contents, using multiple readers, and having each reader parse the portion of the package assets that it's been given, and then by merging the results after all of the partial package readers have completed processing.

    I'm testing this as of this evening, and hope to have the implementation in the upcoming release of the UPAI.

    I have not been able to do much testing with Mac OSX nor Linux, due to the recent setbacks, and the next release will most likely NOT include support for them. Once I have stabilized the current release on Windows, I will then resume dev on MAC and Linux versions of this asset.