Search Unity

  1. Looking for a job or to hire someone for a project? Check out the re-opened job forums.
    Dismiss Notice
  2. Unity 2020 LTS & Unity 2021.1 have been released.
    Dismiss Notice
  3. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

[Released] Ultimate Replay 2.0 - Next generation state-based replay system

Discussion in 'Assets and Asset Store' started by scottyboy805, Sep 8, 2020.

  1. lorewap3

    lorewap3

    Joined:
    Jun 24, 2020
    Posts:
    35
    To anyone reading this in the future, I provided these guys with a stripped down version of my project and within a day they identified a bug and provided a fix. My weird use case brought up another, and even though a much harder to debug issue dealing with the underlying compression library, they implemented a workaround and got it to me yesterday. And I submitted the project to them only 5 days ago including a weekend!

    These guys have a strong dedication to improving their products and fixing any issues you encounter. It is not lost on me and is greatly appreciated!

    I'm currently working on integrating it with the Unity Timeline. The goal is to be able to record play sessions and then be able to scrub the timeline to visually see the replay in the edit mode for the purpose of compositing elements around them. I knew my use case would push the limits of any replay system and these guys have definitely delivered on a system that can handle it!

    -Will
     
    Jmonroe, scottyboy805 and julianr like this.
  2. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    934
    Ultimate Replay 2.1.4 has been submitted to the asset store and will be available for download within a day or two. This version includes the following:
    • Fixed an issue in the replay file target where a CorruptedData exception could be thrown in some rare cases. The problem was caused by a bug in the mono compression library that we use and a suitable workaround is now in place.
    • Fixed an issue in the replay file target where the streamer thread could sometimes report that it had failed to start. This was a simple logic timing issue casued by a race condition and has now been fixed.
    • Fixed a bug in the ReplayAnimator component where some parameters would not be recorded correctly.
    • Fixed an issue where moving the root asset folder into a folder named 'Plugin' would cause script compilation errors.
    • A few othr minor bug fixes.
     
    julianr likes this.
  3. lorewap3

    lorewap3

    Joined:
    Jun 24, 2020
    Posts:
    35
    Hey @scottyboy805 ,

    I'm making progress on the timeline integration, but I have 2 main questions regarding implementing in Edit mode.

    I made my ReplayControls component into a Singleton that executes in both Edit/Play mode. I implemented it such that when the timeline is scrubbed, it starts a coroutine to restore the ReplaySnapshot for the specified time. I'm using coroutines so that in the event I scrub too fast, the SeekToTime operations will be added to the coroutine queue. I'm trying to minimize reading and disposing storage targets excessively, so after a scrub is complete, the storage target is disposed of (as to not incur possible issues later). So with every scrub, the StorageFileTargets are read in. But if there is already a coroutine in progress, the first won't dispose of the storage target. Only the last one will do the disposing.

    Anyway, that's the approach I'm trying to take. Cause I'm not 'Playing' any replay, just scrubbing to a certain time. Maybe there will be a way to integrate with the preview feature of timeline, but that's far down the line and not really necessary.

    Question 1
    What I don't know how to do is how to restore the snapshot directly. I used the code you provided from Jan 11 as a baseline, but it shows how to read the data itself. I really just want to restore it. This is the function so far:
    Code (CSharp):
    1. public IEnumerator SeekToTime(double time)
    2. {
    3.     if (!Application.isPlaying)
    4.     {
    5.         LoadStorageTargets();
    6.         for (int i = 0; i < storageTargets.Count; i++)
    7.         {
    8.             ReplaySnapshot snapshot = storageTargets[i].FetchSnapshot((float)time);
    9.             replayScenes[i].RestoreSnapshot(snapshot, storageTargets[i].InitialStateBuffer);
    10.             yield return null;
    11.         }
    12.     }
    13.     yield return true;
    14. }
    I think I'm using the right method in RestoreSnapshot. But I have no idea what I'm supposed to provide for the 2nd parameter given I'm not using the ReplayManager or in Play mode. I plugged storageTargets.InitialStateBuffer in there but I get null exceptions.

    Question 2
    I'm also interested in preserving the initial state before I enter 'Timeline Editing' mode. Here's the basic 'Directors Chair' window I created. It has 3 states, and shows the replays created. The replay highlighted green is currently loaded.
    upload_2021-3-3_16-44-53.png
    Rehearsal - Nothing happens. No recording and no replay scrubbing.
    Recording - Nothing in Edit mode, but in Play mode records for all loaded scenes.
    Performance - Timeline scrubbing in Edit mode. In Play mode, plays the replay normally.

    So ideally I'd like to save the state of all ReplayObjects before entering 'Performance' mode and restore back to that state after leaving 'Performance' mode. I'm not sure how best to save the state when not using ReplayManager or in Play mode.

    Q1 is really the roadblock at the moment, though. Or perhaps I'm going about this the wrong way and there's a better way to implement this. I did initially try using ReplayManager, but instantly encountered issues as I'd expected. ReplayManager.Awake() gives exceptions and any loading tries to utilize the DoNotDestroyOnLoad scene which Unity is not happy about when in Edit mode. Any advice you have is most appreciated!

    Thanks again!
    Will
     
    Last edited: Mar 3, 2021
  4. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    934
    Hi again,

    1. That code looks fine to me and should work as expected for the most part. The only other thing you need to do (which I am not sure if you are doing in the 'LoadStorageTargets' method) is to prepare the storage target for read operations. It is an important step and can be done by simply calling the following method:

    Code (CSharp):
    1. myStorageTarget.PrepareTarget(ReplayTargetTask.PrepareRead);
    The behaviour of a storage target when attempting to read from a non-prepared target is undefined and would likley explin the NullReferenceException you mention. The initial state buffer data is likley not created at that stage.

    Also, is there any reason why you do not want to use the main replay API for this? It can be done manually if needed, but it should also work fine when called from edit more with a small addition. Replay services wil not be updated in edit mode but you can update them manually if needed like so:

    Code (CSharp):
    1. class Example
    2. {
    3.     float lastTime = 0;
    4.  
    5.     void EditorUpdate()
    6.     {
    7.         float current = EditorApplication.timeSinceStartup;
    8.         float delta =  current - lastTime;
    9.         lastTime = current;
    10.        
    11.         ReplayManager.UpdateState(delta);
    12.     }
    13. }
    Edit - I see now that you tried this but had exceptions from the Awake method? Are you sure it is 'ReplayManager.Awake' and not 'ForceAwake', because the former method has an empty body in the full version of the asset? If force awake is the issue, then I assume the stack trace points back to ReplayControls.Awake? If so, you should be able to simply comment out the call to 'ReplayManager.ForceAwake' and just use the manual update method I suggested. We will add editor defines in the next update so that these issues cannot occur.

    2. Yes this is possible, and infact, the replay system does exactly this to restore the scene to its original state when playback has finished. You can just store a ReplaySnapshot in memory for a give frame and then restore it at a later date when playback has ended.

    Code (CSharp):
    1. class Example
    2. {
    3.     ReplayScene myReplayScene = ...
    4.     ReplaySnapshot snapshot;
    5.     ReplayInitialStateBuffer stateBuffer = new ReplayInitialStateBuffer();
    6.    
    7.     public void BeforeReplay()
    8.     {
    9.         snapshot = myReplayScene.CaptureSnapshot(0, 0, stateBuffer);
    10.     }
    11.    
    12.     public void AfterReplay()
    13.     {
    14.         myReplayScene.RestoreSnapshot(snapshot, stateBuffer);
    15.     }
    16. }
    If you don't see any changes after calling the 'AfterReplay' method, then you may also need to trigger a ReplayUpdate event, which is slightly problematic. The methods used to trigger replay events are marked as internal. You can try the following code to send an update but the compiler may not recognise it if you are using assembly definitions:

    Code (CSharp):
    1. ReplayBehaviour.InvokeReplayUpdateEvent(myReplayScene.ActiveReplayBehaviours, ReplayTime.startTime);
    I hope that makes things a bit clearer for you. Let me know if you have any more questions.
     
  5. lorewap3

    lorewap3

    Joined:
    Jun 24, 2020
    Posts:
    35
    Hey @scottyboy805 , I haven't tried anything yet, but I wanted to respond to this one and give you the error I get when I call ReplayManager.ForceAwake() . (Yes, it was ForceAwake, I mistakenly typed Awake)
    Code (CSharp):
    1. InvalidOperationException: The following game object is invoking the DontDestroyOnLoad method: ReplayManager. Notice that DontDestroyOnLoad can only be used in play mode and, as such, cannot be part of an editor script.
    2. UltimateReplay.ReplayManager.ForceAwake () (at Assets/Plugins/Recording Plugins/Ultimate Replay 2.0/Scripts/ReplayManager.cs:1174)
    3. LorewaPlay.ReplayController.OnEnable () (at Assets/Scripts/Main Assembly/MonoBehaviours/Replay/ReplayController.cs:92)
    I will try the manual update method using ReplayManager as you suggest. Maybe that will allow me to keep the storage targets opened while in the 'Performance' mode and only dispose of them when exiting instead of every scrub operation. I will also try capturing a snapshot before and restoring it. That was the part I was missing:
    Code (CSharp):
    1. myReplayScene.CaptureSnapshot(0, 0, stateBuffer);
    I was trying to call RestoreSnapshot but didn't know how to create the InitialStateBuffer that the RestoreSnapshot was asking for:
    Code (CSharp):
    1. replayScenes[i].RestoreSnapshot(snapshot, storageTargets[i].InitialStateBuffer);
    It's not obvious from the code that CaptureSnapshot is populating the InitialStateBuffer it's taking in as a parameter. I mean it is if you read the function itself but I wouldn'tve guessed otherwise.

    Will try these out and get back to you. Thanks!
     
  6. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    934
    Thanks for checking that.
    We have now fixed the problem on our end by not running that code while in edit mode, but you can just remove the call to 'ForceAwake' for the time being to get around that problem.

    We will add some additional code comments about the initial state buffer so that it is easier to understand what to pass to the snapshot methods.

    Let me know how you get on and if you have any further issues.
     
  7. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    934
    We have just setup a discord server for Ultimate Replay 2.0 and a few other assets of ours so that we can offer better and faster support. We would love for you to come and join and hang our with other users of the asset and maybe share any cool projects you are working on.
     
    Last edited: Mar 21, 2021
    julianr likes this.
  8. lorewap3

    lorewap3

    Joined:
    Jun 24, 2020
    Posts:
    35
    Hey @scottyboy805 ,

    Good news for us about the discord! Thank you for that! As I've told other asset developers, you opened a can of worms you may regret ;).

    I need some more clarification on how to use ReplayManager in Edit mode, if it's possible as you suggest. I'm not calling ReplayManager.ForceAwake() anymore, but I'm still getting the same error using the other ReplayManager methods. I don't want to overload you with unnecessary code, but here's the function calls when entering Performance from the Rehearsal/Recording state:

    Code (CSharp):
    1.  
    2. if (VerifyReplayLoaded())
    3. {
    4.     LoadStorageTargets();
    5.     CaptureInitialState();
    6.     StartPlaybackFrame();
    7. }
    VerifyReplayLoaded ensures that valid replay files exist.
    LoadStorageTargets loads (and prepares) them

    CaptureInitialState: captures Snapshot/InitialState
    Code (CSharp):
    1. private void CaptureInitialState()
    2. {
    3.     initialDataBuffers.Clear();
    4.     initialSnapshots.Clear();
    5.     for (int i = 0; i < replayScenes.Count; i++)
    6.     {
    7.         ReplayInitialDataBuffer replayInitialDataBuffer = new ReplayInitialDataBuffer();
    8.         initialSnapshots.Add(replayScenes[i].CaptureSnapshot(0, 0, replayInitialDataBuffer));
    9.         initialDataBuffers.Add(replayInitialDataBuffer);
    10.     }
    11. }
    and finally:
    StartPlaybackFrame:
    Code (CSharp):
    1.  
    2. private void StartPlaybackFrame()
    3. {
    4.     replayHandles = new ReplayHandle[storageTargets.Count];
    5.     for (int i = 0; i < storageTargets.Count; i++)
    6.         replayHandles[i] = ReplayManager.BeginPlaybackFrame(storageTargets[i], replayScenes[i], true, false, null);
    7.     return;
    8. }
    9.  
    It errors out on
    Code (CSharp):
    1. replayHandles[i] = ReplayManager.BeginPlaybackFrame(storageTargets[i], replayScenes[i], true, false, null);
    with the same "InvalidOperationException: The following game object is invoking the DontDestroyOnLoad method: ReplayManager." error I get from ForceAwake()

    All of those functions execute a single time on an event triggered by switching the mode to Performance. The following SeekToTime function is called externally by the PlayableDirector after entering Performance mode and when the Timeline is scrubbed to a new time.
    Code (CSharp):
    1. public IEnumerator SeekToTime(double time)
    2. {
    3.     if (!Application.isPlaying)
    4.     {
    5.         for (int i = 0; i < replayHandles.Length; i++)
    6.         {
    7.             if (ReplayManager.IsReplaying(replayHandles[i]))
    8.                 ReplayManager.SetPlaybackTime(replayHandles[i], (float)time, PlaybackOrigin.Start);
    9.             yield return null;
    10.         }
    11.         ReplayManager.UpdateState(0.5f); // Edit: Added this late at the start of Edit: at the bottom
    12.     }
    13.     yield return true;
    14.  
    15. }
    I included the SeekToTime function to help understand my thought process but it never gets to that point because the ReplayManager errors out.

    You're right, I don't have ReplayManager.UpdateState() in there, and I'm sure I need to, but I haven't gotten there yet. I believe I understand what you meant by your previous example, but I was a little confused at first by the deltaTime. This doesn't need to run every frame so I don't want to run UpdateState on every editor update as it's unecessary. So please correct me if I'm wrong but in my case I should just be able to do this?
    Code (CSharp):
    1. ReplayManager.UpdateState(0);
    Since deltaTime makes no sense in this context, I'm assuming that UpdateState just needs to be called so it updates all ReplayObjects in the scene and it doesn't matter what you pass it. To be safe I'd just pass it 0, though.

    Do you still think I can use the ReplayManager with some minimal tweaking? I would really prefer to use the built-in ReplayManager methods for all this, but I have no concept of how big of a task it will be to make it Edit mode friendly. Although looking into BeginPlayback it just calls ForceAwake() at the end of the function.

    Edit: (More like I tested this while I was typing and felt it worthwhile to have the history)
    So just for kicks, I tried commenting out the DontDestroyOnLoad(go) line in ReplayManager.ForceAwake(). I don't get any exceptions anymore, but nothing is getting restored either. I've traced the debugging all the way through the SeekToTime, and everything gets executed without errors. The replayHandles are valid, the SetPlaybackTime doesn't error, neither does the UpdateState(), but nothing gets updated in the scene. I also verified the replayScenes were valid as well. I tried 0.5 instead of 0 as a parameter to UpdateState as shown above but that didn't change anything.

    Thanks Scotty!
    Will
     
    Last edited: Mar 6, 2021
  9. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    934
    Hi again,
    Thanks for the detailed report of what you have tried.
    Your setup all looked OK to me so I decided it would be best to put a test project together to run replays in edit mode. After setting up a simple test project, it became clear that there were a couple of issues preventing edit time replays from working correctly. Here are the issues we found:
    • Calling 'ForceAwake' while in edit mode causes issues as you discovered already. We have now added additional checks to prevent that code from running when called in edit mode.
    • ReplayObjects did not have the necessary setup code run while in edit mode. This caused replay components to not be deserialized or updated by the replay system, resulting in static objects with no movement during playback. We modified the code slightly so that initialization code can run once in edit mode if required so that replay components will be updated correctly.
    • Dynamic replay objects that need to be destroyed during playback will generate an error about calling the 'Destroy' method at edit time. We have now added support for the correct method 'DestroyImmediate' at edit time only.
    After discovering and fixing these issues, we can now view a replay file in edit mode with no apparent issues. Replay transfom animations are updated smoothly depending upon how often you call 'UpdateState', and dynamic replay objects work fine also.

    I think you may have misunderstood the 'deltaTime' value for the 'UpdateState' method. It is a required parameter and simply setting it to '0' will cause the function to return without doing anything at all. You should pass a positive value representing the amount of time in seconds since the last update. The replay system uses this value to drive all record and replay timers. if you pass a value of '3' to the method, the replay system will update any recordings or replays by adding '3' seconds to it's internal timers (assuming time scale is set to '1'). For replays this will cause the playback time to be advanced '3'. You can call this as little or as often as you need, but you must still pass ther amount of time in seconds that you want to advance the replay system. I hope that makes sense.

    It is also worth mentioning that you may not need to call UpdateState at all if you are not interested in viewing the replay in sequence. You should just be able to call 'BeginPlaybackFrame' and then use 'SetPlaybackTime' to move to specific frames.

    There were quite a few changes made to the code to get this working which means that we will need to submit an update. We will do that shotly but it will likley take a day or two to be verified and published. If you want the update right away, feel free to send us an email with your invoice number to (info@trivialinteractive.co.uk) or come and message me on the discord server.
     
    Jmonroe likes this.
  10. lorewap3

    lorewap3

    Joined:
    Jun 24, 2020
    Posts:
    35
    Ah this is what I thought. I didn't think it should be necessary to call UpdateState if I was directly setting the playhead with SetPlaybackTime.

    All that makes sense about the deltaTime. That's how I'll do it down the line when I implement being able to 'Preview' a timeline while in edit mode, but right now direct positioning is all I really want.

    I sent you an email with my invoice #. I don't know how long it takes the asset store to approve things but as long as I've been working on this I'm very eager to get it in a working state! If you have the code ready I'd love to get my hands on it to try out.

    Thanks Scott!
    Will
     
  11. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    934
    It usually takes a day or two for updates to be verified in our experience.
    I have sent you the update direct via email though so you have access to it straight away.

    Let me know if you have any more issues or questions.
     
  12. petey

    petey

    Joined:
    May 20, 2009
    Posts:
    1,493
    Hiya! Pretty close to having another play with Ultimate Replay 2. Last time I had a look I ran into a bit of trouble with sounds and particle systems. I realise they could be tricky components to wrangle. Just wondering if you have any tips or best practices for using them in a more replay friendly way?
    Thanks!
    P.
     
  13. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    934
    Hi again,
    The ReplayAudio and ReplayParticle components are still a little limited in their features. We looked into adding better support for things like rewiding audio, moving partice systems, etc. but we could not get a solid working solution with out some tradeoffs. There would always be small (but noticable) audio glitches in the Replay Audio component, and the Replay Particle System component would not always simulate nicely. As a result, we decided to stick to a failry simple implementation, and users can implement a custom more advanced component if needed that is more tailored to their use case. With this in mind, the built in components will work for simple cases, but don't expect perfection in compex use cases. I hope that helps you.
     
  14. petey

    petey

    Joined:
    May 20, 2009
    Posts:
    1,493
    Yeah that’s totally understandable, I think I’ll just have to make a few custom things here and there.
     
  15. lorewap3

    lorewap3

    Joined:
    Jun 24, 2020
    Posts:
    35
    Hey @scottyboy805 ,

    Were the changes made to the codebase that you sent me via email added to the official release yet? I think I clobbered my code. I presumed you guys had published the updates, and went in and updated Ultimate Replay via the package manager. When it gave me the "DoNotDestroyOnLoad was called in Edit mode" error I could tell it wasn't, so I tried reapplying the patch you sent me over email, and the edit previewing no longer works.

    What is the ETA on publishing those changes? I'm not even sure what order to reapply the code changes. I'm working on other things right now, but it does tend to bug me when things are in a non-working state.

    Thanks Scotty,
    Will
     
  16. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    934
    Hi,
    We have not yet submitted an official update to fix that issue. We are currently working on some optimizations that we want to bundle with the same update, and also some other minor bug fixes. There is also a bug going on with the publisher portal where we submit updates, so it may take a little longer than usual for the update to go though. I will see if I can get an update submitted before the weekend.
     
  17. lorewap3

    lorewap3

    Joined:
    Jun 24, 2020
    Posts:
    35
    Ah I see. If you guys are still working on the code I don't want to rush you guys into publishing something you're not comfortable with. I have plenty of work to do elsewhere it's not a huge rush. I probably won't circle back around to this until mid-May.

    I just would like to be notified whenever the changes have been published. So if you could @me here or email me when those updates are published I'd much appreciate it. I do get asset update notifications but those aren't often very prompt.

    Thanks Scotty!
    Will
     
  18. scottyboy805

    scottyboy805

    Joined:
    Apr 10, 2013
    Posts:
    934
    Ok, no worries.
    There is not too much work left to do on the update I believe, just a few more minor fixes and testing.
    I always post on the forum anyway when a new update has been submitted, so hopefully you will get a notification for when the update has been submitted. Hopefully the publisher portal bug will not be an issue for us.
     
unityunity