Feedbacks for experimental feature: GameObjectRecorder (2017.1)

Hi!

In the 2017.1 comes a new editor feature: GameObjectRecorder. It allows to record any properties on a GameObject and its children.

Please keep in mind that this feature is experimental. We know there is lots of room for improvement, but we would first like to get feedback from you, our user, in order to identify use cases that are meaningful/useful to you. This feedback will really help us decide how to evolve this feature, moving forward.

In the example below, there is one root GameObject and all the other GameObject are children of this “Scene Root”. On this Scene Root there is a script using a GameObjectRecorder. This GameObjectRecorder is set up to record the transform changes on the Scene Root and all its children. Once the user stops the recording, it saves everything that’s been recorded into an animation clip.

Click on this link to see the result:
http://imgur.com/a/yiKPR

Here is the code used to produce this example:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using UnityEditor;
5. using UnityEditor.Experimental.Animations;
6.  
7. public class HierarchyRecorder : MonoBehaviour
8. {
9.     // The clip the recording is going to be saved to.
10.     public AnimationClip clip;
11.  
12.     // Checkbox to start/stop the recording.
13.     public bool record = false;
14.  
15.     // The main feature: the actual recorder.
16.     private GameObjectRecorder m_Recorder;
17.  
18.     void Start()
19.     {
20.         // Create the GameObjectRecorder.
21.         m_Recorder = new GameObjectRecorder();
22.         m_Recorder.root = gameObject;
23.  
24.         // Set it up to record the transforms recursively.
25.         m_Recorder.BindComponent<Transform>(gameObject, true);
26.     }
27.  
28.     // The recording needs to be done in LateUpdate in order
29.     // to be done once everything has been updated
30.     // (animations, physics, scripts, etc.).
31.     void LateUpdate()
32.     {
33.         if (clip == null)
34.             return;
35.      
36.         if (record)
37.         {
38.             // As long as "record" is on: take a snapshot.
39.             m_Recorder.TakeSnapshot(Time.deltaTime);
40.         }
41.         else if (m_Recorder.isRecording)
42.         {
43.             // "record" is off, but we were recording:
44.             // save to clip and clear recording.
45.             m_Recorder.SaveToClip(clip);
46.             m_Recorder.ResetRecording();
47.         }
48.     }
49. }

Once saved in a clip, to play it back you can duplicate your scene object, remove any physics components there might be on the duplicated objects (to prevent conflicts with the animation), and drag and drop the clip on the scene object.

Features:

• /!\ Only work in Editor scripts for now
• Can record all the properties of a GameObject
• Can record the properties of one component of a GameObject
• Can record one specific property of a GameObject
• Can record recursively on all the GameObject’s children
• Save the recording in an animation clip (.anim)
• Compute the appropriate tangents
• Reduce the keys

Here is the link to the API: https://docs.unity3d.com/2017.1/Doc...perimental.Animations.GameObjectRecorder.html

 

Sounds interesting, this is for recording cut scenes or similar videos?

 

For instance, yes.
You can record stuff for a video or a future in-game cinematics.

Or you can record a sequence of animations to show in the tutorials of your games.

Or maybe something else that we didn't think of yet

 

I can see this being really useful for recording complex physics simulations and then playing them back. Say having a house collapse - you can set it up on a powerful computer and add a ton of forces and other complex stuff to make the thing fall apart in a natural looking way, record the details, and then play them back cheaply, so you can get as good looking stuff on cheap hardware.

Like how DK Country got away with looking way too good for the snes.

For things like that, it would be nice if there was an easy way to specify a scaling of the recording. Like:

Code (csharp):

1. Time.timeScale = .1f;
2.  
3. //record something that only runs smoothly at 60fps if we're recording at .1 times the normal speed
4.  
5. m_Recorder.SaveToClip(clip, .1f); //this would generate the clip at 10 times the recorded clip

This can be solved by just playing the clip at 10x speed, or by manually multiplying all of the keyframe's time by .1, but having this directly in the API would probably make that smoother (and more obvious to users!)

 

Interesting idea @Baste!

Out of curiosity, I guess you'd like to increase the play speed of a physics because collapsing a huge scene, although physically true, tends to be slower than what we see in the movies?

 

No, I was thinking purely from a performance standpoint.

Like, I'm doing something that's going to make the game run at 5 fps, so I want to record at 1/12th of the normal speed and then speed that recording up by 12 times so it looks like everything's happening smoothly.

I'm not sure if fiddling with the timescale fixes that. I know that some AAA companies (not to be named) record teaser trailers by stepping through frame-by-frame and then playing those frames back at 60fps to make it look like their game runs smoothly.

That's the obvious use of a recorder for me. I mean, record something that's way too expensive and then play it back cheaply, not lying to my customers.



Out of curiosity, how much is it viable to store? I know that eg. Braid did a ton of compression to make it possible to rewind time a lot, so if you're doing something like that, just naively making keyframes every second is not going to give good results. Thoughts on that?

The Super Meat Boy scenario looks like it'd probably just work straight out of the box, which is nice. Ghosts like that are also nice to have in racing games, but that will require the system to handle very long animations. Is there any upper limit to AnimationClips in Unity?

 

About the too expensive scene baked at 60 fps, that's basically what you'll see when it is said "rendered in engine" (compared to "in real-time"). It means that the engine is capable of very impressive stuff, but when all the features are put together, it would need a monster rig to run it

I haven't tried it yet, but I think it would be possible to update the scene at a fixed time using https://docs.unity3d.com/ScriptReference/Time-captureFramerate.html. So you can record at 60 fps with the GameObjectRecorder, and play it back at runtime using way less resources. It might be better than using a time scale since you will have an exact control over the resulting FPS.

About the Braid game (which I love), note that it is done at runtime, which is impossible (yet) with the GameObjectRecorder that works in Editor only (the problem being about creating assets at runtime, we're looking at it).

That being said, the GameObjectRecorder is already compressing the resulting animation. Right now the compression is light and the compression settings aren't exposed in the GameObjectRecorder API, but they could be if requested.

For the go-back-in-time feature, be aware that the game has to be thought that way from the very beginning; it would be very difficult to plug this kind of feature afterward because some events, effects, etc. aren't "symmetric": you can play it forward, but not backward. Think animation events for instance, it triggers something (like a GFX or a sound) over time when played forward, but you don't want it to be triggered at the same time when played backward. Also you'll have to be very careful about what you'll record obviously. If you take all the properties of all the GameObjects of your scene, that's going to use a lot of memory!
But, other than that, I think that recording every relevant GameObject would be the way to go (at least at first) for the go-back-in-time feature. Extra care would have to be given to GFX, like particules... I don't know how "recordable" the particules are.

For the long recording sessions or the ghosting feature, I don't have any figures, but it would be interesting to see how the GameObjectRecorder behaves when we stress-test it like that.

 

Hey this sounds interesting, could it potentially be used to save gameplay replays?

 

For a simple video demo, I think screencasting might be more appropriate.
But if you want to do something more fancy -- say editing videos of the same gameplay sequence but with different skins, terrain, etc.. -- it could indeed be useful.

 

This looks like a great feature! I wish it'd been available last year, would have saved me heaps of time.

Am I right in thinking this can save a hierarchy of transforms, i.e. a skeleton, easily?

 

Thanks!

If by "save" you mean "recording", then yes, the GameObjectRecorder can record any hierarchy you want, it doesn't support humanoid though (yet).

But if by "save" you mean "remove", then no, we don't touch the hierarchy of the recorded scene.

 

I mean as in "recording", sorry that wasn't very clear. I like the word "yet" in your response

Out of curiosity, why would a humanoid rig need to be handled any differently than any other hierarchy of transforms?

Great job again!

 

The binding system (i.e. the system that binds the curves to the bones) is different for the humanoid. The problem is that there can be only one humanoid per animator. So if you record two characters in the same scene, you will get two humanoids in the same .anim and so you'll have two humanoids for one animator which will probably fail to play in very strange ways.

But fear not, we're thinking of a solution as we speak, but it might not come right away. One solution might be to be able to record several GameObject and create a Timeline with the several .anim files generated.

 

Great stuff. Thanks for the response and keep up the good work!

 

Can it be used to bake Cloth physics? example: Record a flag waving for a few seconds and save it as a clip so we could remove the Cloth component and save performance.

 

Yes, that's exactly the kind of use it is meant for!
If it doesn't work, you have a golden ticket to the bug reporter

 

For me this would be really useful if it worked in the editor-less Player Because today I got the task of recording the motion (and spawning) of everything in our project, to be saved to disk. Then later we want to play back exactly what happened, fast forward to a specific part, and then let the project continue running.

It’s a research project, so the goal is to be able to play back how an event went down, change one part, and then see how that affects the outcome.

 

That's an interesting project!

Though, even if the GameObjectRecorder was available in the standalone player, in it's actual implementation, it would probably not be able to record a whole gaming session with all the objects in the scene. It represents a important amount of data and the current implementation keeps everything in memory until the end of the session where it optimize and saves everything in a .anim file.

But for now, the main problem is that we can't create editor assets in the standalone player. So we would have to bring this big piece of code first if we want to have the GameObjectRecorder working in the standalone player. Depending on the popularity of this feature, it could happen in the future...

But in the meantime, for your project, I would advise you to save into your format (known or internal), because eventually, the serializer would probably not be the most important part of your project

 

This looks very promising. Can't wait to get my hands on this.

Question, would this be able to record blendshape values on a skinned mesh renderer component?

 

I implemented a very similar system that stores the movement and arbitrary values on components animation files. Yes, this could be made to record replays as long as you record the time that gameobjects are created and place the animations on them.

However, there is a problem. These replays get really big, really fast. I had a rather simple Asteroids game the replays would just be enormous. I even wrote an algorithm that simplifies the curves when saved but still, even then, they get very large. I wanted to record a number of 5-minute for an attract mode or something, but it was just generating too much data. But Asteroids is kind of a worst case scenario with this type of replay. There are so many little objects being created and tracked with many bullets, small asteroids, etc.

 

Is there any possibility you could use such a system as this to also record / bake particles? I seem to remember an article about the making of Zombieville USA and how they created cheap particle effects for mobile by baking animations in Maya or some such program and exporting that to Unity instead of using a real-time particle system. So it would be handy if we could also bake particle simulations, just like physics simulations, directly in the Unity editor. However, I'm assuming doing this might be difficult and somewhat outside the scope of such a feature because you would probably not be simply recording properties, but instead converting special particle system mesh and transform data into something else entirely in order to "bake it"?

 

The simplest way to find out would be to test
For the blendshape, I'm trying right now to get my hands on an asset so that I can try, I'll keep you posted!

 

Just tested it. Works great. Now we can record real-time animation. This is fantastic. Great work guys.

 

Great! I'm glad you like it! Thanks for having tested it!

 

Any chance you know when this will move out of experimental? We would love to use this in our production plugin.

 

Not in this case, at least not for me anyway . I did actually try binding a ParticleSystem to a GameObjectRecorder already, but it just seems to record the initial ParticleSystem settings which are useless. I'm now guessing you can probably create some kind of advanced script that records each particle's state using ParticleSystem.GetParticles() or something, so I guess it's kind of possible, but it's not exactly simple .

 

Hello @RomainFailliot ,
Do you think that this feature will also work for not-editor scripts as well soon? Or is it possible to achieve something similar thing like creating animation in runtime? I was planning to record game sessions with this feature.

Thanks,
Emre

 

Hi!

No ETA for now, we didn't know actually that this feature would be of interest to so many people!

Hmm... Then it means that the particules system doesn't have bindings and thus we can't create curves out of that. I suspect that it is for performance reasons that they didn't do it, but it would need more investigation on our part. Thanks for trying it out!

Right now it's not planned to have that because it implies a pretty important refactoring in order to be able to serialize editor data at runtime (editor assets don't have the same format as runtime asset, for obvious performance reasons).

Thanks everyone for your curiosity about this feature!

 

Since there seems to be some interest in having the GameObjectRecorder to work in the player, I opened a suggestion:
https://feedback.unity3d.com/suggestions/use-the-gameobjectrecorder-in-the-player

That is the proper way for the users to request for a new feature

 

Hello,

I have some feedback for the recording system. One such application of mine is to record all the 3D transform data of moving full body avatars as well as the 3D transform data of any virtual part of an assembly model that is grabbed and moved by the user in the game. I also want to be able to dynamically load those recordings directly back into the game after finishing recording.

My problem is that I have to use the Animator Controller Mecanim stuff, which doesnt allow me to access the specific key frames of the recorded animation. I also am having a hard time dynamically creating states with necessary transitions in the state machine via scripting. I've searched and found that only in Legacy could you just attach an animation to an object and play it, but now you have to do the Animator Controller stuff which is causing a real headache when it comes to coding for it. I'm seeing you can script entry and exit transitions, but not a simple transition between states? Maybe this exists but I've looked far and wide. I've only found the ability to script a transition between state machines, not just states though. Help?

Also, as for the recording animations in the 2017 beta, might I suggest that it would be incredibly useful to allow a childed object to be recorded in world space, not just local space relative to its parent. Because as you might see, if the child is recorded but not "moving" in respect to the parent, then it's never getting any keyframe data. I can't unchild the object due to my specific uses, and I can't put the GameObjectRecorder script on the parent (because I know it gets the root and all its children). It must specifically be on that one child and that's it. When I grab objects, they become childed to my remote, and it would be nice if I could then do the "add property" function in the animation to start recording the transform data of the object, but the moment you let go of the object, that added property then goes away and you lose it. A fix with this would be incredibly useful.

I would just find it incredibly useful if there was a simpler way to access the keyframe data in the animation itself during the game via script. The ability to rewind, pause, play, and fast-forward these animations at different speeds while in game is incredibly interesting and useful to me.

 

Yeah, I would also be most interested in a run-time implementation of this, since it would open up a lot of possible game-play features, like instant replays. I immediately thought of something like the demo recording tools for the Source Engine when I first heard of this. In fact, I just realized you could probably already use this in a similar manner to the Source Filmmaker to record game action in the editor and then tweak and edit the results in Timeline or something.

 

Hey thanks for the heads up UziMonkey I was thinking it would be cool to have a replay of you could change the camera in after the fact but sounds like it will be a huge pain to build and the file size might be prohibitive.
I did see one cool replay system in a unity game http://turbodismount.com/
My app has a crazy mechanim network though so that would probably be another reason not to try making one

P.

 

@petey @UziMonkey @DreamPower There was a very interesting presentation for the Entitas framework where they created a replay system that required very little data storage, even for pretty complex games. I wont pretend to completely understand how it works, but the key seems to be having a deterministic simulation and only recording the actual user input or events. Then the replay system can use this input/event history to recreate everything at any point in time without any additional data needed. This is however quit different from recording animations and not at all useful for a non-deterministic simulation, like physX.

 

well i tried this for recording animations and using it with new timeline tool. it works quite well but still need some improvements. workflow is crippled right now.

also, i guess we can use this for cheap motion capture animations in engine with Kinect SDK I haven't tested yet but it's promising, I will share results once I test and build a stable and fluent way to record motion capture animations in engine.

 

I'm interested in the fact that it saves properties in the hierarchy. If this can run in play mode, then we can use this as a way to save game state. We would only need to save one frame.

 

So we have finally gotten back to our Unity plugin and started building a feature around this new API. Works well in regards to recording real-time blendshape animation values. However if you try to modify the resulting .anim file or just reload the editor, the transform animation still looks smooth but the blendshape animation no longer is smooth. Rather it only animates on each keyframe, displaying the animation value that was recorded on that frame. All movement between frames is lost for some reason and the resulting animation is extremely choppy. Is there anywhere to alleviate this issue?

 

UPDATE:

After looking at it for the day I noticed that the keyframes created for transforms vs blendshapes are very different. Transforms seem to set keyframes whenever a change to that property occurs while blendshape values, regardless of change at runtime will always keyframe every 50 frames. Please note that I am using the same sample code from the original post, just binding a skin mesh renderer as well as the object transform. It is also worth noting that blendshape keyframes are keyed as "Constant Broken" when the transform keyframes are keyed as "Free Smooth". This could explain why when I reopen the animation the blendshapes values are choppy.

But what I am wondering is why blendshape values are being sampled every 50 frames rather than on every frame their values change, similar to the positional and rotational values.

 

This is how most replay systems work. It's not difficult to set up at all, the problem though is when games get non-deterministic. Frames don't always happen at the same exact time in a game due to different framerates and inconsistent framerates, this makes it very difficult to simply record input and get the same result every time. And if your game uses physics as you mentioned... good luck.

 

i am very interrested by this feature and will try it!

i already used a script found somewhere to record anim but it is better if it is part of unity ! (unityAnimationRecorder) that work great in editor, but without keyframereduction.

it's easy to reduce keyframes on a animation deleting no-change keyframes, or small changes. but i am not a math genious to create smooth curves from linear keyframes!
(i wished the keyframe reducer used in modelImporter could work with a simple generated animation.)

I used it to record VR movement to animate things in cutscenes easily.

 

Is it supposed to record Active property? From my tests, it doesn't.

 

The same happens for values on custom scripts.

 

Hey I was just thinking, pairing this add on up with this > http://docs.unity3d.com/ScriptReference/Animator.StartRecording.html
Could make an awesome replay system.
What do you think?

 

The recording system in Animator isn't exactly the same, it doesn't create an animation file (because of the same constraint that the GameObjectRecorder has: right now, it's impossible to create an animation at runtime), and it doesn't store exactly the same data (it's very animation and state machine related). But if you want to create a sands of time effect, maybe this recording system could be used. I don't know how much memory it can use.

That being said, we're working on integrating the GameObjectRecorder in the new FrameRecorder API. This API is a very generic one, allowing to standardize the different recording systems in Unity (record a video or an animation).

 

Very interesting, I'm trying to create an iOS app, and I need to animate a character at runtime, so the user can move every part of the character and record the movements and store or save on a ".anim" file, pretty similar you are talking about but, it is possible edit those animations on runtime and save or store again?

 

FrameRecorder with animation recorder (alpha) is out: https://github.com/Unity-Technologies/GenericFrameRecorder/releases/tag/0.2.0003

 

@jpatinop80 I'm sorry but it's not possible at the moment, I've explained a bit a the end of this comment: https://forum.unity.com/threads/fee...ameobjectrecorder-2017-1.467852/#post-3143626

 

Thanks mate

 

Did someone manage to bind and record Mesh.vertices?
I've been trying for days but I just can't get it to work.

 

It would be awesome if you could file a bug report
There are some edge cases that we haven't tested yet, and that one might be one of them.

 

Has anyone found a workaround for non-transform values only recording sporadic frames? Is it fixed in newer versions of unity? I'm on 2017.1.0p4

I'll try storing the data while recording and stuffing it into the clip after.