Search Unity

  1. We are migrating the Unity Forums to Unity Discussions. On July 12, the Unity Forums will become read-only.

    Please, do not make any changes to your username or email addresses at id.unity.com during this transition time.

    It's still possible to reply to existing private message conversations during the migration, but any new replies you post will be missing after the main migration is complete. We'll do our best to migrate these messages in a follow-up step.

    On July 15, Unity Discussions will become read-only until July 18, when the new design and the migrated forum contents will go live.


    Read our full announcement for more information and let us know if you have any questions.

[StreamingImageSequence] Create clips with scripts

Discussion in 'Virtual Production' started by GeniusKoala, Aug 3, 2021.

  1. GeniusKoala

    GeniusKoala

    Joined:
    Oct 13, 2017
    Posts:
    97
    Hi,

    We are using the package "SteamingImageSequence" in our TV shows to insert our 3D characters on a live environnement with an image sequences.

    We are using the 0.3.3 version since it's the only one working correctly. Newer versions make the image sequence blinking (dark blinks) and not useable for production. The 0.3.3 was a version we used in a previous project in Unity 2019.4 but we were not creating the clip with a script. We are using Unity 2021.1.16f1 with HDRP11.

    For this new TV show I need to access the API to create clips but I don't find how to do it correctly.

    I know how to create other type of clips like a recorderclip, audioclip. My code to create the clip :

    Code (CSharp):
    1. StreamingImageSequenceRenderer streamingImageSequenceRenderer = FindObjectOfType<StreamingImageSequenceRenderer>();
    2. TimelineClip clip = track.CreateClip<StreamingImageSequencePlayableAsset>();
    3. StreamingImageSequencePlayableAsset asset = (StreamingImageSequencePlayableAsset)clip.asset;
    4. //What can I do with this please?
    5. timeline_director.SetGenericBinding(track, streamingImageSequenceRenderer);

    The result in my timeline:

    upload_2021-8-3_18-43-57.png

    The Console :

    upload_2021-8-3_18-44-52.png

    Please I need help with this as this is urgent for us...

    EDIT: I succeed by exposing all classes and API needed. I take example on the StreamingImageSequenceTest.cs to proceed.

    By the way, is there a way to make the play smoother ? We read images from our server (high speed connection). We do not copy images in our project since it adds more works and duplicates data. For example we also read alembic for our server and it works perfect! We tried with very low resolution images PNG but still it has hard to read the sequence. Is there a good way to make it read a sequence on an external source (not in the Unity project) ?

    Thank you,
     
    Last edited: Aug 3, 2021
  2. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    49
    Version 0.3.3-preview is quite old. There have been quite a lot of new features, bug fixes, and optimizations since that version, so if possible, I'd recommend you to try the latest version.

    Can you please elaborate about this problem, and perhaps let me know how I can reproduce it ?
    If you have a test project, then it will be even more helpful.

    StreamingImageSequence supports playing image files outside of the Unity project.
    For higher performance, you can use a symbolic link inside the Unity project, which refers to your external folder.

    I'd suggest you to take a look at the documentation for folder trade-offs too, but please note that it's only applicable for version 0.9.x and up.
     
    GeniusKoala likes this.
  3. GeniusKoala

    GeniusKoala

    Joined:
    Oct 13, 2017
    Posts:
    97

    Hi!

    I updated to 0.11. I still have randomly black screens (maybe because it needs first to be cached?) but also and it was already like that with 0.33, the last frame of the clip is included contrary to what VCam Can you confirm that? If I want 2 clips sticked one to the other I need to let 1 frame space between otherwise the last frame of the first one override the first frame of the second clip. At this frame, I would need the second SIS clip to be displayed but it's still the first one. You can see the space on my screenshot and the cut of the cam at the bottom is synced with the first frame of the second SIS clip.

    upload_2021-8-12_19-14-52.png

    I am working in 25FPS. The first clip in 1 img and the second one a sequence of 102 images. The first image is 6000x3376 and the sequence 1920x1080. Do you have any hint? Also during the recording there is still random black screen (image not loaded I guess). How could I make sure the image load before the record export? We need to export image sequence and MP4 videos.

    We saw in another thread (not linked to SIS) that the timeline does not step correclty at 25FPS so at the recording I need to shift every clip to 1 frame because otherwise everything is 1 frame late. Do you think 25FPS has an impact while reading the clip in Play mode (needed to use recorder clips)?

    We just need the images to be sure to be loaded before the recorder clip write on the RenderTexture please.

    EDIT : Also what is "CPU mem. usage of tex in Runtime" in the tradeofs? Does it mean we can't play all the images of the sequence if our images are out of the projet?


    Thank you!
     

    Attached Files:

  4. GeniusKoala

    GeniusKoala

    Joined:
    Oct 13, 2017
    Posts:
    97
    @sindharta_at_unity Please it's pretty urgent for us sorry to insist... I debug my problem for the blacks screens. The images can't load smoothly (not a surprise I don't ask for them to load instantly). I found that every black screen returns the code -1 "READ_STATUS_UNAVAILABLE". What does it mean?


    upload_2021-8-12_21-8-55.png

    Is there a way to loop until the loading is a success ? I don't need to load it instantly, it's ok if it needs 5-6 frames, maybe even 10 or 20. I just need the full sequence exported in a sequence images (PNG) and a video (MP4).

    Plus, I guess it uses a cache when playing the SIS in the timeline. I think the cache is cleared once I enter Playmode. A fix could be to not clear that cache since I might assume it is loaded correctly before by hand. BUT this is not the ideal solution and it's just a backup solution.

    The ideal solution is to freeze the scene until the image of the SIS clip is succeedly loaded.

    EDIT : Is there a way to preload all SIS clips ? I would be ok to wait the cache is written and then start the recording. I ask this if it's not possible to make the image loading blocking the timeline. I could forward in the timeline myself only if the current image is loaded but how to know when it's loaded?

    Thanks for your help!
     
    Last edited: Aug 12, 2021
  5. GeniusKoala

    GeniusKoala

    Joined:
    Oct 13, 2017
    Posts:
    97
    Plus the mixer is not working. Sometimes we have 2 tracks in parallel so that the second one override the first one.

    It's working in scrub edit mode, but not when it's playing in Play mode while recording (timeline still). In recording sometimes the first track is still displayed… that’s an urgence for us now if it’s not working.

    upload_2021-8-12_22-18-39.png

    The first track is an image sequence, the second one just a single image. Both are out of the projet.