Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question ARFoundation apply AR VFX on pre-recorded video and AR session playback questions

Discussion in 'AR' started by Renzzauw, Jul 29, 2022.

  1. Renzzauw

    Renzzauw

    Joined:
    Oct 7, 2015
    Posts:
    3
    Hello!

    I'm researching into ARFoundation's capabilities to see if it is possible to create an application that is able to overlay AR VFX over pre-recorded videos on a mobile device and export the result. I'd prefer to use a game engine like Unity as it provides great 2D & VFX design tools as well as AR features.

    As an example, a user should be able to select a video of a football match and I would like to overlay an effect on top of the football (e.g. using the VFX graph or 2D sprites) and generate a new video + VFX.

    Based on my research goal, I have the following questions:
    1. Is it possible to use a pre-recorded video stored on the device as input, instead of running an AR experience in real-time using the device's camera(s)?
    2. Is it possible to store or record AR experiences/sessions and play them back, similar to ARCore's Recording & Playback API (or does ARFoundation support this API in some way)? If so, can the AR session be exported as a video in some way?
    Thanks in advance!
     
    Last edited: Jul 29, 2022
  2. ankur-unity

    ankur-unity

    Unity Technologies

    Joined:
    Aug 12, 2021
    Posts:
    34
    KyryloKuzyk likes this.
  3. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,123
    This is currently not supported not by ARCore, nor by ARKit. And by extension, this is not supported by AR Foundation.

    As far as I understand it, the 'ARCore Session Recording & Playback' feature can't play back an arbitrary video file. It can only play back a video file that was previously recorded with 'ARCore Session Recording & Playback API'. The reason for this is that ARCore saves necessary metadata along the video file.

    Also worth mentioning that ARCore session playback is not deterministic. That is, one session playback may slightly differ from the other in terms of detected planes.
     
  4. Renzzauw

    Renzzauw

    Joined:
    Oct 7, 2015
    Posts:
    3
    Thanks for the replies, these are very insightful!

    Happy to hear ARFoundation supports ARCore session recording & playback, but a shame to hear you cannot directly input an arbitrary video file.

    Would it still technically be possible to create a video recording including the required metadata myself and then feeding this to ARFoundation/ARCore's playback feature? We implemented a custom camera video recording feature in our app that is going to use Unity, I think we can perhaps obtain the relevant information by reading the device sensor data while also recording the footage.