Question ARFoundation apply AR VFX on pre-recorded video and AR session playback questions

Hello!

I'm researching into ARFoundation's capabilities to see if it is possible to create an application that is able to overlay AR VFX over pre-recorded videos on a mobile device and export the result. I'd prefer to use a game engine like Unity as it provides great 2D & VFX design tools as well as AR features.

As an example, a user should be able to select a video of a football match and I would like to overlay an effect on top of the football (e.g. using the VFX graph or 2D sprites) and generate a new video + VFX.

Based on my research goal, I have the following questions:

1. Is it possible to use a pre-recorded video stored on the device as input, instead of running an AR experience in real-time using the device's camera(s)?
2. Is it possible to store or record AR experiences/sessions and play them back, similar to ARCore's Recording & Playback API (or does ARFoundation support this API in some way)? If so, can the AR session be exported as a video in some way?

Thanks in advance!

 

This is currently not supported not by ARCore, nor by ARKit. And by extension, this is not supported by AR Foundation.

As far as I understand it, the 'ARCore Session Recording & Playback' feature can't play back an arbitrary video file. It can only play back a video file that was previously recorded with 'ARCore Session Recording & Playback API'. The reason for this is that ARCore saves necessary metadata along the video file.

Also worth mentioning that ARCore session playback is not deterministic. That is, one session playback may slightly differ from the other in terms of detected planes.

 

Thanks for the replies, these are very insightful!

Happy to hear ARFoundation supports ARCore session recording & playback, but a shame to hear you cannot directly input an arbitrary video file.

Would it still technically be possible to create a video recording including the required metadata myself and then feeding this to ARFoundation/ARCore's playback feature? We implemented a custom camera video recording feature in our app that is going to use Unity, I think we can perhaps obtain the relevant information by reading the device sensor data while also recording the footage.