Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

Showcase Creating a Bond-esque Music Video with Visual Compositor

Discussion in 'Virtual Production' started by simonwittber, Sep 26, 2023.

  1. simonwittber

    simonwittber

    Joined:
    May 13, 2008
    Posts:
    228
    Ever wondered how to create a stunning visual effect reminiscent of a James Bond movie? In this post, we'll delve into the process of using Visual Compositor to make a stylized music video.



    In November 2022, I started a project to produce a video for a short instrumental composition. As I was involved in building a video compositor product in Unity, it was a perfect opportunity to demonstrate the power of this tool.

    ## Step 1. Get something interesting on the screen.

    The basic visuals of the scene were created using two character models animated with data from the CMU animation archive. I slowed the animations down to match the song's tempo and placed the models in the scene, directly in front of the camera.

    ## Step 2. Give it some style.

    The models were transformed into silhouettes by utilizing the matte output port of the rendering node. The silhouettes were then blurred and subtracted from the original matte texture, turning them black. Due to the blurred texture adding extra pixel data around the edges of the silhouette, the outlines shifted closer to a white color, creating a sleek outline effect.

    These outlines silhouettes are then fed into a motion blur node. The motion blur node works like an accumulator buffer, which interpolates the last frame towards the new frame. It is a simple motion blur which is quite effective with black and white images.

    ## Step 3. Add visuals which match the audio.

    The audio visualisation at the bottom of the screen is created by using a custom visual compositor node. The node reads a FFT of the audio stream, and plots it across the bottom of the render texture using a shader. This is then composited on top of a still, public domain photograph of a giant, white stetson hat.

    The Really Simple Audio Spectrum Shader

    Code (CSharp):
    1.  
    2.  
    3.    Shader "AudioSpectrum"
    4.    {
    5.        Properties
    6.        {
    7.            _Color ("Color", Color) = (0,0,0,1)
    8.        }
    9.  
    10.        CGINCLUDE
    11.  
    12.        #include "UnityCG.cginc"
    13.      
    14.        sampler2D _MainTex;
    15.      
    16.        float _AudioSpectrum[1023];
    17.        float _SpectrumCenter, _SpectrumScale;
    18.        int _BucketCount = 1023;
    19.        float4 _Color;
    20.  
    21.        struct appdata
    22.        {
    23.            float4 vertex : POSITION;
    24.            float2 uv : TEXCOORD0;
    25.        };
    26.  
    27.        struct v2f
    28.        {
    29.            float2 uv : TEXCOORD0;
    30.            float2 uv2 : TEXCOORD1;
    31.            float4 vertex : SV_POSITION;
    32.        };
    33.  
    34.        v2f vert (appdata v)
    35.        {
    36.            v2f o;
    37.            o.vertex = UnityObjectToClipPos(v.vertex);
    38.            o.uv = v.uv;
    39.            o.uv2 = v.uv;
    40.            return o;
    41.        }
    42.      
    43.        float4 frag(v2f i): SV_Target
    44.        {
    45.            float s = _AudioSpectrum[i.uv.x*(_BucketCount-1)];
    46.            return abs(i.uv.y-_SpectrumCenter) < s*_SpectrumScale ? _Color:0;
    47.        }
    48.  
    49.        ENDCG
    50.      
    51.        Subshader
    52.        {
    53.            Pass
    54.            {
    55.                ZTest  Always
    56.                Cull   Off
    57.                ZWrite Off
    58.                CGPROGRAM
    59.                #pragma vertex   vert
    60.                #pragma fragment frag
    61.                #pragma target   5.0
    62.                ENDCG
    63.            }
    64.        }
    65.    }

    # Wrap it all up.

    Finally, the differet layers are added together to create the final frame. As the video plays (and is recorded), controls within the compositor graph nodes can be adjusted to create interest and emphasize different parts of the soundtrack. It essentially becomes a visual performance tool.

    The end result is greater than the sum of its parts. Visual Compositor is a great tool for creating unique, interesting visuals.

    Visual Compositor: https://docs.unity3d.com/Packages/com.unity.visual-compositor@0.30/manual/index.html
    Code Repository: https://github.com/simonwittber/music-video
     
  2. newguy123

    newguy123

    Joined:
    Aug 22, 2018
    Posts:
    1,265
    Looks great! Excellent effect.
    I've been meaning to try out Visual Composer
     
  3. shyamarama

    shyamarama

    Joined:
    Dec 13, 2019
    Posts:
    12
    Very cool, thanks for sharing! Would love to see more about the Visual Compositor, and the anime toolbox in general.
     
  4. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    49