Question Retaining cam rotation when removing the free look target

Hello,
I was not able to find a solution by myself so far, nor was I able to find a similar topic, so here is my current issue:
In a cinemachineVirtualCam I have a follow target and the Aim is set to composer, without a LookAt target most of the time, so that the composer is not doing anyting. At some points in the game I am adding a lookAt target and the composer works as inteded. However the issue is that when I remove the lookAt target, the camera changes its rotation - thats the thing - I'm not sure where its being inherited from - I tried overwriting the rotation values of the follow target, the vcam, but its possible that a cached rotation is applied again after removing the LookAtTraget. I want the 3rd person follow from the Follow target to take over from the LookAt target when its removed, but keep the last direction of the lookat target at the time of removing it.
Not sure if that explains the issue, so it there is anything I can add just say the word!
Thanks!