Bug Third Person Camera Collider Jittery

Hello!
I have this problem with my camera. Whenever the camera is looking up and the character walks up a slope, the camera jitters.

I'm pretty sure I narrowed the problem down to the collider.

Heres the code for the Collider:

Code (CSharp):

1. using UnityEngine;
2.  
3. public class CameraCollision : MonoBehaviour {
4.  
5.     public float minDistance = 1.0f;
6.     public float maxDistance = 5.0f;
7.     public float smooth;
8.     public float distance;
9.  
10.     Vector3 dollyDir;
11.  
12.     public GameObject cameraBase;
13.     public FPCamController fpCamController;
14.  
15.     // Use this for initialization
16.     void Start() {
17.  
18.         distance = transform.localPosition.magnitude;
19.     }
20.  
21.     // Update is called once per frame
22.     void Update() {
23.         dollyDir = transform.localPosition.normalized;
24.  
25.         Vector3 desiredCameraPos = transform.parent.TransformPoint(dollyDir * maxDistance);
26.         RaycastHit hit;
27.  
28.         if (Physics.Linecast(cameraBase.transform.position, desiredCameraPos, out hit)) {
29.             distance = Mathf.Clamp(hit.distance * 0.87f, minDistance, maxDistance);
30.  
31.         } else {
32.             distance = maxDistance;
33.         }
34.  
35.         transform.localPosition = Vector3.Lerp(transform.localPosition, dollyDir * (distance - .25f), Time.deltaTime * smooth);
36.         print(transform.localPosition);
37.     }
38. }

Here's a video of the problem:

I borrowed some of this code from this video:

P.S. If some of the code is not the best, forgive me I'm new

 

It seems it stutters only at certain angle from the bottom up.