Panel wont become active

Hi all,
I am trying to set a panel to active when raycast hits an interactable yet it only works for my light switch interactable and not the door no matter what I do. Here is the script:

Code (CSharp):

1.  public class Tofu_PlayerInteraction : MonoBehaviour
2.     {
3.         public Camera playerCam;
4.         public float range = 20f;
5.         public GameObject Panel;
6.         // Use this for initialization
7.         void Start()
8.         {
9.  
10.         }
11.  
12.         // Update is called once per frame
13.         void Update()
14.         {
15.             DetectInteractable();
16.         }
17.  
18.         void DetectInteractable()
19.         {
20.             RaycastHit hit;
21.             if (Physics.Raycast(playerCam.transform.position, playerCam.transform.forward, out hit, range))
22.             {
23.                 Debug.Log(hit.transform.name);
24.                 if (hit.collider.tag.Equals("Door"))
25.                 {
26.                     Panel.gameObject.SetActive(true);
27.                     Debug.Log("Setting Panel");
28.                     if (Input.GetMouseButtonDown(0))
29.  
30.                     {
31.                         hit.collider.GetComponent<Tofu_DoorInteract>().OpenDoor();
32.                        
33.                     }
34.                 }
35.  
36.                 if (hit.collider.tag.Equals("LightSwitch"))
37.                 {
38.                     Panel.gameObject.SetActive(true);
39.                     if (Input.GetMouseButtonDown(0))
40.  
41.                     {
42.                         hit.collider.GetComponent<Tofu_LightSwitchInteract>().Switch();
43.                     }
44.                 }
45.  
46.                 else
47.                 {
48.                     Panel.gameObject.SetActive(false);
49.                  
50.                 }
51.             }
52.            
53.         }
54.  
55.     }

 

The panel is just a simple dot on the screen

 

And btw the raycast works, after pressing down it executes the function opendoor and the door opens. I'm super confused. Also I did some testing and if I start with the panel on then when hovering over the door it will set it off. But where in the script did I ask it to turn of the panel object? its only supposed to turn off when it doesn't hit an object with the 2 tags.