How to create a gizmo

I'm trying to create a gizmo for my level editor (those 3 little arrows that allow you to drag an object, sideways, back and forth, and up and down).

So far I've managed to keep the arrows allways visible with a shader, as shown in the first attached image. There you can see my Y and Z arrows are clearly visible through the boxes because they have the shader applied and my X arrow does not (this was done on purpose obviously).
It was kind of complicated to do this because I had no idea I was searching for a shader to get this function and took me a while to figure it out and find the shader.

Next I've made the gizmo stay the same size regardless of the distance from the main camera. This one wasn't really that complicated compared to the shader.

And now my current problem. Although the arrows are allways visible through objects, the clickable areas (box colliders that I use to detect if the user is using the gizmo) are not. So, if a user selects a box, rotates/adjusts the camera and another object is between the line of sight of the gizmo and the camera, it will block the mouse clicks.
On the second attached image you can see the colliders. If I try to click on an arrow that is overlaping the box, it will not move because the box will absorve the mouse click first.

Any ideas on how to do this?
Should I change my aproach entirely or am I almost there?

 

Maybe, if you're using raycast, you could put the colliders on a different layer, and by doing ~(0 >> yourLayerId) inside it, you can ignore every collider, except for the colliders on that layer
(BTW, I'm not sure in the method's name, but you can get the id of a layer with something like IdFromLayerName or something, and I'm not sure in the direction of the > marks and the order too, so you should look it up, yeah, sorry for that)

 

I'm currently not using raycasts but OnMouseDown. But I guess I can change it up, if I find a way to raycast a single layer

 

Like the way I described?

 

If I understood you correctly, Yes

 

why didn't you just use the Handle class? unless of course this is something that you want the player to be able to do, the handles class takes care of all of this for you.

 

Is it this class? http://docs.unity3d.com/ScriptReference/Handles.html
If it is, it says there it only works on the scene view.
I need it on the game view