Possible to click a button without selecting it?

I have multiple selectables and want to manipulate them by clicking a button. Like deleting one of them when I click the Delete button.

But when I click a button the selected selectable receives an OnDeselect.

Is it possible to change this behavior? Making a button just clickable without selecting it?

 

This was only meant for a first rough version so I can implement Undo/Redo. Changed it out of necessity and now it looks like it was planned this way from the beginning.

The selectable object now shows a local toolbar with the delete button. The toolbar gets hidden in OnDeselect, which hides the button before the click events kicks in. To solve this the toolbar gets hidden x seconds (0.25f at the moment) after the OnDeselect. Enough time for the button to do its work.

Effect: Now the user has a more dominant visual clue about what is selected and a visual clue about the change of selected objects because you see the former selected object for x seconds still having the toolbar.