Question Highpass using Sample Texture 2D LOD

I am doing a highpass by sampling a texture twice, the second time using a LOD sampler set to a higher LOD value and then subtracting the blurry LOD level (approx. the average color) from the regular texture.
I then add a new color, replacing the old average color.

I find that this approach causes sharply increased contrast in the diffuse color as well as wrong colors in the result, does anyone have an idea how to improve this?

The application for this technique is to replace the average color on a texture within a shader (without editing the texture in an image editor).