Best way to apply collision to tilemaps.

Im running into an issue where its difficult to apply collision to tilemaps in an efficient manner.

Image using a tilemap to draw a top down 2d house. I want to apply collision to the house but also leave a spot open for the character to walk through the door for instance. I could do these things:

• Apply colliders and make a composite collider out of all of them.
• I can use a tilemap collider.

The issue with using a tilemap collider is that all tiles are used. Using a tilemap collider also applies innificient colliders to the inside of the object, something that would never be touched.
The issue with composite colliders are that they take a lot of time to produce.

would the best method be using a series of tilemaps as layers?