Clamping tilemaps to screen (without moving camera)?

Hello lovely Unity forum community!
Got a bit of a curveball today that I hope someone might be able to help my team & I with...

Our 2D text adventure project is composed almost entirely out of UI elements, and so there hasn't been any need to move our 2D orthographic camera around for anything at all. But after adding a map system into the game, we decided on using Tilemaps to visually represent it, and then add the ability to pan/zoom the tilemap itself inside of the UI Canvas, over the top of the rest of the UI.

This presented many challenges, the main one being that we can't actually use the main camera for panning and zooming tilemaps (they live in a screen space canvas)... Meaning we would have to do this on the actual tilemaps themselves.

Here is a simplified version of our hierarchy:

Spoiler: Hierarchy

After adding panning/zooming functionality, we also decided it would be best to keep the map within the boundaries of the screen, even when zooming in or out.

Here's how we're handling input:

Spoiler: InputUtils.cs (a singleton class attached to EventSystem)

Code (CSharp):

1. void HandlePan(float panSpeed) {
2.     Vector3 newPanPos = Input.mousePosition;
3.     Vector3 offset = Camera.main.ScreenToViewportPoint(prevPanPos - newPanPos);
4.     Vector3 panDelta = new Vector3(offset.x * panSpeed, offset.y * panSpeed);
5.  
6.     if(move.magnitude > 0.01f) {
7.         if(onPan != null && onPan.Target != null)
8.             onPan.Invoke(panDelta); // Broadcast this event to listeners
9.     }
10.  
11.     prevPanPos = newPanPos; // Store this most recent input to check against it later
12. }
13.  
14. void HandleZoom(float zoomSpeed) {
15.     float offset = Input.GetAxis("Mouse ScrollWheel");
16.     if (offset == 0 || onZoom == null || onZoom.Target == null)
17.         return;
18.  
19.     onZoom.Invoke(offset * zoomSpeed); // Broadcast this event to listeners
20. }

And here's how we're handling panning & zooming:

Spoiler: Areamap.cs (attached to MapPanel)

Code (CSharp):

1. private void Start()
2. {
3.     mapPanel = transform; // This is only here for the sake of clarity
4.     // Called from the InputUtils singleton when the user pans the tilemap
5.     InputUtils.onPan += (Vector3 delta) => {
6.  
7.         // Here, we would need to calculate panBoundsMin & panBoundsMax
8.         //     to keep the map within screen bounds
9.         // This works, but how to calculate panBoundsMin & panBoundsMax,
10.         //     while also taking mapPanel.localScale into account?
11.  
12.         delta.x = Mathf.Clamp(
13.             floorMap.transform.localPosition.x + delta.x,
14.             panBoundsMin.x, panBoundsMax.x
15.         );
16.         delta.y = Mathf.Clamp(
17.             floorMap.transform.localPosition.y + delta.y,
18.             panBoundsMin.y, panBoundsMax.y
19.         );
20.      
21.         // These lines perform the actual tilemap panning
22.         floorMap.transform.localPosition = delta;
23.         overlayMap.transform.localPosition = delta;
24.     };
25.  
26.     // Called from the InputUtils singleton when the user zooms in/out on the tilemap
27.     InputUtils.onZoom += (float delta) => {
28.         zoomValue += delta * zoomScalar; // zoomValue and zoomScalar are private variables
29.         zoomValue = Mathf.Clamp(delta, minZoom, maxZoom); // min & maxZoom are public vars
30.         if(mapPanel != null) {
31.             var newScale = new Vector3(zoomValue, zoomValue, 1);
32.             mapPanel.localScale = newScale; // >> This performs the actual tilemap zoom <<
33.          
34.             // Here, we would need to pan the map back onto the screen,
35.             //     or un-zoom the map if it shows any part outside of map bounds,
36.             //     or simply prevent zooming if we can pre-calculate such an event
37.          
38.         }
39.     };
40. }

How can we calculate panBoundsMin and panBoundsMax (while taking mapPanel.localScale into account) to clamp the map onto the screen while panning/zooming? OR how else can we go about this?