It is difficult I agree that's why I'm trying to flush out the exact details so we can consider implementing it for you and others who have the same concern.
Currently you can 'disconnect' from the camera focus target by using WASD. The camera will move left/right/up/down and no longer follow the unit it may have been attached to.
1. Is it correct that you would want the camera to not change left/right/up/down when you zoom? (i.e. its always looking at the exact same position just from different distances?
2. Clicking to attach the camera to a unit won't work because clicking a unit 'selects' a unit and having the camera change when you are selecting units to give them orders would be disorienting when you are dealing with a lot of stuff in quick succession or issuing orders from other sides of galaxy. Double clicking currently attaches your camera focus to a unit. Is that acceptable?
3. There are some scenarios where "keep looking at where I was unless I choose to move it" may not be desired. Consider this example: you are looking at some empty space around a planet. The planet is moving (it orbits the star). Should the camera continue looking at that empty space or should we break your rule (don't change my view without my permission) and have the camera move with the planet in order to keep looking at that empty space relative to the planet? If not, imagine, trying to place structures while the planet keep shifting out of view of the camera (they move incredibly fast).
I have more to dive into but I will wait to get some clarification / detail on those 3 points first.