The behaviour where it switches from the having panels nicely tabbed to having them all stacked and visible at once has been driving me nuts for years! I've been tempted to roll back from AD21 to Protel for Windows 3.0!
Altium's online help on the subject has been somewhat lacking.
Fortunately a YouTuber recently mentioned holding the SHIFT key down while clicking and dragging the title bar of one panel out of the group.
Drag it out near the centre of the screen, avoiding the arrow icons for docking to screen edges.
Then try dragging the panel back to the center of a screen edge and have it restore the Tabbed panel layout.
To add further confusion, after you drag a panel to the centre, the active display view may flip to another mode or file (eg if editing a Library, it may flip to shematic editor view). This will make some panels like the SCH Library panel disappear from view until you switch back to your library file, etc.
I haven't figured out how to force a tabbed group of panels to change into a merged group, but then I have never found a need to do this!