cgroups and others are the kernel virtualisation subsystems behind it.
If you are already familiar with VMs and with chroot, docker containers are somewhere in the middle. Rather than just the FS being sandboxed a lot of kernel structures are sandboxed too. From the containers point of view anyway. You can see in, but it will struggle to see out. It's not impossible and there are exploits to gain some access beyond the container boundary, but these are usually easily addressed if of concern by forcing all containers to run as "nobody" and use other infra to manage it's actual access. If you don't care, you can just run them as root.
There are alternatives to Docker. The container spec is published independant of docker-io. Obviously it's really what the kernel supports and it's instrumentation. A popular alternative is "Linux Containers" or LXCs. They tend to be used slightly differently to docker, but are much the same. They tend to be more "full on distro" installs. Like for instance, you can spin up a container and install Ubuntu onto it directly. In that they LXCs expect a full boot root setup. Dockers tend to be more a "single process wrapper". LXCs are more heavy weight and actually "mount" or "bind" a rootfs, not just an ephemeral cache one like docker. Note a full distroVM will need 2Gb of RAM or more allocated to it 100% of the time. An LXC running the same distro will probably only use a few hundred meg.
Typically....
Use a docker style container when you want to run an isolated process.
Use an LXC style container when you want to run a monolith "trad" server.
Use a VM when you need actual "virtual" or "real" hardware access.
There is debate on nesting. You can nest VMs/Containers etc. within some limitations. I for instance have LXCs which run docker. VMs will run LXCs and dockers. It tends not to work so well the other way up.
Where a VM is still better:
In my setup I have several "micro-VMs". Little 1Gb or 2Gb or RAM things. In several cases this was choosen because of simple things like USB. While "dev files" map nicely into docker and LXCs some USB devices are a bit more awkward. Things like multi-channel USB devices which don't present their endpoints until you enable them, then they are dynamically named by the driver. These are a nightmare. Not least the dynamic naming, which you can get around, but that the device proxying doesn't fully pass the mapped privelges, so you get "Permission Denined" on access. In the end on these systems I went back to a VM.
The other place I used VMs was in the "storage" layer. Containers, if they are to use any real userId or groupId have to be mapped via a file in /etc. Like all container uids start at 10,000 or something and you need to map VuserIds to realuserIds. For things like "NFS" which uses "client side id" verbatim this causes many problems that I didn't want to solve. So NFS server got changed to a VM with real user Ids.
Also, almost all modern OSes support VMs out of the box without VMWare or other software. Linux the subsystem is called KVM. The kernel virtual machine. I believe qemu is a front end for KVM. Proxmox also. Check the opensource projects for one that fits what you need.
For Windows it's "Enable HyperV" in settings and run HyperVManager. For "docker" it's a bit more complex. You can use plain old docker but it has issues in places. "Docker desktop" which is a commercial "with community edition" application but supports more features. Or... you can isntall "WSL" and just install docker there. Again there are privelege limitations.