Indeed once you have determined the S21, you can simply unwrap the phase and differentiate to get the delay (most VNAs have that built-in). However, depending on the level of accuracy required and the tools available, calibration may not be trivial and people have come up with a lot of different ways to do this. I'll try to give an overview of some relevant methods and their limitations:
1. mechanical measurement
(assumes that mismatch is negligible and that the velocity factor is known)
Simple, measure the mechanical length l, divide by c * VF. This can in fact give excellent results for an air dielectric (e.g. with a precision air line). Unfortunately with a solid dielectric, the permittivity (and hence VF) generally varies due to process variation and also changes with temperature and humidity. Still, at the very least, this is a useful sanity check!
2. open one-port measurement
(assumes that mismatch and fringing effects are negligible,
requires 1-port VNA,
cal-kit with same connector as one side of the DUT)
Calibrate the VNA, connect one side of the adapter, leave the other open. Measure the S11 group delay (round-trip) and half it (and potentially correct for the desired calibration plane). The problem is that an open connector is not an ideal open but is usually somewhat capacitive so it looks longer than it actually is (and may even radiate at high frequencies). If you have a known open or short standard with the correct connector, you can connect that to the open end instead (and subtract its delay). You can also use a TDR instead of a VNA.
3. adapter removal calibration (1-port)
(assumes that adapter is reciprocal,
requires 1-port VNA,
cal-kits with i) same connector as one side and
ii) same connector type and opposite gender as the other side of the DUT)
Calibrate the VNA without the adapter, add the adapter, calibrate again (with the other cal-kit). The complete S-parameters of the adapter can be calculated from the two sets of error terms. A passive coaxial adapter will essentially always be reciprocal (S21 = S12).
4. two-port measurement
(requires a full 2-port VNA,
cal-kits with i) same connector as one side and
ii) same connector as the other side of DUT
iii) a reference adapter with same connectors as DUT)
This is in a way the most straightforward way and givev the full S-parameters of the adapter but unless you already have a precision reference adapter of known length, there is a chicken-and-egg problem.
5. adapter removal calibration (2-port)
(requires a full 2-port VNA,
cal-kits with i) same connector as one side
ii) same connector as other side of DUT
iii) and iv) opposite genders of i) and ii))
Two 2-port calibrations are performed, one with the adapter connected to port 1 (and cal-kits ii) and iv)) and one with the adapter connector to port 2 (and cal-kits i) and iii)). The full S-parameters of the DUT can be calculated from the error terms, similar to 3. and you can calculate a new set of error terms with the adapter removed, i.e. this gives you a full 2-port non-insertable calibration without a known thru-reference.
6. unkown-thru calibration
(requires a full 2-port VNA with 4 receivers,
cal-kits with i) same connector as one side and
ii) same connector as the other side of DUT)
This is certainly the most elegant method and, all other things being equal, probably the most accurate one as well. It works essentially like a standard 2-port SOLT calibration, except that you do not need to enter the length of the thru (our DUT). It will give you a full non-insertable 2-port calibration and the S-parameters of the adapter. Unfortunately, it is based on a different error model and only works with a 4 receiver VNA (you can actually do a second-tier calibration on a 3 receiver VNA to achieve something similar but this requires additional calibration steps and will produce larger uncertainties).
Some of the more advanced calibration methods are not available on all VNAs and you may need to download the raw S-parameters and do the calculations on a PC.