The current sensing part looks complicated. It also seems like the current sense is at the supply side, before the linear regulator, so it will not help in stabilizing the output stage. So one might need something like a 0.1 Ohms resistor between the output stage and the output capacitors. So one might as well have the shunt behind the the output stage (usually low side) instead.
For the capacitors at the output, it is hard to get away without it, but there is a chance to use less output capacitance. The faster the regulator stage, the smaller the caps can be. Something like 2 µF plus another 1 µF in series with 0.1 to 0.5 Ohms might be realistic. To a certain degree the output capacitance (especially the low series resistor part) is a compromise in on how much overshoot / drop on load changes is acceptable. To get a supply with essentially no capacitance a negative supply and than a class AB like output stage would be the obvious path.
However there is also a kind of additional effective capacitance from the constant current loop - depending on the speed of that loop, this can add a simulated capacitance in the µF range. As far as I can tell the current loop as shown is very slow and thus the simulated capacitance could reach the mF range. In addition the slow current limit can in case of a sudden short add an extra current spike before current limiting sets in. This can be as bad as damaging the output.
To get lower in voltage a different OP with a more powerful output stage might be an option (e.g. OPA171).