Whyyy??
Why did you put additional gain, of uncertain magnitude, after the op-amp?
A huge amount of it, no less. Like 40dB excess! It's guaranteed to oscillate that way.
I will tell you a secret. Gather 'round...
Checklist:
- Bounded inputs, bounded outputs (for all stages)
- Linear transfer function, bounds coincide on the curve
- Loop gain is in the stable range
- Phase shift doesn't blow out the op-amp's phase margin
You can tell gain is in the stable range, if the gain outside of the op-amp (in the loop from its output to its feedback input) is less than or equal to the reciprocal of its maximum stable gain. (Which for most, is a gain of 1, i.e., "unity gain stable". Some are "decompensated" and only stable at higher gains, or with compensation capacitors added to other pins. The datasheet will tell.)
The above circuit fails because:
Bounded inputs
------------------
Op-amp inputs: has ESD protection diodes, and inputs are bounded voltage, or limited current (in the worst case, a limited current input will be clamped by the input protection diodes). Good.
Gain stage (Q1): is driven by op-amp (bounded output). Good.
Output stage (Q2): Unbounded. Bad.
Bounded outputs
--------------------
Op-amp output: 0 to 5V. Limited current capacity (10s of mA?). Good.
Gain stage (Q1): 0 to +12V. Not very good: greatly exceeds the input range of the following stage, and if +12V isn't limited elsewhere, neither will this be. Unlimited current capacity: 100s of mA. 2N7000 is quite a beefy little MOSFET. Bad.
Output stage: voltage is suitable for purpose (0 to +V). Current: unlimited (10s of A?). Bad.
Transfer functions
-------------------
Op-amp: only active in a narrow range (a diference of some mV), but that's okay, because we want all the loop gain centered here. Good.
Gain stage (Q1): Nonlinear. Offset (Vgs(th) not accounted for). Saturates well before the full available range. Uses maybe 1.0 to 1.5V out of a 0-5V range. Exponential to quadratic curve, not linear (gain varies with quiescent point!). Bad.
Output stage (Q2): Nonlinear. Offset (Vgs(th) not accounted for). Unlimited output current. Turns on around -1V (relative to +V) and keeps going up from there, until 5-10V range where it's either saturated (Rds(on) resistive region -- in dropout) or at maximum amps, short circuit (10-30A?). Gain varies proportional to load current. Bad.
Loop gain
------------
Who knows? The voltage gain from Q1 may be 10's, and the same for Q2. In the threshold to subthreshold region, where both are just barely turning on, it might even be less than 1.
The wildly varying loop gain is a sure sign of having done something wrong.
Phase shift
------------
Again, who knows? Q1 at least isn't driving much of a load. Q2's gate won't be the tiniest thing ever, but it won't dominate the loop response. It may be relevant towards the cutoff frequency though.
Q2 isn't driving a capacitive load, which is unusual, and probably not a good thing: having little load (especially at high frequency) means the loop gain is very high (especially at high frequencies). So you're also in a bit of a worst-case situation, for the load shown.
Ideally, the phase shift across the transistors should be small (or, put another way, their frequency response as amplifiers should be higher than the op-amp's fT), which shouldn't be hard to arrange.
So, how to fix it?
--------------------
By now, you should be able to guess the most obvious help: reduce the gain of Q1 and Q2, so that they more fully utilize the voltage ranges available.
Simplest solution: add a source degeneration resistor to Q1. This reduces the voltage gain and increases the input voltage range. It doesn't remove the offset, but it makes it less important. (Changing to a BJT like 2N3904 would reduce the offset a bit, and also reduce capacitance.)
The same can be done for Q2, but this worsens dropout at high currents. But a conceptual restructuring is in order, for that one.
Suppose you have a nice big capacitor on the output (which is typical for most applications!). Outside of the resistive range, Q2 is only ever a transconductance amplifier (output current dependent on gate voltage). It would be nice to stabilize its gain, so it is linear, and so its output current can be bounded responsibly.
A current source into a capacitor makes a single pole filter. If we close the loop around that, with the op-amp being a second pole, we get an easily-solved second order system, which can be optimized for step response quite easily.
The challenge, then, is making Q2 have a nice, flat, limited operating range, without compromising its resistance.
Backing up a level -- once you make a clean, linear current source block, all you need to do is level-shift the error amplifier's output to meet the input range of the current source, providing little or no gain in the process. Simply match up the output and input bounds.
Or converting the error amp's output to a constant current sink, if your current-source block works out better as a current amplifier or mirror (current in --> current out). Bounding the input of the current source necessarily gives you a fixed and accurate current limit, thus solving the problem of short-circuit protection, too!
Tim