as for gate voltage after R17 (100ohm), it is 1.1v when no load and gets to 10v when 1.6ohm.
This doesn't sound right. It should be between 4.3v to 6.3v max depending on load. Never 1.1v, and only going to 10v if the mosfet cant drive enough current or it is not a logic level mosfet.
The FET has a very low threshold. It will pass amps of current with just 2V Vgs. So in regulation the Vg should never be more than 5.8V. 10V certainly has the part turned on fully. What is the output voltage with the 1.6 ohm load?
What is V_FB1 with the 1.6 ohm load?
What is the voltage on both sides of R17?
You can probably do some debugging without the 1.6 ohm load. i see you have a 100 ohm resistor on the output, R22, which is enough to not be looking at zero current measurements. The output voltage is a bit high, 3.48V. It should be the same as the reference voltage (3.361V) with a very small margin. When debugging, keep it simple. Try figuring out why the output voltage is high with no added load for starters. The output voltage should depend on nothing other than the reference voltage and the operation of the op amp and FET.
V_opamp_supply: 11.98v
V_3.3_after_7805: 3.361v
V_output3.3v: 3.481v
To find out why the output is so high when the reference is good, measure all the points in the control loop and the power input to the FET, pin 2, the drain. Please present those measurements clearly in a table like you did below. That is everything involved in setting the output voltage and should be enough to understand where a problem might be. Essentially, we need the measurements of pins 1, 2 and 3 on both the FET and the op amp, no load and with load. While you are at it, measure the power and ground pins on the op amp, not the traces, be sure to measure on the pins directly.
I don't understand the AC measurement. Are you saying the voltage outputs are bouncing between half the expected voltage and ground???
ac measurement: keeps doing 0v and 2v each half second on 5v rail, and about 1.4v with 0v on 3.3v rail
I have no idea what this is saying. If it is literally true, how can you present these other measurements??? A problem like this will require opening the feedback loop by removing R19 and taking measurements.