Sorry just realised that the lm317 isnt an opamp but i still like that idea. Was thinking of a different opamp (LM358).
The LM311 is a comparator.
The data sheet for the LM358 op-amp states "• Input Common-Mode Voltage Range Includes Ground"
You can learn things from data sheets.
It is a dual op-amp, so you will need to choose one half and use the appropriate pins.
With the different op-amp, the oscillation problem could come back, so check with your oscilloscope again.
The circuit really should have compensation added to slow it down and improve its stability.