The "on" resistance of a MOSFET is a completely meaningless spec when you're using it in the linear region!!! It only comes into play when you are using it for a switching application, where the transistor is either fully off or fully on.
A linear power supply dissipates the undesired voltage as heat, plain and simple. 20V in, 5V out at 0.5A, you're sending 2.5W to the load, and the other 7.5W is dissipated in the transistor. No ifs ands or buts about it, that's how linear power supplies work.
Linear power supplies are relatively inefficient and dissipate a lot of heat. Some tricks can be used to reduce the amount of heat being dissipated, usually by switching between different input voltages (taps on transformer) depending on the desired output voltage, so you don't have as huge of a voltage drop across the transistor when outputting small voltages. Even with than, you're still not going to get anywhere near the efficiency of a purely switching power supply.
I would recommend looking up some educational electronics tutorials on linear power supply design, rather than trying to copy and modify a manufacturer's switching power supply design into a linear power supply without really knowing what you're doing. Case-in-point, the optocoupler is absolutely not necessary in your schematic! It was there in the original circuit to provide safety isolation in the feedback path between the primary (mains voltage) and secondary side of the switching supply.