For the start, use algorithm that I'm using right now in my projects involving SMPS
Maybe it's not very correct, but until now got me succesful flyback designs
1.Decide with output power (what you've already done)
2.Decide with operating frequency (lower frequency - bigger core, higher frequency - higher core, winding and switching losses. As for me, in my last design I used 115kHz for 20ish watt flyback PSU without using any heatsinks)
3.BUT, in the case when you want to buy assembled flyback transformer, you should adjust frequency as stated in its datasheet.
Deciding with core size and its air gap - it is rather complicated, because it will almost definitely require iterations of calculation
With some given core, you get desired primary inductance, if according to calculations during full power output core saturates -> make air gap bigger -> get more turns to achieve its L -> find out if windings will actually fit in core window, if no -> get bigger core......repeat until you get the best mix 'lowest possible core size and number of windings and no core saturation achieved'
4.Flyback topologies, expecially off-line ones, will definitely require snubber circuit for protecting your main switch. Calculate it too
BTW, it is just safer to use DCM in off-line flybacks
Somewhere between this steps decide with your PWM controller (which should based on current-mode control). Examples of such chips I already gave you earlier.
Finally, don't afraid to look for reference designs and PCB layouts
P.S. Also search how to wind flyback transformers to minimize its parasitics (interwinding capacitance, leakage inductance etc.)