Not somewhere I have played, SDI generally being my starting point, but I can make some informed guesses.
Sync separator then lock a PLL to the input line rate to produce a pixel clock at whatever frequency is appropriate, some ability to slip the input pixel sampling phase to match whatever the video hardware is generating is probably desirable. This could probably be done with some magic involving an NCO and playing with the phase trimming given that a reasonable FPGA will go at way above the required pixel clock rate for 480p
If the input is composite then a well designed comb filter in the digital domain is likely better then trying to deal in the analogue space, and you will want to sort out the various options for chroma subsampling and assumed gamma, a quick enough ADC for this is reasonably trivial these days.
A good deinterlacer is actually one of the hardest things to write, because doing it really properly involves motion estimation, and that just eats gates (and memory bandwidth). Doing it low latency just adds to the joy. Getting below 1 frame of latency is basically impossible if doing it well, but for stuff like old console games there are options.
Scaling however is fairly easy, especially for the integer ratio cases, being a case of applying a 2d polyphase filter to produce the output samples, this can be done with only a few lines of latency depending obviously on the size of the filter kernel.
Memory bandwidth requirements can be reduced by doing it as a two 1D filters at the cost of some loss of accuracy that often does not matter. The hard thing here is making sure that the memory layout works to put the correct pixels on the bus when you need them, not trivial given the nature of DDR3 and friends really needing at least row length reads or writes (Fine for the horizontal processing, pain in the arse for the vertical bit).
Line doubling is of course easy, buffer 1 line, play it out twice at double speed, while buffering the next line, then just ping pong. Not how you write a GOOD scaler (Which is black magic), but very, very low latency.