An then solve the issue of the PLL running with no HDMI video input etc, etc. Hack over hack, complication over complication.
Why not just doing it right from the beginning? There really are available FPGAs for as low as $5. A high performance ARM MCU like you suggested will cost multiple times this figure.
Also, do you have any specific tip on ICs to ser/des the TMDS video data? (Just asking). Dedicated RGB (DPI) to TMDS and back chips are quite rare, often expensive and under NDA (due to the HDMI money suckers).