The IR input is oddball with no pullup/pulldown and it does not show a GND connection. Just two wires, +12V and IN?
With that a photo-transistor (or anything) would not work, nothing to take the 4093 input low.
And it would have low gain, not decent range with a photo-transistor, and 5V would have been enough no need to have a boost converter to 12V. The receiver module must have a pre-amp.
With the remote, if you can measure the carrier frequency or the chipset it uses.