Your feedback resistors of 10K and 16.5k are really high.
The LM317 ADJUST input has an input current and input capacitance.
The website LM317 calculator website I sent you does not take the input current on that pin into account.
This is why I said for R1, use 500Ohm or less to eliminate that figure.
If I had any LM317 on hand, I would have already tested this circuit with a 2N3904 as an NPN buffer to verify final output voltage.
Studying the way the reference works internally, it looks like you will need to add the 0.75v drop of the NPN transistor into your output voltage calculation. IE 3.3v = set regulator for 4v.
Also, I would still add an R1a and R1b, and only mount 1 of them.
Placing the resistor at in the R1b position means the regulator feedback sees only 1/100 the current spikes since it needs to travel through the transistor emitter to the base. This means the circuit is acting like a current multiplier VS more like a regulator.
Note we can switch the LM317 with an op-amp / voltage reverence and make a super precise regulated output, on par with a high quality audio amplifier driving a DC output. This would guarantee dead exact output voltage no matter the load. A dual opamp would replace both LM317s. The voltage reference would be a 78L05 regulator. This would make your output look like a battery with 0ohm ESR. In the past, I have actually made this type of circuit, so I know it works really well and you may swap the BJT for a logic level mosfet here as the opamp will compensate for the Vgs and it's drift as the mosfet temperature changes.
As for your heat-sink. No problem radiating 5 watts, but with 8 watts, it will get warm.
According to ON-Semiconductor's MJB44H11 datasheet, at 5 amps. the Vce saturation voltage is 0.2v. (Actually 0.11v at 3amps)
This means if you want 3.3v out, you can get away with 3.5v in, however, there is a little offset in linearity right at the edge.
Now, your switcher can dip it's output voltage by up to 0.4v during current spikes, so, 0.4v + 0.2 + 0.2 extra margin means minimum = 0.8v above target output voltage.
3 amps *0.8v = 2.4 watts heat *2 = 4.8watts heat total, assuming 2.5amps-12.5 watts of LEDs.
This is as low as I would attempt the circuit with obvious test verification.
Raising the extra margin to 0.4v:
3 amps * 1v = 3 watts heat * 2 = 6 watts heat total, assuming 2.5amps-12.5 watts of LEDs.
Radiating 6 watts will be like having one of those old 120v 5 watt outdoor Christmas light bulbs inside your Dreamcast. If the old PSU had a linear transformer, that transformer alone would radiate that amount of heat at least.
Using the opamp with a mosfet in a DPAK/D2PAK case, you can go down to ~0.6v, ~0.2v above the switcher's worst voltage dip. Since the op-amp will not need to drive current with a mosfet, that part of the circuit will run cooler & only needing a 0.6v headroom, the circuit will radiate a total of 3.6 watts, 1.8 watts per mosfet. You wont need a heatsink here, but, the PCB will still get warm to the touch, but not too hot. I would still prefer at least 0.8v headroom.
Just so we are clear, (3*3.3+3*5)*1.2(loss in switcher supplies) = 29.9 watts at 12v, or 2.5amps at 12V. With no room for the CD drive's 12v supply if you are using a 30 watt 12v power supply.
I've attached an image of using an opamp circuit. Obviously we would use a more modern opamp and something like the mosfet.
Cheap mosfet = IRLR8726TRPBF - (Crss=310pf)
https://lcsc.com/product-detail/MOSFET_International-Rectifier_IRLR8726TRPBF_International-Rectifier-IR-IRLR8726TRPBF_C81137.htmlOr any 50N03 variant should do, like
https://lcsc.com/product-detail/MOSFET_KIA-Semicon-Tech-KIA50N03AD_C112249.htmlYou can also use a TO-220 Mosfet/BJT with a bolted heatsink mounted down on the PCB:
EG:
https://lcsc.com/product-detail/Heat-Sinks_XSD-XSD-heat-sink15-5-10-5-24-Htype_C108928.htmlhttps://lcsc.com/product-detail/Heat-Sinks_Made-in-China-Made-in-China-15-10-20-U_C361579.html (More difficult to accidentally short)
However, they will have active voltage unlike the taped SMD heatsink.
See attached circuit below: ( The + goes to the 78L05 output, for channel 2 on the opamp, 3.3v, well divide the 5v to 3.3v with 2 resistors) This would be an op amp-corrected/compensated/buffered capacitance multiplier.
https://lcsc.com/product-detail/Dropout-Regulators-LDO_Diodes-Incorporated_AS78L05RTR-E1_Diodes-Incorporated-AS78L05RTR-E1_C90471.htmlhttps://lcsc.com/product-detail/General-Purpose-Amplifiers_STMicroelectronics-MC4558CDT_C435907.htmlInput latch-up protection diode not shown in circuit.
https://lcsc.com/product-detail/Switching-Diode_ON-Semiconductor-ON-MMBD914LT1G_C46523.htmlPrecision CMOS opamp:
https://lcsc.com/product-detail/General-Purpose-Amplifiers_Texas-Instruments_TLC272CDR_Texas-Instruments-Texas-Instruments-TLC272CDR_C9374.html