A solderable heatsink is a good idea. The datasheet for that one provides two contradicting unitless numbers for thermal resistance, magical "9" and "11", but thermal product manufacturers are notorious for messing up the basics of documentation, including not understanding the basic units of their trade, so we just need to assume it's 11 degC/W, and finally, test to verify this assumption.
The datasheet says this is valid at 75 degC heatsink dT, in freely moving air. This is a bit demanding condition: your "ambient" would be inside the box I guess, your air probably wouldn't be completely free to move, and you probably couldn't accept dt=75, especially if you are aiming for 100degC transistor temperature - you would need to guarantee your box insides staying at 25degC max! So in practice, the Rth for that particular heatsink would be
higher than 11 degC/W in your conditions.
Still assuming 11 degC/W, the total thermal resistance junction-to-ambient would be 11 + 1 = 12 (degC/W) then. 11W dissipated would cause a 12degC/W * 11W = 132 degC temperature rise, clearly beyond unacceptable. So, you'd need a bigger heatsink.
I guess the thermal resistance number they give already includes what the case of the D2PAK, and their recommended footprint through FR4 already dissipates, so you can't expect any extra from it, unless you add a lot of more copper. With thermal vias out of question, you are starting to lose your options.
Pulse duty with short pulses is helping - D2PAK has quite a lot of thermal mass. But then, you'd need to failsafe your design somehow to prevent duty cycles too high. Add a thermistor closely coupled to the transistor, preventing operation while too hot? This would likely be the cheapest solution.
Finally, I'd look into reducing the dissipation by changing what you are actually doing, and how. 11W is a lot of dissipation in a small integrated device, and very difficult to achieve in SMD design, especially when you only have a single side available. You can look at larger SMD heatsinks, of course, but then you'd need to ventilate the case as well, if you expect the possibility of large operation duty cycles, that is.
can you please provide some links to layouts with similar packages which provide temperature calculations / thermal resistance for a D2PAK package with different layouts?
First result on Google with "D2PAK MOSFET":
https://www.vishay.com/docs/91055/sihf740s.pdfJuction-to-case 1 degC/W (no surprise, it's the same case)
Junction-to-ambient, bare device, 62 degC/W
Junction-to-ambient with example PCB layout of 25.4 x 25.4 mm copper pad on FR-4 - I'd
assume 2 oz copper as they are too lazy to define it, 40 degC/W
Another good one, see Figure 7:
https://www.microsemi.com/document-portal/doc_download/124720-an-208-pd70101-pd70201layout-guidelines-application-noteBasically, on a single layer without thermal vias, you are limited to a bit over 30 degC/W, even reaching infinite copper area, and this is on 2oz copper. You likely have 1oz, I'd guess, making things worse.