But that sends over 2mA from the output through them to ground. Not much, but why don't they recommend much higher values if its just to set a voltage, and not waste power?
A linear voltage regulator needs current for the internal circuits. On the 78xx series of regulators, this current runs from the input, through the IC and back via the ground pin, which is simple enough.
The LM317 is designed to have a stable current running from the ADJ pin (50...100 microamps), otherwise good regulation is not possible. To achieve this, the supply current for the internal circuitry runs from input to output, thus the need for a certain load current (10 mA for a conservative design).
If your LM317 always supplies a load consuming more than 10 mA, you can use higher resistor values.
The 120 ohms for R1 is for cases where your load is unknown or intermittent.