No, I would not remove R4. Don't forget that at startup the ATtiny's pins will default to inputs until your code configures them otherwise, so without a pull-down the gate of the 'FET will be floating momentarily, which may cause spurious activation of the relays.
Also, may I suggest a polarity protection diode on the 12V input? I note you are using a centre-negative barrel jack, which is less common than centre-positive. The diode will protect against someone accidentally connecting the wrong polarity supply.
Oh, and if you want to reduce components, why the LM317? Why not a fixed-ouput 5V regulator? For example 7805, et al.