The package size is probably not the main issue here .... the assembly process is. If the SMD parts are reflowed first and the connector is subsequently hand- or selectively soldered, 0603 should be practical at a 4 mm pitch, provided there is adequate solder-mask clearance between the pads.
If the connector will be wave soldered, however, bottom-side components between its pins could create bridging, shadowing or component-retention problems. PCBWay’s
compact PCB layout guide similarly recommends maintaining manufacturable solder-mask dams and sufficient pad spacing to improve assembly yield.
Between the two sizes, I would retain 0603 for resistors unless clearance requires otherwise. For ceramic capacitors, 0402 has less body length over which PCB strain can act, although placing any MLCC near a large connector still presents some flex-cracking risk. Soft-termination capacitors may be worth considering if reliability is critical.
The best approach is to send the proposed layout to the PCBA supplier and ask which soldering process they will use and whether the clearances pass their DFM review. If they approve the 0603 layout, I see little advantage in changing everything to 0402.