Depends how accurate are the hole positions are in what you are mounting it to.
You want accurate size on one side and oversize on the other to allow for tolerance.
3.5mm would be a generally normal size for an M3 mount, but M3.5 on both pieces may get too sloppy, so you'd normally do 3.2mm on one and 3.5 on the other.
If it's just a general purpose mount hole with no specifiiic 'other bit', use 3.5mm
I'd normally do the pad as 6mm, to correspoind with the typical screw head size, to avoid any tracks ending up under the screw.