Contacts might have thin layers of various metals, so don't use sandpaper or other abrasives.
I used to use pencil eraser, that will only erase the oxides and don't file out the metal. Cut a thin sheet out of a pencil eraser, insert it between the contacts and gently erase the oxides. Then brush away any pencil-eraser debris.