Use Select Outside (S,O) to draw a rectangle around all of the objects you actually want, which should select everything else, then delete?
Altium can get weird about objects that are outside of some invisible boundary of the PCB area, but I don't think there's anyway really around that. I had particular problems with something like that in AD18 the other day but I honestly don't remember how I solved it.
Worst case, you should be able to copy and paste all of the objects you actually want to a new PCB Doc, remove the old PCBDoc from the project and add the new one to the project, then from the PCB editor use Project > Component Links to reconnect the objects in the new file to your schematics. Make sure that your PCB & SCH are synchronized before you do any of that, though, otherwise reestablishing component links will be much harder.