I never said you should be afraid of everything you can smell. I said you should be concerned if you can smell these chemicals. That is rational.
Smell is not used in setting acceptable limits. But that doesn't mean you should ignore the senses everyone is born with, especially if that's the only tool you own that will do the job.
The air we normally breathe is 78% nitrogen, 21% is oxygen. So not toxic, or we'd all be dead. The danger of nitrogen spills is the local displacement of oxygen in the air, which you need to survive.
Scoff all you want, but the dangers are real and studied even before modern anesthetics: "In 1934, Killian gathered all the statistics compiled until then and found that the chances of suffering fatal complications under ether were between 1:14,000 and 1:28,000, whereas under chloroform the chances were between 1:3,000 and 1:6,000." So no, it's not as safe as the alternatives, not even the old alternative of ether.