Let's simplify this.
A simple, single phase mains feed has two wires. Live wire, and neutral wire. Voltage exists as difference between those two wires. When there is a difference in voltage, when connected, current flows.
At this stage, it does not matter which one swings up and down and which one does not. It's all relative. If you base your measurement on neutral wire, live wire swings. If you base your measurement on live wire, neutral swings.
What makes difference is that at power pole and your main feed, neutral wire is connected to ground. Therefore, it is at ground potential. So, it is now simple to think, neutral wire does not swing up or down. It stays at the ground potential. This is why it is neutral. This is why you can grab the neutral wire (white wire in USA) and not be in danger of electrical shock.
I am not going to go into 3 phase stuff. It is not necessary for this discussion. In USA, a regular socket has 3 poles but last one is ground. Just ground. Remaining two is neutral and live. If you measure those two lines ASSUMING neutral is at ground potential (which it is), live swings 120 * 1.4 volt 60 times a second. (peak of AC is higher than what people refer to as "Voltage".)
Since YOU (your body) is coupled weakly to ground, if you touch live, you complete the circuit and feel the shock. If you touch neutral, since you and the wire are both at ground potential, you won't be shocked. If you actually put your feet on the actual ground and touch live wire, you might die, since you are strongly coupled to the ground. Now, if you stand on a perfect insulator and few feet off the ground or away from anything metallic, you can touch live and not feel a thing. This is because with absence of ground, there is no way for live wire to "swing".
Now, the disclaimer.... above is all true assuming everything is done correctly and no fault exists. I have seen many times things are screwed up. Never assume any wires are safe to touch. I am a licensed electrician but I always measure few times before touching anything.