Most likely will be at the power provider side, unless your property has 3 phase power loss of neutral means loss of power, while on the provider side loss of neutral means things go bang as whichever phase has the lowest load gets the highest voltage. Random loss points to the fault likely being at a pole connection somewhere between your house and the distribution transformer.
The likely cause is a poor joint when some tap was made, which resulted in the neutral cable being broken internally, but still connected by contact, and with movement in the wind, load and time it arcs and goes open intermittently, leading to every house downstream of the fault having intermittent neutral connections. If you live in a block of apartments the fault could be in the central power room as well, loose neutral.
Only thing is to have the power company come out and inspect, as all this wiring is their property.