EEVblog Electronics Community Forum

Electronics => Power & Renewable Energy => Topic started by: DuncanCT on September 16, 2019, 01:44:28 pm

Title: Continuous Router Failure
Post by: DuncanCT on September 16, 2019, 01:44:28 pm
Hi all,

First post here. Don't know where to turn, thought this community would be most helpful. If this is posted in the wrong place or inappropriate content admin please move/remove.

We have a site which has now eaten 10 device (routers) over the last couple of months. We are only responsible for the router and nothing else. We have been trying to point the tenant if the right direction for a solution but my knowledge is basic at best.

Background:
They are in a commercial industrial complex.
Failures can sometime happen when they are not there yet.
No heavy electrical equipment on site.
Visual inspection on most the device shows damage to smd capacitors (All are close to the 24v barrel jack input)
Power supplied via UPS and then through factory supplied SMPS.
Electricians have used logging equipment but assume too basic (sample rate too slow?) to help.
Surge suppression device installed but not sure of it's quality.

My assumption is that there might be transients but thought that the surge suppression would prevent this?

Please can anyone point us in the right direction.

Disclaimer: I have no formal qualification in anything electrical so my assumptions could be very wrong.

TIA
Title: Re: Continuous Router Failure
Post by: nuclearcat on September 16, 2019, 05:24:17 pm
Lot of details matter. Not sure i can list all of them here. But at least some, router - some kind of internet router or it is CNC router?
If internet, small SOHO router like "soapbox" NAT with wifi or some beefy cisco router? Does it have PoE? Does it have ethernet devices connected by long cables(especially outdoor)? Can surge come over this cables?

Surge supressors useful only when threat factors is clear and they are installed in proper places.
For example, if you installed protection on the UPS mains, and the EMP comes from one of the connected ethernet devices, this protection is useless, and may even aggravate the situation.

Factory supplied SMPS all different, some are bad quality, some cheapo might leak surges over noise filtering caps, and etc.
Title: Re: Continuous Router Failure
Post by: ejeffrey on September 16, 2019, 11:08:12 pm
Are these devices frequently plugged/unplugged by the users?  Are they placed in an area where people might interact with them or stress them?  It could be something mechanical like stress on the barrel jack flexing the board and cracking ceramic SMD capacitors.

You say there is visual damage to the capacitors but is that the actual problem?  Damaged capacitors will often short circuit, which is easy to diagnose.  Alternately remove some damaged capacitors and measure their capacitance and ability to withstand 24 V.  Its possible that the apparent damage to the capacitors is a red herring.
Title: Re: Continuous Router Failure
Post by: DuncanCT on September 17, 2019, 07:58:17 am
Thanks for the responses.

More details:
Failed network router devices: 9 x Mikrotik RB2011 / 1 x Mikrotik hAP ac2
UPS = Riello Sentinel Pro 1000VA (Supplier suggested that UPS should be protecting from transients)

Setup: All equipment connected to the router is ring fenced/downstream of the UPS and max 2 meters away.
Printer, DSL Modem, PC.

No POE being used, everything power from SMPS via UPS.

The only thing not connect via the USP is the copper DSL line that connects to the DSL modem.

See attached images of typical board damage. All located close to the DC barrel jack.

Also would assume marked area on the board diagram would be the capacitors failing if the problem was coming in over the network, which is not the case.

@ejeffrey - not physical interaction with devices other than just installing and leaving on.
Title: Re: Continuous Router Failure
Post by: Berni on September 17, 2019, 08:29:41 am
That looks weird.

Its like someone ripped the capacitor off with side cutters. I would expect the capacitor to leave some chared remains rather than exploding into nothing but dust without any mark left on the board.
Title: Re: Continuous Router Failure
Post by: Electro Detective on September 17, 2019, 09:17:15 am
If it's not a decent sine wave UPS when in backup mode, its output may be killing the routers during brief outages

and or the UPS is knackered and voltage to the routers is erratic,
with the smps in the routers going nuts trying to establish the voltage and frequency   :-//

I'd be getting an isolation transformer in on the action to feed the routers
as well as an oscilloscope to 'see' wth is currently going into those trashed routers  :o

and give the surge suppression section a real surge or three  >:D to confirm it's not sleeping on the job  -_-
Title: Re: Continuous Router Failure
Post by: DuncanCT on September 17, 2019, 09:49:56 am
This UPS is new (Riello Sentinel Pro 1000VA) and has been setup in "online mode" ("On line: maximum load protection and
output voltage waveform quality").

These problems have occurred with and without the UPS.
Title: Re: Continuous Router Failure
Post by: fourtytwo42 on September 17, 2019, 04:15:13 pm
This looks like mechanical damage from clumsiness or sabotage to me, the first two pics of the same board show scratchmarks where a screwdriver ? has slid across the board. The 3rd pic looks like it was powered up when attacked and a short caused by the tool has burnt the end of the cap. Hard to tell much from the 4th pic except somebody may have shorted the ic legs again with a tool.

Something else to check is the correct wallwart with the correct cable is being used, for example reversed polarity or wrong voltage at the power jack may also be causing problems and could potentially cause smd mlcc capacitors such as these to fail.

I dont go with the UPS being a problem or other equipment would also fail. Might be worth checking the wallwarts have not failed in some devious way (check the output voltage).
Title: Re: Continuous Router Failure
Post by: DuncanCT on September 18, 2019, 08:26:45 am
These devices are in enclosures in a secure area and tend to happen when no one is there. So unlikely it's sabotage.

Wall warts are new with each new device, and they are supplied from the manufactures with the device.

We have 100s of these devices running in other locations without trouble. Something specific to the power supply of this site that is cause the problems, just don't know how to diagnose.
Title: Re: Continuous Router Failure
Post by: nuclearcat on September 19, 2019, 10:39:50 pm
This is IC on photo power mosfet UT4421, so i guess it is power supply area of PCB, it looks like exploded caps from really bad surge. Unfortunately there is no service manuals for RB2011, but even damage is near DC jack, it doesn't mean strike didn't come over ethernet.
This board have PoE ports, including PoE out, so any strike comes over network will reach power supply circuits too. Some components maybe died silently and visually they look ok, but in reality they are fried inside. It is hard to say, who is to blame, DC jack, or surge over network, or surge over network pass over all the board and this poor caps, to some ground point.
Check this topic, same board as yours, similar location of damage: https://www.eevblog.com/forum/repair/thunderstormed-routerboard-mikrotik-rb2011uias-2hnd-in/ (https://www.eevblog.com/forum/repair/thunderstormed-routerboard-mikrotik-rb2011uias-2hnd-in/)
Install surge arresters on all network ports under slightest chance of surge, some protection on DC supply part, (something with fused + TVS between pairs and to ground + reverse polarity protection).
Title: Re: Continuous Router Failure
Post by: bdunham7 on September 23, 2019, 08:19:34 pm
Have you checked the equipment ground?  Or alternatively, can you eliminate it temporarily?  Given that your UPS is in the 'online' mode instead of 'line interactive', the power side is isolated enough anyway.  However, there may be a common ground all the way thru even the wall wart, depending on how it is configured.  If that particular ground plug is wired improperly somewhere back towards the service panel, it might be possible for certain types of equipment--large motors for example--to cause a large surge on the ground line.   

I'd start by connecting a meter, or preferably a scope, between your ground and an absolutely known ground such as a water main or the ground rod at your service panel.  If your customer has large equipment, have them turn it on and off.  If it is a shared premises or you think the problem might be coming from a neighbor, just measure that ground voltage for awhile.
Title: Re: Continuous Router Failure
Post by: Berni on September 24, 2019, 05:12:23 am
Yeah if it is an electrical failure then it is likely something to do with grounding.

Switchmode wallwarts are really easy to kill by overvoltage. Every so often i find one dead due to a thunderstorm. And in pretty much all of the cases whatever the wallwart was powering was still alive.

But wallwarts have EMI suppression capacitors going from the input to the output, These capacitors are typically rated for 1kV or more. Its possible that something causes a large transient on the ground of your equipment that pulls the isolated DC side of the wallwart to high voltages, high enough for that EMI suppression capacitor or something else to arc over, causing your DC ground to get connected to live or neutral for a short moment via the arc. This could cause a huge current to flow back into the DC jack of the equipment and do something bad.

Id check all the ground connections and make sure the rack is grounded properly. If there is PoE being used that could be a source of issues too.
Title: Re: Continuous Router Failure
Post by: nfmax on September 24, 2019, 07:04:47 am
Just maybe, something living at that one site is small enough to crawl in through the hole where the PSU connector fits, and has a taste for eating resistors?
Title: Re: Continuous Router Failure
Post by: ejeffrey on September 27, 2019, 10:44:28 pm
If you are worried about common surges coming over ethernet coupling to the power supply circuit through the POE pickoff, try putting a protection circuit on that?  There are purpose built ethernet surge suppressors or if you have some POE injectors laying around they would isolate the common mode voltage.  Although honestly the POE pickoff should be surge protected on the boards and if you don't see any damage there I think it is unlikely the source.
Title: Re: Continuous Router Failure
Post by: wraper on September 27, 2019, 11:01:53 pm
@ejeffrey - not physical interaction with devices other than just installing and leaving on.
Without someone poking PCB with some sharp object it's only possible if installation was done by apes. 4 ceramic capacitors in 2 different places don't simply disappear from PCB with leftovers of terminals left on the pads and scratches on PCB. Also one resistor is mechanically damaged, part of it missing.
Title: Re: Continuous Router Failure
Post by: wraper on September 27, 2019, 11:22:43 pm
These devices are in enclosures in a secure area and tend to happen when no one is there. So unlikely it's sabotage.
Then someone infiltrates that "safe" area, opens enclosure and does that dirty job. 100% mechanical damage, I've seen plenty of knocked off MLCC to know how it looks like. Ceramic capacitors do not explode or teleport in some other ways. If they fail short and high current passes through them, PCB may scotch. But they won't dissapear with remnants of their terminals left and no single IC without hole in the middle, no burned traces.
Don't forget that this can be easily done through ventilation holes or some other gaps in enclosure.
Title: Re: Continuous Router Failure
Post by: DuncanCT on October 08, 2019, 01:48:49 pm
After a couple more failed devices we have been able to match the time stamps of the failures to the workshop next door turning off their compressor (medium sizes wood working shop).

Electricians have installed surge protection on the incoming lines but they are not helping (not sure of the spec or quality). Have also lost faith in their abilities. What are the tricks to surge protection for sensitive electronics?
Title: Re: Continuous Router Failure
Post by: MagicSmoker on October 08, 2019, 02:19:09 pm
Line reactors are useful at integrating away steeply rising surge voltages, though they are typically intended for flattening out the peaks of current drawn by variable frequency motor drives. See, for example: https://www.mtecorp.com/products/reactors/rl-reactors/ (https://www.mtecorp.com/products/reactors/rl-reactors/)


Title: Re: Continuous Router Failure
Post by: wraper on October 08, 2019, 02:23:28 pm
After a couple more failed devices we have been able to match the time stamps of the failures to the workshop next door turning off their compressor (medium sizes wood working shop).

Electricians have installed surge protection on the incoming lines but they are not helping (not sure of the spec or quality). Have also lost faith in their abilities. What are the tricks to surge protection for sensitive electronics?
How surge protection is supposed to help against someone poking parts with a sharp object? What a joke that you think electric surge can do that  :palm:. If you don't believe me, put security seals on enclosure equipment is located within.
(https://www.airforms.co.uk/ekmps/shops/airforms/images/kndm-seals-red-tamper-evident-security-seals-10cm-x-2.5cm-500-per-roll-26-p.gif)
Title: Re: Continuous Router Failure
Post by: EEEnthusiast on October 08, 2019, 02:33:10 pm
Is there a chance that water can creep into the router somehow. The damage seems to be on the edge of the board where the moisture is likely to come in from the enclosure edges or opening vents.
Title: Re: Continuous Router Failure
Post by: wraper on October 08, 2019, 02:38:21 pm
Is there a chance that water can creep into the router somehow. The damage seems to be on the edge of the board where the moisture is likely to come in from the enclosure edges or opening vents.
And cause capacitors teleport from PCB with no signs of liquid/ corrosion?
Title: Re: Continuous Router Failure
Post by: DuncanCT on October 08, 2019, 03:37:48 pm
Attached photos from two of the latest devices.

I've quiet certain if I wanted to waste money, I could induce a failure. This still doesn't help me find a solution.
Title: Re: Continuous Router Failure
Post by: wraper on October 08, 2019, 03:43:12 pm
Attached photos from two of the latest devices.
Someone poked part with a metal object while router was powered and caused short circuit to enclosure from top of terminal.
Quote
I've quiet certain if I wanted to waste money, I could induce a failure. This still doesn't help me find a solution.
Ensure it's inaccessible. Put a lock, or replace it if there might be other people with a key. Attach security seal on a door of enclosure to know it was tampered with. Or place a network camera FFS to catch that idiot.
Title: Re: Continuous Router Failure
Post by: EEEnthusiast on October 08, 2019, 03:58:00 pm
Is there a chance that water can creep into the router somehow. The damage seems to be on the edge of the board where the moisture is likely to come in from the enclosure edges or opening vents.
And cause capacitors teleport from PCB with no signs of liquid/ corrosion?
The liquid may dry up after some time, but the cap disappearing is not explained. An explosion causing the cap to pulverize to fine dust.???
Title: Re: Continuous Router Failure
Post by: EEEnthusiast on October 08, 2019, 04:02:42 pm
Do you have the part number for those capacitors? What is the voltage rating on them?
Title: Re: Continuous Router Failure
Post by: Monkeh on October 08, 2019, 04:06:41 pm
Do you have the part number for those capacitors? What is the voltage rating on them?

They're unmarked ceramic caps, why would you expect him to have part numbers?

And please, explain the compensation circuit managing to explode in the first picture he posted.. (I'll give you a hint: it involved a screwdriver or a knife)
Title: Re: Continuous Router Failure
Post by: EEEnthusiast on October 08, 2019, 04:11:18 pm
Why would anyone sabotage a router? Unless he wants our friend to go out of business.
Well the recent pictures show the caps being blackened. This is surely due to over voltage or mechanical stress causing the cap to short. The recent photo (2nd one) does not show any mechanical damage to the part.
Title: Re: Continuous Router Failure
Post by: wraper on October 08, 2019, 05:32:51 pm
Why would anyone sabotage a router? Unless he wants our friend to go out of business.
World is full with idiots, vandals, maniacs and all sorts of weirdoes. When people poison neighbors dog and do other stupid shit, why would not someone decide to do this with routers?
Quote
Well the recent pictures show the caps being blackened. This is surely due to over voltage or mechanical stress causing the cap to short.
MLCC don't fail like that. One cap simply has top corner charred. That cap is not even broken with 99% probability. Someone just short circuited it's terminal. Second capacitor corner charred/ melted and crack coming from that. The same thing, just with added mechanical force. You would have whole board covered with burnt semiconductors and traces before ceramic capacitors will start exploding due to overvoltage.
Title: Re: Continuous Router Failure
Post by: bdunham7 on October 08, 2019, 07:04:37 pm
After a couple more failed devices we have been able to match the time stamps of the failures to the workshop next door turning off their compressor (medium sizes wood working shop).

Electricians have installed surge protection on the incoming lines but they are not helping (not sure of the spec or quality). Have also lost faith in their abilities. What are the tricks to surge protection for sensitive electronics?

If you actually have a valid, properly functioning UPS in true online mode (double conversion) there is no possibility of surges reaching your equipment via any mode other than ground.  Please read my previous message about what to do about that...
Title: Re: Continuous Router Failure
Post by: MagicSmoker on October 08, 2019, 07:17:26 pm
Second capacitor corner charred/ melted and crack coming from that. The same thing, just with added mechanical force. You would have whole board covered with burnt semiconductors and traces before ceramic capacitors will start exploding due to overvoltage.

Hmm, yes, it definitely looks like "mechanically induced failure" to me, too. Note to OP: a line reactor won't protect against that; neither will a double-conversion UPS.

Title: Re: Continuous Router Failure
Post by: Mr.B on October 08, 2019, 08:44:19 pm
To the OP.
What @wraper said...
Install tamper proof seals and a hidden webcam.
The cause of your continuous failures will be revealed fairly quickly.
Title: Re: Continuous Router Failure
Post by: EEEnthusiast on October 09, 2019, 03:04:01 am
To the OP.
What @wraper said...
Install tamper proof seals and a hidden webcam.
The cause of your continuous failures will be revealed fairly quickly.
And keep your rifle loaded.
Title: Re: Continuous Router Failure
Post by: drussell on October 09, 2019, 04:01:16 am
I agree that a video camera is the likely solution to your router woes.  :)

That is mechanical damage. 

That is not the way components blow themselves off the board.
Title: Re: Continuous Router Failure
Post by: NiHaoMike on October 10, 2019, 03:09:31 am
Perhaps replace the stock PSU with a cheap metal box PSU of the type commonly found on 3D printers, then mount it over the router. With luck, if it is indeed someone intentionally damaging stuff, adding mains power would greatly increase the chance they get more of a surprise than they were expecting.
Title: Re: Continuous Router Failure
Post by: Berni on October 10, 2019, 05:18:35 am
Yeah a lot of this again looks like someone poked a screwdriver in there.

Id just install a hidden webcam and nothing else so that anyone who is doing this does not notice any counter measures and shows himself. Once you have video evidence show that to your costumer and ask them to fix this or you won't be fixing there routers anymore.
Title: Re: Continuous Router Failure
Post by: nuclearcat on October 10, 2019, 11:51:12 am
Yeah a lot of this again looks like someone poked a screwdriver in there.

Id just install a hidden webcam and nothing else so that anyone who is doing this does not notice any counter measures and shows himself. Once you have video evidence show that to your costumer and ask them to fix this or you won't be fixing there routers anymore.
Hidden survelliance might violate country laws.
Intrusion detection is ok, camera inside box is ok, but hidden cam most likely will put in big trouble.
Title: Re: Continuous Router Failure
Post by: drussell on October 10, 2019, 02:17:55 pm
Hidden survelliance might violate country laws.Intrusion detection is ok, camera inside box is ok, but hidden cam most likely will put in big trouble.

I'm pretty sure that in South Africa, as long as you have your sign at the door that says something like "these premises are monitored by video surveillance," you're fine and will even be able to legally use your recordings to prosecute someone if it comes to that, whether the cameras are "hidden" or in obvious plain sight.  (As long as you don't put them in washrooms, changing rooms, etc.)

"Fairly warned, be thee, says I...."  :)
Title: Re: Continuous Router Failure
Post by: DuncanCT on October 10, 2019, 02:19:38 pm
Latest picture of the surge protection device installed on the DB.

Best theory at the moment is there is a second PC that is not behind a decent UPS and it's passing whatever is incoming via the land cable connecting it to the router. I'm going to install a wireless link and remove the lan cable next.

The only other "exposed" connection would be the copper line for the (A)DSL connection. Assume if a PC can pass this through a lan cable then so can a POTS line via can a modem?
Title: Re: Continuous Router Failure
Post by: drussell on October 10, 2019, 02:30:35 pm
One additional snippet...

Many years ago at my Dad's office downtown, packages of microwave popcorn kept disappearing from the kitchenette area.  We would buy those big flats of like 48 packages from Costco for anyone in the office who wanted a snack to be able to toss one in the microwave.  But they were disappearing at night. 

Now, my dad used to be quite the night owl and would often return to the office at 8-9 PM after and after-dinner-nap and work until the wee hours of the morning before returning home.  Some nights he could smell that popcorn had been recently popped.  It's pretty obvious.

Now, the only people that would have been in our office in the evening would be the (high-rise downtown office) building's cleaning staff who came in each night to vacuum, empty the garbages, paper recycling, etc. so he spoke to the building management who were rightly concerned.  It wasn't that items of significant value were disappearing, but the principle of the thing.

Management's solution was to cut a fairly obvious, crude cutout in an old monitor cardboard box that was up on a top shelf in that room and place a video camera in there.  Sure enough, one of the cleaning staff was helping themselves to a package of popcorn, from inside one of the cabinets.  They were let go from their employment for snooping through cupboards and helping themselves to packages of popcorn.  Who knows how many desk drawers and office nooks and crannies they had been peering into and what else they felt they might have been able to abscond with.

It was very effective.  I believe the video camera was only there for one night.
Title: Re: Continuous Router Failure
Post by: drussell on October 10, 2019, 02:34:38 pm
Latest picture of the surge protection device installed on the DB.

That surge suppressor looks like it might be feeling a tad ill...   :-DD

I still think your power input issues are a red herring.  You are likely chasing two different issues.

Those photos you showed of the router damage is not damage that would be caused by mains-induced surges or spikes.  It simply is not what surge damage at any energy level looks like, from mild spike to lightning strike.
Title: Re: Continuous Router Failure
Post by: Berni on October 10, 2019, 03:23:39 pm
Yep something that could cause a surge protector to turn to charcoal like that would likely also leave its mark on the routers mains power supply.

Nothing wrong with video surveillance as long as you put up a sign on the door or something, tho to be fair unless its a big sign most people won't even notice it due to these signs being all over the place so you sort of start ignoring them. But if anyone objects to being unlawfully filmed then you can still point at the sign on the door. But even if whoever is doing this notices the sign and stops, that's also a win situation since your routers are no longer being destroyed.
Title: Re: Continuous Router Failure
Post by: duak on October 11, 2019, 03:58:21 am
What do these capacitors do?  Do they connect an input jack terminal to chassis or safety ground/earth?  If so, maybe there is a damaged cable or improperly connected neutral or safety ground/earth connection somewhere, eg. conduit connector not tightened.  This allows common mode transient currents to return through a path that is not robust enough to handle it.

I remember a junior engineer added some transorbs between system common and chassis on his board to divert transients.  During system transient susceptibility testing, the transorbs were blown off the boards, unfortunately not cleanly.

Title: Re: Continuous Router Failure
Post by: Jeroen3 on October 11, 2019, 05:59:57 am
You should install a power quality monitor, after that you can move the invoice for component replacement to the grid company.
Eg: PQube or mobile Fluke/Yokogawa.
Title: Re: Continuous Router Failure
Post by: nuclearcat on October 12, 2019, 04:01:18 am
Latest picture of the surge protection device installed on the DB.

Best theory at the moment is there is a second PC that is not behind a decent UPS and it's passing whatever is incoming via the land cable connecting it to the router. I'm going to install a wireless link and remove the lan cable next.

The only other "exposed" connection would be the copper line for the (A)DSL connection. Assume if a PC can pass this through a lan cable then so can a POTS line via can a modem?
Has this PC (or network card) or ADSL modem ever burned down by surge? Usually, when such a surge passes through the equipment, something else than mikrotik may burn out. (unless it passed by shields/ground and pc/dsl modem doesn't have ground/ or they was disconnected from mains).
For example in last incident lightening entered server room over asterisk connected to POTS, from card to "ground" of PC, and because it was not properly grounded, passed over whole rack to device that was has connection to ground, burning everything on the way. After i traced, i found biggest damage on Asterisk FXO port. You can see traces of explosion on card under module 3, and exploded MOV(not sure, lazy to check if it is MOV) on removed module, it was not visible, until i removed it.
[attach=1]

Was there a thunderstorm when Mikrotik failed? (Most likely yes, just noticed surge arrester charred)

And yes, POTS quite common source of surges, unless they are completely underground all the way and no underground high voltage lines nearby. If it is not underground, you definitely need to put surge protector on POTS, they should not be expensive.
Otherwise surge will come over POTS, pass your DSL modem and then, or over DSL modem supply to ground, or more likely over network to closest device that is connected to ground.

P.S. Also avoid shielded cables and plugs.They work only when properly grounded, and when not - they do exactly opposite.