Author Topic: Continuous Router Failure  (Read 1712 times)

0 Members and 1 Guest are viewing this topic.

Offline DuncanCT

  • Contributor
  • Posts: 7
  • Country: za
Continuous Router Failure
« on: September 16, 2019, 01:44:28 pm »
Hi all,

First post here. Don't know where to turn, thought this community would be most helpful. If this is posted in the wrong place or inappropriate content admin please move/remove.

We have a site which has now eaten 10 device (routers) over the last couple of months. We are only responsible for the router and nothing else. We have been trying to point the tenant if the right direction for a solution but my knowledge is basic at best.

Background:
They are in a commercial industrial complex.
Failures can sometime happen when they are not there yet.
No heavy electrical equipment on site.
Visual inspection on most the device shows damage to smd capacitors (All are close to the 24v barrel jack input)
Power supplied via UPS and then through factory supplied SMPS.
Electricians have used logging equipment but assume too basic (sample rate too slow?) to help.
Surge suppression device installed but not sure of it's quality.

My assumption is that there might be transients but thought that the surge suppression would prevent this?

Please can anyone point us in the right direction.

Disclaimer: I have no formal qualification in anything electrical so my assumptions could be very wrong.

TIA
 

Offline nuclearcat

  • Supporter
  • ****
  • Posts: 82
  • Country: lb
Re: Continuous Router Failure
« Reply #1 on: September 16, 2019, 05:24:17 pm »
Lot of details matter. Not sure i can list all of them here. But at least some, router - some kind of internet router or it is CNC router?
If internet, small SOHO router like "soapbox" NAT with wifi or some beefy cisco router? Does it have PoE? Does it have ethernet devices connected by long cables(especially outdoor)? Can surge come over this cables?

Surge supressors useful only when threat factors is clear and they are installed in proper places.
For example, if you installed protection on the UPS mains, and the EMP comes from one of the connected ethernet devices, this protection is useless, and may even aggravate the situation.

Factory supplied SMPS all different, some are bad quality, some cheapo might leak surges over noise filtering caps, and etc.
 

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 1993
  • Country: us
Re: Continuous Router Failure
« Reply #2 on: September 16, 2019, 11:08:12 pm »
Are these devices frequently plugged/unplugged by the users?  Are they placed in an area where people might interact with them or stress them?  It could be something mechanical like stress on the barrel jack flexing the board and cracking ceramic SMD capacitors.

You say there is visual damage to the capacitors but is that the actual problem?  Damaged capacitors will often short circuit, which is easy to diagnose.  Alternately remove some damaged capacitors and measure their capacitance and ability to withstand 24 V.  Its possible that the apparent damage to the capacitors is a red herring.
 

Offline DuncanCT

  • Contributor
  • Posts: 7
  • Country: za
Re: Continuous Router Failure
« Reply #3 on: September 17, 2019, 07:58:17 am »
Thanks for the responses.

More details:
Failed network router devices: 9 x Mikrotik RB2011 / 1 x Mikrotik hAP ac2
UPS = Riello Sentinel Pro 1000VA (Supplier suggested that UPS should be protecting from transients)

Setup: All equipment connected to the router is ring fenced/downstream of the UPS and max 2 meters away.
Printer, DSL Modem, PC.

No POE being used, everything power from SMPS via UPS.

The only thing not connect via the USP is the copper DSL line that connects to the DSL modem.

See attached images of typical board damage. All located close to the DC barrel jack.

Also would assume marked area on the board diagram would be the capacitors failing if the problem was coming in over the network, which is not the case.

@ejeffrey - not physical interaction with devices other than just installing and leaving on.
 

Offline Berni

  • Super Contributor
  • ***
  • Posts: 2526
  • Country: si
Re: Continuous Router Failure
« Reply #4 on: September 17, 2019, 08:29:41 am »
That looks weird.

Its like someone ripped the capacitor off with side cutters. I would expect the capacitor to leave some chared remains rather than exploding into nothing but dust without any mark left on the board.
 

Offline Electro Detective

  • Super Contributor
  • ***
  • Posts: 2218
  • Country: au
Re: Continuous Router Failure
« Reply #5 on: September 17, 2019, 09:17:15 am »
If it's not a decent sine wave UPS when in backup mode, its output may be killing the routers during brief outages

and or the UPS is knackered and voltage to the routers is erratic,
with the smps in the routers going nuts trying to establish the voltage and frequency   :-//

I'd be getting an isolation transformer in on the action to feed the routers
as well as an oscilloscope to 'see' wth is currently going into those trashed routers  :o

and give the surge suppression section a real surge or three  >:D to confirm it's not sleeping on the job  -_-
« Last Edit: September 17, 2019, 09:22:58 am by Electro Detective »
 

Offline DuncanCT

  • Contributor
  • Posts: 7
  • Country: za
Re: Continuous Router Failure
« Reply #6 on: September 17, 2019, 09:49:56 am »
This UPS is new (Riello Sentinel Pro 1000VA) and has been setup in "online mode" ("On line: maximum load protection and
output voltage waveform quality").

These problems have occurred with and without the UPS.
 

Offline fourtytwo42

  • Frequent Contributor
  • **
  • Posts: 799
  • Country: gb
  • Interested in all things green/ECO
Re: Continuous Router Failure
« Reply #7 on: September 17, 2019, 04:15:13 pm »
This looks like mechanical damage from clumsiness or sabotage to me, the first two pics of the same board show scratchmarks where a screwdriver ? has slid across the board. The 3rd pic looks like it was powered up when attacked and a short caused by the tool has burnt the end of the cap. Hard to tell much from the 4th pic except somebody may have shorted the ic legs again with a tool.

Something else to check is the correct wallwart with the correct cable is being used, for example reversed polarity or wrong voltage at the power jack may also be causing problems and could potentially cause smd mlcc capacitors such as these to fail.

I dont go with the UPS being a problem or other equipment would also fail. Might be worth checking the wallwarts have not failed in some devious way (check the output voltage).
 
The following users thanked this post: wraper, nuclearcat

Offline DuncanCT

  • Contributor
  • Posts: 7
  • Country: za
Re: Continuous Router Failure
« Reply #8 on: September 18, 2019, 08:26:45 am »
These devices are in enclosures in a secure area and tend to happen when no one is there. So unlikely it's sabotage.

Wall warts are new with each new device, and they are supplied from the manufactures with the device.

We have 100s of these devices running in other locations without trouble. Something specific to the power supply of this site that is cause the problems, just don't know how to diagnose.
 

Offline nuclearcat

  • Supporter
  • ****
  • Posts: 82
  • Country: lb
Re: Continuous Router Failure
« Reply #9 on: September 19, 2019, 10:39:50 pm »
This is IC on photo power mosfet UT4421, so i guess it is power supply area of PCB, it looks like exploded caps from really bad surge. Unfortunately there is no service manuals for RB2011, but even damage is near DC jack, it doesn't mean strike didn't come over ethernet.
This board have PoE ports, including PoE out, so any strike comes over network will reach power supply circuits too. Some components maybe died silently and visually they look ok, but in reality they are fried inside. It is hard to say, who is to blame, DC jack, or surge over network, or surge over network pass over all the board and this poor caps, to some ground point.
Check this topic, same board as yours, similar location of damage: https://www.eevblog.com/forum/repair/thunderstormed-routerboard-mikrotik-rb2011uias-2hnd-in/
Install surge arresters on all network ports under slightest chance of surge, some protection on DC supply part, (something with fused + TVS between pairs and to ground + reverse polarity protection).
« Last Edit: September 19, 2019, 10:53:28 pm by nuclearcat »
 
The following users thanked this post: fourtytwo42

Offline bdunham7

  • Frequent Contributor
  • **
  • Posts: 309
  • Country: us
Re: Continuous Router Failure
« Reply #10 on: September 23, 2019, 08:19:34 pm »
Have you checked the equipment ground?  Or alternatively, can you eliminate it temporarily?  Given that your UPS is in the 'online' mode instead of 'line interactive', the power side is isolated enough anyway.  However, there may be a common ground all the way thru even the wall wart, depending on how it is configured.  If that particular ground plug is wired improperly somewhere back towards the service panel, it might be possible for certain types of equipment--large motors for example--to cause a large surge on the ground line.   

I'd start by connecting a meter, or preferably a scope, between your ground and an absolutely known ground such as a water main or the ground rod at your service panel.  If your customer has large equipment, have them turn it on and off.  If it is a shared premises or you think the problem might be coming from a neighbor, just measure that ground voltage for awhile.
 

Offline Berni

  • Super Contributor
  • ***
  • Posts: 2526
  • Country: si
Re: Continuous Router Failure
« Reply #11 on: September 24, 2019, 05:12:23 am »
Yeah if it is an electrical failure then it is likely something to do with grounding.

Switchmode wallwarts are really easy to kill by overvoltage. Every so often i find one dead due to a thunderstorm. And in pretty much all of the cases whatever the wallwart was powering was still alive.

But wallwarts have EMI suppression capacitors going from the input to the output, These capacitors are typically rated for 1kV or more. Its possible that something causes a large transient on the ground of your equipment that pulls the isolated DC side of the wallwart to high voltages, high enough for that EMI suppression capacitor or something else to arc over, causing your DC ground to get connected to live or neutral for a short moment via the arc. This could cause a huge current to flow back into the DC jack of the equipment and do something bad.

Id check all the ground connections and make sure the rack is grounded properly. If there is PoE being used that could be a source of issues too.
 

Offline nfmax

  • Frequent Contributor
  • **
  • Posts: 871
  • Country: gb
Re: Continuous Router Failure
« Reply #12 on: September 24, 2019, 07:04:47 am »
Just maybe, something living at that one site is small enough to crawl in through the hole where the PSU connector fits, and has a taste for eating resistors?
 

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 1993
  • Country: us
Re: Continuous Router Failure
« Reply #13 on: September 27, 2019, 10:44:28 pm »
If you are worried about common surges coming over ethernet coupling to the power supply circuit through the POE pickoff, try putting a protection circuit on that?  There are purpose built ethernet surge suppressors or if you have some POE injectors laying around they would isolate the common mode voltage.  Although honestly the POE pickoff should be surge protected on the boards and if you don't see any damage there I think it is unlikely the source.
 

Offline wraper

  • Supporter
  • ****
  • Posts: 10431
  • Country: lv
Re: Continuous Router Failure
« Reply #14 on: September 27, 2019, 11:01:53 pm »
@ejeffrey - not physical interaction with devices other than just installing and leaving on.
Without someone poking PCB with some sharp object it's only possible if installation was done by apes. 4 ceramic capacitors in 2 different places don't simply disappear from PCB with leftovers of terminals left on the pads and scratches on PCB. Also one resistor is mechanically damaged, part of it missing.
 
The following users thanked this post: fourtytwo42

Offline wraper

  • Supporter
  • ****
  • Posts: 10431
  • Country: lv
Re: Continuous Router Failure
« Reply #15 on: September 27, 2019, 11:22:43 pm »
These devices are in enclosures in a secure area and tend to happen when no one is there. So unlikely it's sabotage.
Then someone infiltrates that "safe" area, opens enclosure and does that dirty job. 100% mechanical damage, I've seen plenty of knocked off MLCC to know how it looks like. Ceramic capacitors do not explode or teleport in some other ways. If they fail short and high current passes through them, PCB may scotch. But they won't dissapear with remnants of their terminals left and no single IC without hole in the middle, no burned traces.
Don't forget that this can be easily done through ventilation holes or some other gaps in enclosure.
« Last Edit: September 27, 2019, 11:24:36 pm by wraper »
 
The following users thanked this post: fourtytwo42

Offline DuncanCT

  • Contributor
  • Posts: 7
  • Country: za
Re: Continuous Router Failure
« Reply #16 on: October 08, 2019, 01:48:49 pm »
After a couple more failed devices we have been able to match the time stamps of the failures to the workshop next door turning off their compressor (medium sizes wood working shop).

Electricians have installed surge protection on the incoming lines but they are not helping (not sure of the spec or quality). Have also lost faith in their abilities. What are the tricks to surge protection for sensitive electronics?
 

Offline MagicSmoker

  • Super Contributor
  • ***
  • Posts: 1308
  • Country: us
Re: Continuous Router Failure
« Reply #17 on: October 08, 2019, 02:19:09 pm »
Line reactors are useful at integrating away steeply rising surge voltages, though they are typically intended for flattening out the peaks of current drawn by variable frequency motor drives. See, for example: https://www.mtecorp.com/products/reactors/rl-reactors/


 

Offline wraper

  • Supporter
  • ****
  • Posts: 10431
  • Country: lv
Re: Continuous Router Failure
« Reply #18 on: October 08, 2019, 02:23:28 pm »
After a couple more failed devices we have been able to match the time stamps of the failures to the workshop next door turning off their compressor (medium sizes wood working shop).

Electricians have installed surge protection on the incoming lines but they are not helping (not sure of the spec or quality). Have also lost faith in their abilities. What are the tricks to surge protection for sensitive electronics?
How surge protection is supposed to help against someone poking parts with a sharp object? What a joke that you think electric surge can do that  :palm:. If you don't believe me, put security seals on enclosure equipment is located within.
« Last Edit: October 08, 2019, 02:26:01 pm by wraper »
 

Offline EEEnthusiast

  • Regular Contributor
  • *
  • Posts: 146
  • Country: in
  • RF boards, Precision Analog, Carpentry
    • ZSCircuits
Re: Continuous Router Failure
« Reply #19 on: October 08, 2019, 02:33:10 pm »
Is there a chance that water can creep into the router somehow. The damage seems to be on the edge of the board where the moisture is likely to come in from the enclosure edges or opening vents.
Making products for IOT developers
www.anglercircuits.com
 

Offline wraper

  • Supporter
  • ****
  • Posts: 10431
  • Country: lv
Re: Continuous Router Failure
« Reply #20 on: October 08, 2019, 02:38:21 pm »
Is there a chance that water can creep into the router somehow. The damage seems to be on the edge of the board where the moisture is likely to come in from the enclosure edges or opening vents.
And cause capacitors teleport from PCB with no signs of liquid/ corrosion?
 

Offline DuncanCT

  • Contributor
  • Posts: 7
  • Country: za
Re: Continuous Router Failure
« Reply #21 on: October 08, 2019, 03:37:48 pm »
Attached photos from two of the latest devices.

I've quiet certain if I wanted to waste money, I could induce a failure. This still doesn't help me find a solution.
 

Offline wraper

  • Supporter
  • ****
  • Posts: 10431
  • Country: lv
Re: Continuous Router Failure
« Reply #22 on: October 08, 2019, 03:43:12 pm »
Attached photos from two of the latest devices.
Someone poked part with a metal object while router was powered and caused short circuit to enclosure from top of terminal.
Quote
I've quiet certain if I wanted to waste money, I could induce a failure. This still doesn't help me find a solution.
Ensure it's inaccessible. Put a lock, or replace it if there might be other people with a key. Attach security seal on a door of enclosure to know it was tampered with. Or place a network camera FFS to catch that idiot.
 

Offline EEEnthusiast

  • Regular Contributor
  • *
  • Posts: 146
  • Country: in
  • RF boards, Precision Analog, Carpentry
    • ZSCircuits
Re: Continuous Router Failure
« Reply #23 on: October 08, 2019, 03:58:00 pm »
Is there a chance that water can creep into the router somehow. The damage seems to be on the edge of the board where the moisture is likely to come in from the enclosure edges or opening vents.
And cause capacitors teleport from PCB with no signs of liquid/ corrosion?
The liquid may dry up after some time, but the cap disappearing is not explained. An explosion causing the cap to pulverize to fine dust.???
Making products for IOT developers
www.anglercircuits.com
 

Offline EEEnthusiast

  • Regular Contributor
  • *
  • Posts: 146
  • Country: in
  • RF boards, Precision Analog, Carpentry
    • ZSCircuits
Re: Continuous Router Failure
« Reply #24 on: October 08, 2019, 04:02:42 pm »
Do you have the part number for those capacitors? What is the voltage rating on them?
Making products for IOT developers
www.anglercircuits.com
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf