Tin Whiskers

[sibeen]

Thanks for posting this one, Dave, and hopefully it is educational for many.

I work in the hundreds and thousands of amps, mucking around with data centres, switchboards, large UPS systems, HVAC, power quality etc. Tin whiskers does not really enter my sphere of operation but its closely related cousin 'Zinc Whiskers' kept me well occupied for quite a few years.
I first ran into zinc whiskers around 25 years ago. I'd been called out to a small data centre where they were experiencing an extraordinary number of failures of Sun servers. A few a week where giving up the ghost and the management of the centre were not happy campers. The data centre was typical, a raised floor computer room supporting racks of equipment which were cooled by airflow that came from under the floor. I immediately assumed that there was a power issue on the site and installed a power disturbance monitor, a BMI 8800 for those playing at home, and left this for a week or two to gather data. Came back after the monitoring period to be told that a few more servers had served their last. Went back to the office to begin the examination of data.


The BMI 8800 printed out its reports on thermal paper. It would print out daily status reports and any time that it had captured an event it would print out a list of parameters for the event as well as an oscillogram of the event. So even a short investigation like this could generate quite a ream of paper. The good old days weren’t . I trudged through this ream of data spotting an ‘out of parameter’ event now and then but nothing that could be pointed to as a smoking gun that could have caused any server failures.

Back to site and lots of crawling under the floor, jerking on cables, taking photos, all the while not realising that I was exacerbating the problem. I convinced the client to install some high speed surge suppression (silicon avalanche) devices within one section of the data centre to try to narrow the problem down. Still. no resolution. Servers were still going down for no rhyme nor reason and it didn’t matter where I installed monitoring or suppression. At this point the data centre management were not the only one’s losing hair.

The funny thing is, all these years later, I have no memory of how I discovered what the issue was. Whether it was through a desperate internet search, very unlikely as the internet was still nascent; a technical paper I came across, a wiser older colleague. I really have no memory of the aha! Moment. Turns out that it was the floor tiles. The bottom of them was zinc plated and from this plating zinc whiskers could quickly grow. Lifting a tile and shining a torch along it could quickly show the whiskers being produced. The whiskers were being disturbed, floating around and eventually landing upon some delicate part of the server.

Data centre management were even unhappier when I told them the problem, “your room is killing your computers”. Very expensive to mitigate. Shut everything down and pull it out of the room. Have it extensively cleaned. In the room pull up the floor, have the whole room ‘clean room’ cleaned. Replace the flooring tiles with a new version that does not contain zinc. Replace equipment, start up again and then pray.

Once I got through that first one I then found it popping up regularly for the next ten years or so. I’d get called in to a data centre to resolve a mysterious problem. “We turn on our air-conditioning and our computers start blowing up” would be a common refrain. I’d grab a floor tile lifter, bring up a tile, look at the base and then shake my head in a sorrowful manner. This was a very expensive problem to fix. It got to the stage where I’d get phone calls and I wouldn’t even bother going to site. I’d spend ten minutes on the phone and diagnose it then and there and then hang up before the sobbing and wailing became too extreme. The industry learnt, adapted and moved on.

Continued in part 2...

[sibeen]

Part 2...

Or so I thought . I received a panicked phone call on a Sunday evening two years ago. A client had a facility which had suffered two catastrophic failures within a few days. Could I please gat on a plane first thing in the morning and figure out what the hell had happened.  Got to site the next day and found that upgrade works had been taking place. A new DC power pant was being installed and during these works the original system had suffered a massive failure causing loss of power to the site. The new gear was then quickly brought on-line and the site transferred across to the new power system. This lasted for about a day before the new power system went down (Sunday evening) causing another site shutdown.
 
I went in, kicked the tyres, interrogated the culprits, and quickly determined that there was no problem with the new power system but that the circuit breakers feeding it had been well under sized, had overloaded and then tripped. An easy fix. I then moved onto the original problem. The installers of the new power system swore black and blue that the failure had nothing to do with them. They were working in a completely different room and hadn’t touched the old system.
 
I then got onto the centre management. They told me that an alarm had come up on the old power system and they’d gone to investigate. It was a system that consisted of lots of plug in modules and one had failed. They were standing in front of the unit when another module failed, and then another, and then another. Eventually the rack had gone down and power to the facility was lost. The room the power system was in was nearly empty. It had been cleared out and new gear was slated to be installed in a month or two (see photo). I asked whether anybody was working in the room at the time and they answered ‘”no” but then one of them stated that there was a bloke down the end of the room, but he was only pulling redundant cabling out from under the floor and that couldn’t have had anything to do with it.

A lightbulb moment and a blast from the past. I grabbed a tile lifter, banged it onto a tile, lifted and rotated, and then grinned in recognition (see photo). The company that I was doing this job for was the same company that I’d first come across the problem all those years ago. Different data centre, different state in fact, but the same company. Corporate memory had been lost. New data centre managers had never come across the problem. They were all rather amazed about it all and how it had been resolved fifteen or twenty years before.

Invoked memories
 

[sibeen]

I don't know why but I cannot upload the few photos that I wanted to show with these posts.

Bugger!

[Brumby]

I don't know why but I cannot upload the few photos that I wanted to show with these posts.

Bugger!

How big are they - and while file extension do they have?

[Barny]

My fitst encounter with zinc wiskers was as a little boy when I grabbed bar handed in a box of zinc plated shackles.
(I think I wanted to use them to lift a wight with the winch)

Grabbing this damn shackles felt like touching a cactus.

After this, I never vorgot to put on my gloves again.

[mahwe]

https://nepp.nasa.gov/Whisker/background/index.htm
nice Photos from the NASA to Whiskers

[IanMacdonald]

For those restoring vintage gear, tin whiskers are a big problem with the AF117 and similar four-wire germanium RF transistors. One of the electrodes would short to the case.  A common fix is to disconnect the shield lead. That might only work for a limited time though until another lead gets shorted.

http://www.vintage-radio.com/repair-restore-information/transistor_transistor-faults.html

[coppice]

Is there going to be another video on tin whiskers? I was really disappointed at the end of the first one. He referenced some interesting things he intended to show later, but video stopped before he got to them.

[gildasd]

NASA vid on metal whiskers  in vacuum: