EEVblog #1288 - Synology NAS Dumpster Find!

[EEVblog]

Is a brand new Synology DS415+ NAS in the dumpster too good to be true?
A tale of teardowns, repair, Intel Atom processor silicon degradation, and dodgy clocks.

Subscribe on Library: https://lbry.tv/$/invite/@eevblog:7

[ericloewe]

This was a big deal back when it started. There was some talk of high failure rates for about a year before the problem was officially known, but I think most people were surprised by the failure rate after the story broke - it seemed to be pretty damn close to 100% in the 2.5-3 year timeframe.

What's also interesting is that there still seems to be a large number of failures from stock that should've been fixed. At the time, it looked like manufacturers like ASRock and Supermicro were swapping out entire SoCs, because nobody spotted any mods, so maybe Intel didn't quite fix things completely. I only have a passing familiarity with IC design, but I can imagine that upsizing the transistors driving those pins might have taken a major revision with multiple (all?) new masks.

C2000 was a big seller for embedded applications and some models like the C2758 (a favorite in networking equipment) were specced for 7-year lifecycles (oops).

[johnlsenchak]

I'm a huge  Synology  fanboy,   I have three  of their   NAS  drives, plus one of their WIFI  routers .   To  answer  your  question, yes    that   NAS  drive  is worth  investing the money for the power supply and  hard drives  to make it   fully   functional  in a RAID  configuration

Tech Tip  with Synology,  make sure  you  set up two factor  authentication   with your login and password  with either Google   Authenticator   or  Authy  phone app

[Razor512]

If needed, for the short run, why not use it as an Anime storage NAS while testing for reliability. Just put in a few spare drives that you have laying around, and then use it for a while to store your anime backlog. It can work as a good secondary source for the content for streaming while on the go.
And if you need more anime to properly fill it, then if you used any tracker sites such as anime planet to keep track of which series you have watched, as well as which ones you want to watch, chances are many of them may have been released without you noticing. Overall, you can use this chance to gather anime which you may have forgotten their release dates.

If the NAS fails during your few months of testing, then at least it was only content that you could reacquire.

[Monkeh]

Tech Tip  with Synology,  make sure  you  set up two factor  authentication   with your login and password  with either Google   Authenticator   or  Authy  phone app

Bigger tip, don't allow any external access at all except via suitable means..

If needed, for the short run, why not use it as an Anime storage NAS while testing for reliability. Just put in a few spare drives that you have laying around, and then use it for a while to store your anime backlog. It can work as a good secondary source for the content for streaming while on the go.
And if you need more anime to properly fill it, then if you used any tracker sites such as anime planet to keep track of which series you have watched, as well as which ones you want to watch, chances are many of them may have been released without you noticing. Overall, you can use this chance to gather anime which you may have forgotten their release dates.

If the NAS fails during your few months of testing, then at least it was only content that you could reacquire.

There's uh, more to life than anime.

[Sjokolade]

Also you should hide your serialnumber and mac addresses - if I remember correctly they can be used for virtual Synology's and if they get blacklisted I'm not sure you can have it undone.

[jnissen]

I bet dollars to donuts that the fault internal to the Atom was in the I/O driver PMOS output stage. Likely insufficient metalization for the current of that driver and load. Electro-migration,  ESD, other items likely kills the PMOS output stage. The 100 ohm pull-up is only then using the more robust NMOS output stage of the driver that still is functional. 

I've designed a lot of I/O 's and if the integration team or I/O driver layout sucked then you can get things like this. Should have been caught in a decent design review and or power analysis. EMIR design tools should have captured this but as often is the case the designer or integration team gets caught up into schedule driven madness and corners are cut or complete incompetence was involved. Still for a big name like Intel this is a black eye and should have never went out the door.

[Chromatic]

Hello,

I am the owner of a DS415+ Synology NAS with basically the same issue occuring. I let things happen and now, for every startup, it's the blue led blinking and the disk undetected.
I added this 100ohms resistor from Pin 1 to Pin 6 but without any noteable improvement.
Is there any ressource or additional investigation I could do?
I feed quite bad to throw away a +500 USD device, even if it's 4 years old...

Any help would be appreciated.

Best regards

[thm_w]

Hello,

I am the owner of a DS415+ Synology NAS with basically the same issue occuring. I let things happen and now, for every startup, it's the blue led blinking and the disk undetected.
I added this 100ohms resistor from Pin 1 to Pin 6 but without any noteable improvement.
Is there any ressource or additional investigation I could do?
I feed quite bad to throw away a +500 USD device, even if it's 4 years old...

Measure the signal before and after adding the 100R as Dave has done, if you have an oscilloscope.

[Chromatic]

Hi thm_,

In the meantime, I tested removing the battery, nothing better.
But testing the battery... 1.1V instead of 3V, well... worth the sort to replace it
Will keep you in touch after this

Regards

[Chromatic]

So...
Battery changed, still solid orange LEDs and blinking blue LED... and system reboots after ~5min
I don't know if the battery was powering only the RTC, or also some mandatory data in flash?

No oscilloscope for now, need to bring one from work

Regards

Edit: I can't explain why but after removing and inserting back the battery, then turning on the NAS... it worked !
Like in Dave's video

So... I can't say how long this tweak will work but now it is. Thanks again for the video !

[marsu]

Hello,
I am an owner of a Synology DS415+ NAS that has stopped working and suspect it is due to the issues described here.
I have a large amount of data (videos and music) on this device that I want to recover.
I don't have the equipment or skills to make the mod recommended but am happy to pay cash to get it fixed.
I am located near the Sydney CBD and would be grateful to hear from anyone that may be able to help.
I hope someone may assist me.
Owen

[EEVblog]

I am an owner of a Synology DS415+ NAS that has stopped working and suspect it is due to the issues described here.
I have a large amount of data (videos and music) on this device that I want to recover.

You can recover the data without the mod:
https://www.synology.com/en-us/knowledgebase/DSM/tutorial/Storage/How_can_I_recover_data_from_my_DiskStation_using_a_PC

[Jeroen3]

It's actually much easier if you didn't encrypt your data. Linux Mint can automatically mount the mdadm volumes.
And if you have an USB to SATA dock you can install linux mint in a VM with USB passtrough.

[ericloewe]

I'll add some new details since this thread might have some more visibility, being associated with the video and all:

Why does a hard pull-up work?
It looks like the LPC bus just rests at high using a typical pull-up (tens of kOhm) and gets driven low by the devices. The drivers for the clock lines must degrade to a point where they develop a parasitic resistance in parallel with the driver transistor that's low enough enough to overcome the normal pull-up. Adding a hard pull-up allows for enough current to flow for there to be a usable voltage at the clock output.

Why do systems fail to boot?
The tl;dr is that one of the clock output pins is used at SoC power-on to choose the boot device - either an LPC or SPI flash memory. SPI is the default, LPC can be selected by pulling it down at power-on. Since basically no C2000 system boots from an LPC flash, this places the CPU in a state where it tries to read from a flash memory that does not exist. There are several open questions (the specific pins don't seem to be correct, there are like 4-6 different clock outputs that would be suitable, how is the LPC clock supposed to work if it's pulled down externally, etc.), but that's the clearest explanation anyone's publicly provided.

Is the mod safe?
With a 150 Ohm resistor, that's some 14 mA of current constantly being sunk by the SoC, per bodged pin (many systems use at least two clocks, one for the BMC and one exposed to the TPM header). Not great, but not terrible if each pin has an absolute max rating of 50 mA, which seems to be a standardish rating for Parallel PCI clock buffers (the design of which was likely borrowed wholesale for the LPC bus).

What's the official fix?
Short of a new SoC, which nobody actually seems to have done, it's bodge resistors on any affected LPC clock lines. Supermicro uses 150 Ohm, ASRock uses 120 Ohm.

Clock lines plural?
Many/most systems need more than just the one clock line used for boot:

• TPMs connect over the LPC bus (hence why it's exposed in the first place). Not typically critical, except for the whole wrong boot device thing.
• BMCs connect over the LPC bus for in-band management, to provide sensors info and to provide SuperIO functionality. Typically not critical, but very annoying. Does not break iKVM, but don't lose your BMC password.
• Some systems have additional SuperIO controllers for fans, serial, etc. (ASRock uses a BMC and a SuperIO, because they like wasting money I guess.)

I guess Intel's reference design just puts the TPM on the clock line that controls the boot device and everyone followed along, or something. Fortunate coincidence.

More info (and examples of fixes for Supermicro and ASRock Rack boards):
https://www.eevblog.com/forum/microcontrollers/intel-atom-c2000-failures/
https://www.truenas.com/community/threads/fyi-intel-c2000-family-of-processors-system-fault-may-lead-to-dead-system.50314 (Shameless plug alert)