EEVblog 1690 - Mailbag: Zifnu LED, Quantum Diamonds, Logic Gates, M.2 NAS

[EEVblog]

More Mailbag!

00:00 - Zifnu K-16 Kickstarter LED light
https://www.zifnu.com/
11:02 - Qoool Sensing: Diamond Quantum Fluorescence Demo
https://qoool-sensing.org
16:41 - Logic Gate Educational Demo Board
https://logicgat.es/
20:15 - Beelink ME Mini 6 x M.2 Slot Home Storage NAS
https://www.bee-link.com/products/beelink-me-mini-n150

[coppercone2]

i wanna see the reaction to quantum sensing after ignoring the warning to read the note

its rare to get a device that strange. then it would be like something from a ufo

and it sounds like you missed the color changing function ?

[EEVblog]

Email from a viewer about the Beelink M.2 NAS

You mentioned that although you already have your Synology NAS you might
be using this one for something else. Beelink's Mini ME seems to have
under-powered PSU and/or their single 3.3V rail is not suitable to
handle multiple SSD drives. Especially be careful if you install more
than 4 drives and plan using them simultaneously e.g. in a RAID
configuration.
---

I installed 6 WD Red SN700 2TB SSDs, changed OS to TrueNAS since it's
more suitable for the job and configured all drives to work as a single
RAID storage.

Initial testing looked promising even though thermals weren't that good.
55-58 deg C in idle for a low power device while not horrible wasn't
good either. All drives consumed too much energy while being idle.
Each reported entering P4 (sleep) mode, which according to WD datasheet
should make each that model consume as little as 5mW. Seems however that
each was pulling over 1W while doing nothing.
I don't know if that's issue with Beelink's hardware, BIOS, NVMe driver
in OS, some WD shenanigans or any combination of the above.
With 6 drives installed at idle I measured 15-17W on 240V side. Without
drives installed it was 5-6W.

Aside from that it seemed fine so I decided to give it a go and do some
futher testing. I configured everything and began copying data to my
shiny, new NAS.
After copying few GiB of data from my PC to ME mini ZFS pool crashed as
2 drives failed simultaneously. System logs mentioned drives entered
D3cold state and couldn't get out of it so drives were inaccessible:

Aug 31 00:24:03 nas kernel: nvme nvme0: controller is down; will reset:
CSTS=0xffffffff, PCI_STATUS=0xffff
Aug 31 00:24:03 nas kernel: nvme nvme0: Does your device have a faulty
power saving mode enabled?
Aug 31 00:24:03 nas kernel: nvme nvme0: Try
"nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off"
and report a bug
Aug 31 00:24:03 nas kernel: nvme 0000:03:00.0: Unable to change power
state from D3cold to D0, device inaccessible
Aug 31 00:24:03 nas kernel: nvme nvme0: Disabling device after reset
failure: -19
Aug 31 00:24:04 nas kernel: zio pool=pool
vdev=/dev/disk/by-partuuid/e3a8c2cf-48bf-43f3-8d0f-18b2b062a6d7 error=5
type=2 offset=152168439808 size=4096 flags=3145856
(...)

Reboot helped and since I didn't care about that data, I cleared ZFS
pool state to make TrueNAS show all-green again.
I began copying again and same thing happened but this time 1 drive
failed and to make things worse: a different one to previous two.
I had it running in idle (or at least close to idle) for few hours and
it was fine.
I tried that multiple times and each time situation repeated. Every time
1 to 3 drives "failed". Mostly random, but I very often same 2 ports
failed. I tried switching SSD between ports to rule out that specific
drives are problematic. Again same ports failed so that showed that it's
most likely not the drives themselves. I was also doing read/write tests
before that at their full speed on single drives and it was also fine.
All drives also passed SMART tests.
Since it happened only during load it seemed more like PSU problem than
anything else.
I made a simple test rig using a third-hand tool to hold oscilloscope
probe on 3.3V line going to one of M.2 ports. Luckily the way drives are
oriented makes it easy to access those pins. Fact that 3.3V lines are
grouped on connector edges also helped a lot. I just placed my probe
between two adjacent 3.3V pins. That made whole rig stable enough to do
longer testing.
Already during OS boot I observed voltage drop below 2.5V for at least
140us. You can see that in attached boot_voltage_drop.png screenshot
file. Time base was too short to see how long it really was.
During copying data to NAS voltage instability seemed smaller, but also
went below 3.0V. Keep in mind that I copied data over a 1Gbps ethernet
connection so writing speed was only around 100MiB/s. This is way below
what those drives can handle so I assume average power usage wasn't even
close to maximum.
This can be observed in voltage_drop_during_copying_port3.png file. I
enabled persist mode to see what was happening from past 5 seconds.
Please forgive my newbie setup.

According to SSD manufacturer this model is rated up to 2.8A (peak
power, 10us) at 3.3V. That totals to peak power usage of a bit over 55W.

I tried to find voltage regulators near ports, but couldn't see anything
like that. I did see 6 bigger capacitors. Probably bypass ones for each
port. I didn't want to detach PCB and take heat sink off so I might have
miss something.
Curious, I buzzed-out all 3.3V lines on M.2 ports using my multimeter
and it seems that all 6 ports share a single 3.3V bus.
This suggests that they use a single power source for all 6 ports.
Perhaps even for the whole system including wireless card, eMMC and so on.
That would explain why issue happens more or less at random during load.
Sadly I don't see any easy way to measure 12V line coming out of PSU
without disassembly. I wonder whether it's just 3.3V source that is
under-powered or the whole system is. Or both.
Main PSU is rated for 12V @ 3.75A so just 45W. This doesn't seem enough
for a system sporting six M.2 ports meant to be used for SSDs.
That makes me wonder how did they test this thing.

I followed few suggestions given by Beelink's support. Running OS with
PCI power management effectively disabled (same thing system logs
mentioned) and re-seating all drives. Nothing helped.
One suggestion was to install the OS on 4th SSD port - the faster one.
While this workaround (as I cannot call this a solution) might seemingly
help by making one of SSDs (and eMMC) to be idle most of the time, there
might still be situation when all 6 drives would operate at the same
time: storage ones and OS doing something like writing logs etc. I
consider this workaround to be especially bad as it gives a false sense
of security.

Seeing such instability I doubt even 5 drives would work ok. To me
device is defective. I can't trust it so I'll ask for an RMA.

I just wanted to let you know since you said that you might be using
Mini ME to store your data. Even if it wouldn't be your main NAS,
loosing data could be annoying.