NVME read speed

[PlainName]

Trying to sort out an issue with an NVME drive. The problem is that it's slow. Really slow, like sometimes <1MB/s during backups. This is read speed and is difficult to pin down. A Crystal Disk Mark test shows nothing untoward, but do a backup and after a little bit of full speed stuff it slows to a crawl. Eventually I replaced the SSD with a WDC job, and things are back to being fast again.

Except... I ran the HD Tune benchmark on both the current WDC and previous Silicon Power, and noticed that read speed is slower where there is data. On the SP, which was the problem drive, it is really really pronounced so I thought there was some fault in the drive. But now I notice that the WDC is showing the same symptom, although in use it's not apparent. The attached benchmark for the WDC illustrates the issue.



The HD Tune benchmark is really aimed at spinning disks, so it starts at sector 0 and then reads sequentially to the end sector. It ignores any filesystem and just hits the disk direct, whether there is actual data there or not. The graph shows the start of the drive on the left, the end on the right.

This drive has 5 partitions with a little free space for expansion between them. The dips correspond to partitions, and the peaks between them to the free space. Only the first 500GB is in use (the rest is for expansion/wear). Seems pretty clear to me that where there is actual data the read speed is slower than where there isn't.

I tried to force CDM to use the trough for benchmarking, but it writes the data it's going to read first, so of course it's going to be a relatively unused space. It also only shows an average, which can hide what we can see here.

I find it strange that this effect should exist. Am I missing something that might explain it, or is it a real thing? I note that SATA SSDs don't show this, but then the read spead is substantially slower than the chip capability so may be hiding the same thing.

[Marco]

AFAIK they generally detect formats and zero writes and just mark the block as zero in the metadata, reading just the metadata to return all zeros is obviously going to be faster.

[olkipukki]

Trying to sort out an issue with an NVME drive.

Why do you think you have NVME and not SATA?   

[mariush]

If he gets speeds higher than ~560 MB/s, he has a nvme SSD.

I have some issue with your comments about leaving some room for expansion and for wear.

There's no need to leave unused space for "wear" - the SSD controller doesn't store the data like a mechanical drive, data ends up at some random offset in one of the flash memory chips and there's a translation table stored in the SSD which keeps track of where the data of a particular sector is actually located in the nand chips.

So whether you leave some room for "wear" or not, those flash blocks will be used just like any other flash memory so it's pointless to not have it included in a partition.

Also note that modern SSDs use a portion of the flash memory in pseudo-SLC mode, for caching writes - that's another reason why data is more or less random on the ssd.

For example, in the case of your 1 TB SSD which uses TLC memory (3 bit per cell), the SSD controller can take up to maybe 400 GB worth of TCL memory and flip it to pseudo-SLC memory storing just 1 bit in every cell.
Those 400 GB of TLC end up to maybe 256 GB of SLC memory - these are not the exact numbers, I don't feel like looking the exact values, but it's not that critical - so if you write something to the SSD, it gets quickly dumped in this portion of SLC memory and later when the SSD is idle, the data is moved to regular TLC memory emptying the SLC cache.

As you fill the drive with data, the ssd controller has to "release" chunks of the pseudo-SLC cache, converting that back to TLC ... for example, if there's 50 GB empty space, the controller takes 16 GB of pseudo-slc memory and converts it back to 20-24 GB of TLC memory and now you have 70-74 GB of free space.

So HDTach doesn't do the SSD a favor by requesting data sector by sector, in a linear fashion. The SSD controller has to constantly look up into its internal translation table to see where exactly is sector 0 stored, then sector 1, and so on...

[PlainName]

Trying to sort out an issue with an NVME drive.

Why do you think you have NVME and not SATA?   

Because it's not an SATA interface, it's sold as NVMe, the connector is NVMe, it's reported as NVMe, and when it's fast it's MUCH faster than SATA.

Other than that, dunno

[PlainName]

AFAIK they generally detect formats and zero writes and just mark the block as zero in the metadata, reading just the metadata to return all zeros is obviously going to be faster.

OK, I could go with that for the inter-partition stuff. But surely then the used blocks would return broadly similar speeds. We'd expect to see two levels (used, unused), both reasonably flat.

[wraper]

Trying to sort out an issue with an NVME drive.

Why do you think you have NVME and not SATA?   

Because it's not an SATA interface, it's sold as NVMe, the connector is NVMe, it's reported as NVMe, and when it's fast it's MUCH faster than SATA.

Other than that, dunno

There is no NVMe connector. There is M.2 connector and depending in implementation it may support NVMe, SATA or both types of SSD. Your dive probably is a DRAM-less garbage.

[Marco]

OK, I could go with that for the inter-partition stuff. But surely then the used blocks would return broadly similar speeds. We'd expect to see two levels (used, unused), both reasonably flat.

Well, there's also trim. So deleted files will create empty blocks all over.

[PlainName]

There's no need to leave unused space for "wear"

It's for dynamic over-provisioning.

By 'expansion' I mean increasing the partition size as I run out of space. Why don't I allocate it all at once? Well, I don't know which partition is going to fill up, so I might end up with one full and next door empty. Also, not have oodles or space doesn't tempt me to fill it up with rubbish that's then too much hassle to figure out what can be deleted when I eventually run out space.

So HDTach doesn't do the SSD a favor by requesting data sector by sector, in a linear fashion. The SSD controller has to constantly look up into its internal translation table to see where exactly is sector 0 stored, then sector 1, and so on...

Yeah, I can see that. But... that's how backups are done, so it's a real-world problem if that's the cause. And remember that what kicked this off was backups taking 4 hours instead of 20 minutes.

[PlainName]

For reference, this is the Silicon Power SSD benchmark when I pulled it. Note that it's being accessed via a USB3.1 enclosure, hence the max rate is low for NVMe. But the low rate is pukka.

[AntiProtonBoy]

Except... I ran the HD Tune benchmark on both the current WDC and previous Silicon Power, and noticed that read speed is slower where there is data. On the SP, which was the problem drive, it is really really pronounced so I thought there was some fault in the drive. But now I notice that the WDC is showing the same symptom, although in use it's not apparent.

Have you tried replicating the issue under a different OS, like a bootable Linux image?

Also worth trying is updating the chipset drivers.

[PlainName]

Have you tried replicating the issue under a different OS, like a bootable Linux image?

No, not least because I don't think HD Tune runs on non-Windows. However, in case it was something in the OS getting in the way (and because I might have to send it back) I did a full erase with random data. The graph you see of the SP is after doing that, so the partitions and original data (and blank space) no longer exist  but still show up in read performance.

[olkipukki]

What is firmware version/revision? Does it start from 21 or 23?

[PlainName]

Revision of...?

If it's the SP, I don't know. The Silicon Power tool doesn't recognise the drive.

[olkipukki]

WDC one

[PlainName]

2112 according to SMART.

Thought I had the WDC tools, but apparently not. I'll remedy that in a bit

Edit: Found out after downloading 300MB of it that I did have it after all, but it's called the Sandisk SSD Dashboard. Hmmm. Anyway, that agrees that it is 2112. And is up to date.

[olkipukki]

In short, just replace it…

https://www.overclock3d.net/news/storage/western_digital_promises_to_do_better_following_sn550_qlc_scandal/1

[PlainName]

Yeah, they all seem to be doing that. Crucial did it with their P2, etc. But that affects write speeds and here it's read speeds that are noticeably adrift.