Author Topic: Strange hard disk performance issue  (Read 2549 times)

0 Members and 1 Guest are viewing this topic.

Offline HalcyonTopic starter

  • Global Moderator
  • *****
  • Posts: 5629
  • Country: au
Strange hard disk performance issue
« on: January 25, 2023, 12:11:51 am »
I have a bunch of HGST 2TB 7K4000-series SATA drives (exact model is HUS724020ALA640) which were pulled from either a NetApp or IBM server some time ago (I don't remember which).

I've been attempting to securely erase them, but have found the performance is absolutely shocking, like abnormally so. Chopping it up to a potentially dead drive, I tested a few of the others, same thing. They seem to be reading/writing OK but just extraordinarily slowly (like it would take 3+ days just to format 2TB). Obviously something isn't right here.

My only hypothesis so far is that the drives have some kind of custom firmware from NetApp/IBM to prevent disk shucking and re-use in other machines and/or to ensure their customers only buy their "special" disks, but this is just pure speculation at this point.

This is not something I've ever run into before, particularly on SATA disks (these aren't even SAS). Has anyone got any thoughts? I have about 10 of these drives and would rather not shred them if they are still perfectly good.

 

Offline MrMobodies

  • Super Contributor
  • ***
  • Posts: 1906
  • Country: gb
Re: Strange hard disk performance issue
« Reply #1 on: January 25, 2023, 12:26:08 am »
Have you tried MHDD (dos) to see if the sector readings are indeed slow?


It reminds me a bit of Connor drives that I found very slow.

Back in 2005, I brought 4x 36GB 10,000 rpm scsi drives from an Ebay seller.

I connect them up, formatted all them and two failed.
I told the seller this and refunded me.
Now when I say formatted I mean by the controller before joining them on raid.

I noticed that the two that failed were HP branded on the sticker whereas the others were Seagate. Exactly the same drive, model number, shape and pcb board no difference in appearance.
« Last Edit: January 25, 2023, 06:55:20 am by MrMobodies »
 

Offline JeremyC

  • Regular Contributor
  • *
  • Posts: 135
  • Country: us
Re: Strange hard disk performance issue
« Reply #2 on: January 25, 2023, 01:15:53 am »
I have a bunch of HGST 2TB 7K4000-series SATA drives (exact model is HUS724020ALA640) which were pulled from either a NetApp or IBM server some time ago (I don't remember which).

I've been attempting to securely erase them, but have found the performance is absolutely shocking, like abnormally so. Chopping it up to a potentially dead drive, I tested a few of the others, same thing. They seem to be reading/writing OK but just extraordinarily slowly (like it would take 3+ days just to format 2TB). Obviously something isn't right here.

My only hypothesis so far is that the drives have some kind of custom firmware from NetApp/IBM to prevent disk shucking and re-use in other machines and/or to ensure their customers only buy their "special" disks, but this is just pure speculation at this point.

This is not something I've ever run into before, particularly on SATA disks (these aren't even SAS). Has anyone got any thoughts? I have about 10 of these drives and would rather not shred them if they are still perfectly good.

I would try connect the drive to a Linux machine and try write zeros.
You must be root to do it.
Assuming that this drive is /dev/sdb:

   dd if=/dev/zero of=/dev/sdb bs=10M

Detailed information about "dd": https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html#dd-invocation
« Last Edit: January 25, 2023, 01:24:45 am by JeremyC »
 

Offline edpalmer42

  • Super Contributor
  • ***
  • Posts: 2260
  • Country: ca
Re: Strange hard disk performance issue
« Reply #3 on: January 25, 2023, 02:51:37 am »
1.  Use a program like CrystalDiskInfo to check the SMART parameters to see if the drives are just worn out.  Look for 'uncorrectable sectors' or 'reallocated sectors'.

2.  When you say 'format' do you mean a Windows format without the quick format option?  If so, you could be bumping up against various OS limitations.  Either use the previously mentioned dd under Linux or a utility program from one of the drive manufacturers.  I always use Western Digital's Data Lifeguard program.  They've discontinued it, but it's widely available at various download sites.

3.  Do you have any security software running that might be interfering with the format process?

4.  How are you connecting the drive to your system?  Are your drives connecting at SATA 2 or 3 speeds?  This drive has a sustained throughput of 171 MB/s which translates to ~1.4 Gbps.  If your drive is connected through a USB2 or SATA1 connection your performance will be poor, but shouldn't be as poor as you describe.  CrystalDiskInfo will tell you what type of connection you have.
 

Offline HalcyonTopic starter

  • Global Moderator
  • *****
  • Posts: 5629
  • Country: au
Re: Strange hard disk performance issue
« Reply #4 on: January 25, 2023, 05:40:36 am »
SMART checks out. In any case, I'm experiencing the same issue across multiple (identical) drives.

The issue occurs regardless whether I connect it via a USB interface or directly on the SATA controller. It also happens across multiple machines.

Windows format, or a utility like DBAN or MHDD produce the same slow results when erasing (about 8 MB/sec). Strangely the MHDD "scan" option appears to read the surface at proper speeds.

Doing some further investigation as we speak.
 

Offline DrGeoff

  • Frequent Contributor
  • **
  • Posts: 793
  • Country: au
    • AXT Systems
Re: Strange hard disk performance issue
« Reply #5 on: January 25, 2023, 05:42:28 am »
Try running iometer HDD profiling and diagnostics to see if they are any use.

Iometer project
http://www.iometer.org

Was it really supposed to do that?
 

Offline edpalmer42

  • Super Contributor
  • ***
  • Posts: 2260
  • Country: ca
Re: Strange hard disk performance issue
« Reply #6 on: January 25, 2023, 06:05:11 am »
If they're trash anyway, maybe you could try updating the firmware and see what happens.   :-//

https://www.dell.com/support/home/en-ca/drivers/driversdetails?driverid=xw1yg

http://files.hddguru.com/download/Firmware%20updates/Hitachi/

etc.
 

Offline HalcyonTopic starter

  • Global Moderator
  • *****
  • Posts: 5629
  • Country: au
Re: Strange hard disk performance issue
« Reply #7 on: January 25, 2023, 08:06:32 am »
If they're trash anyway, maybe you could try updating the firmware and see what happens.   :-//

https://www.dell.com/support/home/en-ca/drivers/driversdetails?driverid=xw1yg

http://files.hddguru.com/download/Firmware%20updates/Hitachi/

etc.

Good suggestion, I tried the Dell firmware, it fails to install as the "system doesn't meet specifications" (even on an actual Dell machine).

I'll try some other sources.

From what I've been able to tell, these drives are probably vendor-locked. They work just enough for you to recover data from them, but not usable for actual real-world applications as the write speed is limited.
« Last Edit: January 25, 2023, 08:15:10 am by Halcyon »
 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 16544
  • Country: us
  • DavidH
Re: Strange hard disk performance issue
« Reply #8 on: January 25, 2023, 03:08:13 pm »
I have used the same drives without problems on my ARECA raid controllers, and with MHDD and HDAT2.  One or the other of MHDD or HDAT2 usually runs slow, but the ARECA controller has its own throughput tests which show full speed operation.

Take a close look at the SMART diagnostics to see if UDMA error are present which would indicate a problem with the SATA interface.

If the drives are going bad, then the reallocated sector count should be high, or the available sectors low, or there should be pending sector reallocations.
 

Online PA0PBZ

  • Super Contributor
  • ***
  • Posts: 5121
  • Country: nl
Re: Strange hard disk performance issue
« Reply #9 on: January 25, 2023, 04:19:20 pm »
A vendor lock usually works the other way around, it's not to prevent you from using their drives in another system, it's there so you can only use their drives (bios) in their system and not the oem version.
Keyboard error: Press F1 to continue.
 

Offline AndyBeez

  • Frequent Contributor
  • **
  • Posts: 855
  • Country: nu
Re: Strange hard disk performance issue
« Reply #10 on: January 25, 2023, 08:59:26 pm »
I'm not sure that I'm bringing anything new to the game here, but one thought is is some hidden or hard to kill EFI (or similar) partition info stuck in a graveyard orbit around the spindle. To burn up the junk requires using the Windows diskpart utility; with this you can carry out a (highly) destructive clean of the disc's information. A clean should considered an option of last resort. A cleansed disc is then reinitialised with a fresh MBR and reformatted as NTFS.

btw I had a look at the HGST (Western Digital) datasheet for this model and there is nothing to suggest an OEM or vendor locking option - either onboard or BIOS based. Maybe re-flashing the EEPROM might work, as this could be some unpublished RAID setup.

About diskpart:

https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/diskpart

https://www.windowscentral.com/how-clean-and-format-storage-drive-using-diskpart-windows-10

 

Offline nightfire

  • Frequent Contributor
  • **
  • Posts: 576
  • Country: de
Re: Strange hard disk performance issue
« Reply #11 on: January 25, 2023, 11:03:49 pm »
Also a known fact is that Netapp uses 520 bytes sectoring along with special firmware to do so, at least in the classic machines that used FC based drives or SAS.
Could also be a contributing factor to the whole hiccup, as they also use their proprietary WAFL filesystem on them.
 

Offline HalcyonTopic starter

  • Global Moderator
  • *****
  • Posts: 5629
  • Country: au
Re: Strange hard disk performance issue
« Reply #12 on: January 26, 2023, 02:23:06 am »
Take a close look at the SMART diagnostics to see if UDMA error are present which would indicate a problem with the SATA interface.

If the drives are going bad, then the reallocated sector count should be high, or the available sectors low, or there should be pending sector reallocations.

Zero errors according to SMART and surface read tests check out totally fine. I don't think the drives are faulty at all, it's looking more and more like custom firmware designed to be used in certain servers only.

Also a known fact is that Netapp uses 520 bytes sectoring along with special firmware to do so, at least in the classic machines that used FC based drives or SAS.

I haven't considered sector size. I'll check that out.

A vendor lock usually works the other way around, it's not to prevent you from using their drives in another system, it's there so you can only use their drives (bios) in their system and not the oem version.

Yes usually, but it can work the other way around. For example, some manufacturers are vendor-locking AMD CPU's so you can only use their CPU's in their machines (and vice versa). There is some vague information out there about vendor-locked hard disk drives as well. Whilst HGST/WD might not mention it in public specs, that doesn't mean they don't provide the mechanism and firmware to vendors to be able to spin their own custom firmware under some kind of agreement that is not known to the public. Similar kind of agreement to what Dave has with Brymen and flashing/programming his custom multimeters.
« Last Edit: January 26, 2023, 02:29:08 am by Halcyon »
 

Online Ed.Kloonk

  • Super Contributor
  • ***
  • Posts: 4000
  • Country: au
  • Cat video aficionado
Re: Strange hard disk performance issue
« Reply #13 on: January 26, 2023, 05:43:25 am »
Such shitcanery.

Wasn't it Compaq who used to keep the bios on the HDD instead of on the mainboard?

I thought we'd all grown up and stopped doing this rotten bullshit.
iratus parum formica
 

Offline Haenk

  • Super Contributor
  • ***
  • Posts: 1076
  • Country: de
Re: Strange hard disk performance issue
« Reply #14 on: January 26, 2023, 09:06:15 am »
As Test-Software, I'd highly recommend "Hard Disk Sentinel".
The odd sector size could really be an issue, chances are you are not able to change it. However another controller might be worth a try.
Did you check, if these are OEM drives (with no manufacturer warranty)? Maybe check firmware revision as well?
 

Offline AndyBeez

  • Frequent Contributor
  • **
  • Posts: 855
  • Country: nu
Re: Strange hard disk performance issue
« Reply #15 on: January 26, 2023, 09:51:41 am »
Have you taken a forensic look at the SMART feature set using Linux smartctl?

https://linux.die.net/man/8/smartctl
 

Offline HalcyonTopic starter

  • Global Moderator
  • *****
  • Posts: 5629
  • Country: au
Re: Strange hard disk performance issue
« Reply #16 on: January 27, 2023, 05:01:58 am »
Everything looks good on SMART. Disk checks out 100% health. Also checked the sector size, definitely 512 KB.

The strangest thing though, I just plugged in a disk to test using HDD Sentinel (to the same machines I've been using) and it seems to be writing at normal speed ~170 MB/sec. I then tested the 3 disks that I have been testing this whole time, and same thing, nil issues. Tried again with MHDD's erase command, everything is fine.

Now I can't be certain which disk is which now because I wasn't making note of the serial numbers, but I'm completely confused why all of a sudden things are working. I can't explain it. This isn't the first time I've had this issue with these exact drives, I remember when I first got them about a year ago I was seeing similar behavior so I set them aside and didn't do anything with them until now.
« Last Edit: January 27, 2023, 05:08:44 am by Halcyon »
 

Online Ed.Kloonk

  • Super Contributor
  • ***
  • Posts: 4000
  • Country: au
  • Cat video aficionado
Re: Strange hard disk performance issue
« Reply #17 on: January 27, 2023, 05:10:10 am »
Bad SATA cable?
iratus parum formica
 

Offline HalcyonTopic starter

  • Global Moderator
  • *****
  • Posts: 5629
  • Country: au
Re: Strange hard disk performance issue
« Reply #18 on: January 27, 2023, 05:22:04 am »
Bad SATA cable?

On multiple machines? Unlikely. Also, I haven't swapped out any cables and on one of them, the drive goes directly into a dock with a USB 3 interface.

Part of me is now sitting here thinking, did I just imagine this whole thing?
 

Online Ed.Kloonk

  • Super Contributor
  • ***
  • Posts: 4000
  • Country: au
  • Cat video aficionado
Re: Strange hard disk performance issue
« Reply #19 on: January 27, 2023, 05:42:40 am »
Bad SATA cable?

On multiple machines? Unlikely. Also, I haven't swapped out any cables and on one of them, the drive goes directly into a dock with a USB 3 interface.

Part of me is now sitting here thinking, did I just imagine this whole thing?

Drives are funny.

A couple of months ago I had to go away for work. I took along a laptop and WD Red choc full of cat videos and a couple of USB SATA converters. The drive was fine before I left. After a 7 hour drive north from here, I livened up the drive and got all kinds of read error messages, so severe I thought that the car trip stuffed the drive (My driving isn't -that- bad).

Since I had considered the drive lost, I kept poking at it every night with SMART short test, which failed each time until one day it didn't. And it hasn't thrown a single error since.

Only thing I can put it down to is the drive didn't like the change of location.
iratus parum formica
 

Offline rdl

  • Super Contributor
  • ***
  • Posts: 3665
  • Country: us
Re: Strange hard disk performance issue
« Reply #20 on: January 27, 2023, 06:09:55 am »
Yes, they are.
I just set up a new encrypted external drive and when copying files to it the speed was all over the place. The average was barely 30 MB/sec. That's just not right. Internal SATA 7200 rpm HDD to external USB 3 SSD should be more like 80 MB/sec I think. I suspect there's something not right with the front panel USB 3 port, but I haven't had a chance to really look into it.

Drives are funny.
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6171
  • Country: fi
    • My home page and email address
Re: Strange hard disk performance issue
« Reply #21 on: January 27, 2023, 09:43:49 am »
It could be the drive advertises a combination of properties – like NCQ, FUA, 1.5G/3G link speed etc. – that results in most current SATA drivers choosing a subset that the drive itself supports poorly.  I'd be particularly suspicious of the power management options, because servers (where these kinds of drives were mostly used) typically mostly ignore those, but desktop machines tend to try to implement.  Basically, I suspect the actual firmware on the drives is older, and advertises features that weren't commonly used at the time (on servers at least), but now are, but severely degrade the drive performance.

In Linux, I'd look at what properties/parameters your SATA chipset driver provides, and check if overriding (downgrading some of) them yields better performance.  If you have both ACPI and libata drivers for your chipset or USB adapter, I'd also check if switching between the two yields a difference in performance.

 

Offline HalcyonTopic starter

  • Global Moderator
  • *****
  • Posts: 5629
  • Country: au
Re: Strange hard disk performance issue
« Reply #22 on: January 27, 2023, 09:50:23 am »
So weirdly, I've now been able to replicate the problem (kind of). I just don't know how (nor do I know how I fix it).

A drive which was suddenly "working" and erasing at full speed using MHDD suddenly reverted back to the slow 8 MB/sec writing speed (but reads at ~170 MB/sec) after I disconnected it from my test PC running MHDD and moved it over to wipe on my normal rig which runs Windows and an older version of the Active@ KillDisk application (the new versions suck).

It ran slow on KillDisk, then I moved it back over to the MHDD machine, same cables and everything and it's slow once again there too.

EDIT: For shits and giggles I re-ran the erase on the "good" disk (without plugging it into a different machine) and it too is back to the slow pace again.
So then, without touching anything, rebooted the system, started a full format using Windows 7, everything is good again.
Reboot back into MHDD, full speed once again.

This makes absolutely zero sense.
« Last Edit: January 27, 2023, 10:05:50 am by Halcyon »
 

Online Ed.Kloonk

  • Super Contributor
  • ***
  • Posts: 4000
  • Country: au
  • Cat video aficionado
Re: Strange hard disk performance issue
« Reply #23 on: January 27, 2023, 09:57:25 am »
It's punishing you for all that Janet Jackson music.
iratus parum formica
 

Offline HalcyonTopic starter

  • Global Moderator
  • *****
  • Posts: 5629
  • Country: au
Re: Strange hard disk performance issue
« Reply #24 on: January 27, 2023, 09:57:59 am »
It's punishing you for all that Janet Jackson music.

 :-DD

Cheeky bitch.
 
The following users thanked this post: Ed.Kloonk

Offline HalcyonTopic starter

  • Global Moderator
  • *****
  • Posts: 5629
  • Country: au
Re: Strange hard disk performance issue
« Reply #25 on: January 27, 2023, 10:14:39 am »
Repeated this again (without moving, or touching any cables or drives)...

Erase under MHDD = Slow
Rebooted into Windows 7, ran format = Fast
Rebooted into MHDD = Fast

 :scared:
 

Offline edpalmer42

  • Super Contributor
  • ***
  • Posts: 2260
  • Country: ca
Re: Strange hard disk performance issue
« Reply #26 on: January 27, 2023, 05:11:04 pm »
When you move drives around are you hot-plugging them or do you power everything down first?  Maybe the custom firmware in these drives doesn't like hot-plugging.  This could even apply to a USB3 connection.

Ed
 

Offline Monkeh

  • Super Contributor
  • ***
  • Posts: 7990
  • Country: gb
Re: Strange hard disk performance issue
« Reply #27 on: January 27, 2023, 05:31:35 pm »
Repeated this again (without moving, or touching any cables or drives)...

Erase under MHDD = Slow
Rebooted into Windows 7, ran format = Fast
Rebooted into MHDD = Fast

 :scared:

Are you erasing the entire device or partitions?
 

Offline MrMobodies

  • Super Contributor
  • ***
  • Posts: 1906
  • Country: gb
Re: Strange hard disk performance issue
« Reply #28 on: January 27, 2023, 08:57:05 pm »
It's punishing you for all that Janet Jackson music.

 :-DD

Cheeky bitch.
Joke: Janet Jackson is hurting their heads so they can't think properly and therefore slow down. I wonder what the heads will have to say about that, calling them cheeky for their high tolerance and choosing wisely not to make mistakes at the expense of speed.

Repeated this again (without moving, or touching any cables or drives)...

Erase under MHDD = Slow
Rebooted into Windows 7, ran format = Fast
Rebooted into MHDD = Fast

 :scared:

Have you noticed or tried taking note to see if the drive parameters have changed when it works fast and when it goes slow like the ones here:


I remembered about 7 years ago there was some DMicros video recorders that had an IDE interface and couldn't get many IDE drives new anymore. We tried different sata converters where it didn't pick them the drive up or it just hung on detecting the drive. I brought in a few and this Silicon Image controller (I found them very reliable) worked where it did detect the drive and loaded up but as soon as formatting them it would never finish.

Some brand new IDE Hitachi drives were found and still being made but the recorders wern't detecting them. I tested the drives and they were fine. I compared a used Western Digital 500GB that worked and the new Hitachi drives that weren't being detected and noticed a difference in "ATA version". I  am not sure myself if that had something to do with it.
« Last Edit: January 27, 2023, 09:17:46 pm by MrMobodies »
 

Offline HalcyonTopic starter

  • Global Moderator
  • *****
  • Posts: 5629
  • Country: au
Re: Strange hard disk performance issue
« Reply #29 on: January 27, 2023, 11:11:46 pm »
When you move drives around are you hot-plugging them or do you power everything down first?  Maybe the custom firmware in these drives doesn't like hot-plugging.  This could even apply to a USB3 connection.

Always powering down. The USB3 dock I use is a Tableau forensic dock with it's own power switch to power off drives before removing them.

Are you erasing the entire device or partitions?

Entire device using MHDD/DBAN/DiskKill, and partition when we're talking about Windows format.

Are you sure that the MHDD erase is writing with 4K alignment?

Are you letting it use BIOS functions?  If so, maybe there's a BIOS issue.

Maybe you should try writing with badblocks instead.


I've tried with the BIOS functions disabled previously and it didn't seem to make a difference. As I mentioned before, this behaviour has been observed across 3 different computers. I'm 99% certain it has something to do with the drives. I just don't know what. There doesn't seem to be an obvious reason but whatever it is I believe is deliberate (unless a whole batch of drives suddenly became faulty somehow).

Have you noticed or tried taking note to see if the drive parameters have changed when it works fast and when it goes slow like the ones here:

That was going to be my next task, was to try and view the drive parameters and compare.
« Last Edit: January 27, 2023, 11:14:38 pm by Halcyon »
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf