Author Topic: SSD slow WriteFile() with "FAST IO DISALLOWED" (Win11)  (Read 4369 times)

0 Members and 1 Guest are viewing this topic.

Offline Georgy.MoshkinTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: hk
  • R&D Engineer
    • Electronic projects, modules and courses on Arduino and STM32
SSD slow WriteFile() with "FAST IO DISALLOWED" (Win11)
« on: December 03, 2023, 02:24:40 pm »
Observing some strange slowdowns when copying files...

1. AS SSD Benchmark write speed around 200mb/s
Process Monitor shows that benchmark writes to file and WriteFile () result is always SUCCESS. Procmon also shows, that file is created with "nocache" attribute.

2. Copying single 90gb file to external SSD using FAR Manager, write speed is 100mb/s. Haven't checked procmon yet. Was wrong about this one, speed is much lower. Disk goes to 100% load in task manager and speed drops.

3. Copying a lot of files using 7-zip (zero level compression) 12mb/s to external SSD, 20mb/s if writing to same disc. ProcMon shows that large 7z archive has a lot of WriteFile() calls, and some of then has FAST IO DISALLOWED result. 99% sure will work fast after clean install, but then something may be broken again. No defender, indexing, stopped all services with clickable stop button. Want to avoid reinstalling the system.

So what's strange is that making 100gb zero level compression file on the same disk as input files and then copying big file to external ssd is faster than outputting 7z archive directly on external ssd. Using this backup process since windows 8, and it's always something new with slowdowns. My 10-year old notebook does same backup with 7z at 90mb/s, and slowdowns was due to indexing or defender and resolved fast enough by disabling them.

Maybe something to do with sata ahci controller vid_8086 pid_4dd3. I couldn't find any driver, so currently it's Microsoft standard sata ahci driver.

Computer is noname n5095 laptop, 16gb ram, replaced SSD with a faster one.

What could it be? I tried to disable "MSI" through registry key, monitored disk writes and reads, disabled/enabled caching policy, but it's still unclear what makes copying process so slow.
« Last Edit: December 06, 2023, 03:50:58 pm by Georgy.Moshkin »
 

Offline Georgy.MoshkinTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: hk
  • R&D Engineer
    • Electronic projects, modules and courses on Arduino and STM32
Re: SSD slow WriteFile() with "FAST IO DISALLOWED" (Win11)
« Reply #1 on: December 04, 2023, 01:46:27 pm »
I couldn't stand it, made a backup and reinstalled Windows. There is an improvement, backup speed is around 60mb/s, but not reaching 100mb/s.  It turns out that "fast io disallowed" for writes is not related to this problem. Most likely it is somehow related to SSD firmware, I'll try to search for updates. This doesn't explain why AS SSD benchmark always shows the same good results though.

Offline Haenk

  • Super Contributor
  • ***
  • Posts: 1091
  • Country: de
Re: SSD slow WriteFile() with "FAST IO DISALLOWED" (Win11)
« Reply #2 on: December 06, 2023, 10:59:39 am »
Too little info.
How is the external SSD connected? eSATA? USB? (level?) to SATA or m.2 SSD?
Which model of SSD?

Just a note on SSDs: A lot of cheaper SSDs even come without DRAM cache these days and reach their advertised speed by using multilevel flash as single level. But only a small amount of the storage capacity is used this (fast) way. The regular quad level memory is quite slow.
So when copying a really large file, you immediatly saturate the DRAM cache, very soon saturate the single level area (a couple of GB, max) and then run into quad level writing, which is really slow. Depending on the model it might be way slower than a mechanical HDD.

Benchmarking usually runs only on DRAM or single cell has no meaning when doing large transfers.

And, another thing, the SSD controllers get very hot when running a maximal transfer - going into thermal throtteling is common. (This is rather an issue with fast M.2 and not so much on SATA SSDs...)


 
The following users thanked this post: SeanB, 5U4GB

Offline DavidAlfa

  • Super Contributor
  • ***
  • Posts: 5912
  • Country: es
Re: SSD slow WriteFile() with "FAST IO DISALLOWED" (Win11)
« Reply #3 on: December 06, 2023, 11:28:11 am »
Which SSD? Is everything OK in SMART?
Cheap ones have slow NAND, no DRAM cache, can perform like a slow USB drive!

Hantek DSO2x1x            Drive        FAQ          DON'T BUY HANTEK! (Aka HALF-MADE)
Stm32 Soldering FW      Forum      Github      Donate
 

Offline Georgy.MoshkinTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: hk
  • R&D Engineer
    • Electronic projects, modules and courses on Arduino and STM32
Re: SSD slow WriteFile() with "FAST IO DISALLOWED" (Win11)
« Reply #4 on: December 06, 2023, 02:19:05 pm »
Internal ssd is Kingspec nt-512 (2280) ngff. I will check smart. Something tells me that if I use ReadFile() and WriteFile() the same way as AS SSD Benchmark, then copy speed will be around 100-200mb/s for big files. I think that something if broken in Windows when using this chipset.

External SSD:
1) Cuso c5s-evo 480gb in Jmicron usb 3 generic enclosure. This one gave around 100mb/s during backups when connected to an old Inspiron-3542 laptop.
2) Product page vanished, I'll check later. Jmicron usb 3 enclosure. This one was main SSD of Inspiron-3542. CD-ROM was replaced by another 128gb SSD. Copy speeds always was satisfactory.

Now, using a newer PC I observe slightly better AS SSD Benchmark results, but copy speeds became much worse. I always used 7-zip's "store compression" level for backups, because it generates single file on external drive and provides significant copy speed boost compared to normal copy/paste.
« Last Edit: December 06, 2023, 02:37:33 pm by Georgy.Moshkin »
 

Offline DavidAlfa

  • Super Contributor
  • ***
  • Posts: 5912
  • Country: es
Re: SSD slow WriteFile() with "FAST IO DISALLOWED" (Win11)
« Reply #5 on: December 06, 2023, 02:25:04 pm »
Have you tried documenting yourself googling that error?
There's plenty of info about it.
Hantek DSO2x1x            Drive        FAQ          DON'T BUY HANTEK! (Aka HALF-MADE)
Stm32 Soldering FW      Forum      Github      Donate
 

Offline Georgy.MoshkinTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: hk
  • R&D Engineer
    • Electronic projects, modules and courses on Arduino and STM32
Re: SSD slow WriteFile() with "FAST IO DISALLOWED" (Win11)
« Reply #6 on: December 06, 2023, 03:38:15 pm »
Have you tried documenting yourself googling that error?
There's plenty of info about it.
Yes, my hypothesis was that 100% disk C: load is somehow related to this error (copying from C: to C:). That's the only difference between AS SSD Benchmark and system copy routine in process monitor (plus nocache flag for AS SSD Benchmark generated temp files). Actually, topics on Microsoft forum and serverfault are not helping to solve this slowdown mystery. I observe these errors for writes (not reads!), and most of time it returns "SUCCESS". I still think that it may be related. Most importantly, I do backups all the time and know the speeds. Full backup of 128gb old ssd to 1gb toshiba hdd took less than 25 minutes around 10 years ago. System + data on SSD was around 70gb. When I bough bigger 256gb ssd and performed backup to an old 128gb ssd, I was pleased by 15 minute backup speeds.  Now it's 2023 and I copy 90gb and it takes almost 2 hours, but should take 30 min maximum.

Simple readfile/writefile test with FILE_FLAG_NO_BUFFERING can help with checking my hypothesis: If I am right, reading 100gb file from C: and writing it to external D: should show something around 100mb/s minimum.  Circular bufer, two threads, reading/writing by chunks chasing one another.
« Last Edit: December 06, 2023, 03:47:01 pm by Georgy.Moshkin »
 

Offline Georgy.MoshkinTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: hk
  • R&D Engineer
    • Electronic projects, modules and courses on Arduino and STM32
Re: SSD slow WriteFile() with "FAST IO DISALLOWED" (Win11)
« Reply #7 on: February 28, 2024, 09:49:40 am »
Finally solved! 14-minutes backups are back! It seems that source of the problem was Window Event Log service (now disabled).
I have used resmon (windows built-in), ProcessMonitor and ProcessExplorer. It seems that frequent "trickle charge" Event Logs slowdowns copying speeds. Today my backup speed dropped to 11mb/s and took 2+ hours! I never believed "experts" who told that SSD is very slow, and it's normal. It's simply not! I clearly remember that 10 years ago I already had fast daily backups.

1. First, I've noticed that pagefile start burst writes during backup. I tried to disable pagefile - no effect. Enabled it back.

2. Second, I've noticed that there are lots of tiny system writes from NTFS. I found some "NTFS hacks" page and tried this:
fsutil behavior set disable8dot3 1
fsutil behavior set mftzone 2
fsutil behavior set disablelastaccess 1
backup speed is still slow, but Task Manager now shows 60%-80% for source disk usage instead of 100%. No speed improvement, but still interesting.

3. Tried
fsutil usn deletejournal /d /n c:
no effect

4. The winner!
Disabled "Window Event Log service" and wow! Speeds goes up from 11 mb/s to  146 mb/s
Somehow this "trickle" logging slowdowns SSD significantly. Had same problem ~10 years ago because multi-functional printer driver constantly trickled event log with 20 bytes of some "could not find some compatible Twain" bs error.

Update: unfortunately this only worked for one or two weeks. I reinstalled everything to get speeds I need...
« Last Edit: April 01, 2024, 08:35:48 am by Georgy.Moshkin »
 

Offline DavidAlfa

  • Super Contributor
  • ***
  • Posts: 5912
  • Country: es
Re: SSD slow WriteFile() with "FAST IO DISALLOWED" (Win11)
« Reply #8 on: February 28, 2024, 02:38:42 pm »
Don't delete the journal, you'll regret it when something unexpected happens to your FS.
Make sure you didn't lit a match near a gas tank when setting those "hacks".
Hantek DSO2x1x            Drive        FAQ          DON'T BUY HANTEK! (Aka HALF-MADE)
Stm32 Soldering FW      Forum      Github      Donate
 
The following users thanked this post: SeanB

Offline Georgy.MoshkinTopic starter

  • Regular Contributor
  • *
  • Posts: 146
  • Country: hk
  • R&D Engineer
    • Electronic projects, modules and courses on Arduino and STM32
Re: SSD slow WriteFile() with "FAST IO DISALLOWED" (Win11)
« Reply #9 on: April 01, 2024, 08:34:01 am »
Ok... It came back few days ago. Copying became slow again. I've made a backup, deleted all partitions on SSD and created a new one using third-party partition manager bootable flash drive, installed latest WIN11 and now it works great again with >100mb/s copying speeds.

Online radiolistener

  • Super Contributor
  • ***
  • Posts: 3385
  • Country: ua
Re: SSD slow WriteFile() with "FAST IO DISALLOWED" (Win11)
« Reply #10 on: April 11, 2024, 08:16:20 am »
Did you tried to execute blkdiscard command for that SSD drive before testing?

The speed slowdown is possible due to many not erased sectors on SSD, it requires controller to perform erase operation during write operation and it takes additional time. If you do blkdiscard before testing it will erase all unused sectors, so the write operation will be executed faster.

I'm not familiar with details on how it's implemented, but blkdiscard helps to improve write speed performance in my case.
« Last Edit: April 11, 2024, 08:28:25 am by radiolistener »
 

Offline 5U4GB

  • Frequent Contributor
  • **
  • Posts: 391
  • Country: au
Re: SSD slow WriteFile() with "FAST IO DISALLOWED" (Win11)
« Reply #11 on: April 12, 2024, 09:52:02 am »
Various comments, firstly check everything @Haenk said, that was also my first reaction to seeing your post.  In particular Kingspec drives are cheap and nasty so the problem could just be the drive, see if you get the same problem with something like an EVO 8x0 series.  Finally, how much have you written to the drives?  The temporary reinstatement of performance after a reinstall, i.e. a complete redo of storage, followed by a slowdown again may just mean the flash wear limit has been reached.
 

Offline mariush

  • Super Contributor
  • ***
  • Posts: 5029
  • Country: ro
  • .
Re: SSD slow WriteFile() with "FAST IO DISALLOWED" (Win11)
« Reply #12 on: April 12, 2024, 10:33:44 am »
I would suspect the constant writing (trickle writing, small writes) prevents the firmware from entering into garbage collection mode and erase/recover blocks of flash memory marked for erase

Flash memory is arranged in blocks (ex 24/32/64 MB or higher blocks of flash, which are then arranged in pages of 512 bytes or 4096 bytes or other sizes) - the controller can write to an empty page, but can't overwrite that page - in order to make the page available for writing the whole block (64 MB or whatever size it is) has to be erased (and that erase process wears out the flash memory).

So each time something has to be overwritten, the controller just finds an empty page somewhere in other parts of the flash drive, copies the old page with the changed bytes to a new page , marks the old page for erase, and updates a lookup table that says contents previously on block x , page y is now in block m, page n .... (off topic that lookup table is usually cached in ram at startup and makes dram based ssds a bit faster, dram isn't used to cache writes)
 
When there's a threshold of erased pages out of a block reached, for example 90% of pages from a 64 MB block, the controller copies the remaining 10% of pages in random places and then erases the block and makes it available to the system.

If the controller doesn't detect some idle time, it's possible it never erases blocks so the more time it takes the harder it is for the controller to find empty pages to put content into, and it may not have enough empty blocks to convert from TLC to pseudo-SLC to faster write speeds.

One potential solution would be to run the TRIM command (Windows defrag tool will do it on SSDs instead of defragmenting, TRIM will force the drive to do cleanup)

Most drives use pseudo-SLC to cache writes - what they do is take one of those blocks of 24/32/64 MB of flash memory where each cell holds 2 bits for MLC, 3 bits for TLC and 4 bits for QLC, and convert the block to SLC mode storing just one bit in each cell - so for example 64 MB of QLC becomes 16 MB of fast SLC cache, 64 MB of TLC becomes around 20 MB of SLC cache etc etc.

You write stuff, controller puts it in SLC cache and afterwards at idle time, it will slowly move the data to TLC memory areas.

Blocks in pseudo-SLC mode wear out slower ... maybe 10k erases QLC is rated for 200-600 erases, TLC goes from around 1000 to 4000 erases, MLC goes up to around 10-15k erases, SLC goes from 10k to maybe 30-50k erases, some small slc chips can do even 100k erases.

-

OP , if your backups are a few GB or less, I would suggest seeing if you can spin up a ram disk before your backup - for example on Windows I use ImDisk toolkit and can create ram drives with or without physical backup (disk image).  Without a disk image, I can set up a 10 GB ram drive (because I have 16 GB of ram) in seconds and quick format it to NTFS and you could compress the data to the RAM drive, then simply copy the archive to the SSD in a burst.

You may also take advantage of this to sort your file types and then pack together your files in a TAR archive (7zip can create TAR archives) and you could run a DIFF between previous backups and current backup. I like XDelta  (open source tool) but there's other binary diff tools.

Keep each day's archive for 30 days, after that you could keep only 1st day of the week or month and then for the next days keep the diffs only (so if you want friday's backup, generate from diff between monday and friday). 7z in store mode may also work, but 7z format may shift some bytes around that could make it less efficient to do binary diffs.  TAR is simpler, 512 byte blocks, if your files are sorted in same order then a binary diff would easily store only the differences.


 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf