tips and tricks, on using compact flash under linux

[legacy]

so, I happen to have a 64Gbyte CF attached to a PCMCIA adaptor, and I need to back up the first slice of my laptop

I am on linux 2.6.39, I can't upgrade it, the CF is handled by "ide-cf"(1) (statically compiled inside the kernel), and dmesg is reporting the CF handled as "HDA"
(cat /proc/devices is confirming the major device, equal to 3, which is "HDA" in term of "device name" under linux)

thus, I'd like to type the following

Code: [Select]

kanojo #sudo dd if=/dev/sda1 of=/dev/hda

which actually works, but ... it's damn too slow

and forking a bigger buffer doesn't solve

Code: [Select]

kanojo #sudo dd if=/dev/sda1 of=/dev/hda bs=32Mbyte

my Compact Flash is made by SanDisk, it's called "Exteme", and claimed to be able to read/write things with a bandwidth of 160Mbyte/sec
thus I'd like to know why? on my laptop I am going something like ~10Mbyte/sec 

there is no filesystem in the middle on the compact flash (which can cause overhead due to the flash needs, e.g. page erase before write)


any trick ? advice ? suggestion ? are the welcome

thanks




(1) ide-cf is p/ATA PCMCIA hard drive support
there is also a s/ATA PCMCIA hard drive support
unfortunately it doesn't work stable on my hardware

[legacy]

linux 2.6.39

• Device Drivers -> ATA/ATAPI/MFM/RLL support (DEPRECATED) -> PCMCIA IDE support, "A driver for PCMCIA IDE/ATA disk cards", drivers/ide/ide-cs.c
• Device Drivers -> Serial ATA and Parallel ATA drivers -> PCMCIA PATA support, "PCMCIA PATA controller driver", drivers/ata/pata_pcmcia.c

edit:
mac-fdisk doesn't recognize the correct amount of blocks in the CF card (it says 2Gbyte, wrong)
while cfdisk and fdisk recognize it correctly (they say 64Gbyte, correct)

[legacy]

PCMCIA Storage
Today's PCMCIA/CF storage support densities in the gigabyte realm. The storage cards come in different flavors:

• Miniature IDE disk drives or microdrives. These are tiny versions of mechanical hard drives that use magnetic media. Their data transfer rates are typically higher than solid state memory devices, but IDE drives have spin-up and seek latencies before data can be transferred. The IDE Card Services driver ide_cs, in conjunction with legacy IDE drivers, is used to communicate with such memory cards.
• Solid-state memory cards that emulate IDE. Such cards have no moving parts and are usually based on flash memory, which is transparent to the operating system because of the IDE emulation. Because these drives are effectively IDE-based, the same IDE Card Services driver (ide_cs) can be used to talk to them.
• Memory cards that use flash memory, but without IDE emulation. The memory_cs Card Services driver provides block and character interfaces over such cards. The block interface is used to put a filesystem onto card memory, whereas the character interface is used to access raw data. You may also use memory_cs to read the attribute memory space of any PCMCIA card.

[legacy]

my SanDisk CF card is described by case2, Solid-state memory cards that emulate IDE

[legacy]

I do not know the state of art with kernel 3.x and 4.x, with 2.6.39 "drivers/ata/pata_pcmcia.c" seems to be "experimental" for CF
here it doesn't work as expected

thus I have chosen "drivers/ide/ide-cs.c", which is known to be rock (even if old and deprecated)

feel free to let me know your experiences with such a kernel modules and devices 

[stmdude]

Does hdparm give you any clues?

If I recall correctly, 2.6.x was a bit careful about enabling UltraDMA, Multi-sector transfers and other features needed for maximum performance.

[legacy]

Code: [Select]

*  sys-apps/hdparm
      Homepage:      http://sourceforge.net/projects/hdparm/
      Description:   Utility to change hard drive performance parameters
      License:       BSD GPL-2


Code: [Select]

# hdparm -I /dev/hda

/dev/hda:

CompactFlash ATA device
Model Number:       SDCFXS-064G                             
Serial Number:      GAZ020116120307
Firmware Revision:  HDX12.03
Standards:
Likely used: 6
Configuration:
Logical max current
cylinders 65535 65535
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors:   66059280
LBA    user addressable sectors:  125059072
LBA48  user addressable sectors:  125059072
Logical/Physical Sector size:           512 bytes
device size with M = 1024*1024:       61064 MBytes
device size with M = 1000*1000:       64030 MBytes (64 GB)
cache/buffer size  = 1 KBytes (type=DualPort)
Capabilities:
LBA, IORDY(may be)(cannot be disabled)
bytes avail on r/w long: 4
Standby timer values: spec'd by Vendor
R/W multiple sector transfer: Max = 1 Current = 0
Advanced power management level: disabled
DMA: not supported
PIO: pio0 pio1 pio2 pio3 pio4
     Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
Enabled Supported:
    Power Management feature set
    Write cache
    WRITE_BUFFER command
    READ_BUFFER command
    NOP cmd
    CFA feature set
    Advanced Power Management feature set
   * 48-bit Address feature set
    Mandatory FLUSH_CACHE
    FLUSH_CACHE_EXT
CFA max advanced io_udma cycle time: 80ns
CFA max advanced mem_udma cycle time: 80ns
   * CFA Power Level 1  (max 500mA)
HW reset results:
CBLID- above Vih
Device num = 1
Integrity word not set (found 0x0000, expected 0xe4a5)


[legacy]

125059072 LBA address is correct

see mac-fdisk

Code: [Select]

# mac-fdisk /dev/hda
Command (? for help): i
size of 'device' is 4194304 blocks:
new size of 'device' is 4194304 blocks
Command (? for help): p
/dev/hda
        #                    type name                length   base    ( size )  system
/dev/hda1     Apple_partition_map Apple                   63 @ 1       ( 31.5k)  Partition map
/dev/hda2              Apple_Free Extra              4194240 @ 64      (  2.0G)  Free space

Block size=512, Number of Blocks=4194304
DeviceType=0x0, DeviceId=0x0

4194304 instead of 125059072 

[legacy]

Code: [Select]

R/W multiple sector transfer: Max = 1 Current = 0
 DMA: not supported

this is interesting for an UDMA device 

[stmdude]

Can you run hdparm without parameters?  It should just spit out the performance related settings then.

[legacy]

Code: [Select]

# hdparm /dev/hda

/dev/hda:
 multcount     =  0 (off)
 IO_support    =  0 (default)
 unmaskirq     =  0 (off)
 using_dma     =  0 (off)
 keepsettings  =  0 (off)
 readonly      =  0 (off)
 readahead     = 256 (on)
 geometry      = 65535/255/63, sectors = 125059072, start = 0


[legacy]

Code: [Select]

# hdparm -a512 -d1 -m16 -u1 /dev/hda

/dev/hda:
 setting fs readahead to 512
 setting multcount to 16
 HDIO_SET_MULTCOUNT failed: Invalid argument
 setting unmaskirq to 1 (on)
 setting using_dma to 1 (on)
 HDIO_SET_DMA failed: Operation not permitted
 multcount     =  0 (off)
 unmaskirq     =  1 (on)
 using_dma     =  0 (off)
 readahead     = 512 (on)

it seems I have to recompile the kernel adding more support

[stmdude]

Yup, that would do it.

That's the most safe settings it could choose, but definitely not the fastest.

Code: [Select]

multcount     =  0 (off)
 IO_support    =  0 (default)
 using_dma     =  0 (off)


So, those are the important ones.

Right now, each 512byte sector is read like this:
1. Send READ_SECTOR command to the ATA controller
2. Twiddle thumbs until IRQ fires
3. Issue 256 IO operations to read the data. (16 bits per read)
4. Done

Now, the "multicount" setting changes how many sectors that can be transfered in one go. I.e, to transfer (for example) 4 sectors (sequential sectors), you only have to issue one command, and wait for one IRQ.  This produces a noticable increase in performance.
You can adjust this value with the "-m" parameter to hdparm

"IO_support" says if the IO operations are 16 or 32 bits (0 versus 1). Going 32 bits halves the number of IO operations needed to read out the data This also produces a noticable increase in performance.You can adjust this value with the "-c" parameter to hdparm

You can adjust this value with the "-c" parameter to hdparm

"using_dma" eliminates the IO operations all together. The data is DMAd into your RAM for you, and ready to use by the time the IRQ fires.
You can adjust this value with the "-d" parameter to hdparm

It seems your controller or disk doesn't like DMA though..

The mismatch in sectors reported from various fdisks is an issue I've seen myself, even on 4.4.x kernels.  Don't know what's up there..

[stmdude]

it seems I have to recompile the kernel adding more support

That's possible. It's been many years since I configured a 2.6.x kernel, and used (what effectively is) IDE disks.

[ade]

I'm pretty sure one cannot do IDE UDMA over PCMCIA.

[legacy]

I am on IBM X61S, it's an intel core duo2 laptop 
going to recompile the kernel

[legacy]

it's PCMCIA v2.1

[legacy]

Industrial SLC PCMCIA ATA Card HERCULES-N Series
SLC - NAND type flash technology
Data transfer mode support PIO 0~6, MWDMA 0~4, UDMA 0~7
Compatible with PC card standard and PCMCIA V.2.1
Compatible with ATA/ATAPI-8 standard
PCMCIA Type II form-factor
S.M.A.R.T. (Self-Monitoring, Analysis and Reporting Technology)
Supports PC Card Memory mode or True IDE mode
Optional for standard casing and rugged metal casing
Capacities from 8GB up to 64GB
Performance up to: Read 117.4 MB/sec ; Write 83.8 MB/sec

an example of a product that should work that way
it might be a linux matter (kernel module, configuration),
or an hardware matter (my PCMCIA controller)
who knows? 

[legacy]

and here it is CF technology
Performance up to: Read 37.8 MB/sec ; Write 27.0 MB/sec

is there any valid reason to consider a bottleneck between the CF interface (inside the card) and PCMCIA?

[ade]

is there any valid reason to consider a bottleneck between the CF interface (inside the card) and PCMCIA?

CF basically supports three different interfaces:  PIO (Port I/O), MMIO (Memory-Mapped I/O) and DMA.

PIO is the slowest, < 10 MB/s.  MMIO can go between 10 MB/s and 20 MB/s depending on the exact transfer method.  DMA is much faster, potentially > 100 MB/s.

However, the interface we typically call "PCMCIA" does not support CF/IDE/ATA DMA (*), so it is limited to MMIO at best.

DMA support only came with "PCMCIA 5.0" which is a completely different 32-bit interface more commonly called CardBus.

So from the above it seems that you are getting MMIO throughput.  There may be a way to optimize it a bit by playing with different timings / settings but not to DMA speeds.

(*) Note: I believe PCMCIA 2.1 optionally specified a type of rudimentary DMA, but it is not directly compatible with IDE DMA and hence rarely implemented.

[Rasz]

solution http://www.ebay.com/itm/CF-to-44-Pin-IDE-HDD-Hard-Disk-Drive-Adapter-Converter-for-Notebook-PC-/162025970497

but since http://www.amazon.com/SanDisk-Extreme-Compact-Memory-SDCFXPS-064G-X46/dp/B00ECEVFFO = $85.95 
better solution is $30 proper sata ssd like http://www.amazon.com/Transcend-MSA370-mSATA-Solid-TS64GMSA370/dp/B00K67E5DA

[Kilrah]

I played with a CF and PCMCIA maybe 5-6 years ago to find a use for the 32GB CF I had for my camera after I changed to a new one that used SD instead, if I remember well the PCMCIA to CF adapter had to explicitely support UDMA if you wanted that to work and it was very rare, there were like only one or 2 models available that did and they were "complex" devices, i.e. included a controller chip with drivers, not a simple passive adapter.

[Kilrah]

Found the reference of the adapter I had in my old emails (was actually 8 years ago):
http://www.mydigitaldiscount.com/delkin-compact-flash-pcmcia-cardbus-32-adapter-ddcfprocb-ad/

I remembered correctly about a driver being needed, so might not be of much use for you as it's likely not supported on linux.

[Rasz]

Found the reference of the adapter I had in my old emails (was actually 8 years ago):
http://www.mydigitaldiscount.com/delkin-compact-flash-pcmcia-cardbus-32-adapter-ddcfprocb-ad/

I remembered correctly about a driver being needed, so might not be of much use for you as it's likely not supported on linux.

its not pcmcia (essentially ISA), its CardBus (pretty much PCI), thats why it needs a driver for a full blown ide controller inside

[ade]

Kilrah, the terminology is a bit confusing but what you've linked to is what we'd commonly call "CardBus", not "PCMCIA".

So that adapter will only work on laptops with a 32-bit CardBus slot, not ones with a "PCMCIA" port.

It is confusing because CardBus was technically described in the "PCMCIA 5.0" specification.

PCMCIA stands for "People Can't Memorize Computer Industry Acronyms".   

Edit: Rasz beat me to it!