linux low IO performance on /dev/ram ?!?

[legacy]

"top" says

Code: [Select]

CPU:  0.0% usr  0.5% sys  0.0% nic 99.4% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.00 0.01 0.12 1/30 1194


[nctnico]

One way to find out is to create a program which toggles an I/O pin without writing to /tmpfs and one with writing to /tmpfs. That way you can see if there is something wrong in general or there is a filesystem problem.
I guess any kind of swap space (virtual memory) is disabled? Any dmesg output?

[legacy]

swap space (virtual memory) is disabled?

enabled, not used

Any dmesg output?

Code: [Select]


kernelinfo compiled by root@OrangeCube
kernelinfo compiled with gcc version 4.2.4 (Gentoo 4.2.4-r1 p1.4)
kernelinfo compiled ticket #226 Thu Mar 1 21:47:26 CET 2018
Found legacy serial port 0 for /plb/opb/serial@ef600300
  mem=ef600300, taddr=ef600300, irq=0, clk=11110000, speed=9600
Found legacy serial port 1 for /plb/opb/serial@ef600400
  mem=ef600400, taddr=ef600400, irq=0, clk=11110000, speed=9600
Top of RAM: 0x3000000, Total RAM: 0x3000000
Memory hole size: 0MB
Zone PFN ranges:
  DMA      0x00000000 -> 0x00003000
  Normal   empty
  HighMem  empty
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0: 0x00000000 -> 0x00003000
On node 0 totalpages: 12288
free_area_init_node: node 0, pgdat c0472178, node_mem_map c04b7000
  DMA zone: 96 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 12192 pages, LIFO batch:1
MMU: Allocated 1088 bytes of context maps for 255 contexts
pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
pcpu-alloc: [0] 0
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 12192
Kernel command line: console=ttyS0,9600 mem=48M rdinit=/sbin/init pci=assign-busses pci=nobios root=/dev/mtdblock1 rw init
=/sbin/init video=matroxfb:vesa:0x11B,mem:4M,pciretry,nobios
PID hash table entries: 256 (order: -2, 1024 bytes)
Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
High memory: 0k
Memory: 43808k/49152k available (4448k kernel code, 5344k reserved, 148k data, 222k bss, 172k init)
Kernel virtual memory layout:
  * 0xfffcf000..0xfffff000  : fixmap
  * 0xff800000..0xffc00000  : highmem PTEs
  * 0xff600000..0xff800000  : consistent mem
  * 0xff600000..0xff600000  : early ioremap
  * 0xc4000000..0xff600000  : vmalloc & ioremap
SLUB: Genslabs=13, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
NR_IRQS:512 nr_irqs:512 16
UIC0 (32 IRQ sources) at DCR 0xc0
time_init: decrementer frequency = 266.640000 MHz
time_init: processor frequency   = 266.640000 MHz
clocksource: timebase mult[f00625] shift[22] registered
clockevent: decrementer mult[444284df] shift[32] cpu[0]
Console: colour dummy device 80x25
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
devtmpfs: initialized
NET: Registered protocol family 16
PCI host bridge /plb/pci@ec000000 (primary) ranges:
 MEM 0x0000000080000000..0x000000009fffffff -> 0x0000000080000000
  IO 0x00000000b0000000..0x00000000b000ffff -> 0x0000000000000000
4xx PCI DMA offset set to 0x00000000
PCI: Probing PCI hardware
pci_bus 0000:00: scanning bus
pci 0000:00:00.0: [1014:0156] type 0 class 0x000600
pci 0000:00:00.0: calling quirk_mmio_always_on+0x0/0x24
pci 0000:00:00.0: reg 14: [mem 0x00000000-0x7fffffff pref]
pci 0000:00:00.0: calling pcibios_fixup_resources+0x0/0xe0
pci 0000:00:00.0: calling fixup_ppc4xx_pci_bridge+0x0/0x11c
PCI: Hiding 4xx host bridge resources 0000:00:00.0
pci 0000:00:00.0: calling quirk_resource_alignment+0x0/0x184
pci 0000:00:00.0: supports D1
pci 0000:00:04.0: [105a:0d30] type 0 class 0x000101
pci 0000:00:04.0: calling quirk_mmio_always_on+0x0/0x24
pci 0000:00:04.0: reg 10: [io  0x01f0-0x01ff]
pci 0000:00:04.0: reg 14: [io  0x03f4-0x03f7]
pci 0000:00:04.0: reg 18: [io  0x0170-0x017f]
pci 0000:00:04.0: reg 1c: [io  0x0374-0x0377]
pci 0000:00:04.0: reg 20: [io  0x83fa40-0x83fa7f]
pci 0000:00:04.0: reg 24: [mem 0x80000000-0x8001ffff]
pci 0000:00:04.0: reg 30: [mem 0x000dc000-0x000dffff pref]
pci 0000:00:04.0: calling pcibios_fixup_resources+0x0/0xe0
pci 0000:00:04.0: calling fixup_ppc4xx_pci_bridge+0x0/0x11c
pci 0000:00:04.0: calling quirk_resource_alignment+0x0/0x184
pci_bus 0000:00: fixups for bus
irq: irq 31 on host /interrupt-controller mapped to virtual irq 31
pci_bus 0000:00: bus scan returning with max=00
pci 0000:00:04.0: BAR 5: assigned [mem 0x80000000-0x8001ffff]
pci 0000:00:04.0: BAR 5: set to [mem 0x80000000-0x8001ffff] (PCI address [0x80000000-0x8001ffff])
pci 0000:00:04.0: BAR 6: assigned [mem 0x80020000-0x80023fff pref]
pci 0000:00:04.0: BAR 4: assigned [io  0x1000-0x103f]
pci 0000:00:04.0: BAR 4: set to [io  0x1000-0x103f] (PCI address [0x1000-0x103f])
pci_bus 0000:00: resource 0 [io  0x0000-0xffff]
pci_bus 0000:00: resource 1 [mem 0x80000000-0x9fffffff]
bio: create slab <bio-0> at 0
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Switching to clocksource timebase
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 2048 (order: 2, 16384 bytes)
TCP bind hash table entries: 2048 (order: 1, 8192 bytes)
TCP: Hash tables configured (established 2048 bind 2048)
TCP reno registered
UDP hash table entries: 256 (order: 0, 4096 bytes)
UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
NET: Registered protocol family 1
pci 0000:00:00.0: calling quirk_cardbus_legacy+0x0/0x44
pci 0000:00:00.0: calling quirk_usb_early_handoff+0x0/0x5f0
pci 0000:00:04.0: calling quirk_cardbus_legacy+0x0/0x44
pci 0000:00:04.0: calling quirk_usb_early_handoff+0x0/0x5f0
PCI: CLS 0 bytes, default 32
irq: irq 0 on host /interrupt-controller mapped to virtual irq 16
irq: irq 1 on host /interrupt-controller mapped to virtual irq 17
irq: irq 11 on host /interrupt-controller mapped to virtual irq 18
irq: irq 12 on host /interrupt-controller mapped to virtual irq 19
irq: irq 10 on host /interrupt-controller mapped to virtual irq 20
irq: irq 13 on host /interrupt-controller mapped to virtual irq 21
irq: irq 14 on host /interrupt-controller mapped to virtual irq 22
irq: irq 2 on host /interrupt-controller mapped to virtual irq 23
irq: irq 15 on host /interrupt-controller mapped to virtual irq 24
irq: irq 9 on host /interrupt-controller mapped to virtual irq 25
OF_RTC: /plb/ebc/nvram@1,0 is a rtc-ds1742 @ 0xf0000000-0xf0001fff
msgmni has been set to 85
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
io scheduler noop registered
io scheduler deadline registered (default)
io scheduler cfq registered
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
serial8250.0: ttyS0 at MMIO 0xef600300 (irq = 16) is a 16550A
console [ttyS0] enabled
serial8250.0: ttyS1 at MMIO 0xef600400 (irq = 17) is a 16550A
ef600300.serial: ttyS0 at MMIO 0xef600300 (irq = 16) is a 16550
ef600400.serial: ttyS1 at MMIO 0xef600400 (irq = 17) is a 16550
brd: module loaded
loop: module loaded
nbd: registered device at major 43
Found: AMD AM29LV040B
fff80000.flash: Found 1 x8 devices at 0x0 in 8-bit bank
number of JEDEC chips: 1
myramfs: devname=myramfs, devstart=0x3000000, devlength=0x4000000
myramfs: Registered device myramfs from 49152KiB to 114688KiB
myramfs: Mapped from 0xc4180000 to 0xc8180000
myramswap: devname=myramswap, devstart=0x7000000, devlength=0x1000000
myramswap: Registered device myramswap from 114688KiB to 131072KiB
myramswap: Mapped from 0xc8200000 to 0xc9200000
PPC 4xx OCP EMAC driver, version 3.54
MAL v1 /plb/mcmal, 1 TX channels, 1 RX channels
eth0: EMAC-0 /plb/opb/ethernet@ef600800, MAC de:00:b1:ef:ae:a3
eth0: found Generic MII PHY (0x09)
aoe: AoE v47 initialised.
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
mousedev: PS/2 mouse device common for all mice
I2O subsystem v1.325
i2o: max drivers = 8
I2O ProcFS OSM v1.316
i2c /dev entries driver
ibm-iic ef600500.i2c: using standard (100 kHz) mode
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
IPv4 over IPv4 tunneling driver
TCP bic registered
TCP cubic registered
TCP westwood registered
TCP highspeed registered
TCP hybla registered
TCP htcp registered
TCP vegas registered
TCP veno registered
TCP scalable registered
TCP lp registered
TCP yeah registered
TCP illinois registered
NET: Registered protocol family 10
IPv6 over IPv4 tunneling driver
NET: Registered protocol family 17
NET: Registered protocol family 15
802.1Q VLAN Support
VFS: Mounted root (ext2 filesystem) on device 31:1.
devtmpfs: mounted
Freeing unused kernel memory: 172k init
userland init
eth0: link is up, 100 FDX, pause enabled


[legacy]

Code: [Select]

myramfs: devname=myramfs, devstart=0x3000000, devlength=0x4000000
myramfs: Registered device myramfs from 49152KiB to 114688KiB
myramfs: Mapped from 0xc4180000 to 0xc8180000


Code: [Select]

myramswap: devname=myramswap, devstart=0x7000000, devlength=0x1000000
myramswap: Registered device myramswap from 114688KiB to 131072KiB
myramswap: Mapped from 0xc8200000 to 0xc9200000

these two kernel-modules are physical ram handlers.
in the dmesg:
- myramswap (/dev/myramdev2) has allocated ram, the MMU can't touch it, but it's not used.
- myramfs (/dev/myramdev1) has allocated ram, the MMU can't touch it, and it's used as ram-rootfs.

Code: [Select]

/dev/myramdev1 on / type ext2 (rw)
mpme on /dev/pts type devpts (rw)
/mnt/disk0 on /var type none (rw,bind)
/var/tmp on /tmp type none (rw,bind)


[Bruce Abbott]

tmpfs stays around 9Mbyte/sec
/dev/ram0 stays around 4.5Mbyte/sec

Is there some other test you can do to measure the SDRAM speed (eg. create a large array and read data from it)?

I can't understand why the  PPC405GP's ram controller (built-in) is so slow.

I have re-created the toolchain, and gcc is now using a special flag for the 405GP, but .. it's still slow.

SDRAM controller properly initialized? Data cache enabled?

 

[legacy]

Is there some other test you can do to measure the SDRAM speed (eg. create a large array and read data from it)?

Code: [Select]

~ # ramspeed -b3 -m2
RAMspeed/SMP (GENERIC) v3.5.0 by Rhett M. Hollander and Paul V. Bolotoff, 2002-09

8Gb per pass mode, 2 processes
INTEGER   Copy:      104.74 MB/s
INTEGER   Scale:     99.29 MB/s
INTEGER   Add:       86.03 MB/s


SDRAM controller properly initialized? Data cache enabled?

That is hard to be verified at the moment since it involves both the firmware and the kernel.

uboot on bootstrap says

Code: [Select]

U-Boot 1.2.0 (Nov  7 2010 - 18:54:57)

CPU:   AMCC PowerPC 405GP Rev. E at 266.640 MHz (PLB=66, OPB=33, EBC=33 MHz)
       Internal PCI arbiter enabled, PCI async ext clock used
       16 kB I-Cache 8 kB D-Cache

and the linux kernel doesn't say more 

my doubt: if the devicetree has something to do with the correct re-initialization of the dram controller

[Bruce Abbott]

104.74 MB/s when copying from one array to another indicates that your memory is fine. The fact that your ramdisk is slow and doesn't speed up with larger transfer sizes suggests a problem in the driver or how it was tested.

Is there some way you can profile the code to find out what proportion of the time is spent in memory transfers to/from the ramdisk memory, and how much in other stuff?
 
 

[legacy]

Is there some way you can profile the code

that is the question to be addressed to "linux experts". Anyone here? 
DENX's guys are not answering my emails (who? they were THE guys whose signature you can still find in Linux sources)

[legacy]

Code: [Select]

dd if=/dev/zero of=10Mbyte bs=1k count=10000
time cp 10Mbyte there

time says ~ 10Mbyte/sec on kernel 2.6.39 (it's around 0.99 sec), and 5Mbyte/sec on kernel 4.11 (it's around 1.99sec), and it's more interesting than the output of dd because it's a "true" application

[VintageTekFan]

I am curious which I/O scheduler you are using.  Some of them are optimized for high (relatively) latency devices.

Typical choices are: NOOP, CFQ, Anticipatory, Deadline...

A good read on them: https://www.safaribooksonline.com/library/view/linux-system-programming/0596009585/ch04s06.html

[legacy]

which I/O scheduler you are using

kernel 2.6.39: Deadline IO scheduler
kernel 4.11: Deadline IO scheduler

[Bruce Abbott]

kernel 2.6.39: Deadline IO scheduler
kernel 4.11: Deadline IO scheduler

Try NOOP.

Choosing I/O Scheduling

The Linux 2.6 release provides four disk I/O schedulers:

Anticipatory, deadline, completely fair queuing (CFQ), and noop along with an option to select one of these four at boot time or runtime...

Noop is often the best choice for memory-backed block devices (e.g. ramdisks) and other non-rotational media (flash) where trying to reschedule I/O is a waste of resources

[bson]

You might be getting large numbers of alignment traps, either in the kernel ram block device driver or dd.

Even if the memory bus is 8-bit wide the PPCs still don't won't permit unaligned accesses, and those will trap into the kernel and get emulated.  You can make them produce a SIGBUS for user space programs, see e.g. (it's been over 10 years since I had to track down similar problems so this may be out of date):
https://www.kernel.org/doc/Documentation/arm/mem_alignment

Verify that your ram disk starts at an 8-byte alignment, although in reality the driver should page align it unless it's buggy.

[legacy]

I had to track down similar problems so this may be out of date

I will investigate, but ... on kernel 2.6.39? kernel 4.11? shouldn't they be a bit "modern"- enough to have those problems already fixed  ?