Author Topic: PCI Gigabyte I-RAM without the PCI part :)  (Read 20282 times)

0 Members and 1 Guest are viewing this topic.

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
PCI Gigabyte I-RAM without the PCI part :)
« on: January 15, 2015, 05:34:51 am »
Hi All,

I have always wanted a hardware ram disk to play with, and when the Gigabyte I-RAM came out I was amazed at the performance that was possible with this device. These days its performance is not that great in comparison with SSDs. The reason for this hack/mod was because I recently had to rebuild the RAID array in my home server after a drive failed, since this was an opportunity to update to ZFS, I took it. After learning about ZFS and how you can provide it with even a small USB (slow) cache disk for massive disk performance improvements, I decided I would see if I could obtain a 2nd hand I-RAM.

After a bit of searching I found one supplier in the US that had a brand new I-RAM listed as unsold stock, I snapped this up for $40USD. I already had 4x 1GB DDR modules from an old upgrade (Yes, I hoard), so I did not have to purchase those. Next problem was that my server has no more PCI slots remaining on the board, they are all in use. After a close look at the I-RAM (about 2 seconds), it was very obvious that the device is only powered from the PCI slot, there are no signal lines connected except for RESET.

I dug out an old faulty motherboard (yes, again, I hoard stuff), and did my best to de-solder a PCI slot. Now I realise I could have just soldered onto the I-RAM, but I would like to keep the card in good condition, it is an item of interest :). I got the PCI slot out, but lost 1/2 the pins in the process, which was no big deal since we only needed to provide the +12v, +5V, +3.3V and ground connections, and there were enough pins remaining for this after rearranging them :).

It should be noted that this thing has a battery on it, and charges it, also DDR can be pretty power hungry, so to be sure I could supply enough current every ground and voltage pin must be connected (later I confirmed this theory). In total about 20 pins had to be wired up, and a small DC-DC switch mode regulator to provide the +3.3V rail.

After plugging in a test card (random PCI sound card) to ensure there were no faults (there were, had +5v shorted to GND), I installed the I-RAM into my adaptor. LEDs came on... great! After noting that it draws 1.2A, I was glad I opted to connect every power pin. Next I wired up the Molex connector and attached it to the PC, lights came on, and an extra one came on when the SATA cable was connected, but no joy, the PC would not see the device. I did a quick double check of things and I decided to pull up the RESET pin to +3.3V. Success, the device was now accessible.

Code: [Select]
[618884.466560] ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[618884.467076] ata6.00: configured for UDMA/133
[618884.467084] ata6: EH complete
[618884.467186] scsi 5:0:0:0: Direct-Access     ATA      GIGABYTE i-RAM   8    PQ: 0 ANSI: 5
[618884.467398] sd 5:0:0:0: Attached scsi generic sg3 type 0
[618884.467409] sd 5:0:0:0: [sdd] 8386559 512-byte logical blocks: (4.29 GB/3.99 GiB)
[618884.467561] sd 5:0:0:0: [sdd] Write Protect is off
[618884.467565] sd 5:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[618884.467647] sd 5:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[618884.469121]  sdd: unknown partition table
[618884.469430] sd 5:0:0:0: [sdd] Attached SCSI disk

Now as I said earlier, the I-RAM is nothing to brag about these days, benchmarks show a consistent 120MB/s and 0.1ms access time. Any decent modern SSD can outperform this device, but the down side of the SSD would be the write life as the I-RAM should electrically fail before it hits any write limits, making it the perfect RAID-Z cache device on a budget.

Since these photos were taken I have covered the connector pins in hotglue to prevent any shorts or movement as this connector has seen better days after my treatment of it while de-soldering (levering out with a crowbar?) it :). I have also clearly marked the front end of the connector as the card can be plugged in backwards, don't want to blow it up should I need to change things later :).

Code: [Select]
xen:~# zpool status
  pool: datapool
 state: ONLINE
  scan: scrub repaired 0 in 5h45m with 0 errors on Sun Jan 11 07:45:19 2015
config:

NAME                                                STATE     READ WRITE CKSUM
datapool                                            ONLINE       0     0     0
  raidz1-0                                          ONLINE       0     0     0
    ata-WDC_WD40EFRX-68WT0N0_WD-XXXXXXXXXXXX-part2  ONLINE       0     0     0
    ata-WDC_WD40EFRX-68WT0N0_WD-XXXXXXXXXXXX-part2  ONLINE       0     0     0
    ata-WDC_WD40EFRX-68WT0N0_WD-XXXXXXXXXXXX-part2  ONLINE       0     0     0
cache
  ata-GIGABYTE_i-RAM_XXXXXXXXXXXXXXXXXXXX     ONLINE       0     0     0
« Last Edit: January 15, 2015, 06:38:02 am by gnif »
 

Offline Fsck

  • Super Contributor
  • ***
  • Posts: 1157
  • Country: ca
  • sleep deprived
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #1 on: January 15, 2015, 06:43:51 am »
if your ARC throughput is 120MB/s, that doesn't seem like it'd accelerate transfers much unless you're doing small size, high iops predominantly.
"This is a one line proof...if we start sufficiently far to the left."
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #2 on: January 15, 2015, 06:49:34 am »
if your ARC throughput is 120MB/s, that doesn't seem like it'd accelerate transfers much unless you're doing small size, high iops predominantly.

You are correct, this pool is for a development server and local caching proxy, lots of small files :). Honestly though I do not need this, it is just for the cool factor :)
 

Offline Scrts

  • Frequent Contributor
  • **
  • Posts: 797
  • Country: lt
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #3 on: January 15, 2015, 09:36:12 am »
Thumbs up, nice work! :-+

I wonder did anyone ever made a faster device with faster memory? Maybe something for servers? E.g. DDR2 memory stuff before SSDs came in.
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #4 on: January 15, 2015, 11:28:33 am »
Thumbs up, nice work! :-+

I wonder did anyone ever made a faster device with faster memory? Maybe something for servers? E.g. DDR2 memory stuff before SSDs came in.

Thanks. Yes there are other offerings that use DDR2 and DDR3, but they are extremely expensive, usually only used in high end database servers that require insane IOPS. These cards interface directly over the PCIe bus, no SATA interface on them, I believe Intel recently started offering cards that give throughput in excess of 1GB/s.
 

Offline suicidaleggroll

  • Super Contributor
  • ***
  • Posts: 1453
  • Country: us
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #5 on: January 15, 2015, 01:03:21 pm »
Thumbs up, nice work! :-+

I wonder did anyone ever made a faster device with faster memory? Maybe something for servers? E.g. DDR2 memory stuff before SSDs came in.

I don't really see the point.  Just throw more RAM in the server and mount some of it as a ramdisk in the OS.  10+ GB/s for just the cost and complexity of more/bigger RAM modules.  I do this all the time on my servers.

Back "in the day" when motherboards could only support a few GB of memory (or less) max, this device made sense, but that hasn't been an issue for a long time.
 

Offline TiN

  • Super Contributor
  • ***
  • Posts: 4543
  • Country: ua
    • xDevs.com
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #6 on: January 15, 2015, 02:56:23 pm »
Been there, did that.  ;D
More straight forward tho, just SATA power plug to cut pieces of MB.











There is another option, with DDR2 and dual SATA3 ports - ACARD ANS-9010, based on Xilinx Spartan FPGA.
« Last Edit: January 15, 2015, 02:58:45 pm by TiN »
YouTube | Metrology IRC Chat room | Let's share T&M documentation? Upload! No upload limits for firmwares, photos, files.
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #7 on: January 15, 2015, 08:55:49 pm »
Thumbs up, nice work! :-+

I wonder did anyone ever made a faster device with faster memory? Maybe something for servers? E.g. DDR2 memory stuff before SSDs came in.

I don't really see the point.  Just throw more RAM in the server and mount some of it as a ramdisk in the OS.  10+ GB/s for just the cost and complexity of more/bigger RAM modules.  I do this all the time on my servers.

Back "in the day" when motherboards could only support a few GB of memory (or less) max, this device made sense, but that hasn't been an issue for a long time.

These devices are battery backed so can be used for persistent data. Also you may be (like me) running in older hardware that is already running with the maximum ram capacity. And finally, software such as  ZFS already use as much RAM as it can to improve performance, this just gives it some more to work with.
 

Offline mattaw

  • Contributor
  • Posts: 12
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #8 on: January 19, 2015, 02:41:07 am »
It is great to see another person do this mod - as well and see all our different approaches (TiN's I saw on HardForum? but somewhere first). I documented mine here and also the need to keep some 2N7000's around in case of mistakes - had to replace the same one twice so far due to floating the reset until I got that figured out. When I read that you hadn't pulled it up I was worried for a while that you would burn it too.

Here is the link to my solution, similar to yours but I used a linear from TI hotplate soldered onto a larger scrap of PCB for heatsinking and hardwired to the card itself mounted to the case lid with standoffs.

http://mattaw.blogspot.com/2014/06/modding-gigabyte-i-ram-gc-ramdisk-to.html

Still a fantastic amount of IOPS for not much money. Would love to get my sticky fingers on a A-Data box. I should add that there are some marvell PCI-e RAM/NAND Flash cards on eBay for about 30-40 USD but driver support is very lacking AFAIK. They were designed to plug into Dell Equalogic storage systems. Might try picking one up anyway for my new system.
« Last Edit: January 19, 2015, 02:47:34 am by mattaw »
 

Offline Boost

  • Newbie
  • Posts: 1
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #9 on: January 19, 2015, 07:08:06 am »
I'm qurious. What OS are you running ZFS on?
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #10 on: January 19, 2015, 07:23:13 am »
I'm qurious. What OS are you running ZFS on?

This is on Debian Linux using ZFS On Linux
 

Offline mattaw

  • Contributor
  • Posts: 12
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #11 on: January 20, 2015, 04:31:37 am »
Ubuntu with zfsonlinux. However I am looking at linux mint for an OS without all that Canonical "Magic" I am so unfond of. Especially the integration with amazon and other stuff which sends all your search results to commercial companies. I understand they want to make money but I will be making other choices of distribution in future.

M
 

Offline chickenHeadKnob

  • Super Contributor
  • ***
  • Posts: 1054
  • Country: ca
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #12 on: January 20, 2015, 05:42:52 am »
Ubuntu with zfsonlinux. However I am looking at linux mint for an OS without all that Canonical "Magic" I am so unfond of. Especially the integration with amazon and other stuff which sends all your search results to commercial companies. I understand they want to make money but I will be making other choices of distribution in future.

M

Mint also monetizes their users.
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #13 on: January 20, 2015, 02:05:10 pm »
Ubuntu with zfsonlinux. However I am looking at linux mint for an OS without all that Canonical "Magic" I am so unfond of. Especially the integration with amazon and other stuff which sends all your search results to commercial companies. I understand they want to make money but I will be making other choices of distribution in future.

M

Just use Debian, no Canonical Magic, still pure APT, and widely supported. Ubuntu IMHO is a load of bloat ware.
 

Offline JJalling

  • Supporter
  • ****
  • Posts: 246
  • Country: dk
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #14 on: January 20, 2015, 02:57:51 pm »
Ubuntu with zfsonlinux. However I am looking at linux mint for an OS without all that Canonical "Magic" I am so unfond of. Especially the integration with amazon and other stuff which sends all your search results to commercial companies. I understand they want to make money but I will be making other choices of distribution in future.

M

Just use Debian, no Canonical Magic, still pure APT, and widely supported. Ubuntu IMHO is a load of bloat ware.

My server is currently running with 4x2TB disks in RAID-5 and ext-3. I plan on changing the disks to 4*4TB instead.
Would you recommend that I switch over to RAID-Z instead of just building a new RAID-5? I have read that ZFS requires a lot of memory. I currently have 16GB installed - is that enough?

BR Jonas
 

Offline macboy

  • Super Contributor
  • ***
  • Posts: 2252
  • Country: ca
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #15 on: January 20, 2015, 07:19:15 pm »
Ubuntu with zfsonlinux. However I am looking at linux mint for an OS without all that Canonical "Magic" I am so unfond of. Especially the integration with amazon and other stuff which sends all your search results to commercial companies. I understand they want to make money but I will be making other choices of distribution in future.

M

Just use Debian, no Canonical Magic, still pure APT, and widely supported. Ubuntu IMHO is a load of bloat ware.

My server is currently running with 4x2TB disks in RAID-5 and ext-3. I plan on changing the disks to 4*4TB instead.
Would you recommend that I switch over to RAID-Z instead of just building a new RAID-5? I have read that ZFS requires a lot of memory. I currently have 16GB installed - is that enough?

BR Jonas
What is your server doing? Storing and serving media files for you and your family, or for a neighborhood of ten thousand people? If the former, then yes you have enough memory. If the latter, then no you don't.

I run ZFS on OpenSolaris with 2 GB RAM. I have 5 x 1.5T disks + separate ZFS root disk, no cache device, and it performs extremely well as a home storage server. ZFS is amazing. You will love it compared to RAID-5. When one of my disks started to go and sectors started becoming unreadable, ZFS detected which sectors on which disk were bad and repaired the affected files automatically, with zero data loss. RAID-5 will not do that. ZFS can rebuild the array onto a new replacement disk while the old (failing but usable) disk is still working away, providing redundant data to 99.999% of the filesystem. RAID-5 can't. ZFS can do 2 or 3 redundant disks, hot spares, etc. Very cool. And don't even get me started on how cool snapshots are, or partition-less file systems. When sharing zfs file systems to Windows using Opensolaris, snapshots appear as "previous versions" of a disk just like restore points do for NTFS disks. You can use Windows explorer to browse through snapshots and copy deleted or old versions of files from them.
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1672
  • Country: au
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #16 on: January 21, 2015, 10:37:27 am »
Ubuntu with zfsonlinux. However I am looking at linux mint for an OS without all that Canonical "Magic" I am so unfond of. Especially the integration with amazon and other stuff which sends all your search results to commercial companies. I understand they want to make money but I will be making other choices of distribution in future.

M

Just use Debian, no Canonical Magic, still pure APT, and widely supported. Ubuntu IMHO is a load of bloat ware.

My server is currently running with 4x2TB disks in RAID-5 and ext-3. I plan on changing the disks to 4*4TB instead.
Would you recommend that I switch over to RAID-Z instead of just building a new RAID-5? I have read that ZFS requires a lot of memory. I currently have 16GB installed - is that enough?

BR Jonas
What is your server doing? Storing and serving media files for you and your family, or for a neighborhood of ten thousand people? If the former, then yes you have enough memory. If the latter, then no you don't.

I run ZFS on OpenSolaris with 2 GB RAM. I have 5 x 1.5T disks + separate ZFS root disk, no cache device, and it performs extremely well as a home storage server. ZFS is amazing. You will love it compared to RAID-5. When one of my disks started to go and sectors started becoming unreadable, ZFS detected which sectors on which disk were bad and repaired the affected files automatically, with zero data loss. RAID-5 will not do that. ZFS can rebuild the array onto a new replacement disk while the old (failing but usable) disk is still working away, providing redundant data to 99.999% of the filesystem. RAID-5 can't. ZFS can do 2 or 3 redundant disks, hot spares, etc. Very cool. And don't even get me started on how cool snapshots are, or partition-less file systems. When sharing zfs file systems to Windows using Opensolaris, snapshots appear as "previous versions" of a disk just like restore points do for NTFS disks. You can use Windows explorer to browse through snapshots and copy deleted or old versions of files from them.

I could not agree more, I just moved from a RAID6 array to RAID-Z, this is the first time I have used ZFS and I must say it is incredible. It also avoids the raid write hole issue. My server only has 8GB of RAM, most of which is dedicated to xen virtual machines leaving only 2GB for the host machine, and ZFS. This performs quite well even before I added the ZIL device.

A small note though, making ZFS your root device is rather hard to get right initially, it took me 4-5 hours to figure out how to make Debian happy to use ZFS as it's root file system. I have also used a software (mdadm) RAID1 partition on the disk for /boot, this is because not all versions of GRUB support booting directly from a ZFS volume, should you need to perform recovery it is good to have third party tools such as sysrescuecd to be able to see your /boot volume.
 

Offline JJalling

  • Supporter
  • ****
  • Posts: 246
  • Country: dk
Re: PCI Gigabyte I-RAM without the PCI part :)
« Reply #17 on: January 21, 2015, 10:56:22 am »
Ubuntu with zfsonlinux. However I am looking at linux mint for an OS without all that Canonical "Magic" I am so unfond of. Especially the integration with amazon and other stuff which sends all your search results to commercial companies. I understand they want to make money but I will be making other choices of distribution in future.

M

Just use Debian, no Canonical Magic, still pure APT, and widely supported. Ubuntu IMHO is a load of bloat ware.

My server is currently running with 4x2TB disks in RAID-5 and ext-3. I plan on changing the disks to 4*4TB instead.
Would you recommend that I switch over to RAID-Z instead of just building a new RAID-5? I have read that ZFS requires a lot of memory. I currently have 16GB installed - is that enough?

BR Jonas
What is your server doing? Storing and serving media files for you and your family, or for a neighborhood of ten thousand people? If the former, then yes you have enough memory. If the latter, then no you don't.

I run ZFS on OpenSolaris with 2 GB RAM. I have 5 x 1.5T disks + separate ZFS root disk, no cache device, and it performs extremely well as a home storage server. ZFS is amazing. You will love it compared to RAID-5. When one of my disks started to go and sectors started becoming unreadable, ZFS detected which sectors on which disk were bad and repaired the affected files automatically, with zero data loss. RAID-5 will not do that. ZFS can rebuild the array onto a new replacement disk while the old (failing but usable) disk is still working away, providing redundant data to 99.999% of the filesystem. RAID-5 can't. ZFS can do 2 or 3 redundant disks, hot spares, etc. Very cool. And don't even get me started on how cool snapshots are, or partition-less file systems. When sharing zfs file systems to Windows using Opensolaris, snapshots appear as "previous versions" of a disk just like restore points do for NTFS disks. You can use Windows explorer to browse through snapshots and copy deleted or old versions of files from them.

That sounds great. The server is mainly for storage, so I will give it a go.
Now I'm just waiting for the disks to arrive :)

Thanks

BR Jonas
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf