Modernising an array of spinning rust?

[dexters_lab]

This is really just a hypothetical look at an unusual problem, i don't actually expect to implement anything, it's just to explore the different ways this problem could be tackled.

Sat in my lab is a large (40kg!) disk array for the quantel hal video compositing system known as a dylan 2, the entire array and it's interface is entirely proprietary to quantel but it does use standard Fujitsu 3.5" 50 pin SCSI disks of 1.2Gb capacity each. The array is used to record and playback uncompressed bt.601 video.

In total there are 20 disks, each one is a single drive on it's own SCSI bus on SCSI ID0, data i believe it's written to each disk sequentially to achieve the bandwidth needed, one of the disks is a parity disk. The array is able to rebuild a failed disk using the data on the other disks but i doubt it's any kind of standard raid implementation. The array uses 20 QLogic FAS236 SCSI processors to interface with the disks.

My question is, what are the options to modernise this and use solid state media instead of the 20 spinning disks?

So far i have come up with the following, but wanted to see what other ideas others might come up with...

1) replace the disks with SCSI2SD adaptors and uSD cards:
https://amigakit.amiga.store/catalog/product_info.php?products_id=1264
Pros: easy to use, no modifications needed, off the shelf
Cons: prohibitive cost (£1100), 1Mb/sec transfer is 10x slower than the original disks maybe insufficient, though there is a 10Mb/sec version for £100 each.

2) custom device to plug into each of the 20 channels and consolidate the data into a single standard sata ssd
Pros/cons: 

[Berni]

You can get microSD cards to go pretty fast. Its pretty reasonable to get 30MB/s of read speed from a decent card. Fancy cards can go faster but require weird DDR signaling modes on the bus to get there. Write speeds on SD cards are quite crap tho. Even fast cards are tough to get over 10MB/s consistently. But for speed CF cards are a better choice anyway.

I suppose you could also build your own SCSI SSD drive by connecting a FPGA to some parallel NAND flash chips, could be fairly cost effective and fast but it will need quite a bit of programming to make the FPGA do its magic.

[grumpydoc]

SCSI SSD's exist http://www.bitmicro.com/products/e-disk-altima-2-5%E2%80%B3-scsi-narrow-ssd/?gclid=EAIaIQobChMIhduz94Px2QIVWmQZCh2iJgUxEAAYASAAEgKG-vD_BwE though as they are targeted at the military I suspect that the bill for 20 would be eye-watering.

There are also SCSI to SATA adapters https://www.span.com/category/ATA-SATA-to-SCSI-Adapters~1202 but, again, 20 is going to need a new mortgage.

The SCSI to SD looks the best bet, if you can get the bandwidth.

Do you know if  it will take any old SCSI disks or is there a dependancy on either the 1.2G size or the manufacturer

EDIT: there's a SCSI to full-sized SD adapter here http://shop.codesrc.com/index.php?route=product/product&path=59&product_id=62 more dosh but the bandwidth is there.

[dexters_lab]

Do you know if  it will take any old SCSI disks or is there a dependancy on either the 1.2G size or the manufacturer

i've not tried yet, but i am expecting it will take any disk that is at least 1.2gb. i have not seen any evidence it's locked to a particular make/model, but tbh the SCSI2SD adaptors allow the make/model/serial number fields of the emulated disks to be customised. The disks are also an off-the-shelf model from Fujitsu

[DC1MC]

Contact those guys for some 2GB disks:

https://www.ebay.de/itm/SILICON-MEDIA-4GB-SLC-3-5-SCSI-II-50pin-SOLID-STATE-DISK/152517919178

The price is almost similar with a a SCSI2SD adapter + an extreme good quality SD, and for 20pcs you'll get a discount.

 Cheers,
 DC1MC

[dexters_lab]

Contact those guys for some 2GB disks:

https://www.ebay.de/itm/SILICON-MEDIA-4GB-SLC-3-5-SCSI-II-50pin-SOLID-STATE-DISK/152517919178

The price is almost similar with a a SCSI2SD adapter + an extreme good quality SD, and for 20pcs you'll get a discount.

 Cheers,
 DC1MC

erm, that would cost nearly 5x as much!

[senso]

Ask for a quote 20x vs single quantity should have a decent difference in price.

[DC1MC]

Contact those guys for some 2GB disks:

https://www.ebay.de/itm/SILICON-MEDIA-4GB-SLC-3-5-SCSI-II-50pin-SOLID-STATE-DISK/152517919178

The price is almost similar with a a SCSI2SD adapter + an extreme good quality SD, and for 20pcs you'll get a discount.

 Cheers,
 DC1MC

erm, that would cost nearly 5x as much!

What will cost 5x as much ?

With the TOY SCSI adapters (

Write speeds achievable to a maximum of 2.5Mb/sec (approximately). Supports asynchronous transfers only.

) and let's put 5GPB for a card you have:
(55 + 5) * 20 = 1200GPB
There is 56% chance that the stuff will actually NOT be compatible to a RAID for video recording, but this is secondary.

Let's assume that the seller will give you a price 60% for the 2GB disk that is close to the original Fujitsu, say 150GPB/disk, maybe lower.

That is 150 * 20 = 3000GPB , that is more than 1200, but actually has a good chance to work for more than 2hrs until the SD cards will go bad.

 So, where is the 5 times more, 'cause I don't see it ?

 Cheers,
 DC1MC

[BrianHG]

Just a short reminder course for me, doesn't uncompressed bt.601 require 27 megabytes a second.  (OK, I realize a tad less due to missing video data during blanking period, let's approximate this extra data as the parity bits).  If the device had a custom method to spread out the data across 20 controllers we are talking 1.3 megabytes/sec per controller.  I'll assume the drives read and write in almost straight lines.  Your drives all need to be media grade still, all it takes is a thermal calibration cycle, and the video record/playback will be messed up.

I'm assuming this server, or the device it's attached to has at least a few frames of cache buffer ram.
I would guess that each drive would need to burst 2mb/sec when reading or writing, and sustain 1.3mb/sec.  In straight lines without a file system, on an SD card, this may be possible depending on the SD card's controller.

I know, if writing non-stop to high density high speed m.2 flash drives, they do heat up and throttle down, but, the fact that all these flashes are isolated chips on separate boards, each running slow, they might save you thermal problems here.

I think you are sitting on the border with the Amiga device, but, in the coments, user's say it works on SCSI-I buss as well as SCSI-II and this one is a shit load faster with 10mb/sec, worst case 5.5mb/sec support: http://amigakit.amiga.store/catalog/product_info.php?products_id=1314  Though, it costs more, you now have huge safety margin.

Here is the board you selected, but used in a PC environment instead of an Amiga.

The video claims 1mb/sec, though, this may be a bottleneck of the PC.
Many others have used this device in a PC, MAC, NeXT, Roland digital synths/sound mixers, and of course Amigas.  It seems compatibility wont be a problem, it's will the throughput be good enough.

The array is used to record and playback uncompressed bt.601 video.

If your array did any minimal lossless signal compression at all, like a 1.5:1, or 2:1, or 1.25:1 (4:2:0 video) then, these SCSI2SD devices would clear your speed hurdles instead of going right to the edge of just making it.

[BrianHG]

Yup, it's gonna cost you, but, the V6 of SCSI2SD is 5x faster at 10mb/sec Vs 2mb/sec.
The v5 you have chosen has the following speed:

Up to 2.6MB/s read, 2.3MB/s write.

http://www.codesrc.com/mediawiki/index.php?title=SCSI2SD

This should be fast enough for you digital video recorder running 20 of these in parallel.  the one + about these boards, is in the setup, you can exactly mimic the existing drives model names, serial numbers and sector count & size.

The more expensive v6 board is overkill.  However, this is till an expensive endeavor.

[BrianHG]

2) custom device to plug into each of the 20 channels and consolidate the data into a single standard sata ssd
Pros/cons: 

You are now bordering on the line of making a box, with the same plugs in the back as the dylan 2, feed the plugs with a big FPGA and a m.2 or SATA3 SSD drive and emulate the dylan 2.  Just all the consolidation of 20 SCSI channels back down to 1 buss alone will cost you more time, but, maybe a little less money than the 20x SCSI2SD boards, while just simulating the entire dylan 2, which is basically a sequential 27mhz 8/10 bit parallel port reader & writer where you can address the beginning of each frame, is doable on a sub 400$ FPGA eval board with 1 high performance SSD, and a fraction the coding time compared to telling the existing dylan 2 you have reconnected 20 valid drives.  The only reason the FPGA board is so expensive is for the high speed LVDS needed to talk to the m.2 / SATA3 drive.  (I'm going on Altera prices, Xilinx may have cheaper FPGA boards with 3GB/sec LVDS lines.  1.5gb SATA1 should still be fast enough.) Otherwise, a bottom end below 100$ FPGA board will do it with many parallel flash cards, though, the speed wont be as guaranteed as a high performance single drive.

The other advantage to using a SATA3/2/1 interface, is that you can use cheap spinning HDs as well if you are worried about burning out your flash from multiple writing.  But, unless you are recording video, over-and-over agian-and-again day in and day out, like a hundreds of recordings over the same area, you will just need to replace the SSD.

[dexters_lab]

thanks everyone

i do have one of those SCSI2SD v5 adaptors (the cheaper ones) in my Paintbox Express, they do top out at 1Mb/sec from what i have seen, which is a bit of a shame as the original Fujitsu disks it replaced could sustain 10Mb/sec so i don't think that model will cut the mustard in terms of speed.

I have asked for a quote from the supplier of SCSI SSDs that was found on ebay, i am waiting to hear back.

i spent some time with forum user OzOnE the other day looking at this from our previous experiences with quantel and although we have not turned on the 'scope yet and checked we are coming up with some educated ideas about how this array works which might mean making an emulator for the entire array somewhat more viable than i'd previously thought.

The array might be setup as video input/output links and a serial control link, video data might actually be bt.656, the parallel version of SDI and the serial link could be using the Inmos link used on the array's CPU. This arrangement mirrors the Paintbox & Ramcorder setup of the Quantel Harriet

[xani]

Take care:

http://dexterslab2013.blogspot.com/2018/03/hacking-quantel-dylan-disk-array.html

Seems like array have hardcoded serials of drives, presumably so you have to pay $$$ for technican to replace drive instead of doing it yourself

[dexters_lab]

Take care:

http://dexterslab2013.blogspot.com/2018/03/hacking-quantel-dylan-disk-array.html

Seems like array have hardcoded serials of drives, presumably so you have to pay $$$ for technican to replace drive instead of doing it yourself

 

that's my own blog, the serial number issue is not a problem

[technix]

What the array uses is very likely some kind of RAID, be it a dedicated-parity RAID-4 system or a distributed-parity RAID-5 system. The specific system design of using multiple SCSI controllers points me at a software RAID solution. Either way, if you know the stripe size, you have a chance of recovering all original data after imaging all drives and reassemble the original contents on a modern computer. RAID-4 can be reassembled by putting the data disks together ad RAID-0 after a consistency check. RAID-5 may need some more trial and error at figuring out what is the parity for each stride.

[dexters_lab]

What the array uses is very likely some kind of RAID, be it a dedicated-parity RAID-4 system or a distributed-parity RAID-5 system. The specific system design of using multiple SCSI controllers points me at a software RAID solution. Either way, if you know the stripe size, you have a chance of recovering all original data after imaging all drives and reassemble the original contents on a modern computer. RAID-4 can be reassembled by putting the data disks together ad RAID-0 after a consistency check. RAID-5 may need some more trial and error at figuring out what is the parity for each stride.

thankfully i don't need to recover any data, this is really to look at making it reliable with the removal of the old hard disks