EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1

[kyndal]

just use a 3 pin high connector. block /disregard  bottum row of pins

Kyndal

[Fungus]

They seem to require heatsinks (according to one of his reviews), so how big is big enough? 

I doubt it will be very much if they sell them without heatsinks.

Don't be so sure about it. I'm using a 'SoC on a module' for a commercial project and it needs a huge friggin heatsink which is sold seperately to keep the module within specs and it is mounted in a casing with plenty of natural convection.

I re-checked the previous video where Dave ran one of these at 100%. It got up to 90 degrees but it didn't die.

I guess the size of the heatsink will come down to airflow in the case.

[CM800]

I think this has the same-ish problem as the flat-packed Pies: lots of wasted board space under them.

My first thought (echoed early on by several others) was to make a simple board that converted the dual-row pin-header to a card-edge connector. These adapter boards would be super simple and small: barely a dual-row header and one side with card-edge fingers. I very much like the idea of going with a very common connector (e.g. PCI, even though it's more pins than necessary.) For that matter, it wouldn't be much more effort to design the board with a second set of holes for another female header to handle the other rotation of the board (e.g. Raspberry vs. Orange).

The question is, should you make the adapter board include things like LEDs, pass-through pins, a local LDO regulator, or the SPI-Ethernet adapter.

I'd beg to differ there, Wasted board space is not the biggest issue, cores per cm^3 is something more to think about. PCBs are cheap! Each of these boards could even possibly be done on a two layer, having four, or even eight of the Pis on them (1 on each side, using SMT header connectors)

The heatsink would be in the middle, then each board can be placed on a back-plane directly next to each-other, it would be the most compact solution we've discussed here no doubt.

All that work to achieve the ethernet I/O?
I'd be SORELY tempted to outlay a HUGE extra US$2.01 per board and use the Orange Pi Lite which has onboard WiFi!
Whether I would actually USE the supplied WiFi antennae or not is another question.
I'm willing to bet they could all 'cross-communicate' without ANY antenna being connected if they were all co-resident in a single enclosure with just a 50ohm resistor soldered across the u.fl connector?
The FIRST 'PI' within any such 'array' / 'cluster' could then serve as a wired-to-wireless protocol converter as well as a basic local WiFi 'Access Point'
It would probably work out cheaper overall and significantly reduce the internal wiring complexity.

Few things to think about there: You're creating lots of traffic in the wifi spectrum, this thing isn't infinite, believe it or not, if you set a lot of networked devices over wifi sending data constantly, you will find all the wireless networks slow down in the vicinity. (You're broadcasting on all the available channels in the local air) I might be incorrect here but from what I've heard it's true and a real effect.

The other thing is that holds no interest to most of us I'd imagine, if your going to do it with wi-fi, you might as well do it wired. I feel the main goal (or at least what I, and others are  getting out of this project is learning electrical design knowledge, not how to turn on a load of pies then configure them to communicate with each-other through wifi. Most, if not all of us here could do that in a few hours after they first get their hands on them and read up on google. There is much more to learn developing a compact baseboard.

[tszaboo]

What I would do:
Make a Eurocard PCB with the DIN41612 connector at the end (lets call it adapter board). Design the board, so it can take 2 Orange PIs, 5ish raspberry Pi zeros, 4 CHIP. or your choice of computing. Orange PI is mounted upside down on the adapter board, RPi zero has the connector soldered on the bottom, and USB pins tapped with a pogo pin. Route 24V, maybe 12V to the adapter board, have an onboard POL converter. Design a backplane, it routes power + ethernet to each board. Each adapter board has an ethernet switch, in case of necessary. Maybe it even has hot swap control. The backplane has 21 connectors (AFAIK, that is standard). I reckon, you can fit 42 RPis, 105 RPi zeros into a 3U rack. All mechanical parts are off the shelf. Cooling vertically. Hot plugging the cards, if you feel like. Mix different cards, if you feel like.

[MisterDiodes]

I think that if you can bypass PHY entirely you may be able to connect the RGMII interfaces together directly.

RGMII is generally not thought of being perfectly symmetrical in both the hardware and protocol sense - although I know some Micrel (et al) PHY chips can be connected back to back like that if you're building a repeater or media converter.  Usually that isn't the case for a generic connection - Normally RGMII is meant for a MAC interfacing to PHY... Not MAC to MAC.

[Towger]

Few things to think about there: You're creating lots of traffic in the wifi spectrum, this thing isn't infinite, believe it or not, if you set a lot of networked devices over wifi sending data constantly, you will find all the wireless networks slow down in the vicinity.

I also suggested wifi, but using the small (ebay type) dongle which fit into a usb socket.  Bandwidth should not be a problem, except if all machines first start Bonic at the same time.  After Bonic downloads a block of data it (depending on cpu power) can take days before it finishes and needs more.

In saying that I like the mineral oil cooling idea, totally impractical, but leads to a more interesting video (more views).  I don't think 2.4ghz will have much of a range in oil [emoji3]

[nctnico]

Few things to think about there: You're creating lots of traffic in the wifi spectrum, this thing isn't infinite, believe it or not, if you set a lot of networked devices over wifi sending data constantly, you will find all the wireless networks slow down in the vicinity.

I also suggested wifi, but using the small (ebay type) dongle which fit into a usb socket.  Bandwidth should not be a problem, except if all machines first start Bonic at the same time.  After Bonic downloads a block of data it (depending on cpu power) can take days before it finishes and needs more.

If you want to run tasks in parallel they are likely to need data pumped in&out so network traffic could be massive. I don't know the Allwinner's performance when it comes to compressing video but if it is any good Dave could use the cluster to compress his Youtube videos.

edit: typo

[KenGaler]

I agree with the suggestion of a 90Deg adapter board and get rid the slots.  They can be designed on the same layout as the mother board and v-scored.  This way they are essentially free.

Ken

[onarch]

How about using the microUSB OTG to connect all the boards together?

You can get microUSB plug connector for PCB mount. Like ZX80-B-5SA from Hirose, see [3]. (Didn't know these existed before I checked)

Use one board as the master (USB host) and connect the other ones (USB devices) to it using a USB hub chip[2]. USB OTG can act as either a device or host depening the state of the ID pin. In Linux you can then USB Ethernet gadget on the devices to create a Ethernet connection to the master (host).  See: http://linux-sunxi.org/USB_Gadget
The master can then setup a network bridge between its external Ethernet interface and the USB Ethernet connections from the devices, and will thus act like a switch.

8x NanoPi           $7.99      $63.92 [1]
8x ZX80-B-5SA       $1.60      $14.40 [2]
1x TUSB2077         $4.79      $ 4.79 [3]
+ PSU and board

Advantages:

• Only 4 wires needed for both signal and power for each board
• USB is easy to route on a PCB
• USB hubs use less power than Ethernet switches
• Standard microUSB interconnect allows you to basically use any board you want
• USB hubs can be cascaded (like Ethernet)

Disadvantages:

• Requires a master board
• Boards will most likely need more mechanical support than just the microUSB connector
• Requires somewhat advanced Linux configuration (gadget+bridge setup)

[1] http://www.friendlyarm.com/index.php?route=product/product&product_id=132
[2] http://www.digikey.com/product-detail/en/texas-instruments/TUSB2077APTR/296-37871-1-ND/4878718
[1] http://www.digikey.com/product-detail/en/hirose-electric-co-ltd/ZX80-B-5SA/H11612-ND/1963857

[nctnico]

I think that if you can bypass PHY entirely you may be able to connect the RGMII interfaces together directly.

RGMII is generally not thought of being perfectly symmetrical in both the hardware and protocol sense - although I know some Micrel (et al) PHY chips can be connected back to back like that if you're building a repeater or media converter.  Usually that isn't the case for a generic connection - Normally RGMII is meant for a MAC interfacing to PHY... Not MAC to MAC.

Connecting RGMII back to back should be possible either with an external clock or one interface as master but the point is rather moot. Routing ethernet is going to be much easier. Because of the short distance on a board you don't need matched impedance traces so as long as they are routed as pairs away from noise everything is fine (100Mbit ethernet has frequency components up 100MHz so up to 40cm long traces don't behave as transmission lines yet).

[claus]

I think I understood that performance is not so important here, but a "supercomputer" (even if it's a pocket size supercomputer is (wikipedia): A supercomputer is a computer with a high-level computational capacity compared to a general-purpose computer. Performance of a supercomputer is measured in floating-point operations per second (FLOPS). Have in mind that a "general purpose computer" in this context is a PI.

So essentially you want to build something faster combining various cpus. The problem here is that if communications are slow, the resulting computer might be slower, and this would just be a waste of energy. Those Pi's are not so terribly bad at crunching numbers, especially the 4-cores cpus, see http://www.roylongbottom.org.uk/Raspberry%20Pi%20Multithreading%20Benchmarks.htm, they get around 2 GFlops for the multithreaded version.

So if you want the final thing to be faster (on an application that is not embarrassingly parallel) than a single PI you need good (fast) communication, otherwise a multithreaded program might run slower on 4 Pi's than on 2. The orange pi has a gigabit Ethernet port which should give a reasonable connection for parallel computation, I would use it with a gigabit switch for a model "supercomputer". If all applications will be embarrassingly parallel the SPI/Ethernet approach might make sense, but only as an application specific "supercomputer".

If on the other hand the design of a sexy SPI/Ethernet interface without cables for various Pi's is the main pointl, the Pi-zero would be the better candidate, imho, as each cpu is much slower and so are communication requirements. You might get some speedup adding Pi-zeros and thus at least get a "zero-Supercomputer"

[free_electron]

i have one question : what are you going to run on it ? not operating system , what APPLICATION are you going to run on it ?
What ? why ? and is it made for a cluster setup ?

[rob77]

i have one question : what are you going to run on it ? not operating system , what APPLICATION are you going to run on it ?
What ? why ? and is it made for a cluster setup ?

i assume it will be boinc client on every board. it's not a computing cluster.... it's a cluster of independent single board computers

[Poolkeeper]

Interesting project to learn from.

Found this about running Ethernet connection without magnetics:

http://www.ti.com/lit/an/snla088a/snla088a.pdf

[rob77]

of course it's not a real supercomputer... a crucial component of a real supercomputer is RDMA (remote DMA) as far as i know only 2 technologies provide native RDMA , it's Infiniband and 10Gb Ethernet , and neither of those is present. (of course you could write a software RDMA implementation on top of any connection.. but that's  not the real thing )

[Brumby]

It may not be a 'supercomputer' - but I think it's a super computer idea.

Engineering solutions that deliver is what enthuses me.  Who cares if performance doesn't stack up against an Intel flagship CPU - it was never meant to.  Getting the thing to work as intended is the magic.

[technix]

I think that if you can bypass PHY entirely you may be able to connect the RGMII interfaces together directly.

RGMII is generally not thought of being perfectly symmetrical in both the hardware and protocol sense - although I know some Micrel (et al) PHY chips can be connected back to back like that if you're building a repeater or media converter.  Usually that isn't the case for a generic connection - Normally RGMII is meant for a MAC interfacing to PHY... Not MAC to MAC.

Connecting RGMII back to back should be possible either with an external clock or one interface as master but the point is rather moot. Routing ethernet is going to be much easier. Because of the short distance on a board you don't need matched impedance traces so as long as they are routed as pairs away from noise everything is fine (100Mbit ethernet has frequency components up 100MHz so up to 40cm long traces don't behave as transmission lines yet).

RGMII carries a 1Gbps link at 125MHz DDR, so not that much more difficult to route than 100BASE-TX. And those are single-ended signals so only length matching is required for longer runs.

Twisted pair PHY have much higher latency than other PHY due to its line encoding. This latency is not significant to 100Mbit Ethernet but when you are coming to loading down a 1Gbps connection it can become a significant portion of the system latency. There are existing backplane Ethernet standards, but those are either based on fiber PHY or use GMII/XAUI (GMII is functionally identical to RGMII but requires almost 2x the pins, XAUI is a functional sucessor to [R]GMII but carries a 10/40/100Gbps link) direct attach.

[haybailes]

i have just finished watching the video and dont have time to read all the post
i dont know if anyone has said anything along these lines but sorry if someone has

instead of holes in the PCB and having the 90° header pins
you could you just normal straight female header pins going to a PCB with 90° header pins
then the raspberry pi attaches to that

pros: the PCB can be changed to handle other mini computer like C.H.I.P or orange pi, can be placed from the top down, more room on mother board,...
cons: more PCBs more and more components cost, may sway a lot in the slots,...

[xDR1TeK]

How is the Ethernet to SPI being done?
Emulation of TCP/IP to SPI?
Does the Linux kernel have a module to skip over the OSI physical layer which is connected to ethernet port?
Would the ARM controller be built with the whole digital TCP/IP circuitry or just the analog circuitry part to operate the port channel on the output?
This has been something eating at me for a long time, and finding books that cover this much is impossible.

[xDR1TeK]

Ok figured out this much, Application, Transport, Network layers are all in software. Then the Link layer is going to be SPI interface. The active components with the wires are the physical layer.
So the encapsulation is being routed in software to port over the SPI.
then the SPI to ethernet would be just a dummy converter.
Very interesting.

[iXod]

Dave mentioned a price of $10 for pi. Where are these to be found at this price?

[Fungus]

How is the Ethernet to SPI being done?

Those Ethernet ships do everything - complete IP stack on a chip. You just give them a MAC address and tell them to start.

You communicate with them over SPI just to check for incoming connections and receive/send the data.

[rob77]

How is the Ethernet to SPI being done?

Those Ethernet ships do everything - complete IP stack on a chip. You give them a MAC address

You just communicate with them over SPI to check for incoming connections and grab the data as it arrives.

actually there are 2 different kind of ethernet chips for micros... ethernet only and ethernet + IP stack

the mentioned ENC28J60 is a MAC + PHY with SPI interface... so it's just a plain ethernet "network card" connected via SPI and it does have a driver in the linux kernel.

the other kind of chips are with IP stack implemented but those are mainly targeted for small microcontrollers - so you don't have to implement the IP stack on your small micro. an example of such a chip is Wiznet W5100.

[Fungus]

actually there are 2 different kind of ethernet chips for micros... ethernet only and ethernet + IP stack

the mentioned ENC28J60 is a MAC + PHY with SPI interface... so it's just a plain ethernet "network card" connected via SPI and it does have a driver in the linux kernel.

the other kind of chips are with IP stack implemented but those are mainly targeted for small microcontrollers - so you don't have to implement the IP stack on your small micro. an example of such a chip is Wiznet W5100.

OK, I got them mixed up.

I've used the Wiznet W5100 but not the ENC28J60. I thought they were similar.

[eneuro]

How is the Ethernet to SPI being done?

Those Ethernet ships do everything - complete IP stack on a chip. You just give them a MAC address and tell them to start.

Is it gigabit ethernet? If not, than forget about any supercomputing and we should... SMELL BULLSHIT  instead of supercomputing
[youtube][/youtube]

Update: Yep it is    Orange Pi One – 10/100M Ethernet