EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1

[chris_leyson]

Hi Dave, interesting project. I've used the METZ AJP92A8813 pcb mount connector for Gige 1000BASE-T in the past and they work OK, Element 14 stock them. Also been using 0.5mm pitch flat flex cable as co-planar waveguide for Gige ethernet with no problems, I tried to measure the differential cross talk in 50mm and 200mm long flat flex cables at 125MHz but my home brewed test jig was limited to about -50dB cross talk. The connecting cable made very little difference <0.25dB so I guess cross talk in a 200mm long flat flex cable is perhaps better than -65dB or -70dB. Shouldn't be a problem with 10/100 ethernet and you might even be able to get away ribbon cable.

[frank_gamefreak]

Hello Community,
first of all my English is not the best, so please excuse that I don't read the complete thread. Maybe someone already mention my ideas but I would like to give my opinion to the product.
 
I like the idea of staking Pis together to a compact cluster. And I wish we could make a kickstarter out of it.

My first suggestion for the backplain is, could we make a „master bus“. I think of a pin that is zero if no Pi is alive and the first one that it to one. My idea is that the first Pi knows it is the first and starts a DHCP server. Also the other Pis can scan this pin and recognize if the „master Pi“ go offline and the next one have to start a dhcp server.

My second suggestion would be a switch to change the pin-out between raspberry and orange Pi. I think someone like you have an idea how to make this possible. 

And my last one for now is to make the connectors different to your idea mentioned in the video. I think it is possible to place the connectors into the free space. Ok, it have do be a double sided board for this but I think it should be possible. I can't make a good picture of this. But think about a strait connector with one side of pins of each side of the board and the female part of the connector in the space for the Pi.

Please think about it and tell me your opinion about it.
Thank you.
 

[rs20]

My first suggestion for the backplain is, could we make a „master bus“. I think of a pin that is zero if no Pi is alive and the first one that it to one. My idea is that the first Pi knows it is the first and starts a DHCP server. Also the other Pis can scan this pin and recognize if the „master Pi“ go offline and the next one have to start a dhcp server.

Nice idea. Earlier I suggested using a series of pins to allow the Orange Pi to detect which slot it was in, and choose a static IP accordingly. This has the advantage that all of them would have a well-defined IP address.

My second suggestion would be a switch to change the pin-out between raspberry and orange Pi. I think someone like you have an idea how to make this possible. 

Earlier suggestions included using little adapters with a straight female pin header + a card edge connector; one advantage of this approach is that different adapters could support different boards.

And my last one for now is to make the connectors different to your idea mentioned in the video. I think it is possible to place the connectors into the free space. Ok, it have do be a double sided board for this but I think it should be possible. I can't make a good picture of this. But think about a strait connector with one side of pins of each side of the board and the female part of the connector in the space for the Pi.

I couldn't follow what you were trying to express here?

[frank_gamefreak]

My first suggestion for the backplain is, could we make a „master bus“. I think of a pin that is zero if no Pi is alive and the first one that it to one. My idea is that the first Pi knows it is the first and starts a DHCP server. Also the other Pis can scan this pin and recognize if the „master Pi“ go offline and the next one have to start a dhcp server.

Nice idea. Earlier I suggested using a series of pins to allow the Orange Pi to detect which slot it was in, and choose a static IP accordingly. This has the advantage that all of them would have a well-defined IP address.

I see a problem here with multiple backplains in a stack. The advantage of my idea is that you only connect the master bus to the next board and all pis know ther is a dhcp.

And my last one for now is to make the connectors different to your idea mentioned in the video. I think it is possible to place the connectors into the free space. Ok, it have do be a double sided board for this but I think it should be possible. I can't make a good picture of this. But think about a strait connector with one side of pins of each side of the board and the female part of the connector in the space for the Pi.

I couldn't follow what you were trying to express here?

I hope this "CAD" helps you to understand.

[rob77]

for those who think this project is a waste of time... no it's not... it's a fun project which will lead to a usable product at the end.
if you go for the highly optimized off-the-shelf solution, then where is the fun and learning ? 
i bet my bottom dollar this very thread already helped to share a lot of ideas and knowledge, despite the project is in it's infancy yet

[doobedoobedo]

Performance was specifically stated as NOT the primary aim. So my question is why make performance the primary focus of your criticism? Now that's an epic facepalm moment for you.

then dont call it supercomputer. Dave mentions it not being as fast as latest modern intel cpu while in fact the it wont even beat 2 year old budget product.

Think of it as a model of a supercomputer. It uses the same cluster architecture of many actual supercomputers, but at a fraction of the cost and power consumption.

I hope Dave finds time to build it to scale and paint it .

[Fungus]

Think of it as a model of a supercomputer. It uses the same cluster architecture of many actual supercomputers, but at a fraction of the cost and power consumption.

It might actually be better in terms of bang per buck.

Yes, an i7 is probably faster than this but an i7 needs external RAM, a fancy motherboard, etc. It all costs money. For the price of an i7 plus support hardware you could buy a lot of Pis.

It will be interesting to see the final numbers if/when this thing gets built.

[cv007]

My 2 cents-

You already have the orange pi's,  I would think step 1 would be to just mount them together via standoffs (or not at all), hook up power, connect to a switch via ethernet, get them booting, working together, etc.- then see what you have. If all works as planned, go to step 2. Step 1 is 10 times easier than step 2, and you most likely have everything needed on hand to do it. (You could make a video about it- would give viewers an idea of what the end result is all about, and if step 2 never happens for whatever reason at least you will have had something to show).

You will find out what is important, what is not, you will get actual power measurements, heat measurements, what works, what doesn't. You may discover many things that could be useful to know. Let the set of pi's run for a week and see if they are what you expect- maybe they are flaky and full of little problems, or maybe they are great- you will at least get a good idea before putting in a lot of time on a board design.

Designing a board before you know you have a working 'circuit' seems like a 'trap for young players' 

Just a thought. (There is still 2 cents left, so must have been free advice).

[Fungus]

You already have the orange pi's,  I would think step 1 would be to just mount them together via standoffs (or not at all), hook up power, connect to a switch via ethernet, get them booting, working together, etc.

I'm pretty sure they'll boot up and run BOINC - Dave already did this in his review of the OrangePi IIRC.

It might be a good idea to get a couple of ENC28J60 modules off eBay and try that part of it though, before making a PCB. 

[nctnico]

How about this: create a full size ISA/PCI style board which has an onboard ethernet switch and two external network ports which can be daisy chained. The pies (mounted flat) can be connected to the onboard switch using short RJ45 cables soldered into the board. Ditto for the power. I think each board can hold 8 to 10 Orange Pi zeroes. This would address the cabling problems Dave is trying to avoid. Also the board would fit in a standard PC casing (which provides features like forced air cooling and a power supply) using a standard PCI or ISA backplane (cheap!).

[Frant]

Before the project becomes too complicated and overengineered, it may be a good idea to take a breath and think about the actual goal. For example, I would probably choose to make the simplest possible prototype (proof of concept), just to play with it and see what can be expected from such a system in terms of its computing power and practical usability. Significant effort in order to design a professional grade hardware only makes sense if the prototype shows that the expected functionality and/or performance can be achieved. A decent off-the-shelf power supply and a 16-port Ethernet switch will be sufficient for the start. The software aspect of the project can be much more challenging than it seems at this point.

[Fungus]

Before the project becomes too complicated and overengineered

The engineering is the project.

Significant effort in order to design a professional grade hardware only makes sense if the prototype shows that the expected functionality and/or performance can be achieved.

The performance really doesn't matter. It's obvious it's a complete waste of time if "ultimate performance" is your goal. A MicroATX PC with an i7 in it will be much faster/easier to build

The "design effort" is what makes it worthwhile to Dave.

(I'm putting words into Dave's mouth as I understand this. Correct me if I'm wrong...)

[nctnico]

I understand the same thing. According to the first few minutes of the video the goal is to come up with a solution which doesn't need a lot of wiring and external ethernet switches as shown in the systems which already exist. IMHO going the SPI to ethernet route is not the best one because it takes a lot of effort to build, create drivers for and it will still be slow.

[Fungus]

IMHO going the SPI to ethernet route is not the best one because it takes a lot of effort to build, create drivers for and it will still be slow.

The video clearly says:
a) The OrangePi kernel already has a driver for those particular SPI-to-Ethernet chips - just edit a text file and enable it
b) Network speed isn't important.

(start at 11:40 in the video if you missed it)

[cv007]

I'm pretty sure they'll boot up and run BOINC - Dave already did this in his review of the OrangePi IIRC.

Booting up and running is one thing, running continuously is possibly another. They seem to require heatsinks (according to one of his reviews), so how big is big enough?  Will the required size heatsinks affect the spacing between boards (physical spacing because of heatsink size)?  I'm sure a lot could be learned by getting a flock of pi's setup and working as intended before finalizing any board design, which is my only point.

[Fungus]

I'm pretty sure they'll boot up and run BOINC - Dave already did this in his review of the OrangePi IIRC.

Booting up and running is one thing, running continuously is possibly another. They seem to require heatsinks (according to one of his reviews), so how big is big enough? 

I doubt it will be very much if they sell them without heatsinks. If it was a big problem they'd have them welded onto the chips.

But that's just speculation. We need engineering data and I'm sure there's a video on this topic in the pipeline. 

[nctnico]

They seem to require heatsinks (according to one of his reviews), so how big is big enough? 

I doubt it will be very much if they sell them without heatsinks. If it was a big problem they'd have them welded onto the chips.

Don't be so sure about it. I'm using a 'SoC on a module' for a commercial project and it needs a huge friggin heatsink which is sold seperately to keep the module within specs and it is mounted in a casing with plenty of natural convection.

[CM800]

They seem to require heatsinks (according to one of his reviews), so how big is big enough? 

I doubt it will be very much if they sell them without heatsinks. If it was a big problem they'd have them welded onto the chips.

Don't be so sure about it. I'm using a 'SoC on a module' for a commercial project and it needs a huge friggin heatsink which is sold seperately to keep the module within specs and it is mounted in a casing with plenty of natural convection.

Exactly right. These modules are designed to be as flexible as possible for as many customers as possible. Some customers can't use fans due to noise requirements, others can't have a heatsink, it takes up too much space or there won't be airflow inside the case, it'll have to be thermally coupled to said case. Others may be running in a very hot environment where the only way they could possibly cool it (along with other components) would be water cooling (say the unit is out in the desert in the sun, some electronics will be cooled via a large heat exchange unit.

[jolshefsky]

I think this has the same-ish problem as the flat-packed Pies: lots of wasted board space under them.

My first thought (echoed early on by several others) was to make a simple board that converted the dual-row pin-header to a card-edge connector. These adapter boards would be super simple and small: barely a dual-row header and one side with card-edge fingers. I very much like the idea of going with a very common connector (e.g. PCI, even though it's more pins than necessary.) For that matter, it wouldn't be much more effort to design the board with a second set of holes for another female header to handle the other rotation of the board (e.g. Raspberry vs. Orange).

The question is, should you make the adapter board include things like LEDs, pass-through pins, a local LDO regulator, or the SPI-Ethernet adapter.

[mariush]

Another idea just came to me 

use four bars to space the pi's evenly.  Fish tank , mineral oil , put the pi's all in the oil hanging down by the network and power and io cables all  going up onto a top board . add a couple of fans to circulate the mineral oil around. done.

Since they run on low voltages and relatively low currents, you probably don't have to insulate each cpu (with regular computers some hard to pour insulating material around the socket because they got shots or arcs between contacts near the cpu sockets.

[Frant]

IMHO going the SPI to ethernet route is not the best one because it takes a lot of effort to build, create drivers for and it will still be slow.

Although a software driver for the SPI/Ethernet chip exists (Dave mentioned it in the video), I would rather try to stick with Ethernet. The idea to use right-angle pcb-mount Ethernet plugs was a good one and it remains unclear to me why it was dismissed.

[TheRevva]

All that work to achieve the ethernet I/O?
I'd be SORELY tempted to outlay a HUGE extra US$2.01 per board and use the Orange Pi Lite which has onboard WiFi!
Whether I would actually USE the supplied WiFi antennae or not is another question.
I'm willing to bet they could all 'cross-communicate' without ANY antenna being connected if they were all co-resident in a single enclosure with just a 50ohm resistor soldered across the u.fl connector?
The FIRST 'PI' within any such 'array' / 'cluster' could then serve as a wired-to-wireless protocol converter as well as a basic local WiFi 'Access Point'
It would probably work out cheaper overall and significantly reduce the internal wiring complexity.

[MisterDiodes]

A couple "Head's Up" suggestions for Dave:

1.  Check out Samtec for a right angle female header socket connector - part of their SSQ line I believe.  The female header clips on to your Pi header, then the square leads come out and turn right angle, and you can get these in a variety of lengths - say .3" or .5" long.  No need to remove existing 40 pin header on the Pi, and you can use any variety of vertical-entry sockets on the backplane - and that makes it a breeze to layout without slots in the way.  For a couple bucks you turn your Pi into a module than can stand on edge and plug into your backplane, no slots required. 

2.  Watch out connecting the Ethernet PHY ports together without magnetics!  Ask me how I know.  It works sometimes, but not always - just depends on what chips you'll use.  What happens is the current drivers get confused if they are trying to negotiate auto-MDI crossover when first making the connection - and if you're using an un-managed port switch chip that can be an issue.  Those PHY port drivers are designed from ground zero to see the mags attached.  You'll also see weird noise pickup effects on a crowded board if you're not careful.

I suggest you lay in the pads for the 1:1 magnetics ALONG WITH your resistor pads on your proto board, and then you can easily experiment to see what does and doesn't work.  You definitely want to have your magnetics in place for the first few tests, and then you can always remove them to start playing with direct-connect resistors.

Remember those current driver resistors will have to be changed for the direct-connect situation.

[technix]

A couple "Head's Up" suggestions for Dave:

1.  Check out Samtec for a right angle female header socket connector - part of their SSQ line I believe.  The female header clips on to your Pi header, then the square leads come out and turn right angle, and you can get these in a variety of lengths - say .3" or .5" long.  No need to remove existing 40 pin header on the Pi, and you can use any variety of vertical-entry sockets on the backplane - and that makes it a breeze to layout without slots in the way.  For a couple bucks you turn your Pi into a module than can stand on edge and plug into your backplane, no slots required. 

2.  Watch out connecting the Ethernet PHY ports together without magnetics!  Ask me how I know.  It works sometimes, but not always - just depends on what chips you'll use.  What happens is the current drivers get confused if they are trying to negotiate auto-MIDI crossover when first making the connection - and if you're using an un-managed port switch chip that can be an issue.  Those PHY port drivers are designed from ground zero to see the mags attached.  You'll also see weird noise pickup effects on a crowded board if you're not careful.

I suggest you lay in the pads for the 1:1 magnetics ALONG WITH your resistor pads on your proto board, and then you can easily experiment to see what does and doesn't work.  You definitely want to have your magnetics in place for the first few tests, and then you can always remove them to start playing with direct-connect resistors.

Remember those current driver resistors will have to be changed for the direct-connect situation.

I think that if you can bypass PHY entirely you may be able to connect the RGMII interfaces together directly.

[optoisolated]

2.  Watch out connecting the Ethernet PHY ports together without magnetics!  Ask me how I know.  It works sometimes, but not always - just depends on what chips you'll use.

One of the reasons I liked the KSZ8895MQX chip is that you could configure the important registers with pull-ups. The MDI/MDIX Negotiation just happens to be switchable by pulling up pin 1. Winning! It's also programmable, but for the project Dave had in mind, that wouldn't be worth the effort. Not bad for 8 bucks.