EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2

[m98]

As for airflow optimization for the Raspberry Pi, just remove the FPC connector at the front, as it causes the flow to instantly separate, creating a dead zone right above the whole PCB. Remove it, and you'll get a significant improvement.

[technix]

Since you are spinning for riser boards already, you might as well put interfacing circuitry on it directly: SPI to Ethernet (or just an Ethernet bypass,) some EEPROM for configuration data (so you can interchange Pi's without having to reconfigure it) and most importantly, 12V to 5V DC-DC converter. The host board connector will have both 12V and 5V pins, but only one of the two is connected at a time. This allows you to shift boards around without minding the power rail voltage.

As of the SATA trays, you can put a USB to SATA chipset on the board so you can still access those drives from one of the Pi's.

Raspberry Pi 3 now supports network booting, so if you have one Pi running off the backup power (so it is guaranteed to be online when the rest of the nodes are on) you can boot everything else off that Pi. This, coupled to the configuration EEPROM, allows you to use only one SD card in the entire system, significantly cutting down the cost.

[technix]

I wonder if ethernet switch(es) will consume more power than Pis'..

May not be far off it, certainly some older ones. But those little domestic switches don't seem to generate much heat. 
The simplest solution to build, but not the neatest would a be 48 port switch and stack the pies up using PCB standoffs. With short patch and power cables to interconnect them.  Just be aware the fans if the switch has any (older models) may be loud.  It should be possible to remove the switch from its case and just use the PCs case fans to cool it and maybe power from the case's psu as well.

Any ideas on how to monitor the individual pies status, even if just a flashing 'Alive' LED on each unit?

You can use a network-based protocol to monitor it, no need for special hardware. The Pi also comes with a built-in hardware watchdog so there is no worry for it locking up (as the watchdog would just go ahead and reset it)

[technix]

Naked TPlink pcbs on the top part or maybe there is enough space for two 24 ports switches

Came here to write exactly this. There is plenty of space to wire normal Ethernet and not bother with SPI adapters.

Yeah, there is. SPI Ethernet is neat from a wiring point of view, but that's it's only advantage.
But then what's left for a motherboard, power and some LED's?
Guess you could MUX the serial port from each board or something.

Once you get the networking up there is no need for serial ports. The Pi can be kept alive using the built-in hardware watchdog. To monitor the status of all Pi's you can use SSH. SSH tunnels a console (effectively a serial port, albeit a pure software implementation) through networking infrastructure and as long as your monitoring machine still have the system resources (aka RAM and network capacity) you can run as many instances of SSH as you like.

[olafnew]

Ditch the out of the box ethernet switches.
Making an ethernet switch does not require any magnetics on inter-board interconnects. Magnetic coupling is only needed if there is an issue with ground offsets, which, in case of inter-board interconnects - is no issue at all. So you do not need any transformer or optoisolation for your ethernet when you're designing an interposer board(thats the term i believe is used).

I, for one - would design TWO boards that plug into the PCB. Something like this(see pic attached)
When one board connects the power and etc to the boards, and a small, another board is connecting the ethernet ports, with those nifty pcb mountable connectors you've mentioned earlier in the video, from the rpi to the motherboard via a different header.

Power - it is reasonable to ditch the compatibility of the connector from the plug in card module onto which the rPi is plugged, to the motherboard itself, and use something like 3x40 connector, and that way - it has more rigidity, and more possibilities. 12v->5v DC-DC conversion i would be doing on the card module, and NOT on the motherboard. It's much more servicable this way. The motherborad is a great place for the ethernet switching ICs, interconnects, and etc.

I'd really hate to see any cabling when designing such a heffty project.

[f4eru]

Hi

The "transistor on wires" is in effect used as a thermal sensor. ( I also suspected random number generator at first)

from http://tim.id.au/laptops/apple/powermac/powermac_g5.pdf
p119

[manu]

Why not getting rid of the gpio connector on the main board?

Design a PoE power supply on the riser board and use only the RJ45 connector and plug it directly on the main PCB with the PCB mount RJ45 jack you showed in the video.
In this config, you wouldn't need no cables at all  you need only a small rj45 cable between the rj45 on the rpi and the rj45 on the poe (the riser board+RPi should be light enough to be supported by the RJ45 connector) .

[ataradov]

PRi does not support PoE.

[rrinker]

 For monitoring, one extra Pi mounted up in the drive area with a display mounted where the drive tray comes out, as Dave mentioned. That standalone Pi would have all the various sensors connected and display status on the display.

[BobC]

A couple of ideas.  The first assumes every lily will be gilded...

Every REAL supercomputer must have high-throughput board-to-board communication!  This will permit applications requiring lots of communication between between boards, such as distributed real-time video analysis, or massive neural networks.

Remember, the Pi already uses a USB-Ethernet adapter on-board: The Broadcom chip lacks native Ethernet.

1. Dump the on-board Ethernet, use USB instead.  It'll get you higher bandwidth (480 MBps vs ~200 MBps).  This will let you do things like real-time distributed video processing (totally not needed by BOINC).

2. Place 4 PIs per riser.  2 per side with heat sinks out, oriented to have the SD card accessible.  With 12 risers, that's 48 PIs.

3. On the riser, have 4 USB-to-(R)GMII interface and a 5-port GigE chip (perhaps the NXP SJA1105ELY).  Route the GMII switch output to the riser connector (no magnetics needed).

4. On the motherboard have 12-ports of switchable GigE (probably not a single chip, but a tree of smaller switch chips).  CAREFULLY route the 12 riser GMII interfaces to these chips.

If you avoid using GigE PHYs (use GMII) then you pay the GigE PHY Power Penalty *only* for the single rear connector.

Now, you may not want to have to raid Fort Knox to get that much gold.  So let's leave the lilies alone and look at another path: Do everything with USB instead (taking a bandwidth and latency hit).

But if BOINC is your only application, then an all-USB solution should suffice, but it will need a dedicated PI to manage the network:

1. Still 4 PIs per riser.  USB port in slave mode.  (And/or use usbnet or usbip driver.)

2. 1 5-port USB hub chip per riser.  (Again, no magnetics needed.)

3. Mainboard has a 13-port USB3 hub (or a tree of smaller hubs).

4. Connect mainboard hub to Master PI.

This all-USB solution should be both easier to route (fewer riser connector pins) and lower cost than using Ethernet.

And, heck, with USB3 on the riser, you *could* power the PIs through it (though a separate USB3 power manager chip would be needed on each riser).

To get more bandwidth, instead of using a RasPi, the mainboard could use an SBC with multiple USB3 ports, which will simplify the mainboard USB hub.

[Zad]

Beware of being over-ambitious, it is the graveyard of many projects... Remember, KISS. (No, not being over-affectionate to Dave)

Sure it can be made efficient, with lots of compute nodes, good airflow, good communications and looking sexy, but I suspect getting all these would hammer the "on time" and "on budget" part of the engineers' trinity. Here are some thoughts to throw against the wall and see what sticks:

A mahoosive monolithic motherboard is going to be expensive and a pain to re-spin, even if only double sided. It would need to be thicker than usual to be mechanically rigid enough to support risers and all the modules. How about mounting some form of substrate (e.g. an aluminium plate) on the stand-offs, and have (say) four or 8 smaller PCBs mount on there. This would allow easier development and hardware debugging than messing around with a huge motherboard. It would allow for much freer placement of your own PCB mount screws etc. Also very useful for hiding messy hardware such as Ethernet patch cables. Is there any access via the other side of the case?

That power supply will be quite crusty now after so many years of hard work, I'd like to see inside it. Teardown! Teardown!

How are you going to support the riser cards on the end farthest from the motherboard? The risers will be quite heavy, especially if populated with Ethernet patches, heatsinks etc. This is one problem when mounting horizontally. PCs solve it by bolting it to the rear of the case. Mounting the cards vertically promotes convection cooling, in a commercial unit, air would be drawn in from the bottom of the case and vented at the top/rear.

The information about the new RPi 3 Compute Module is out. Basically, it is very very similar to the old one, just faster with more memory: https://www.raspberrypi.org/documentation/hardware/computemodule/RPI-CM-DATASHEET-V1_0.pdf   "Raspberry Pi guarantee availability of CM1, CM3 and CM3 Lite until at least January 2023" Which is reassuring.

Compute Modules can be populated more densely, incorporate their own flash, can boot over USB, don't have the horrendous power overhead of the RPi's standard Ethernet chip, and already fit nice standard 200 pin memory module sockets. They do, however, need 3.3V and 1.8V, and the comms problem is still there, although it supports USB OTG, meaning each node can be a device, and could be controlled by a master Pi, which is known to be capable of boot programming CMs.

Which ever node type you go with, I would keep it as simple as possible, at least in the first instance, just to get a prototype up and running and capable of being hacked around with to see what it possible. Sexy can wait until it is working. For example, mount 8 RPi3s flat on a daughterboard / riser PCB on stand-offs, use a short patch cables to connect power and ethernet to the riser. The riser just having 1 buck reg per Pi node, and purely passive traces to bring the Ethernet to the motherboard via 0.1in headers, PCI or whatever. For development purposes, a simple breakout board could be made with PC Molex power connector and Ethernet sockets. I wouldn't mess around integrating a switch, in the final design, the motherboard can break out standard (non-magnetics) sockets to the rear of the unit, through the rigid aluminium substrate, via standard patch cables to a network switch mounted at the top of the unit.

Final thought (for now). If you are going to see the insides of the sexy compute monster, then the Apple panel will have to go, presumably being replaced by a laser cut Perspex (acrylic) panel with an Apple and a RPi logo. This will mean cutting a hole in the old panel I'm afraid!

I'm going to have to draw some ideas for myself, to make sense of all the possibilities!

[Stonent]

I wanted to add something from a software standpoint.  Set up the Pi's to use a root file system on a network share.
I think there's a project out there to make a PXE client for the Pi. You'd just a have a tiny microSD card that had the bootloader and then it would just load some kind of ramdisk image with a pretty much read only FS.  It would just use scratch space on the ramdisk to function.  That is I guess if you are just using BOINC.

That way you could cut the power to the cluster and never have to worry about a corrupt system.

https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net_tutorial.md

For the hardware, I would almost say design the Pi's in a stacked configuration of say 7. (Like a skyscraper.) Then also integrate the board from the network switch into the stack.  The 7 Pi's would then plug into 7 of the 8 network ports and the 8th port would be the uplink to the main router.  The Pi's would connect to the switch with standard 3inch network patch cables.

[Stonent]

Final thought (for now). If you are going to see the insides of the sexy compute monster, then the Apple panel will have to go, presumably being replaced by a laser cut Perspex (acrylic) panel with an Apple and a RPi logo. This will mean cutting a hole in the old panel I'm afraid!

Or just buy a damaged side panel and route it out. Oh and if you could make it in any way resemble ORAC, that would just bring it to the next level.

[Stonent]

This guy did 15 in a toolbox. I thought the scrolling LED light was neat.  What would really be cool I think would be to somehow adjust its speed based on current draw from the Pi's. More current draw means they are working harder and then you'd have the LEDs move faster.

[manu]

PRi does not support PoE.

That's why I proposed to design a poe supply on the riser board, like the one attached below, for example.

Then you use the second rj45 to have both power and network from the main board.
As a plus, you probably could use 12V on the poe injector which gives plenty of current from the G5 supply.

The riser board would be : a gpio connector, 2 rj45, a 12V to 5V buck converter. The main board would be a bunch of pcb mounted male rj45's and a beefy 12V trace to the different riser boards.

[EEVblog]

The information about the new RPi 3 Compute Module is out. Basically, it is very very similar to the old one, just faster with more memory: https://www.raspberrypi.org/documentation/hardware/computemodule/RPI-CM-DATASHEET-V1_0.pdf   "Raspberry Pi guarantee availability of CM1, CM3 and CM3 Lite until at least January 2023" Which is reassuring.

Price?

[bigdog989]

Two quick thoughts:

1) can you power a rpi straight from the 3v3 gpio pin?

2) you could place one of those RJ45 connectors and a usb port on the mother board to power the pi! this way you just push the board onto the mother board. that side of the board is also shorter allowing more to fit! it does remove the compatibility with different pi types but would save a lot of money on riser boards and other connectors....

[PODF]

... SPI Ethernet is neat from a wiring point of view, but that's it's only advantage...

As far as i understood is the SPI-Ethernet thingy only useing the IO-Ports of the Raspberry. So you should consider deactivating the USB-/Ethernet-Chip. Because you dont need USB and Ehternet with the SPI-Network solution. That would give an extra powersaving advantage!

The USB-Chip does some interrupts (about 8'000 times a second) and really gets hot. Even if doing "nothing". I used a RPi for Video-Recording with a Raspi-Camera-Module. In Recording-Mode my Rpi B+ consumed average 384mA. With the USB-Chip deactivated it only consumed 269mA. Thats cool when you run this thing out of batteries.

You can Power-Off the Chip with this linux command:

Code: [Select]

echo 0x00 > /sys/devices/platform/bcm2708_usb/buspower

maybe you also want to turn off the HDMI output on the RPi for further powersaving.
My RPi B+ went down to 251mA in videorecording mode (recording-indicator-led off, usb-chip off, hdmi off)

Code: [Select]

/opt/vc/bin/tvservice -off

For easy servicing you could implement something that the power-saving script will only run when the RPi is attached to your mainboard and if you plug it out and do some tests with a particular Raspberry it would detect that it is not connected to the mainboard an so it will not poweroff usb/ethernet/hdmi so you can debug the thing (with keyboard, ethernet and monitor).

[CJay]

As for airflow optimization for the Raspberry Pi, just remove the FPC connector at the front, as it causes the flow to instantly separate, creating a dead zone right above the whole PCB. Remove it, and you'll get a significant improvement.

What happens if you angle the board?

[nctnico]

I wouldn't use the SPI ethernet solution, because except from a nicer wiring it only has drawbacks:
- It is more expensive (you need to add another ethernet chip instead of a cheap RJ45 connector/cable)
- It draws more power (0.5W for ENC28J60, that's a 14% increase in power consumption).
- It is slower (limited by SPI instead of USB)

But wiring 32 or 64 boards using RJ45 cables will not be easy unless you use many cable ties making access to a single module difficult.

The wiring can be made easier by creating custom cables which only have 2 pairs instead of 4. Lose the PVC cable sleeving and you are left with a bunch of twisted pair wires. If you combine a bunch of ethernet connections into a braided sleeve then the cabling will look neat and use little space. It just takes measuring and a piece of wood with nails to layout the wiring looms.

[tobia]

I just wanted to mention that there are several kinds of 90° header connectors especially designed to mount RPies vertically on a mainboard.

The Orange Pi has a 180° rotated header, but you just need to mount it upside down and the IO pins align perfectly.

Excuse the crudity of the ASCII art model:

Code: [Select]


+------+---+            +------+---+     +------+---+
|      |1 2|  2 1       |      |40 |     |      |2 1|  2 1
|      |3 4|  4 3       |      |. .|     |      |4 3|  4 3
|      |. .|  . .       |Orange|. .|     |Orange|. .|  . .
| RPi2 |. .|  . .       |  PI  |. .|     |  PI  |. .|  . .
| top  |. .|  . .       | top  |4 3|     |bottom|. .|  . .
|      | 40|  40        |      |2 1|     |      |40 |  40
|      +---+            +------+---+     +------+---+
|          |  main-      (no good)        (flipped)    main-
+----------+  board                                    board
              header                                   header



Code: [Select]


      |                                             |
      |   right-                            right-  |Orange
  RPi2|   angle                             angle   |PI
      |   connector                         conn.   |
      |-1 ---+                               +--- 2-|
      |-2 -+ |                               | +- 1-|
           2 1     <- same mb layout! ->     2 1
==================                       ==================
    mainboard                                mainboard



[Romain]

The RaspberryPI does ethernet (a star topology) through a USB to ethernet adapter.
1. If using the GPIO, a neat solution would be to recreate this topology with one of the available buses. Problem is : I2C and SPI are not star, and would need a master to handle every board trying to access the Internet. Could be one of the Pi handling the cluster, but bandwidth will be shared.  I think it is overkilled to add an extra ethernet switch for each board.

2. If male RJ45 connectors exist, so does USB male connector. The 5V could be supplied through it, and make the GPIO connector redundant. Then the motherboard would be a simple interconnect board with just connectors and caps.

[Zad]

The information about the new RPi 3 Compute Module is out. Basically, it is very very similar to the old one, just faster with more memory: https://www.raspberrypi.org/documentation/hardware/computemodule/RPI-CM-DATASHEET-V1_0.pdf   "Raspberry Pi guarantee availability of CM1, CM3 and CM3 Lite until at least January 2023" Which is reassuring.

Price?

No idea, as a potential bulk purchaser, you might be able to get a quote off them more easily than I could. The current CM sells for £20 at RS Components and Farnell, I wouldn't be surprised if the CM3 came in at the same price, but I only base that on what RPi have done in the past and not hard information.

[technix]

The information about the new RPi 3 Compute Module is out. Basically, it is very very similar to the old one, just faster with more memory: https://www.raspberrypi.org/documentation/hardware/computemodule/RPI-CM-DATASHEET-V1_0.pdf   "Raspberry Pi guarantee availability of CM1, CM3 and CM3 Lite until at least January 2023" Which is reassuring.

Price?

No idea, as a potential bulk purchaser, you might be able to get a quote off them more easily than I could. The current CM sells for £20 at RS Components and Farnell, I wouldn't be surprised if the CM3 came in at the same price, but I only base that on what RPi have done in the past and not hard information.

I do have a super high density cluster design based on CM that can achieve more than 300 Pi's per 1U rack mount unit, and have clear vertical air flow across the rack when packed to the brim.

[onlyrgu]

Watching your videos for years, I taught I will chip some ideas to this Project.

I support idea of using Ethernet instead of complex SPI IC + Switch IC + supporting components. It will cut down your BOM cost.
I also like the idea of universal plugin modules so you can use OrangePi/RPi/etc
Most Ethernet switches comes with the power connector (eg. 5.5mm) on the back close to the RJ45 female connectors. You also need a power traces on the motherboard for powering the Ethernet switch modules, normally they are 9V/12 V.
Those fans run on 12V. You can also have a dedicated controller for fans on your motherboards
I would suggest ditching the Bulky SMPS out of G5 case that will give you more room for more RPi. Plug in modules.
You can use an off the shelf power supply (or easy to design one) that can provide 5V for the Pi and 9/12V for the Ethernet switches. And you can accommodate this power supply module at the hard disk area.
Looking forward for this awesome project!

P.S.
OT: That SMD connector must have costed them 4 times the cost of a 2N3904, Maybe Apple have a fancy patent to monitor ultrasonic heat waves using a NPN transistor