Author Topic: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2  (Read 30750 times)

0 Members and 1 Guest are viewing this topic.

Online EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« on: November 22, 2016, 07:17:22 am »
Part 2 of building the Raspberry Pi computer cluster.
Dave strips down an old Apple G5 PowerMac to use as the enclosure.

 

Offline djlorenz

  • Contributor
  • Posts: 43
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #1 on: November 22, 2016, 08:01:19 am »
Being in Australia for work helps being in the first watcher of the video :D
The idea of a "apple pi" is great!

The 2 devices board is good, wiring that one is already interesting, cabling also ethernet will be a great engineering experience but imho cabling and tplink switches are good and cheap enough for the ethernet wiring. Naked TPlink pcbs on the top part or maybe there is enough space for two 24 ports switches (the switches are probably taking more space than the PIs?).

I will re-use the front switch for power up of the system, looking forward to see updates on this! good luck!
 

Offline Towger

  • Super Contributor
  • ***
  • Posts: 1645
  • Country: ie
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #2 on: November 22, 2016, 08:02:37 am »
A part 2 of a project...  Yipee.  But will it be finished! 

Says he who has 5:1 ratio of half finished to finished projects.
 

Offline Barny

  • Frequent Contributor
  • **
  • Posts: 311
  • Country: at
  • I'm from Austria, not Australia ;)
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #3 on: November 22, 2016, 08:06:58 am »
When the lid is closed nobody will see whats inside.
Specially, when it stands somewhere under a desk.

Because of that, I would focuse on function instad of the look.
Build the way you think this project work the best and most flexible.

PS.: Mmmmmmm, Apple - Bannana - Raspberry pie *Insert Homer voice here*
« Last Edit: November 22, 2016, 08:09:22 am by Barny »
 

Online ataradov

  • Super Contributor
  • ***
  • Posts: 11228
  • Country: us
    • Personal site
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #4 on: November 22, 2016, 08:07:05 am »
Naked TPlink pcbs on the top part or maybe there is enough space for two 24 ports switches
Came here to write exactly this. There is plenty of space to wire normal Ethernet and not bother with SPI adapters.
Alex
 

Offline rs20

  • Super Contributor
  • ***
  • Posts: 2317
  • Country: au
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #5 on: November 22, 2016, 09:15:09 am »
I'd still like to see the SPI ethernet approach, it's vastly more interesting than ethernet, since everyone else does ethernet solutions of varying inelegance. As you've mentioned before, Dave, the bandwidth limitations of SPI are not really relevant; since most cluster computing is not bandwidth-limited.

I'd consider a solution that mounted ethernet switch ICs directly onto the motherboard as a fair compromise though. I don't think you'd need magnetics on the board if they're already built-in to the RPi/OPi's. There's no rule saying that you need more than 1 set of magnetics between two devices; and indeed as discussed in the last video you don't even need that 1, can use capacitors instead -- the magnetics provide protection against high voltage damage from 100m long cables running in harsh environments, not really relevant to a connection within a metal case.

I'd suggest designing the riser boards so that the OPi/RPi connector is always closest to the motherboard. In other words, have both riser boards be the "short"/"squat" variety. It ought to be trivial to deal with the routing challenges that result*; I doubt you'll be actually using very many of those pins? The majority can be no-connects or power pins.

* To clarify, I meant the routing challenges within the riser boards. You'd still have a totally consistent pinout on the motherboard of course (although if I were you I'd design the motherboard card edge connector pinouts to be easiest to route as far as the motherboard is concerned, and then deal with whatever challenges emerge in the riser boards later.)
« Last Edit: November 22, 2016, 09:38:00 am by rs20 »
 

Offline Geerant101

  • Contributor
  • Posts: 14
  • Country: au
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #6 on: November 22, 2016, 10:26:29 am »
I like the idea of having 2 Raspberry Pi's on a single riser board, makes better use of the space. Perhaps even using a PCI express edge connector back to the main board supported by PCB card guides. The PCB rails can then all be joined together for additional support. A double row header may hold the pcb in place better without needing a PCB guide but I think it would still need additional support, especially orienting the boards horizontally. Lining up pins on a double row header would be a difficult using a card guide - it only takes one slightly bent pin to ruin your day ;)
 

Offline bktemp

  • Super Contributor
  • ***
  • Posts: 1616
  • Country: de
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #7 on: November 22, 2016, 10:29:36 am »
I wouldn't use the SPI ethernet solution, because except from a nicer wiring it only has drawbacks:
- It is more expensive (you need to add another ethernet chip instead of a cheap RJ45 connector/cable)
- It draws more power (0.5W for ENC28J60, that's a 14% increase in power consumption).
- It is slower (limited by SPI instead of USB)

But wiring 32 or 64 boards using RJ45 cables will not be easy unless you use many cable ties making access to a single module difficult.

If you use riser boards, you could add a stepdown with enable to each board and add a large microcontroller on the mainboard as a system status controller. Maybe you could also add UART or some other communication bus, so it can talk to each module an cycle power if it does not respond.

I like the idea of using the pcb mount RJ45 connectors. It would simplifiy the wiring, especially when you also plug the ethernet switch into the mainboard.
 
The following users thanked this post: flextard

Offline Krokkodillo

  • Newbie
  • Posts: 1
EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #8 on: November 22, 2016, 11:51:14 am »
Hi guys, I'm new to electronics but I wanted to share my thoughts on how to set the modules. In terms of what to use spi or rj45 i would prefer simpler implementation, for rj45 i would definitely go for plug in option no cables, for spi version i would say that the modules should be connected horizontal way so it won't restrict the airflow from the front of the panel, i would also suggest to have smaller adapter connector to the mainboard.


Sent from my iPhone using Tapatalk
 

Online EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #9 on: November 22, 2016, 12:12:13 pm »
Naked TPlink pcbs on the top part or maybe there is enough space for two 24 ports switches
Came here to write exactly this. There is plenty of space to wire normal Ethernet and not bother with SPI adapters.

Yeah, there is. SPI Ethernet is neat from a wiring point of view, but that's it's only advantage.
But then what's left for a motherboard, power and some LED's?
Guess you could MUX the serial port from each board or something.
 

Offline Fungus

  • Super Contributor
  • ***
  • Posts: 16561
  • Country: 00
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #10 on: November 22, 2016, 12:26:45 pm »
Hi guys, I'm new to electronics but I wanted to share my thoughts on how to set the modules. In terms of what to use spi or rj45 i would prefer simpler implementation

If you want "simpler implementation" then just don't do it at all. A single i7 chip will probably be faster than this whole rig.

Loads of cables will cover up all the pretty electronics and obstruct airflow.
 

Offline Barny

  • Frequent Contributor
  • **
  • Posts: 311
  • Country: at
  • I'm from Austria, not Australia ;)
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #11 on: November 22, 2016, 12:31:52 pm »
Yes, a I7 is faster.
But the Rasperrys are more efficient then the I7.

And the question isn't why, the question is why not.
 

Offline UpLateGeek

  • Contributor
  • Posts: 16
  • Country: au
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #12 on: November 22, 2016, 01:34:49 pm »
I would avoid socketing the switch to the motherboard. Not only does it lock you into an exact brand and model of switch (so a switch failure would mean a motherboard replacement if you can no longer get the exact same switch, and with cheapie switches it's not a question of if the switch will fail, but when), but it will take up room on the motherboard that would better be used for the RPi's.

The neatest solution would be the SPI Ethernet, but this would be very expensive, as it would require duplicating the SPI Ethernet circuit for each of the 32-64 RPi's. Not to mention how slow it would be compared to the 100Mbit Ethernet, let alone to the 1Gbit on the OPi 2.

However, it looks like you'll have plenty of room under the motherboard, so I would route out a small slot next to each board and run the Ethernet cable through the motherboard, up behind it and over to the Ethernet switches above. You'll need jam in a whole bunch of those little switches up top, but there should be enough room if you rip out the optical drive and everything else. For 64 RPi's you'll need 10 of the 8 port switches - 7 RPi's + 1 uplink per-switch. 16, 24, or even 48 port switches would be better, but I guess it depends what you can get for a reasonable price, and the total number of RPi's you want. If you're running OPi 2's or other gigabit boards, you probably don't want to run all the switches to an aggregation switch, with just the one uplink to feed the lot, since that would effectively give you a maximum of 64:1 contention ratio. Even if you're using 100Mbit boards like the RPi 2/3 or the OPi 1, you'd still have 6.4:1 contention ratio. With 8 port switches, you'd have 10 uplinks, so I'd just bring all the uplinks to the back panel. I would cut out the connector panel, either to the right size to snap in those Keystone Cat6 Ethernet couplers, or mount sockets along the top of the motherboard to pass the Ethernet signals through to sockets on the back panel. You could even 3D print a custom connector panel to get the exact port layout you want.

But then again, I am a network engineer rather than an electronics engineer, so obviously I'm a bit biased when it comes to network connectivity!

As for power, here's the G5 "service manual". The power supply pinouts are on pages 179-180. The cheapest solution would be a Wun Hung Lo buck converter off ebay (massively over-rated to allow for their huge power rating fudge factor). Building one into the motherboard wouldn't cost much more though. The toughest part would be finding somewhere out of the way of all the RPi's to mount the heatsink for the mosfet and power diode.
 

Offline Towger

  • Super Contributor
  • ***
  • Posts: 1645
  • Country: ie
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #13 on: November 22, 2016, 02:24:09 pm »
Those bodge wires on the back of the motherboard must have been slipped past Steve Jobs.  You don't see that sort of crap on a PC motherboard. 

Don't trust those power connector pins are standard, the left connector looks standard, but it is a bloody Apple!

Copious use of the offical EEVBlog J Cloth and the watch magically changes mid video.
 

Offline CJay

  • Super Contributor
  • ***
  • Posts: 4136
  • Country: gb
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #14 on: November 22, 2016, 02:28:10 pm »
Lots of switches tend to use the same 'harmonica' port connectors at that price band so it's highly likely that a failed switch would render the whole board useless IMHO

I've commented on the video but I'd expand on that comment, I like the idea of a pluggable sub board to carry the Pi and mates with a 'standard' connector on the motherboard, one that perhaps mates USB, GPIO, Power, Ethernet to the motherboard.

That way the backplane/motherboard can be re-used many times as new versions of Pi/Orange/Banana etc. become available, you just need to respin the sub board to accomodate new GPIO, power and USB requirements.

Designing the backplane/motherboard to route Ethernet and USB signals could be a challenge but far from impossible, add in a little 'expansion room' with extra GPIO and alternative power rails (can you 'back feed' the Pi with 3.3V through the GPIO?) and it would be future proof.

 

Offline LightPathVertex

  • Newbie
  • Posts: 1
  • Country: de
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #15 on: November 22, 2016, 04:07:17 pm »
For the networking, I think the nicest solution would be to design a long PCB with 8 PCB-mount connectors that just directly plug into the RJ45s of the PIs and an ethernet switch IC.
Since you have no actual cable, I guess you could even get rid of the magnetics and just connect them straight into the IC - that way, you're just left with a strip-shaped PCB with 8 plugs with the right spacing, a IC like the KSZ8999 and one jack at the top.

For a 8x6x2 arrangement of PIs, you'd need 12 of those strips, and could then plug them all into a 16-port switch at the top of the case.
« Last Edit: November 22, 2016, 04:08:49 pm by LightPathVertex »
 
The following users thanked this post: richfiles

Online mariush

  • Super Contributor
  • ***
  • Posts: 4983
  • Country: ro
  • .
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #16 on: November 22, 2016, 04:49:52 pm »
Dave, @ 20:00

You CAN'T use both 5v @ 19A AND 3.3v @ 22 A at same time.  Look at the drawing on the label, like with ATX power supplies 3.3v and 5v rails can each output up to that much current but combined they're limited to some limit, like 100-150 watts.
Then, the total power on 3.3v+5v COMBINED with 12v rails can not exceed 340 watts.

There's two 12v rails with max 14A on each rail, but total for both is only 23A .. so 12v x 23A = 276 watts on 12v, which means there's 340w-276w =  64w on 3.3v+5v... which is obviously not right.  So my guess there's more power available for 5v + 3.3v, maybe 100-120w, but the transformer inside or something isn't suitably sized (or maybe it's an efficiency/heat dissipation thing) inside that limits the total to 340 watts.

Anyway, point is I wouldn't rely on more than 15A from that 5v.  I wouldn't use that power supply in the first place, because chances are it's probably around 70% efficient. You could just buy an industrial 5v 40-60A power supply from Digikey for $40-60 running at 75-80% efficiency and take up less space and be less noisy.

 

Offline kcozens

  • Contributor
  • Posts: 44
  • Country: ca
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #17 on: November 22, 2016, 04:59:34 pm »
I also like the look of the Apple case. It was amazing to see examples of how other people have re-purposed them. BTW, early on in the video you said 1.6GHz instead of 1.8GHz.

The riser idea is an interesting one. When you started talking using a riser and then about a DC/DC converter I had a thought. If the height is available (or if you have enough space on the riser board) you could add a connector that would accommodate a DC/DC converter, or build a DC/DC converter as part of the riser. You could add some jumpers on to the riser so that the two Pi boards connected on the riser can use either the 5V rail from the power supply or the 12V rail via the on-board DC/DC converter to power the Pi boards.
 

Offline lpickup

  • Regular Contributor
  • *
  • Posts: 98
  • Country: us
  • Uncle Bobby Dazzler
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #18 on: November 22, 2016, 05:48:56 pm »
If you are looking to maximize the number of Pi's, how about a staggered arrangement that may let you squeeze in a few more rows (assuming the predominantly horizontal arrangement that you described) such that the connectors of one row wind up being above the narrower part (or maybe the space between) of the row below.
 

Offline mxmarek

  • Contributor
  • Posts: 18
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #19 on: November 22, 2016, 06:05:59 pm »
I wonder if ethernet switch(es) will consume more power than Pis'..
 

Offline Lightages

  • Supporter
  • ****
  • Posts: 4313
  • Country: ca
  • Canadian po
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #20 on: November 22, 2016, 06:29:53 pm »
I am not too sure how to handle the inter-connectivity. I am sure of one thing though. I would use two motherboards instead of double capacity riser cards. I would put one motherboard in its usual place and mount the other just inside the cover with the cards facing in. This board would need to be on hinges or some other easy way to access it. This way you have less complexity in the adapter/riser boards and have a nice wind tunnel for cooling,  guess if my approach were to be used then the SPI approach for the networking would be much less difficult to manage.
 

Offline timgiles

  • Regular Contributor
  • *
  • Posts: 236
  • Country: se
  • Programmer, DB architect
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #21 on: November 22, 2016, 06:37:45 pm »
Dave,

Great video, glad you are moving forwards with the project. I thought that the vertical male eth connectors looked fun but I worry they will lock in place and you will need 8 fingers pressing on 8 tabs to release the switch once it is pushed on. Personally, I hope you go down the route of either adding a few ethernet bridge ICs on to the board or the SPI route. This being an EE blog, I would rather see electrical design than just plugging wires in to ports (!).... although I do temper my argument that the PCB design and smaller daughter boards will not be a trivial effort either way.

Hope you are going to do a video with some discussions on BOM, PCB design and maybe choice of parts. Seeing how a prof does these in Altium or one of the entry level CAD packages would be great.

Also, the power connector decision. Surely having 8 (was it 8?) connectors you to screw off is a PITA compared to a single Molex type connector. At least when it comes to disconnecting. But I do accept, the other connection method oozes class! Just a shame the dodge wires killed the appreciation later on in the video!

 

Online ataradov

  • Super Contributor
  • ***
  • Posts: 11228
  • Country: us
    • Personal site
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #22 on: November 22, 2016, 06:46:44 pm »
But then what's left for a motherboard, power and some LED's?
In that case the whole thing can actually be updated to be a wired harness instead of a solid motherboard. Plus some mechanical design for mounting the boards. If anything, that presents a bigger challenge.

BTW, there are flat Ethernet cables (all twisted pairs are molded in parallel), I used them to wire 150 devices to 24-port switches and the result was pretty neat, they are very easy to route in a controllable way, unlike regular Ethernet cables.
Alex
 

Offline Towger

  • Super Contributor
  • ***
  • Posts: 1645
  • Country: ie
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #23 on: November 22, 2016, 07:00:54 pm »
I wonder if ethernet switch(es) will consume more power than Pis'..

May not be far off it, certainly some older ones. But those little domestic switches don't seem to generate much heat. 
The simplest solution to build, but not the neatest would a be 48 port switch and stack the pies up using PCB standoffs. With short patch and power cables to interconnect them.  Just be aware the fans if the switch has any (older models) may be loud.  It should be possible to remove the switch from its case and just use the PCs case fans to cool it and maybe power from the case's psu as well.

Any ideas on how to monitor the individual pies status, even if just a flashing 'Alive' LED on each unit?
 

Offline bktemp

  • Super Contributor
  • ***
  • Posts: 1616
  • Country: de
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #24 on: November 22, 2016, 07:16:47 pm »
Any ideas on how to monitor the individual pies status, even if just a flashing 'Alive' LED on each unit?
I would use UART and connect it to a medium sized microcontroller with an own ethernet interface.
Writing a simple script or program running on each Pi to querry the status (temperature, CPU load, etc.) from UART should be easy.
If the microcontroller starts the query and waits for the answer it can query all Pies sequentially using a cheap multiplexers for the UART signals.
That allows to monitor the cluster without having to connect to each Pi.
You could also add some more advanced features like current monitoring of each Pi and remote shutdown and power cycle capability.
 

Offline m98

  • Frequent Contributor
  • **
  • Posts: 613
  • Country: de
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #25 on: November 22, 2016, 08:56:26 pm »
As for airflow optimization for the Raspberry Pi, just remove the FPC connector at the front, as it causes the flow to instantly separate, creating a dead zone right above the whole PCB. Remove it, and you'll get a significant improvement.
« Last Edit: November 22, 2016, 09:00:19 pm by m98 »
 
The following users thanked this post: richfiles, Doofenshmirtz

Offline technix

  • Super Contributor
  • ***
  • Posts: 3507
  • Country: cn
  • From Shanghai With Love
    • My Untitled Blog
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #26 on: November 22, 2016, 09:10:51 pm »
Since you are spinning for riser boards already, you might as well put interfacing circuitry on it directly: SPI to Ethernet (or just an Ethernet bypass,) some EEPROM for configuration data (so you can interchange Pi's without having to reconfigure it) and most importantly, 12V to 5V DC-DC converter. The host board connector will have both 12V and 5V pins, but only one of the two is connected at a time. This allows you to shift boards around without minding the power rail voltage.

As of the SATA trays, you can put a USB to SATA chipset on the board so you can still access those drives from one of the Pi's.

Raspberry Pi 3 now supports network booting, so if you have one Pi running off the backup power (so it is guaranteed to be online when the rest of the nodes are on) you can boot everything else off that Pi. This, coupled to the configuration EEPROM, allows you to use only one SD card in the entire system, significantly cutting down the cost.
 
The following users thanked this post: richfiles

Offline technix

  • Super Contributor
  • ***
  • Posts: 3507
  • Country: cn
  • From Shanghai With Love
    • My Untitled Blog
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #27 on: November 22, 2016, 09:15:25 pm »
I wonder if ethernet switch(es) will consume more power than Pis'..

May not be far off it, certainly some older ones. But those little domestic switches don't seem to generate much heat. 
The simplest solution to build, but not the neatest would a be 48 port switch and stack the pies up using PCB standoffs. With short patch and power cables to interconnect them.  Just be aware the fans if the switch has any (older models) may be loud.  It should be possible to remove the switch from its case and just use the PCs case fans to cool it and maybe power from the case's psu as well.

Any ideas on how to monitor the individual pies status, even if just a flashing 'Alive' LED on each unit?

You can use a network-based protocol to monitor it, no need for special hardware. The Pi also comes with a built-in hardware watchdog so there is no worry for it locking up (as the watchdog would just go ahead and reset it)
 

Offline technix

  • Super Contributor
  • ***
  • Posts: 3507
  • Country: cn
  • From Shanghai With Love
    • My Untitled Blog
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #28 on: November 22, 2016, 09:19:24 pm »
Naked TPlink pcbs on the top part or maybe there is enough space for two 24 ports switches
Came here to write exactly this. There is plenty of space to wire normal Ethernet and not bother with SPI adapters.

Yeah, there is. SPI Ethernet is neat from a wiring point of view, but that's it's only advantage.
But then what's left for a motherboard, power and some LED's?
Guess you could MUX the serial port from each board or something.
Once you get the networking up there is no need for serial ports. The Pi can be kept alive using the built-in hardware watchdog. To monitor the status of all Pi's you can use SSH. SSH tunnels a console (effectively a serial port, albeit a pure software implementation) through networking infrastructure and as long as your monitoring machine still have the system resources (aka RAM and network capacity) you can run as many instances of SSH as you like.
 

Offline olafnew

  • Newbie
  • Posts: 5
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #29 on: November 22, 2016, 09:26:23 pm »
Ditch the out of the box ethernet switches.
Making an ethernet switch does not require any magnetics on inter-board interconnects. Magnetic coupling is only needed if there is an issue with ground offsets, which, in case of inter-board interconnects - is no issue at all. So you do not need any transformer or optoisolation for your ethernet when you're designing an interposer board(thats the term i believe is used).

I, for one - would design TWO boards that plug into the PCB. Something like this(see pic attached)
When one board connects the power and etc to the boards, and a small, another board is connecting the ethernet ports, with those nifty pcb mountable connectors you've mentioned earlier in the video, from the rpi to the motherboard via a different header.

Power - it is reasonable to ditch the compatibility of the connector from the plug in card module onto which the rPi is plugged, to the motherboard itself, and use something like 3x40 connector, and that way - it has more rigidity, and more possibilities. 12v->5v DC-DC conversion i would be doing on the card module, and NOT on the motherboard. It's much more servicable this way. The motherborad is a great place for the ethernet switching ICs, interconnects, and etc.

I'd really hate to see any cabling when designing such a heffty project.
 

Offline f4eru

  • Super Contributor
  • ***
  • Posts: 1086
  • Country: 00
    • Chargehanger
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #30 on: November 22, 2016, 09:28:10 pm »
Hi

The "transistor on wires" is in effect used as a thermal sensor. ( I also suspected random number generator at first)

from http://tim.id.au/laptops/apple/powermac/powermac_g5.pdf
p119

Offline manu

  • Regular Contributor
  • *
  • Posts: 84
  • Country: fr
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #31 on: November 22, 2016, 09:55:09 pm »
Why not getting rid of the gpio connector on the main board?

Design a PoE power supply on the riser board and use only the RJ45 connector and plug it directly on the main PCB with the PCB mount RJ45 jack you showed in the video.
In this config, you wouldn't need no cables at all  you need only a small rj45 cable between the rj45 on the rpi and the rj45 on the poe (the riser board+RPi should be light enough to be supported by the RJ45 connector) .
« Last Edit: November 22, 2016, 10:06:20 pm by manu »
 
The following users thanked this post: Geerant101

Online ataradov

  • Super Contributor
  • ***
  • Posts: 11228
  • Country: us
    • Personal site
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #32 on: November 22, 2016, 11:14:46 pm »
PRi does not support PoE.
Alex
 

Offline rrinker

  • Super Contributor
  • ***
  • Posts: 2046
  • Country: us
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #33 on: November 23, 2016, 01:30:29 am »
 For monitoring, one extra Pi mounted up in the drive area with a display mounted where the drive tray comes out, as Dave mentioned. That standalone Pi would have all the various sensors connected and display status on the display.

 

Offline BobC

  • Supporter
  • ****
  • Posts: 119
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #34 on: November 23, 2016, 01:47:24 am »
A couple of ideas.  The first assumes every lily will be gilded...

Every REAL supercomputer must have high-throughput board-to-board communication!  This will permit applications requiring lots of communication between between boards, such as distributed real-time video analysis, or massive neural networks.

Remember, the Pi already uses a USB-Ethernet adapter on-board: The Broadcom chip lacks native Ethernet.

1. Dump the on-board Ethernet, use USB instead.  It'll get you higher bandwidth (480 MBps vs ~200 MBps).  This will let you do things like real-time distributed video processing (totally not needed by BOINC).

2. Place 4 PIs per riser.  2 per side with heat sinks out, oriented to have the SD card accessible.  With 12 risers, that's 48 PIs.

3. On the riser, have 4 USB-to-(R)GMII interface and a 5-port GigE chip (perhaps the NXP SJA1105ELY).  Route the GMII switch output to the riser connector (no magnetics needed).

4. On the motherboard have 12-ports of switchable GigE (probably not a single chip, but a tree of smaller switch chips).  CAREFULLY route the 12 riser GMII interfaces to these chips.

If you avoid using GigE PHYs (use GMII) then you pay the GigE PHY Power Penalty *only* for the single rear connector.

Now, you may not want to have to raid Fort Knox to get that much gold.  So let's leave the lilies alone and look at another path: Do everything with USB instead (taking a bandwidth and latency hit).

But if BOINC is your only application, then an all-USB solution should suffice, but it will need a dedicated PI to manage the network:

1. Still 4 PIs per riser.  USB port in slave mode.  (And/or use usbnet or usbip driver.)

2. 1 5-port USB hub chip per riser.  (Again, no magnetics needed.)

3. Mainboard has a 13-port USB3 hub (or a tree of smaller hubs).

4. Connect mainboard hub to Master PI.

This all-USB solution should be both easier to route (fewer riser connector pins) and lower cost than using Ethernet.

And, heck, with USB3 on the riser, you *could* power the PIs through it (though a separate USB3 power manager chip would be needed on each riser).

To get more bandwidth, instead of using a RasPi, the mainboard could use an SBC with multiple USB3 ports, which will simplify the mainboard USB hub.
 

Offline Zad

  • Super Contributor
  • ***
  • Posts: 1013
  • Country: gb
    • Digital Wizardry, Analogue Alchemy, Software Sorcery
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #35 on: November 23, 2016, 04:32:54 am »
Beware of being over-ambitious, it is the graveyard of many projects... Remember, KISS. (No, not being over-affectionate to Dave)

Sure it can be made efficient, with lots of compute nodes, good airflow, good communications and looking sexy, but I suspect getting all these would hammer the "on time" and "on budget" part of the engineers' trinity. Here are some thoughts to throw against the wall and see what sticks:

A mahoosive monolithic motherboard is going to be expensive and a pain to re-spin, even if only double sided. It would need to be thicker than usual to be mechanically rigid enough to support risers and all the modules. How about mounting some form of substrate (e.g. an aluminium plate) on the stand-offs, and have (say) four or 8 smaller PCBs mount on there. This would allow easier development and hardware debugging than messing around with a huge motherboard. It would allow for much freer placement of your own PCB mount screws etc. Also very useful for hiding messy hardware such as Ethernet patch cables. Is there any access via the other side of the case?

That power supply will be quite crusty now after so many years of hard work, I'd like to see inside it. Teardown! Teardown!

How are you going to support the riser cards on the end farthest from the motherboard? The risers will be quite heavy, especially if populated with Ethernet patches, heatsinks etc. This is one problem when mounting horizontally. PCs solve it by bolting it to the rear of the case. Mounting the cards vertically promotes convection cooling, in a commercial unit, air would be drawn in from the bottom of the case and vented at the top/rear.

The information about the new RPi 3 Compute Module is out. Basically, it is very very similar to the old one, just faster with more memory: https://www.raspberrypi.org/documentation/hardware/computemodule/RPI-CM-DATASHEET-V1_0.pdf   "Raspberry Pi guarantee availability of CM1, CM3 and CM3 Lite until at least January 2023" Which is reassuring.

Compute Modules can be populated more densely, incorporate their own flash, can boot over USB, don't have the horrendous power overhead of the RPi's standard Ethernet chip, and already fit nice standard 200 pin memory module sockets. They do, however, need 3.3V and 1.8V, and the comms problem is still there, although it supports USB OTG, meaning each node can be a device, and could be controlled by a master Pi, which is known to be capable of boot programming CMs.

Which ever node type you go with, I would keep it as simple as possible, at least in the first instance, just to get a prototype up and running and capable of being hacked around with to see what it possible. Sexy can wait until it is working. For example, mount 8 RPi3s flat on a daughterboard / riser PCB on stand-offs, use a short patch cables to connect power and ethernet to the riser. The riser just having 1 buck reg per Pi node, and purely passive traces to bring the Ethernet to the motherboard via 0.1in headers, PCI or whatever. For development purposes, a simple breakout board could be made with PC Molex power connector and Ethernet sockets. I wouldn't mess around integrating a switch, in the final design, the motherboard can break out standard (non-magnetics) sockets to the rear of the unit, through the rigid aluminium substrate, via standard patch cables to a network switch mounted at the top of the unit.

Final thought (for now). If you are going to see the insides of the sexy compute monster, then the Apple panel will have to go, presumably being replaced by a laser cut Perspex (acrylic) panel with an Apple and a RPi logo. This will mean cutting a hole in the old panel I'm afraid!

I'm going to have to draw some ideas for myself, to make sense of all the possibilities!
 
The following users thanked this post: onlyrgu

Offline Stonent

  • Super Contributor
  • ***
  • Posts: 3824
  • Country: us
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #36 on: November 23, 2016, 05:17:48 am »
I wanted to add something from a software standpoint.  Set up the Pi's to use a root file system on a network share.
I think there's a project out there to make a PXE client for the Pi. You'd just a have a tiny microSD card that had the bootloader and then it would just load some kind of ramdisk image with a pretty much read only FS.  It would just use scratch space on the ramdisk to function.  That is I guess if you are just using BOINC.

That way you could cut the power to the cluster and never have to worry about a corrupt system.

https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net_tutorial.md

For the hardware, I would almost say design the Pi's in a stacked configuration of say 7. (Like a skyscraper.) Then also integrate the board from the network switch into the stack.  The 7 Pi's would then plug into 7 of the 8 network ports and the 8th port would be the uplink to the main router.  The Pi's would connect to the switch with standard 3inch network patch cables.
The larger the government, the smaller the citizen.
 

Offline Stonent

  • Super Contributor
  • ***
  • Posts: 3824
  • Country: us
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #37 on: November 23, 2016, 05:21:00 am »

Final thought (for now). If you are going to see the insides of the sexy compute monster, then the Apple panel will have to go, presumably being replaced by a laser cut Perspex (acrylic) panel with an Apple and a RPi logo. This will mean cutting a hole in the old panel I'm afraid!


Or just buy a damaged side panel and route it out. Oh and if you could make it in any way resemble ORAC, that would just bring it to the next level.

« Last Edit: November 23, 2016, 05:55:09 am by Stonent »
The larger the government, the smaller the citizen.
 

Offline Stonent

  • Super Contributor
  • ***
  • Posts: 3824
  • Country: us
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #38 on: November 23, 2016, 05:42:38 am »
This guy did 15 in a toolbox. I thought the scrolling LED light was neat.  What would really be cool I think would be to somehow adjust its speed based on current draw from the Pi's. More current draw means they are working harder and then you'd have the LEDs move faster.





« Last Edit: November 23, 2016, 05:50:47 am by Stonent »
The larger the government, the smaller the citizen.
 

Offline manu

  • Regular Contributor
  • *
  • Posts: 84
  • Country: fr
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #39 on: November 23, 2016, 07:34:54 am »
PRi does not support PoE.

That's why I proposed to design a poe supply on the riser board, like the one attached below, for example.

Then you use the second rj45 to have both power and network from the main board.
As a plus, you probably could use 12V on the poe injector which gives plenty of current from the G5 supply.

The riser board would be : a gpio connector, 2 rj45, a 12V to 5V buck converter. The main board would be a bunch of pcb mounted male rj45's and a beefy 12V trace to the different riser boards.
« Last Edit: November 23, 2016, 07:38:06 am by manu »
 

Online EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #40 on: November 23, 2016, 07:50:48 am »
The information about the new RPi 3 Compute Module is out. Basically, it is very very similar to the old one, just faster with more memory: https://www.raspberrypi.org/documentation/hardware/computemodule/RPI-CM-DATASHEET-V1_0.pdf   "Raspberry Pi guarantee availability of CM1, CM3 and CM3 Lite until at least January 2023" Which is reassuring.

Price?
 

Offline bigdog989

  • Newbie
  • Posts: 3
  • Country: au
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #41 on: November 23, 2016, 08:57:11 am »
Two quick thoughts:

1) can you power a rpi straight from the 3v3 gpio pin?

2) you could place one of those RJ45 connectors and a usb port on the mother board to power the pi! this way you just push the board onto the mother board. that side of the board is also shorter allowing more to fit! it does remove the compatibility with different pi types but would save a lot of money on riser boards and other connectors....
 

Offline PODF

  • Contributor
  • Posts: 13
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #42 on: November 23, 2016, 10:49:45 am »
... SPI Ethernet is neat from a wiring point of view, but that's it's only advantage...

As far as i understood is the SPI-Ethernet thingy only useing the IO-Ports of the Raspberry. So you should consider deactivating the USB-/Ethernet-Chip. Because you dont need USB and Ehternet with the SPI-Network solution. That would give an extra powersaving advantage!

The USB-Chip does some interrupts (about 8'000 times a second) and really gets hot. Even if doing "nothing". I used a RPi for Video-Recording with a Raspi-Camera-Module. In Recording-Mode my Rpi B+ consumed average 384mA. With the USB-Chip deactivated it only consumed 269mA. Thats cool when you run this thing out of batteries.

You can Power-Off the Chip with this linux command:
Code: [Select]
echo 0x00 > /sys/devices/platform/bcm2708_usb/buspower

maybe you also want to turn off the HDMI output on the RPi for further powersaving.
My RPi B+ went down to 251mA in videorecording mode (recording-indicator-led off, usb-chip off, hdmi off)
Code: [Select]
/opt/vc/bin/tvservice -off

For easy servicing you could implement something that the power-saving script will only run when the RPi is attached to your mainboard and if you plug it out and do some tests with a particular Raspberry it would detect that it is not connected to the mainboard an so it will not poweroff usb/ethernet/hdmi so you can debug the thing (with keyboard, ethernet and monitor).
« Last Edit: November 23, 2016, 11:06:11 am by PODF »
 

Offline CJay

  • Super Contributor
  • ***
  • Posts: 4136
  • Country: gb
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #43 on: November 23, 2016, 04:51:47 pm »
As for airflow optimization for the Raspberry Pi, just remove the FPC connector at the front, as it causes the flow to instantly separate, creating a dead zone right above the whole PCB. Remove it, and you'll get a significant improvement.

What happens if you angle the board?
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 26755
  • Country: nl
    • NCT Developments
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #44 on: November 23, 2016, 08:45:05 pm »
I wouldn't use the SPI ethernet solution, because except from a nicer wiring it only has drawbacks:
- It is more expensive (you need to add another ethernet chip instead of a cheap RJ45 connector/cable)
- It draws more power (0.5W for ENC28J60, that's a 14% increase in power consumption).
- It is slower (limited by SPI instead of USB)

But wiring 32 or 64 boards using RJ45 cables will not be easy unless you use many cable ties making access to a single module difficult.
The wiring can be made easier by creating custom cables which only have 2 pairs instead of 4. Lose the PVC cable sleeving and you are left with a bunch of twisted pair wires. If you combine a bunch of ethernet connections into a braided sleeve then the cabling will look neat and use little space. It just takes measuring and a piece of wood with nails to layout the wiring looms.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline tobia

  • Contributor
  • Posts: 24
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #45 on: November 23, 2016, 11:18:59 pm »
I just wanted to mention that there are several kinds of 90° header connectors especially designed to mount RPies vertically on a mainboard.

The Orange Pi has a 180° rotated header, but you just need to mount it upside down and the IO pins align perfectly.

Excuse the crudity of the ASCII art model:

Code: [Select]

+------+---+            +------+---+     +------+---+
|      |1 2|  2 1       |      |40 |     |      |2 1|  2 1
|      |3 4|  4 3       |      |. .|     |      |4 3|  4 3
|      |. .|  . .       |Orange|. .|     |Orange|. .|  . .
| RPi2 |. .|  . .       |  PI  |. .|     |  PI  |. .|  . .
| top  |. .|  . .       | top  |4 3|     |bottom|. .|  . .
|      | 40|  40        |      |2 1|     |      |40 |  40
|      +---+            +------+---+     +------+---+
|          |  main-      (no good)        (flipped)    main-
+----------+  board                                    board
              header                                   header

Code: [Select]

      |                                             |
      |   right-                            right-  |Orange
  RPi2|   angle                             angle   |PI
      |   connector                         conn.   |
      |-1 ---+                               +--- 2-|
      |-2 -+ |                               | +- 1-|
           2 1     <- same mb layout! ->     2 1
==================                       ==================
    mainboard                                mainboard

« Last Edit: November 23, 2016, 11:30:28 pm by tobia »
 

Offline Romain

  • Regular Contributor
  • *
  • Posts: 72
  • Country: fr
  • HW & SW consultant
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #46 on: November 24, 2016, 12:43:36 am »
The RaspberryPI does ethernet (a star topology) through a USB to ethernet adapter.
1. If using the GPIO, a neat solution would be to recreate this topology with one of the available buses. Problem is : I2C and SPI are not star, and would need a master to handle every board trying to access the Internet. Could be one of the Pi handling the cluster, but bandwidth will be shared.  I think it is overkilled to add an extra ethernet switch for each board.

2. If male RJ45 connectors exist, so does USB male connector. The 5V could be supplied through it, and make the GPIO connector redundant. Then the motherboard would be a simple interconnect board with just connectors and caps.
 

Offline Zad

  • Super Contributor
  • ***
  • Posts: 1013
  • Country: gb
    • Digital Wizardry, Analogue Alchemy, Software Sorcery
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #47 on: November 24, 2016, 02:01:38 am »
The information about the new RPi 3 Compute Module is out. Basically, it is very very similar to the old one, just faster with more memory: https://www.raspberrypi.org/documentation/hardware/computemodule/RPI-CM-DATASHEET-V1_0.pdf   "Raspberry Pi guarantee availability of CM1, CM3 and CM3 Lite until at least January 2023" Which is reassuring.

Price?

No idea, as a potential bulk purchaser, you might be able to get a quote off them more easily than I could. The current CM sells for £20 at RS Components and Farnell, I wouldn't be surprised if the CM3 came in at the same price, but I only base that on what RPi have done in the past and not hard information.


Offline technix

  • Super Contributor
  • ***
  • Posts: 3507
  • Country: cn
  • From Shanghai With Love
    • My Untitled Blog
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #48 on: November 24, 2016, 02:40:18 am »
The information about the new RPi 3 Compute Module is out. Basically, it is very very similar to the old one, just faster with more memory: https://www.raspberrypi.org/documentation/hardware/computemodule/RPI-CM-DATASHEET-V1_0.pdf   "Raspberry Pi guarantee availability of CM1, CM3 and CM3 Lite until at least January 2023" Which is reassuring.

Price?

No idea, as a potential bulk purchaser, you might be able to get a quote off them more easily than I could. The current CM sells for £20 at RS Components and Farnell, I wouldn't be surprised if the CM3 came in at the same price, but I only base that on what RPi have done in the past and not hard information.
I do have a super high density cluster design based on CM that can achieve more than 300 Pi's per 1U rack mount unit, and have clear vertical air flow across the rack when packed to the brim.
 

Offline onlyrgu

  • Newbie
  • Posts: 7
  • Country: nl
  • Engineer by Nature!!
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #49 on: November 24, 2016, 05:54:10 pm »
Watching your videos for years, I taught I will chip some ideas to this Project.

I support idea of using Ethernet instead of complex SPI IC + Switch IC + supporting components. It will cut down your BOM cost.
I also like the idea of universal plugin modules so you can use OrangePi/RPi/etc
Most Ethernet switches comes with the power connector (eg. 5.5mm) on the back close to the RJ45 female connectors. You also need a power traces on the motherboard for powering the Ethernet switch modules, normally they are 9V/12 V.
Those fans run on 12V. You can also have a dedicated controller for fans on your motherboards
I would suggest ditching the Bulky SMPS out of G5 case that will give you more room for more RPi. Plug in modules.
You can use an off the shelf power supply (or easy to design one) that can provide 5V for the Pi and 9/12V for the Ethernet switches. And you can accommodate this power supply module at the hard disk area.
Looking forward for this awesome project!

P.S.
OT: That SMD connector must have costed them 4 times the cost of a 2N3904, Maybe Apple have a fancy patent to monitor ultrasonic heat waves using a NPN transistor

 

Offline boz

  • Regular Contributor
  • *
  • Posts: 75
  • Country: nz
    • Roving Dynamics Ltd
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #50 on: November 24, 2016, 06:10:56 pm »

1) can you power a rpi straight from the 3v3 gpio pin?

Yes, I have also done it on a few boards now (eg http://rodyne.com/?p=754) and it works real good, you can also provide a shutdown signal (and UPS) if needed through a GPIO
Fearless diver and computer genius
 

Offline bktemp

  • Super Contributor
  • ***
  • Posts: 1616
  • Country: de
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #51 on: November 24, 2016, 06:55:03 pm »

1) can you power a rpi straight from the 3v3 gpio pin?

Yes, I have also done it on a few boards now (eg http://rodyne.com/?p=754) and it works real good, you can also provide a shutdown signal (and UPS) if needed through a GPIO
I think you mixed up 5V and 3.3V. Powering from 5V pin is possible, from 3.3V does not work, because the processor core voltage is supplied by a stepdown powered from the 5V rail.
 

Offline Nobody2

  • Contributor
  • Posts: 17
  • Country: de
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #52 on: November 26, 2016, 02:48:06 pm »
Hi Dave,

considering you would want the "minicomputers" (as a general term independent of the exact model you want to use) mount horizontally (for airflow), have you also considered to mount them rotated? Either by using the network and USB ports to plug into your board directly (USB for power, I don't think they have protection to prevent that) or so that USB and network remain accessible on the top. Doing so should increase the number of pis you can have in each row.

I don't think using the SPI-Ethernet bridge would be a problem in you case, because the the spi port is not that slow (megabit/s range) compared to an average internet connection, but having the network switch integrated on the board would be cool.

As for potential "daughter"-boards. I don't think there is a reason to connect through all 40 pins. Just power and the network connection should be sufficient. So even two to one, or three to one adapter boards should not require big connectors, although mechanical stability might be a major concern.

If you don't require easy access to the individual "minicomputers" later on, you also might want to consider to stack two mainboards on top of each other as a second layer, instead of using 20-30 2-to-1 adapters, in order to get a better "fill-factor" of the case.
 
The following users thanked this post: boz

Offline Artlav

  • Frequent Contributor
  • **
  • Posts: 750
  • Country: mon
    • Orbital Designs
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #53 on: November 26, 2016, 05:30:17 pm »
As for airflow optimization for the Raspberry Pi,
Neat. What software was that made with?
 

Offline wesphillips

  • Newbie
  • Posts: 5
  • Country: us
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #54 on: November 27, 2016, 01:33:29 am »
I Agree with Tobia. This right angle connector would allow just such an option:

https://www.adafruit.com/product/2823


I just wanted to mention that there are several kinds of 90° header connectors especially designed to mount RPies vertically on a mainboard.

The Orange Pi has a 180° rotated header, but you just need to mount it upside down and the IO pins align perfectly.

Excuse the crudity of the ASCII art model:

Code: [Select]

+------+---+            +------+---+     +------+---+
|      |1 2|  2 1       |      |40 |     |      |2 1|  2 1
|      |3 4|  4 3       |      |. .|     |      |4 3|  4 3
|      |. .|  . .       |Orange|. .|     |Orange|. .|  . .
| RPi2 |. .|  . .       |  PI  |. .|     |  PI  |. .|  . .
| top  |. .|  . .       | top  |4 3|     |bottom|. .|  . .
|      | 40|  40        |      |2 1|     |      |40 |  40
|      +---+            +------+---+     +------+---+
|          |  main-      (no good)        (flipped)    main-
+----------+  board                                    board
              header                                   header

Code: [Select]

      |                                             |
      |   right-                            right-  |Orange
  RPi2|   angle                             angle   |PI
      |   connector                         conn.   |
      |-1 ---+                               +--- 2-|
      |-2 -+ |                               | +- 1-|
           2 1     <- same mb layout! ->     2 1
==================                       ==================
    mainboard                                mainboard

 

Offline kc8apf

  • Regular Contributor
  • *
  • Posts: 103
  • Country: us
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #55 on: November 28, 2016, 06:34:09 am »
I chuckled when I saw Dave pointing out the bodge on the G5 motherboard.  A good friend of mine who still works at Apple was responsible for that bodge.  There was a timing bug in the first production run of northbridge ASICs.  That wire is the proper length to insert the a delay to "fix" the bug.  Ah, memories of weeks spent in lab figuring that bug out.
 
The following users thanked this post: BillyD, thm_w, exuvo

Offline Doofenshmirtz

  • Contributor
  • Posts: 13
  • Country: us
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #56 on: November 29, 2016, 05:58:12 am »
Hi, Dave.

I know your focus tends to be hardware, but have you considered a more 'software' alternative to this project?  That is, something different than Ethernet to connect your boards?

Thinking being...
* You don't need high performance, you just need connectivity
* You've got a lot of header pins, SPI, etc to create a shared comms bus
* One Pi on the bus could act as a gateway to the rest of the world via its Ethernet (using NAT)
* The missing link would seem to be a network driver implemented on SPI (i.e., SPI between Pis, not to an Ethernet controller)
[Edit 1/7: I've learned that no SPI slave driver has been developed for the RPi, so you'd have to bit-bang your own bus+network driver using I/O pins.]

I'm on a similar quest, but I haven't yet gone hunting for such a driver, so I don't have a turnkey solution for you.  But boy, would it simplify the design - I've got to think someone has coded this by now.  You could practically use an IDC ribbon cable for the bus and some standoffs to stack the Pis.  (Or, for a PCB design, a right-angle female 2x20 header pairs nicely with an IDC shrouded pin header - no PCB slot required.  Cost for headers from Adafruit is cheaper than Digikey, at least for the US.)

The above would have a bonus of being a low-cost solution for Pi Zero, which might make it $/GHz more attractive.  If you could circumvent the need for an SD card (which would be a whole separate project), that practically cuts the Pi Zero cost in half.

Speaking of Zero, the new Orange Pi Zero looks pretty darned attractive at USD$7 with quad core 1.2GHz, 256MB RAM, and wireless.  All it'd need is a couple pins for power.  OS support seems sketchy out the gate, but I expect that'll resolve itself in due time.  If it works well, it seems like game-over for this kind of 'clustering' project.

And Pi Compute Module... has previously offered small form-factor, but higher cost.  Since the CM3 is claimed to be backward-compatible at the connector with CM1, I expect it to similarly be a subset of Pi3.  But it does have MMC instead of SD, which is nice.  It's handier for integrating into a commercial product, but if you can make a Pi3 work it'll probably be a cheaper 'Raspberry' option (i.e., if you ignore Orange).

Cheers,
Richard
« Last Edit: January 08, 2017, 02:33:44 am by Doofenshmirtz »
 

Offline jeff.moye

  • Newbie
  • Posts: 1
  • Country: gb
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #57 on: November 29, 2016, 10:50:48 am »
Hi Dave,

Why not design your riser boards so that the Pi's all have their ethernet ports facing you?

(see attached JeffCad printout - had to find an envelope especially for this project!)

This will have he following advantages:
  • narrower footprint - should allow an extra column of pi's
  • ethernet cables won't obstruct airflow over the boards
  • usb ports are exposed - allowing usb drives or other devices to be connected

Jeff
 

Offline CaptCrash

  • Regular Contributor
  • *
  • Posts: 50
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #58 on: November 29, 2016, 11:39:13 am »
If the point of this is to process SETI@HOME or other things via BOINC, then the focus on the ethernet controllers seems to be a waste of time, Its not a component of the design that matters.

On the other hand making a sexy board design that the Pi's/OrangePi's can connect into would be great and from that point of view, maximising density, minimising parts counts (ethernet switches, cables, USB power etc) would be a great goal.  Especially with the use of SPI networking or similar, where the ethernet and wireless chips can be shutdown to save power and more importantly heat on each Pi.

I thought I would have a go with this as well (not the electronics part yet), with a bit more of a focus on the computing side for a bit of fun.  To start I went with the following

3 x raspberry pi 3 model B's running Raspbian Jessie Lite
3 x 8gb Micro SD cards
1 x USB high power hub

Other accessories to setup
HDMI to VGA adapter
Keyboard and monitor

I setup the three nodes, as rp3-0, rp3-1 and rp3-2, currently they all attach to my wireless network and directly collect work units from the internet, my next change is to run the rp3-0 as an access point with a seperate SSID and route the network data via this node to my cabled network.  The end result will be one ethernet connection with however many Pi's that I add as I go forward connected wirelessly, inside of a metal PC case with fans and a single power supply.

The network bandwidth that the PI's need is insignificant as they download a work unit, then process it for some time before uploading the result to the internet.  In my case, my ADSL has a download rate of approx. 3mbits/s so even the worst wireless network performance (11mbits/s) easily outpaces this.

On the other hand if you want to run actual cluster applications (not what has been mentioned in the video's, something like the video below), using MPI or an equivalent, then having a common file system (NFS or NAS share), better network performance etc would be important. 


One item that I haven't really noticed you mention or focus on in the videos as yet is what you will do to assist with the cooling of the Pi's other than just some fans.  In my case I have stacked the three Pi's with 30mm risers between them and have a fan blowing across them.  I found that they were running at 60-70c when running 4 BOINC threads per Pi (it was about 26-28c in the room) after a couple of hours.  I then added some small heat sinks to the processor, dropping the processor temps to 50-52c.  note: Temps are from the systems software reported temps, not actual measurements.

Then I had a bit of a play with the overclocking parameters on the Pi's, increasing the CPU to 1.325GHz, increasing the memory bus speed to 450 and increasing the core voltage setting to 4.  This lifted the operating temps to 52-54c, but with an improvement of about 17% in the processing performance of the Einstein@Home work units across all three devices.

Im keenly following your progress.  I am also starting to have a play with MPI on the cluster to see if I can use it to do something useful with it, other than its current task of monitoring the temps on all nodes in the cluster.
« Last Edit: November 29, 2016, 11:44:41 am by CaptCrash »
 
The following users thanked this post: Doofenshmirtz

Offline boz

  • Regular Contributor
  • *
  • Posts: 75
  • Country: nz
    • Roving Dynamics Ltd
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #59 on: November 30, 2016, 03:27:11 am »

1) can you power a rpi straight from the 3v3 gpio pin?

Yes, I have also done it on a few boards now (eg http://rodyne.com/?p=754) and it works real good, you can also provide a shutdown signal (and UPS) if needed through a GPIO
I think you mixed up 5V and 3.3V. Powering from 5V pin is possible, from 3.3V does not work, because the processor core voltage is supplied by a stepdown powered from the 5V rail.

Correct, too quick on the keyboard..
Fearless diver and computer genius
 

Offline richfiles

  • Regular Contributor
  • *
  • Posts: 156
  • Country: us
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #60 on: December 02, 2016, 05:16:21 am »
http://hackaday.com/2016/12/01/running-intel-tbb-on-a-raspberry-pi/

I'm no expert in this stuff, but I'll just leave this link here, and let those who understand it decide if it's worth looking into. Basically, someone got Intel’s open source Threading Building Blocks running on the Raspberry Pi to "achieve better parallelism in his OpenCV project". Intel TBB wasn’t available in Raspbian, due to some kind of issue with creating a build to run on ARM, but this dude was able to create a working build, and made it available to download.

I don't know if it's relevant for this build, but I'll just leave that there for those interested.

As far as the build itself goes, I have to say i rather like the idea of using multi-level risers that mount at least 2 boards deep, and PCB "strips" that have the male RJ45 plugs acting as board to board interconnects. For airflow reasons, I see no reason that the Ethernet PCB strips need to be rectangular either. Routing the boards at follows:

[_]_[]_[]_[]_[]_[]_[]_[]

This would optimize airflow, since you only need boards as wide as the traces themselves need to be. You might even be able to optimize manufacturing by having two boards mirrored against one another, so the protrusions for each male RJ45 connector intermesh during manufacture, to minimize wasted board space. One would assume one end would be larger, so as to support the Ethernet switch chip, and an RJ45 connection to either an external Ethernet switch, or to another riser board, designed to bridge all the ethernet PCB strips to the motherboard.

____
btw... What the heck is up with this forum not supporting extended unicode characters... I had a GREAT diagram for that PCB made up from unicode, and it failed to post properly. Had to redo it in basic ASCII. What is this? forum software from 2002?  :palm:
« Last Edit: December 02, 2016, 05:21:44 am by richfiles »
 

Offline PODF

  • Contributor
  • Posts: 13
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #61 on: December 15, 2016, 09:09:49 am »
Hey Dave

I will help you with the Linux side of this Project. Maybe i can write one or two bash scripts for you. Help setting up the software side. Just ask.

And thanks for your great videos!
 

Offline technix

  • Super Contributor
  • ***
  • Posts: 3507
  • Country: cn
  • From Shanghai With Love
    • My Untitled Blog
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #62 on: December 17, 2016, 01:05:33 am »
RPF have relelased information on some new features in Pi 3 bootloader - mainly netboot is interesting here. This means you can set up the cluster to boot off a single Pi over network, without a mess of SD cards.
 

Offline johnkeates

  • Contributor
  • Posts: 21
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #63 on: January 16, 2017, 05:10:34 pm »
Not sure how alive project is, but: https://www.raspberrypi.org/blog/compute-module-3-launch/

Makes for a much better modular cluster I'd say. Technically you could combine this with other form factors too, just make SO-DIMM carrier boards for them.
 

Offline Zad

  • Super Contributor
  • ***
  • Posts: 1013
  • Country: gb
    • Digital Wizardry, Analogue Alchemy, Software Sorcery
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #64 on: January 16, 2017, 06:25:28 pm »
I just came here to post that too. Stock seems limited though (Farnell UK prices):
Data sheet: http://www.farnell.com/datasheets/2175850.pdf

(Version with eMMC)  1+ £24.75 100+ £23.50     
4,475 will be available for delivery on 05-Feb-2017   
More stock available week commencing 24/04/17

(Without eMMC)  1+ £20.50 100+ £19.50   
15 will be available for delivery on 20-Jan-2017 
4,420 will be available for delivery on 05-Feb-2017
More stock available week commencing 24/04/17

Dev kit £164
43 will be available for delivery on 20-Jan-2017
1,062 will be available for delivery on 05-Feb-2017
More stock available week commencing 24/04/17

RS Components price: £26:99  £21:99  £125:99  (RS seem to have a lot more in stock at the time of checking)

Seems kinda expensive to me, maybe it is early adopter tax?

Online thm_w

  • Super Contributor
  • ***
  • Posts: 6277
  • Country: ca
  • Non-expert
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #65 on: January 16, 2017, 08:20:31 pm »
Seems kinda expensive to me, maybe it is early adopter tax?

It was always going to be expensive as its expected to be used in commercial products. The Rpi/Rpi zero are subsidized so price is lower than expected.
Rpicompute 1 is available for a few pounds less.

Of course compared to orange pi etc. its a lot more. But I have yet to see a plug in module from them, they could easily design one and sell for half the price.
Profile -> Modify profile -> Look and Layout ->  Don't show users' signatures
 

Offline KiwiDaz

  • Newbie
  • Posts: 2
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #66 on: January 17, 2017, 12:27:03 am »
Now no connection issues Dave, standard SODIMM edition : http://tinyurl.com/haydble
« Last Edit: January 17, 2017, 12:29:04 am by KiwiDaz »
 

Offline Lux_

  • Contributor
  • Posts: 11
  • Country: nl
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #67 on: January 30, 2017, 06:08:59 pm »
Does someone know if this project will be continued by Dave?

Lux,
 

Offline Lux_

  • Contributor
  • Posts: 11
  • Country: nl
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #68 on: February 26, 2017, 07:41:54 pm »
nobody?
i really liked the project.
 

Offline stuckat1

  • Newbie
  • Posts: 1
  • Country: us
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #69 on: July 19, 2017, 03:37:58 pm »
Bueller?  Bueller?  Bueller? 
 

Offline jack-daniels

  • Contributor
  • Posts: 36
  • Country: gb
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #70 on: July 19, 2017, 06:15:27 pm »
Bueller?  Bueller?  Bueller?

You're still here? It's over. Go home.
 
The following users thanked this post: Fungus, kingsolmn

Offline kingsolmn

  • Contributor
  • Posts: 18
  • Country: us
  • 220, 221 - whats the difference?!
    • TeamRampage Services
Re: EEVblog #946 - Apple (Raspberry) Pi Cluster - PART 2
« Reply #71 on: August 15, 2019, 06:21:54 am »
Forgive me if it's a no-no to bump this thread, but I'd really like to hear if Dave's done any more with this. I'd really like to see the finished project as it sounds like it's going to be quite the sight when done.

I originally came across these videos about a year ago and got rolled by life ( :-//) and came over to have a look to see if there's something here about an update to the project, but alas no :-BROKE.
I run a small IT/Electronics repair shop out of my garage. The electronics part came about because my son was getting old enough to use a soldering station by himself and does some pretty interesting things (and he has plenty of ideas!) and e-waste recycling for parts (except for the eBay stuff).
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf