Author Topic: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1  (Read 54442 times)

0 Members and 1 Guest are viewing this topic.

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #25 on: October 15, 2016, 11:16:59 am »
Another option is the ODROID-C0 http://www.hardkernel.com/main/products/prdt_info.php?g_code=G145326484280
It comes without populated GPIO and USB connectors. Not ultra cheap but it might suit some uses.

Someone on youtube pointed me to the nanopi-neo:
http://nanopi.io/nanopi-neo.html
It's only $8 for a 4 core H3
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #26 on: October 15, 2016, 11:18:03 am »
What about use a small FPGA to connect all the SPI lines with a master node?
My thoughts exactly - either emulate multiple ENC28J60 chips and a swithc in the FPGA, or if there is a mechanism in the ENC protocol to add waits, maybe mux the SPIs into a single ENC chip.

Because that's another complex step. Far easier to just use the chips.
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #27 on: October 15, 2016, 11:22:07 am »
Adding an ENC28J60 will waste a lot of power: Those things get pretty hot, because they draw 120mA when idle and 160mA when transmitting. That's an additional 0.5W per chip!
ENC424J600 is faster and draws less current.

Hadn't looked at the power consumption yet, thanks for the heads up.
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #28 on: October 15, 2016, 11:24:00 am »
Might I suggest this special snowflake of a connector?

PCB mount Ethernet Plug!  :-DD

I couldn't find one of those!  :o

10 bucks says the height does not match the required vertical USB micro PCB plug!
« Last Edit: October 15, 2016, 11:26:45 am by EEVblog »
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #29 on: October 15, 2016, 11:28:33 am »
If you want to go smaller you might want to look at the NanoPi NEO: http://www.friendlyarm.com/index.php?route=product/product&path=69&product_id=132. It uses the same AllWinner H3 CPU and it's only 40x40mm!

Yeah ,someone else pointed this out and I really like it. Very cheap, available by the looks of it, and tiny.
 

Offline CM800

  • Frequent Contributor
  • **
  • Posts: 882
  • Country: 00
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #30 on: October 15, 2016, 11:30:21 am »
Might I suggest this special snowflake of a connector?

PCB mount Ethernet Plug!  :-DD

I couldn't find one of those!  :o

Went on RJ45 on Farnell, then clicked plug and solder / through hole, this came up..


Think you might use it? Much more elegant solution.

It might even be possible to make a fancy footprint to use this rightangle USB Type A right angle SMT connector as a THT vertical:

http://uk.farnell.com/multicomp/mc000991/usb-2-0-type-a-plug-smt/dp/2476092?MER=sy-me-pd-mi-alte

Don't know how well the boards would line up however... I suspect not so well. May not matter. (depends what board you go for)



If you want to go smaller you might want to look at the NanoPi NEO: http://www.friendlyarm.com/index.php?route=product/product&path=69&product_id=132. It uses the same AllWinner H3 CPU and it's only 40x40mm!

Yeah ,someone else pointed this out and I really like it. Very cheap, available by the looks of it, and tiny.

They look small enough you could use them horizontally with standoffs and use pogo pins for the ethernet connection... that could be quite elegant.



EDIT:

Brought 4 of them! Look perfect to get antiquated with ARM / Linux. Maybe in the future I can fit a nice CPLD or FPGA under it too.
« Last Edit: October 15, 2016, 11:43:33 am by CM800 »
 

Offline bktemp

  • Super Contributor
  • ***
  • Posts: 1616
  • Country: de
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #31 on: October 15, 2016, 11:43:56 am »
Someone on youtube pointed me to the nanopi-neo:
http://nanopi.io/nanopi-neo.html
It's only $8 for a 4 core H3
You could use the PCB mount ethernet plug next to an PCB mound USB A connector and plug the NanoPi NEO into both connectors: Ethernet for communication and USB for power (you will be using the USB output as power input, but it should be ok, the schematics shows the 5V USB connected directly to the internal 5V rail).
 

Offline rob77

  • Super Contributor
  • ***
  • Posts: 2085
  • Country: sk
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #32 on: October 15, 2016, 11:57:22 am »
regarding all the networking suggestions for a "fun" project like this ... the top priority is the question: does linux have a working and usable driver for that solution ? if the answer is no, then you should ditch the idea right away because it would increase the complexity of the project (and not everyone is able to write network device drivers for linux).

so the viable options are (simplest to most complicated):
- wifi dongles + dedicated wifi router
- pppd over serial to a "master" with ethernet connection and acting as a router/bridge
- dedicated SPI ethernet chip for each board (ENC28J60 or enc424j600) + ethernet switch (without magnetics of course)

the other possibilities are countless , virtually you can do anything.. but the question is the drivers. of course you can write you application specific communication routines but that would be single purpose...  if you are able to write the network device drivers for your solution, then go ahead with it... otherwise go for a solution which is already available.
« Last Edit: October 15, 2016, 11:59:04 am by rob77 »
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #33 on: October 15, 2016, 12:04:40 pm »
They look small enough you could use them horizontally with standoffs and use pogo pins for the ethernet connection... that could be quite elegant.

Yep, that was my first thought. Although I wouldn't use pogo pins, just suck out the RJ45 and USB and solder them in place. Could have multiple flat motherboards stacked, along with thermal sheets going over the boards and processors maybe. density would be super high. 5 per board, 5 boards stacked would be bugger-all volume for a 100 core ARM system.
Best passive thermal coupling would be 5 boards mounted back to back with thermal pads going to a thins case top and bottom.
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #34 on: October 15, 2016, 12:07:07 pm »
You could use the PCB mount ethernet plug next to an PCB mound USB A connector and plug the NanoPi NEO into both connectors:

Murphy will ensure it's not that easy and the two are not compatible height wise.
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #35 on: October 15, 2016, 12:08:31 pm »
regarding all the networking suggestions for a "fun" project like this ... the top priority is the question: does linux have a working and usable driver for that solution ?

Doesn't matter, it's just like having dozens of RPi's plugged into a switch. Linux doesn't know any  different.
 

Offline rob77

  • Super Contributor
  • ***
  • Posts: 2085
  • Country: sk
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #36 on: October 15, 2016, 12:13:42 pm »
regarding all the networking suggestions for a "fun" project like this ... the top priority is the question: does linux have a working and usable driver for that solution ?

Doesn't matter, it's just like having dozens of RPi's plugged into a switch. Linux doesn't know any  different.

i was referring to the solutions suggested above - like the multiplexed SPI... if there is no driver in the kernel for it, then it will over-complicate the project because you will have to write the divers for it.
 

Offline rs20

  • Super Contributor
  • ***
  • Posts: 2317
  • Country: au
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #37 on: October 15, 2016, 12:22:34 pm »
i was referring to the solutions suggested above - like the multiplexed SPI... if there is no driver in the kernel for it, then it will over-complicate the project because you will have to write the divers for it.

Mike's suggestion explicitly mentioned emulating the ENC28J60 on the FPGA, in which case the pre-existing ENC28J60 drivers would work. However, I'd personally prefer just to use real chips though, impersonating chips on an FPGA sounds like quite the rabbit-hole.
 

Offline CM800

  • Frequent Contributor
  • **
  • Posts: 882
  • Country: 00
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #38 on: October 15, 2016, 12:26:45 pm »
They look small enough you could use them horizontally with standoffs and use pogo pins for the ethernet connection... that could be quite elegant.

Yep, that was my first thought. Although I wouldn't use pogo pins, just suck out the RJ45 and USB and solder them in place. Could have multiple flat motherboards stacked, along with thermal sheets going over the boards and processors maybe. density would be super high. 5 per board, 5 boards stacked would be bugger-all volume for a 100 core ARM system.
Best passive thermal coupling would be 5 boards mounted back to back with thermal pads going to a thins case top and bottom.

Could machine a beautiful long aluminum or copper block under them all with raises in them to touch the processor, then throw a hole through the length of it and pump water to keep each stack cool. I think this could be a fun project, even competition.... how many cores per square cm can we fit in with using only off-the-shelf ARM boards (no custom ARM boards)
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #39 on: October 15, 2016, 12:30:19 pm »
Could machine a beautiful long aluminum or copper block under them all with raises in them to touch the processor, then throw a hole through the length of it and pump water to keep each stack cool. I think this could be a fun project, even competition.... how many cores per square cm can we fit in with using only off-the-shelf ARM boards (no custom ARM boards)

Water cooling would not be required, passive would work fine.
 

Offline EEVblogTopic starter

  • Administrator
  • *****
  • Posts: 37661
  • Country: au
    • EEVblog
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #40 on: October 15, 2016, 12:34:39 pm »
Mike's suggestion explicitly mentioned emulating the ENC28J60 on the FPGA, in which case the pre-existing ENC28J60 drivers would work. However, I'd personally prefer just to use real chips though, impersonating chips on an FPGA sounds like quite the rabbit-hole.

Exactly. Why you'd bother to go to that effort just for a project like this is beyond me.
It's a huge job, look at the datasheet and the hundreds of registers and whatnot. What worthwhile benefit d you get for all that effort?
Time and effort much better spent on other aspects.
« Last Edit: October 15, 2016, 12:37:46 pm by EEVblog »
 

Offline rob77

  • Super Contributor
  • ***
  • Posts: 2085
  • Country: sk
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #41 on: October 15, 2016, 12:39:27 pm »
i was referring to the solutions suggested above - like the multiplexed SPI... if there is no driver in the kernel for it, then it will over-complicate the project because you will have to write the divers for it.

Mike's suggestion explicitly mentioned emulating the ENC28J60 on the FPGA, in which case the pre-existing ENC28J60 drivers would work. However, I'd personally prefer just to use real chips though, impersonating chips on an FPGA sounds like quite the rabbit-hole.

i was more talking about this one

What about use a small FPGA to connect all the SPI lines with a master node?
My thoughts exactly - either emulate multiple ENC28J60 chips and a swithc in the FPGA, or if there is a mechanism in the ENC protocol to add waits, maybe mux the SPIs into a single ENC chip.

muxing into a single ENC chip would mean you have to implement some kind of virtualization in the driver - the driver would have to hold a "virtual" ENC chip instance and it would have to re-inicialize the real ENC oince it get it's "machine time". at least the MAC address would have to be re-initialized on every context switch of the real ENC - otherwise the whole cluster would appear as a single node on the network (single MAC address).
 

Offline hans

  • Super Contributor
  • ***
  • Posts: 1626
  • Country: nl
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #42 on: October 15, 2016, 12:39:57 pm »
Multiplexed SPI sounds like a big can of worms. You'd need to multiplex 4 I/O's in 2 directions 8 times, which does not cut down on the number of chips and just a little on cost (ENC28J60 is 2.35 EUR each, a 4ch mux is probably more like 0.50 EUR).

Also there are no drivers. And I think this is the biggest problem; because I doubt it's even possible to create a pleasing workable system. The ENC28J60 is a nice chip because it has filters like unicast and such, so it drops any frame from the buffer that is not targetted at it's own MAC. If you have 8 linux boxes talking through the same ENC28J60 they would need to be assigned individual software MAC's. They then need to poll the ENC one by one to check if there is a packet they can take. And how are you going to handle stuff like broadcasts, multicasts and packets that do not belong to the cluster? Again: can of worms.

Emulating the ENC28J60 sounds doable, but probably a lot of work even for a dumbed down emulation. And I imagine you need a pretty large FPGA to emulate the frame buffers for each ENC28J60. If you have 8 emulations running, times 8kB of buffer, that's 64kB of RAM + all logic required to basically implement an ethernet switch. All this work for 25 euro worth of chips. Could be fun, but it is an enormous undertaking and ultimately does not add unique business logic that doesn't already exist and you absolutely need to implement in a FPGA.

ENC28J60 and ENC424J600 drivers are already in the kernel and can be used via SPI, but like I said earlier both are limited to around 10-14 Mbit/s respectively.
 

Offline CM800

  • Frequent Contributor
  • **
  • Posts: 882
  • Country: 00
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #43 on: October 15, 2016, 12:46:14 pm »
Could machine a beautiful long aluminum or copper block under them all with raises in them to touch the processor, then throw a hole through the length of it and pump water to keep each stack cool. I think this could be a fun project, even competition.... how many cores per square cm can we fit in with using only off-the-shelf ARM boards (no custom ARM boards)

Water cooling would not be required, passive would work fine.

True... I'm just looking for an excuse to use a tiny radiator like this:




Adorable  :scared: :scared:!


Quote
Although I wouldn't use pogo pins,

Also, is there any reason you don't like the idea of pogo pins? (to save desoldering / enhance plug / playability)

« Last Edit: October 15, 2016, 12:49:52 pm by CM800 »
 

Offline microcircuit

  • Newbie
  • Posts: 8
  • Country: gb
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #44 on: October 15, 2016, 01:01:03 pm »
Hi Dave,
Just a thought, rather than routing slots in your motherboard, if you were to fit the right angle connector to a Pie before soldering it to the MB this would provide the required spacing stand off between the MB and Pie.  Not having a connector or Pie to test I'm unsure if the connector pins are long enough for this to work.
Excellent videos,
Phil   
 

Offline crasbe

  • Newbie
  • Posts: 2
  • Country: de
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #45 on: October 15, 2016, 01:54:44 pm »
Hi Dave,

I don't know if you've seen my comment on YouTube, but maybe the NXP SC16IS740/750/760 Slave UART might be a solution to look into.
That's a I2C or SPI-attached Slave UART and you can put up to 16 of them onto one I2C-bus.

You'd have to dedicate one Orange Pi as a master for PPP over serial. There is also support for the SC16IS7x0 in the Linux kernel, so the software side shouldn't be too complex.

Greetings,
Chris
 

Offline CM800

  • Frequent Contributor
  • **
  • Posts: 882
  • Country: 00
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #46 on: October 15, 2016, 02:18:29 pm »
I've made a nice CAD model of this now, the only inaccurate dimensions are the usb and ethernet height, the length is correct, as is the height of the BGA relitive to the pcb.

I'm not sure about the thickness of the PCB, I made it 1.5mm.




 

Offline BurtyB

  • Regular Contributor
  • *
  • Posts: 66
  • Country: gb
    • 8086 Consultancy
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #47 on: October 15, 2016, 02:28:57 pm »
Someone on youtube pointed me to the nanopi-neo:
http://nanopi.io/nanopi-neo.html
It's only $8 for a 4 core H3

I've been playing with the NanoPi NEO in my v1.3 Cluster HAT (http://clusterhat.com/dist/img/ClusterHAT-Zero-NanoPi.jpg) which uses USB Gadget mode to provide Ethernet, Serial Console and powering it over a single USB connector.

I can't see the NanoPi NEO without headers on the FriendlyARM site anymore which is a shame but on the up side it should be possible to boot the NEO over USB without an SD card (http://linux-sunxi.org/FEL/USBBoot) making it even cheaper to use and easier to deploy. I'm hoping the Pi Zero will also support it properly one day but it doesn't currently work (https://github.com/raspberrypi/tools/tree/master/usbboot).

Chris.
 

Offline mikeselectricstuff

  • Super Contributor
  • ***
  • Posts: 13694
  • Country: gb
    • Mike's Electric Stuff
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #48 on: October 15, 2016, 02:41:39 pm »
Mike's suggestion explicitly mentioned emulating the ENC28J60 on the FPGA, in which case the pre-existing ENC28J60 drivers would work. However, I'd personally prefer just to use real chips though, impersonating chips on an FPGA sounds like quite the rabbit-hole.

Exactly. Why you'd bother to go to that effort just for a project like this is beyond me.
It's a huge job, look at the datasheet and the hundreds of registers and whatnot. What worthwhile benefit d you get for all that effort?
Time and effort much better spent on other aspects.
I was just thinking about cost ( and power)  if you wanted to do this on  a bigger  scale. Though it might be easier to write an ethernet driver that did parallel comms over the IO lines...
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Offline @rt

  • Super Contributor
  • ***
  • Posts: 1051
Re: EEVblog #934 - Raspberry Pi Supercomputer Cluster PART 1
« Reply #49 on: October 15, 2016, 02:45:22 pm »
Turning the boards 45 degrees would solve the routing problem if the CAD can rotate the footprint at arbitrary angles.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf