EEVblog Electronics Community Forum

Electronics => Projects, Designs, and Technical Stuff => Topic started by: pbgben on April 11, 2014, 02:30:57 am

Title: DISCUSSION: Raspberry Pi Mass Cluster
Post by: pbgben on April 11, 2014, 02:30:57 am
With the recent release of the Raspberry Pi Compute Module I've been thinking of what new possibilities this could bring to the hosting market.

So, What I had in ind was a discussion about how many of these modules you could get working and in what space, E.G. a 1U rack space.

(http://www.raspberrypi.org/wp-content/uploads/2014/04/Untitled.png)

Ill start,

I'm not very technical but you could easily fit 300+ in a 1u space. Im not sure how you would network them or power them in the same space though.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: T0mn0mn0m on April 11, 2014, 02:49:55 am
A cluster with the new SODIMM form factor raspberry pis would certianly be cool. You probably could fit 300+ in 1u of rack space if you had the funds. Correct me if I'm wrong but I think the new form factor rasppberry pis lack ethernet, or at least have the ethernet chip but lack the jack with the ethernet transformers. So networking them could be tricky, you would probably need a seperate switch and a whole bunch of ethernet transformer chips for that.

Or if fast networking isn't required,  I suppose they could be networked over GPIO with say 20 raspberry pis per mini cluster, and 5 networked with faster communication to a controller pi, or a little mini itx board or something.

Apart from just for the fun of it I can't see any real application, and as 20 of these boards will set you back $600 it's an expensive project.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: pbgben on April 11, 2014, 02:59:19 am


Networking will be a massive issue in my eyes, Maybe someone with the know how can shed some light.
As for the practicality, say we can fit 250 per U, and we use a 42U rack thats 10,000 RasPis with 2U to use for networking etc,
10,000 Pis = to 10,000ghz CPU and 4.8tb RAM. all from $300,000 plus the hardware to link them together
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: GeoffS on April 11, 2014, 03:56:28 am
The new compute module has no USB or ethernet ability. That's provided by  the motherboard into which the compute module plugs.
The motherboard shown in the video (http://www.raspberrypi.org/raspberry-pi-compute-module-new-product/) provides USB and HDMI but not ethernet.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: theatrus on April 11, 2014, 05:00:58 am
The RPi SoC has native USB, but does not have native Ethernet (it is after all a mobile phone SoC). The main Rpi boards run an on-board USB<->Ethernet chip (http://www.microchip.com/wwwproducts/Devices.aspx?product=LAN9512 (http://www.microchip.com/wwwproducts/Devices.aspx?product=LAN9512))
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: EEVblog on April 11, 2014, 05:26:52 am
Apart from just for the fun of it I can't see any real application, and as 20 of these boards will set you back $600 it's an expensive project.

How does the combined computer power of these 20 boards compare other existing compute solutions costing that same $600?
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: EEVblog on April 11, 2014, 05:35:36 am
How does the combined computer power of these 20 boards compare other existing compute solutions costing that same $600?

The answer seems to be that 20 barely equals a modern PC:
http://raspberrywebserver.com/raspberrypicluster/raspberry-pi-cluster.html (http://raspberrywebserver.com/raspberrypicluster/raspberry-pi-cluster.html)
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: theatrus on April 11, 2014, 05:56:44 am
How does the combined computer power of these 20 boards compare other existing compute solutions costing that same $600?

The answer seems to be that 20 barely equals a modern PC:
http://raspberrywebserver.com/raspberrypicluster/raspberry-pi-cluster.html (http://raspberrywebserver.com/raspberrypicluster/raspberry-pi-cluster.html)

If the efficiency was good, a lot of serving is strictly I/O bound or wants lots of little failure domains (where managing 1M little servers is not a big deal). However, the RPi isn't going to excel in either area.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: ejeffrey on April 11, 2014, 06:34:36 am
If the efficiency was good, a lot of serving is strictly I/O bound or wants lots of little failure domains (where managing 1M little servers is not a big deal). However, the RPi isn't going to excel in either area.

Except that when you plug these into a backplane with networking and a power supply you add a bunch of single points of failure that can take out dozens of 'independent' failure domains.  Other than a gee-whiz factor, there isn't a lot to recommend this approach over a software based container / virtualization system.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: theatrus on April 11, 2014, 06:37:51 am
In theory, you've fully isolated software (OS/Kernel) faults, but I agree, this is just gee-whiz. Intel hardware ends up being amazingly efficient (performance/$, performance/watt, performance/m3).
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: Someone on April 11, 2014, 09:53:17 am
In theory, you've fully isolated software (OS/Kernel) faults, but I agree, this is just gee-whiz. Intel hardware ends up being amazingly efficient (performance/$, performance/watt, performance/m3).
performance/watt usually gets topped by FPGA or DSP computing.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: tszaboo on April 11, 2014, 12:04:18 pm
There are already ARM srevers in the pipeline:
http://armdevices.net/2011/11/06/arm-servers-getting-ready-to-disrupt-intels-50billionyear-server-market/ (http://armdevices.net/2011/11/06/arm-servers-getting-ready-to-disrupt-intels-50billionyear-server-market/)
I've seen news from HP, IMB about it. But I dont think the Pi would be really a cost effective product for this. The processing power is way too low for that. The same day the Pi was released at Embedded world 2012, I was holding a Tegra 2 development board in my hands, which did everything what the Pi did and more. To start with, it is based on the old ARM11 architecture, not the Cortex. It is more hyped than what it deserves IMHO.
The new Pi computemodule also looks good on pictures. "Integrate the Pi in your system", good hype. But, as already written, id doesnt have ethernet, it doesnt have proper USB (at least 1 for debug, one for user, and one for hosting). The reason to chose this kind of module is to keep the costs down on the mainboard. You outsource the 6-8 layer BGA work, and you dont have to worry about it, it is tested, OS is ported to it. Integration time is low. That is the idea.
Now, if you want ethernet and USB, you need a controller for each. And your mainboard is not cheap anymore.
So it will be a hit and miss. Miss because of the lacking features, and a hit because it will be most likely cheaper than the Pi itself, and half the price than anything we already have. I hope it will remind the manufacturers for the need of the engineers, and at some point we get some "pi killer" usable product at a great price. Like an Alwinner module from Olimex.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: gxti on April 11, 2014, 12:44:32 pm
I agree that the most interesting thing about the "Compute" module is that it will encourage other manufacturers to do better. However, despite being an overall Pi-hater I still think it's a nifty board considering the price and I'd consider using it. It's not the first module in that niche, but to my knowledge it's the first at such a low price point.

As for filling a rack with them, that's just silly. Let's say you can fit 600 of them in 2U as someone threw out earlier. That's $18,000 in modules alone, not counting the backplane and its supporting circuitry, redundant mains power supplies, USB-to-Ethernet and Ethernet switch ICs, management console of some sort (how do you multiplex 600 UARTs? FPGA probably). Plus you still have to cool it all.

$18,000 can buy a seriously ballin' real server that fits in the same space and has all of that, including the 1TiB of RAM and enterprise SSDs. Add virtualization and you can run as many "nodes" as your heart desires.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: Rigby on April 11, 2014, 12:58:29 pm
Doesn't this question belong on SlashDot?  A running joke comment there for a long time was "Imagine a Beowulf cluster of [very slow computing device]!"
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: pbgben on April 12, 2014, 12:12:50 pm
Ok, so it may not be the best performance for your money, but I'm sure there is still a market for a "Clustered" setup.

I've created a simple design concept.
(http://i.imgur.com/KiSz876.jpg)
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: pbgben on June 21, 2014, 01:07:53 pm
Are their any PCB designers here that can lend me a hand? I'd like to get some prototypes done so I can start crowd funding.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: Monkeh on June 21, 2014, 01:16:54 pm
 :-DD
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: sync on June 21, 2014, 01:33:12 pm
Ok, so it may not be the best performance for your money, but I'm sure there is still a market for a "Clustered" setup.
Which markets?
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: pbgben on June 21, 2014, 01:43:38 pm
Ok, so it may not be the best performance for your money, but I'm sure there is still a market for a "Clustered" setup.
Which markets?

Educational and Hobby users mainly, But this system can be used buy anyone that is in need of a low power cluster.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: sync on June 21, 2014, 01:57:08 pm
Educational and Hobby users mainly, But this system can be used buy anyone that is in need of a low power cluster.
How much hobby users will spend >1000 bucks for such a toy?
And professionals won't touch this. For example no one sane uses 100s of GB non-ECC RAM. Too much hassle.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: pbgben on June 21, 2014, 02:23:58 pm
Educational and Hobby users mainly, But this system can be used buy anyone that is in need of a low power cluster.
How much hobby users will spend >1000 bucks for such a toy?
And professionals won't touch this. For example no one sane uses 100s of GB non-ECC RAM. Too much hassle.

Not all hobbyists will be able to afford one for them self, but a group of them could easily afford one. As for ECC RAM, I don't see it being much of an issue.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: tom66 on June 21, 2014, 02:33:36 pm
You have never designed a PCB before but you expect to be able to manage a crowdfunding campaign? This will be another Mu Optics. Please don't.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: sync on June 21, 2014, 02:39:52 pm
Not all hobbyists will be able to afford one for them self, but a group of them could easily afford one.
So you will sell a dozen worldwide.

Quote
As for ECC RAM, I don't see it being much of an issue.
It's basically a requirement for professional use. Non-ECC RAM increases the maintenance cost and for a lot of use cases it's not suitable because you can't trust the data. But there are other requirements like (on site) hardware support and software for managing the thing. So you are limited to the hobby users.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: rob77 on June 21, 2014, 02:45:46 pm
my 2 cents...

clustering R-pi boards is useless for real world applications... end of the story.

why ?
for parallel computing you need remote DMA (one node is able to write to the other node's memory) this is achieved by either InfiniBand, or  if the latency is not SO critical then 10G Ethernet with remote DMA capability.

if you think about hosting tens or hundreds of R-pi boards in a single case/rack - that makes sense - but that's NOT clustering ;)
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: Monkeh on June 21, 2014, 02:57:22 pm
if you think about hosting tens or hundreds of R-pi boards in a single case/rack - that makes sense

It does?

They're slow, power hungry for their performance, have no NICs..
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: rob77 on June 21, 2014, 03:02:09 pm
if you think about hosting tens or hundreds of R-pi boards in a single case/rack - that makes sense

It does?

They're slow, power hungry for their performance, have no NICs..

actually the R-pi hosting services are quite popular - therefore it makes sense (there is a demand for it). with the compute modules it would be harder in terms of connectivity , but still doable. if it has USB then IP over USB (works in linux - androids are tethered that way)  would be a good candidate ;)
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: Monkeh on June 21, 2014, 03:03:35 pm
*shudder*
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: pbgben on June 21, 2014, 03:05:22 pm
You have never designed a PCB before but you expect to be able to manage a crowdfunding campaign? This will be another Mu Optics. Please don't.
This is why I am asking to talk to someone that can, obviously I must do my research before hand. Also, just because I personally don't know how to do something doesn't mean I'll fail.
Not all hobbyists will be able to afford one for them self, but a group of them could easily afford one.
So you will sell a dozen worldwide.

Quote
As for ECC RAM, I don't see it being much of an issue.
It's basically a requirement for professional use. Non-ECC RAM increases the maintenance cost and for a lot of use cases it's not suitable because you can't trust the data. But there are other requirements like (on site) hardware support and software for managing the thing. So you are limited to the hobby users.

Raspberry Pi's are weak compared to a computer or ARM equivalent like the Odroid, but they are still successful.

I can see the point you are making about the ECC, but you are seeing this wrong. You wouldn't use this for high accuracy math computes, and they don't even have to be used as a cluster. Each Pi is a separate device with its own CPU, Memory and Storage.

my 2 cents...

clustering R-pi boards is useless for real world applications... end of the story.

why ?
for parallel computing you need remote DMA (one node is able to write to the other node's memory) this is achieved by either InfiniBand, or  if the latency is not SO critical then 10G Ethernet with remote DMA capability.

if you think about hosting tens or hundreds of R-pi boards in a single case/rack - that makes sense - but that's NOT clustering ;)
People have had an interest in clustering Pi's https://www.youtube.com/results?search_query=Rpi+cluster (https://www.youtube.com/results?search_query=Rpi+cluster)

I see that its no good for advanced use, and thats the reason real clusters cost a s**t ton of cash to build and maintain. This setup is for people that need the features of a distributed compute cloud for a low cost.

Since the first time this idea popped into my head, I have looked deeper into the practicality and uses, I have noticed that, as you said, they would be used more for their density/u then the compute-ability,  I see this as more of a system for learning and low power distributed cloud set-ups.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: theatrus on June 21, 2014, 03:06:32 pm

if you think about hosting tens or hundreds of R-pi boards in a single case/rack - that makes sense

It does?

They're slow, power hungry for their performance, have no NICs..

actually the R-pi hosting services are quite popular - therefore it makes sense (there is a demand for it). with the compute modules it would be harder in terms of connectivity , but still doable. if it has USB then IP over USB (works in linux - androids are tethered that way)  would be a good candidate ;)

The RPi already uses a USB to Ethernet chip. It's also dog slow - any other SoC with onboard Ethernet will have better network performance.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: tom66 on June 21, 2014, 03:07:51 pm
The Compute Module is about $30 and a Raspi Model B board is $35, it is going to be  cheaper to use separate Pi boards when you account for the necessary ethernet interface IC (if you so insist on a Pi cluster in the first place.)
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: pbgben on June 21, 2014, 03:12:55 pm
The Compute Module is about $30 and a Raspi Model B board is $35, it is going to be  cheaper to use separate Pi boards when you account for the necessary ethernet interface IC (if you so insist on a Pi cluster in the first place.)

The model B board is bulky, you don't need and video/audio/gpio/etc on the board for a compute setup. How much room would they take up compared to the compute modules
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: rob77 on June 21, 2014, 03:20:32 pm

if you think about hosting tens or hundreds of R-pi boards in a single case/rack - that makes sense

It does?

They're slow, power hungry for their performance, have no NICs..

actually the R-pi hosting services are quite popular - therefore it makes sense (there is a demand for it). with the compute modules it would be harder in terms of connectivity , but still doable. if it has USB then IP over USB (works in linux - androids are tethered that way)  would be a good candidate ;)

The RPi already uses a USB to Ethernet chip. It's also dog slow - any other SoC with onboard Ethernet will have better network performance.

if considering IP over USB then you would skip the Ethernet completely - using a software router/switch running on a more powerfull SoC (e.g. the one you are suggesting with the onboard ethernet)  terminating all the IP over USB lines for the R-pi modules (couple of USB controllers with couple of USB hubs would be needed).

the R-pi hosting is never used for network inntensive applications - therefore a high performance network connectivity is not needed at all.  so a 1:16 aggregation (networking Soc with ethernet providing downstream IP over USB  to 16 R-pi boards)  would be still more than enough bandwidth for the R-pi boards.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: Monkeh on June 21, 2014, 03:24:01 pm
if considering IP over USB then you would skip the Ethernet completely - using a software router/switch running on a more powerfull SoC (e.g. the one you are suggesting with the onboard ethernet)  terminating all the IP over USB lines for the R-pi modules (couple of USB controllers with couple of USB hubs would be needed).

the R-pi hosting is never used for network inntensive applications - therefore a high performance network connectivity is not needed at all.  so a 1:16 aggregation (networking Soc with ethernet providing downstream IP over USB  to 16 R-pi boards)  would be still more than enough bandwidth for the R-pi boards.

Alternatively, one could replace the whole damn mess with one of these and get better performance: http://www.supermicro.co.uk/products/system/1U/5018/SYS-5018A-MHN4.cfm (http://www.supermicro.co.uk/products/system/1U/5018/SYS-5018A-MHN4.cfm)
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: rob77 on June 21, 2014, 03:32:42 pm
if considering IP over USB then you would skip the Ethernet completely - using a software router/switch running on a more powerfull SoC (e.g. the one you are suggesting with the onboard ethernet)  terminating all the IP over USB lines for the R-pi modules (couple of USB controllers with couple of USB hubs would be needed).

the R-pi hosting is never used for network inntensive applications - therefore a high performance network connectivity is not needed at all.  so a 1:16 aggregation (networking Soc with ethernet providing downstream IP over USB  to 16 R-pi boards)  would be still more than enough bandwidth for the R-pi boards.

Alternatively, one could replace the whole damn mess with one of these and get better performance: http://www.supermicro.co.uk/products/system/1U/5018/SYS-5018A-MHN4.cfm (http://www.supermicro.co.uk/products/system/1U/5018/SYS-5018A-MHN4.cfm)

the idea behind the R-pi hosting is to have separate standalone small systems rented to /owned by individual users. some users simply prefer the "feeling" of their own little hardware over a virtual server/container.
for example if you run a DNS server for a single domain along with a mailserver with a traffic of 5 mails per day... then the R-pi is still a huge overkill for that :D

p.s. if you don't like the idea - then it doesn't necessarily mean the idea is bad or useless ;)
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: pbgben on June 21, 2014, 03:33:55 pm
if considering IP over USB then you would skip the Ethernet completely - using a software router/switch running on a more powerfull SoC (e.g. the one you are suggesting with the onboard ethernet)  terminating all the IP over USB lines for the R-pi modules (couple of USB controllers with couple of USB hubs would be needed).

the R-pi hosting is never used for network inntensive applications - therefore a high performance network connectivity is not needed at all.  so a 1:16 aggregation (networking Soc with ethernet providing downstream IP over USB  to 16 R-pi boards)  would be still more than enough bandwidth for the R-pi boards.

Alternatively, one could replace the whole damn mess with one of these and get better performance: http://www.supermicro.co.uk/products/system/1U/5018/SYS-5018A-MHN4.cfm (http://www.supermicro.co.uk/products/system/1U/5018/SYS-5018A-MHN4.cfm)

Thats not distributed computing though, is it. Nice CPU though.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: Monkeh on June 21, 2014, 03:35:06 pm
some users simply prefer the "feeling" of their own little hardware over a virtual server/container.

Well, there are always odd people around.

Quote
p.s. if you don't like the idea - then it doesn't necessarily mean the idea is bad or useless ;)

Quite true! The performance, power consumption, and MTBF are what make it bad.

Thats not distributed computing though, is it. Nice CPU though.

No, it's a sensible idea, not a load of cheap chips nobody wanted to buy shoved in a box, because hype.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: pbgben on June 21, 2014, 04:05:17 pm
some users simply prefer the "feeling" of their own little hardware over a virtual server/container.

Well, there are always odd people around.

Quote
p.s. if you don't like the idea - then it doesn't necessarily mean the idea is bad or useless ;)

Quite true! The performance, power consumption, and MTBF are what make it bad.

Thats not distributed computing though, is it. Nice CPU though.

No, it's a sensible idea, not a load of cheap chips nobody wanted to buy shoved in a box, because hype.

70Ghz in <600Watts and in 1U Thats not bad for performance/power (Don't forget cooling)
If anything, they are better in regards to failure, as multiple modules can fail with no major effect to the "cluster"

People do want, Pi's sell don't they? http://raspberrycolocation.com/ (http://raspberrycolocation.com/)
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: rob77 on June 21, 2014, 04:13:27 pm
some users simply prefer the "feeling" of their own little hardware over a virtual server/container.

Well, there are always odd people around.

Quote
p.s. if you don't like the idea - then it doesn't necessarily mean the idea is bad or useless ;)

Quite true! The performance, power consumption, and MTBF are what make it bad.

Thats not distributed computing though, is it. Nice CPU though.

No, it's a sensible idea, not a load of cheap chips nobody wanted to buy shoved in a box, because hype.

70Ghz in <600Watts and in 1U Thats not bad for performance/power (Don't forget cooling)
If anything, they are better in regards to failure, as multiple modules can fail with no major effect to the "cluster"

People do want, Pi's sell don't they? http://raspberrycolocation.com/ (http://raspberrycolocation.com/)

forget the 600W in 1U ;) and the Ghz is useless too ;) better to count them in standalone servers per 1U ;)
the fact that you could physically fit a certain amount of those modules into a 1U case doesn't mean it's possible - not at all ! you have to consider all the supporting infrastructure around the modules. so basically divide the number of modules by 2 or 3 and you'll end up with a more realistic figure ;)
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: Monkeh on June 21, 2014, 04:13:56 pm
Clock rate is not a measure of performance.

Anyone else want a stick to go at this horse with?
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: sync on June 21, 2014, 04:18:59 pm
70Ghz in <600Watts and in 1U Thats not bad for performance/power (Don't forget cooling)
:palm: :palm:
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: rob77 on June 21, 2014, 04:22:38 pm
Clock rate is not a measure of performance.

Anyone else want a stick to go at this horse with?

of course it's not... a cpu's clock rate is the measure how good the design of the core is and how good the silicon manufacturing process is :D

the only valid measure is some kind of real-world end-to-end computing test (the CPU itself is just one piece of the puzzle).
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: linux-works on June 21, 2014, 04:43:09 pm
I have a few rasp pi's and beaglebones, as well.  I plan to buy a few more and put them into some kind of unified box, single psu, backplane to select ethernets or something (maybe just a crossbar/vlan switch).

my use-case is network testing.  I'm doing some SDN research and need a bunch of 'end nodes'.  rpi's are good enough for that.  they run a full linux distro (ipv4, v6, etc) and even openflow and openvswitch is there on them (I have yet to try but that's my plan).  with ttcp and iperf, I've seen a reliable 80Mbps thru them and that's fair enough to do some flow testing.

even a stack of mini itx boxes starts to take up room once you build a pool of nodes.  the rpi/bbb form factor is great if you need to do some non-VM based network testing.  compute power on these sucks big-time but they're fine as fanless and small end-nodes.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: sync on June 21, 2014, 04:51:25 pm
Linux-works, would you buy an 1U ras pi housing for a few 100s dollars alone for this?
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: pbgben on June 21, 2014, 04:54:50 pm
some users simply prefer the "feeling" of their own little hardware over a virtual server/container.

Well, there are always odd people around.

Quote
p.s. if you don't like the idea - then it doesn't necessarily mean the idea is bad or useless ;)

Quite true! The performance, power consumption, and MTBF are what make it bad.

Thats not distributed computing though, is it. Nice CPU though.

No, it's a sensible idea, not a load of cheap chips nobody wanted to buy shoved in a box, because hype.

70Ghz in <600Watts and in 1U Thats not bad for performance/power (Don't forget cooling)
If anything, they are better in regards to failure, as multiple modules can fail with no major effect to the "cluster"

People do want, Pi's sell don't they? http://raspberrycolocation.com/ (http://raspberrycolocation.com/)

forget the 600W in 1U ;) and the Ghz is useless too ;) better to count them in standalone servers per 1U ;)
the fact that you could physically fit a certain amount of those modules into a 1U case doesn't mean it's possible - not at all ! you have to consider all the supporting infrastructure around the modules. so basically divide the number of modules by 2 or 3 and you'll end up with a more realistic figure ;)

I see, my mistake. ;)
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: linux-works on June 21, 2014, 05:09:32 pm
Linux-works, would you buy an 1U ras pi housing for a few 100s dollars alone for this?

no, I don't want a rackmount, actually.  it will sit on my desk or nearby.  being low noise, I don't have to put this into a lab, I can have a 'mini lab' on my desk.  that's part of the attraction.

it may likely be more vertical than horizontal, but I'm not ready to think about physical form factor quite yet.

for cost, I can make a case from plexiglass and have it custom cut to size (I have access to a laser) and so it will cost very little, for me.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: josem on June 21, 2014, 05:22:40 pm
Forget clusters, that's so 90's Slashdot.

Stick one Pi module inside each Solar Roadaways tile.  That's your power sorted then just add a few connections and you'll have a Solar Raspberry Pi IoT CoR (Cluster on Roadway) system.

Imagine the possibilities. Smart environment friendly roads AND a climate simulation platform!


Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: sync on June 21, 2014, 05:27:44 pm
for cost, I can make a case from plexiglass and have it custom cut to size (I have access to a laser) and so it will cost very little, for me.
So you are not a potential customer for pbgben's idea. I think most users of a small scale ras pi "cluster" will build their case them self instead of buying an expensive 1U frame.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: rob77 on June 21, 2014, 05:28:06 pm
Forget clusters, that's so 90's Slashdot.

Stick one Pi module inside each Solar Roadaways tile.  That's your power sorted then just add a few connections and you'll have a Solar Raspberry Pi IoT CoR (Cluster on Roadway) system.

Imagine the possibilities. Smart environment friendly roads AND a climate simulation platform!

i hope you're busy with preparing your kickstarter and indiegogo campaigns for this ! ;) i bet you'll outperform (in terms of collected funds) the solar roadway guys ! :D
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: rob77 on June 21, 2014, 05:31:33 pm
for cost, I can make a case from plexiglass and have it custom cut to size (I have access to a laser) and so it will cost very little, for me.
So you are not a potential customer for pbgben's idea. I think most users of a small scale ras pi "cluster" will build their case them self instead of buying an expensive 1U frame.

potential customers would be the companies/individuals running the R-pi hosting services - but those would rather pay a group of engineers ( read engineering students - cheap workforce) to design and develop the platform for them ;)
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: sync on June 21, 2014, 05:38:14 pm
potential customers would be the companies/individuals running the R-pi hosting services - but those would rather pay a group of engineers ( read engineering students - cheap workforce) to design and develop the platform for them ;)
How big is the ras pi hosting market?
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: rob77 on June 21, 2014, 05:43:20 pm
potential customers would be the companies/individuals running the R-pi hosting services - but those would rather pay a group of engineers ( read engineering students - cheap workforce) to design and develop the platform for them ;)
How big is the ras pi hosting market?

does it matter ? :) i already answered your next question in my previous post ;)
no it's definitely not a good candidate for a crowd funding project ;)
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: linux-works on June 21, 2014, 05:47:57 pm
no one would seriously 'host' on a rpi.

the only thing these are good for is experimenting, playing, learning.

if you plan to do serious and reliable thruput with these very underpowered boards, you are in for a surprise.

a rack of atoms (AMD does this, or did this with their buying of the company sea-(something); I forget that name but they did high density of atoms meant for rackmount and datacenter use).  it was power efficient and had mgmt tools all set for it.

for me, being able to keep costs down and use 'commodity' cheap fanless linux 'blades' is a plus, but I would only use them for informal testing and experimenting.

remember: the usb system (and ethernet) on the rpi is pathetic.  I see constant syslog messages about dropped events.  this is NOT a reliable platform, folks.  its a hack.  but its cheap and for some things, its ok enough.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: SirNick on June 24, 2014, 09:14:52 pm
I've designed a new multi-tenant hardware hosting platform called Blueberry Cobler.  Looking for engineers to create the PCBs.  And ICs.  And chassis.

Anyone in for a Kickstarter campaign?   :-DD

(I keed, I keed.)
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: fusionmkx on November 07, 2014, 02:19:29 am
Been thinking about creating a cluster myself, some ideas were being toyed here, http://www.raspberrypi.org/forums/viewtopic.php?f=98&t=74447 (http://www.raspberrypi.org/forums/viewtopic.php?f=98&t=74447)
120 modules in a 1U rack. Well, it could be even denser and some water cooling channels to cool all the modules. Since it's sodimm form factor, you could probably use off-the-shelf heatspreaders.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: blackice504 on November 13, 2014, 12:45:49 am
This is going to be really short and i will try to be sweet.

Any SOC that is on the market does not even come close to the servers that are in Data centers especially the new ones

Example of a Production web server "specs of single server" (these are normally in a cluster so depending on cluster size x the info by the following specs).

2 x E5-2697V3 << latest socket LGA2011-3

14 Cores "28 Threads"
9.6GT/s QPI
2 x Quad Channel DDR4 "per cpu 1.5TB RAM"
68GB/s
40 PCIe 3.0 Lanes.
145W

10Gb Network "keeping in mind i have seen 40Gb and 100Gb" << just not common down under.

SSD Raid

you got a shit load of bandwidth so much that it would make that cluster of Raspberry PI look like an old Atari or C64.

Plus software i don't think Vmware / Citrix / Oracle / M$ / KVM. would work out of the box on the PI sure it could be modded ect but in hosting everything is about deployment and Bandwidth as well as low latency.

Then keep in mind this machine is just 1 out of a rack that is clustered so that makes the PI even more useless for this application not to mention Processing Power per 1, after all there is a reason why nearly every data center uses on brand Intel. "sorry if your a fan of AMD or what ever but Intel Dominates the Server Market"

Intel Costs more cheaper to run, more bang per watt and more bandwidth.

Stuff to look forward to Intel and AMD and Nvidia are all set on an idea to make a super computer that will fit in the palm of your hand but even then i am sure that a desktop / server will have more power as its larger , has more bandwidth "more channels" ect.

So in short sorry its not going to happen.





Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: pbgben on May 30, 2015, 08:09:45 am
Looks like someone else hase created a similar product...

(https://www.scaleway.com/img/blade.91ac.png)

https://www.scaleway.com/features/ (https://www.scaleway.com/features/)
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: linux-works on May 30, 2015, 05:04:07 pm
there was a company SeaMicro that tried this with intel atom chips, maybe 5 years ago.  I had an interview there twice: both before amd bought them and after.  funny, since they started off with atom chips for low power and then amd bought them (lol).  they offered high density 500 cpu systems (iirc) that were mostly for i/o, thinking that business apps and webserver apps were more io bound than compute bound, at least many of them were.  so, they tried to build configurable clusters of atom cpus with 'patchable' fabric, io, compute, backplanes, etc.  kind of a neat idea, really.

but they never made a big enough impact and they went out of business (amd killed the project, iirc).

they had some serious pcb's, power supplies, chassis and over 100 people there at one time and they could not sustain this concept long-term, even with amd's funding!

the concept of a pi cluster is for fun and learning, but NEVER more than that.  sad but true; this 'imagine a beowulf cluster of ...' meme never wants to die, but other than for a proof of concept, its still not an efficient (cost or otherwise) way to build a cluster.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: pbgben on January 27, 2016, 06:36:00 am
(http://www.cnx-software.com/wp-content/uploads/2016/01/Raspberry_Pi_Zero_Cluster.jpg)

A "Cluster" of the latest Pi's

http://www.cnx-software.com/2016/01/22/this-is-what-a-16-raspberry-pi-zero-cluster-board-looks-like/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+cnx-software%2Fblog+%28CNXSoft+-+Embedded+Software+Development%29 (http://www.cnx-software.com/2016/01/22/this-is-what-a-16-raspberry-pi-zero-cluster-board-looks-like/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+cnx-software%2Fblog+%28CNXSoft+-+Embedded+Software+Development%29)
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: Monkeh on January 27, 2016, 04:41:17 pm
What a ridiculous waste of money.
Title: Re: DISCUSSION: Raspberry Pi Mass Cluster
Post by: suicidaleggroll on January 27, 2016, 07:00:07 pm
If you read the details, they did NOT develop that board for clustering/HPC, they built it to rapidly program and test their deployable sensors.  It lets them program and test 16 boards at a time in a convenient and self-contained package, instead of having to individually wire them up one by one.

Quote
We are making a sensing device that uses Raspberry Pi compute module. So we need many Pi’s for the development and tests. Since we will use Pi’s GPU for image processing, deep learning, etc. We need real Pis but not just Linux machines. Another reason. It can be used for flashing eMMCs of our devices via USB ports when we have to do that by ourselves.

Depending on how many of those sensors they plan to build/sell, that "clustering" platform could easily end up paying for itself in reduced labor costs.  Of course the "fan boys" are jumping on it and begging to be able to buy it as a compute cluster, but that's not the company's fault.