Author Topic: Microcontroller mesh network  (Read 14993 times)

0 Members and 1 Guest are viewing this topic.

Offline sirhaggisTopic starter

  • Regular Contributor
  • *
  • Posts: 54
Microcontroller mesh network
« on: January 24, 2015, 12:49:41 pm »
G'day guys,
I'm working on a project where I am trying to connect many micro-controllers together in such a way that they can talk to their neighbours in a mesh network arrangement. Each controller will be a 32 bit entry level, something like an ARM Cortex M0 or similar. The idea is that every MCU can have up to 6 neighbours and the number of controllers can potentially be very large (e.g. 100s). Controllers need to be able to form communication channels across the mesh to arbitrary other controllers in the network, and also form groups of controllers to which members can broadcast to each other. If that were not complicated enough, I would like to be able to get some decent amount of bandwidth and low latency, even when controllers are at distant points in the network. Power usage is also a concern - e.g. constantly running polling code is out of the question, and I would expect that most controllers should sit largely idle in low power mode when they are not performing intensive tasks (e.g. listening for traffic).

So the main design decision I am struggling with at the moment is how to implement networking in such a way as to meet reasonable speed/latency/power requirements. Some immediate possibilities that I am considering and my thoughts on them are as follows:

- Multiplex a I2C or SPI connection to each neighbour controller and switch between them for communication. I'm not sure how well this would work in practice and whether there would be issues with lost data whilst switched to a different channel. Obviously the max bandwidth for any controller is limited by the bus speed shared across all connections, but that is workable as long as there is no data loss. Apparently there are some I2C multiplexing chips out there that I have not looked into yet. The problem with this approach seems to be potential latency issues because packet forwarding needs to be done in software and associated power costs. Also I'm not really sure how well I2C/SPI would operate in a multiplexed scenario.

- Shared bus. This option would create a shared bus (single wire) between all controllers with each having a unique address. This is obviously great for latency, but means the bandwidth of all communications is limited to the bus speed. I think there would also be more significant issues when the number of controllers start hitting the hundreds, such as address space issues and parasitic capacitance/noise on the bus, etc.

- Multiplexed channels. This is an idea I had that could allow multiple independent closed networks to be dynamically configured on the fly. It would involve using multiplexers/crosspoint chips at each controller to create direct links/channels through that point by switching appropriately (like a switchboard). So if you wanted to create a direct link between the controller on the left and right you would set the switches as appropriate using the controller, and a direct connection will be created between them. By setting the switches (with GPIO pins) you could create arbitrary groups of controllers that share the same bus and have multiple independent buses communicating with each other. This potentially has great latency and bandwidth properties but would come at the cost of complexity (and probably way too many multiplexers).

Given the severe practical concerns with having a shared bus, I think the other options are the way to go. Fully software routing has a certain appeal but I suspect the power and latency issues may make it impractical (although perhaps some combination of interrupts and DMA could practically address this?). A multiplexing arrangement is potentially very powerful, but I'm not sure the added complexity is worth it and there a potentially issues with the number of control pins available to set the multiplexers.

Something else I haven't looked into is whether there are exists cheap ICs that can offload the networking/switching logic for each controller and and only bother the MCU when there is traffic it needs to process. I'm not sure if such a thing exists that is a) physically compact and b) cheap (think $1-$2 range as there are potentially going to be hundreds of nodes). Does anybody know if such a thing exists?

Thoughts?
 

Offline Maxlor

  • Frequent Contributor
  • **
  • Posts: 565
  • Country: ch
Re: Microcontroller mesh network
« Reply #1 on: January 24, 2015, 01:10:45 pm »
Can you give some numbers for the average and max distance between nodes, as well as the latency and bandwidth requirements?

Btw, SPI, I2C and similar buses are problematic when used as a field bus, they'll pick up a lot of noise and achievable speed will be limited. I differential bus will probably perform better.
 

Offline sirhaggisTopic starter

  • Regular Contributor
  • *
  • Posts: 54
Re: Microcontroller mesh network
« Reply #2 on: January 24, 2015, 01:26:19 pm »
As an approximation I would say max distance would be 40 nodes, average about 5-10. Latency/bandwidth requirements are essentially 'as good as I can get', but as a figure perhaps 100-200Mbps between each node (although if there were a shared bus for all nodes it would need to be considerably higher), latency perhaps <10ms per link.

I agree on the noise issues, which is why I don't think I will be pursuing a shared bus over the whole network. In a switched channel arrangement there would obviously have to be limits on how large the networks are to avoid noise. A software approach would not have to worry about noise because the controllers would relay messages at each node to their neighbour.
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 26682
  • Country: nl
    • NCT Developments
Re: Microcontroller mesh network
« Reply #3 on: January 24, 2015, 01:45:27 pm »
ETHERNET!
You can run your own protocol over existing hardware if implementing UDP or even TCP/IP is making things too complicated but you could be re-inventing the wheel here.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline sirhaggisTopic starter

  • Regular Contributor
  • *
  • Posts: 54
Re: Microcontroller mesh network
« Reply #4 on: January 24, 2015, 02:48:24 pm »
Yes I have thought about ethernet and it would be an awesome solution if I could get it to work. I'm not so sure whether it would be feasible with the hardware constraints I have however - I am using entry level MCUs which don't natively support ethernet. I suppose it might be possible to implement it using GPIO's but that sounds hairy. Other than that there might be ethernet-on-a-chip type products that I could interface with, but I doubt they would fit in the $1-$2 each price range I am looking for (there will be many controllers implemented as modules that can be stacked).

Definitely something I will look into though.
Thanks
 

Offline sirhaggisTopic starter

  • Regular Contributor
  • *
  • Posts: 54
Re: Microcontroller mesh network
« Reply #5 on: January 24, 2015, 02:55:13 pm »
"Other than that there might be ethernet-on-a-chip type products that I could interface with, but I doubt they would fit in the $1-$2 each price range"

Scrap that, I just found some contenders - some less than $1 in bulk!
To the bat mobile!
 

Offline sirhaggisTopic starter

  • Regular Contributor
  • *
  • Posts: 54
Re: Microcontroller mesh network
« Reply #6 on: January 24, 2015, 03:54:06 pm »
There are a bunch of ICs that do something what I want here:
http://www.digikey.com/product-search/en?FV=fff40027%2Cfff801a4%2C908000d&k=ethernet&mnonly=0&newproducts=0&ColumnSort=1000011&page=1&stock=0&pbfree=0&rohs=0&quantity=&ptm=0&fid=0&pageSize=25

On further thought however, it would seem that I would need one ethernet transceiver for each neighbour connection, which means six of these per node. I'm looking into the possibility of ethernet multiplexing, but I'm not sure if that would actually work or not.
 

Offline sirhaggisTopic starter

  • Regular Contributor
  • *
  • Posts: 54
Re: Microcontroller mesh network
« Reply #7 on: January 24, 2015, 03:59:50 pm »
Something like this with built int TCP/IP stack is also possible, but its in the $3-$5 category, not $1ish category
http://wizwiki.net/wiki/lib/exe/fetch.php?media=products:w5500:w5500_ds_v105e_141111.pdf
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 26682
  • Country: nl
    • NCT Developments
Re: Microcontroller mesh network
« Reply #8 on: January 24, 2015, 04:50:11 pm »
There are a bunch of ICs that do something what I want here:
http://www.digikey.com/product-search/en?FV=fff40027%2Cfff801a4%2C908000d&k=ethernet&mnonly=0&newproducts=0&ColumnSort=1000011&page=1&stock=0&pbfree=0&rohs=0&quantity=&ptm=0&fid=0&pageSize=25

On further thought however, it would seem that I would need one ethernet transceiver for each neighbour connection, which means six of these per node. I'm looking into the possibility of ethernet multiplexing, but I'm not sure if that would actually work or not.
Try to find a switch chip which is MII ports instead of an analog interface. If the MII port on that switch chip can be put into slave mode even better: you can bitbang the MII interface so you don't need a microcontroller with a MAC.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline BlueBill

  • Regular Contributor
  • *
  • Posts: 169
  • Country: ca
Re: Microcontroller mesh network
« Reply #9 on: January 24, 2015, 05:44:08 pm »
Might be helpful if the OP would tell us what they're trying to build.
 

Offline Dave Turner

  • Frequent Contributor
  • **
  • Posts: 447
  • Country: gb
Re: Microcontroller mesh network
« Reply #10 on: January 24, 2015, 11:15:23 pm »
My first thought is a new take on a neural net  ;D
 

Offline sirhaggisTopic starter

  • Regular Contributor
  • *
  • Posts: 54
Re: Microcontroller mesh network
« Reply #11 on: January 25, 2015, 02:38:25 am »
Might be helpful if the OP would tell us what they're trying to build.

It is something of a thought experiment that I am considering taking further and actually seeing if it would work in practice. The idea is that you can have many cheap (think a few dollars) modules that have some processing capacity and can connect to other modules to create a highly parallel computing framework. It would probably use something like the MapReduce approach to allow problems to be distributed over the nodes and solved using the combined power of all nodes. The idea would be that you can keep adding modules (preferably on the fly while the system is running) to increase the processing power of the system. Nodes would share power and data connections with each of their neighbors.
 

Offline Richard Crowley

  • Super Contributor
  • ***
  • Posts: 4317
  • Country: us
  • KJ7YLK
Re: Microcontroller mesh network
« Reply #12 on: January 25, 2015, 02:48:04 am »
Quote
Build A Simple And Inexpensive Controller Area Network Node

Shown here is the schematic for a simple, inexpensive controller area network (CAN) node (Fig. 1). The PIC12C672 was chosen because of its low pin count (8 pins) and powerful feature set. This set includes an internal oscillator, on-board multichannel 8-bit analog-to-digital converter (ADC), multiple interrupt sources, and a low-power sleep mode.

A full CAN 2.0 implementation with message filtering is supplied by the MCP2510 14-pin standalone CAN controller. Therefore, the host microcontroller is relieved from performing any CAN-bus-related overhead. This is a key feature given the limited available code space of the PIC12C672. The PCA-82C251 transceiver was selected arbitrarily because this article focuses on implementing the CAN protocol, and the physical layer is undefined by CAN.

http://electronicdesign.com/boards/build-simple-and-inexpensive-controller-area-network-node

The code is available for download from the URL cited above.

Note that the concept could be applied to other sub-$1 8-pin microcontrollers (Atmel/Arduino, for example).

« Last Edit: January 25, 2015, 02:52:01 am by Richard Crowley »
 

Offline sirhaggisTopic starter

  • Regular Contributor
  • *
  • Posts: 54
Re: Microcontroller mesh network
« Reply #13 on: January 25, 2015, 03:52:10 am »
Great link Richard, I will definitely be looking into a CAN approach. My only concern with the given example is that all nodes would have to share the same network, limiting the total combined bandwidth of all nodes to 1Mbps. When we are considering hundreds of nodes this is probably going to be a severe constraint in most cases.

Currently I am thinking about whether an ethernet transceiver could be combined with a multiplexer to allow multiple ethernet links to be maintained while only requiring a single, cheap ethernet transceiver (rather than 6). The main issue I can see with this approach is how to deal with 'normal link pulses' which are sent every 8-24ms to check if the connection is up (I wonder how fast a transceiver can be activated/deactived to send data on request? Presumably auto-negotiation would have to be disabled?). In my mind, you could have a line from each controller to its neighbor that you set high to indicate you want to send it some data. When the neighbor node is ready to receive data it switches its transceiver with the mux and sets its line high to indicate it is ready to receive data. The sender also switches its multiplexer to connect the ethernet transceiver to that neighbor and a data exchange can occur. Pass through traffic would be handled by the MCU so there would be a latency/power cost, but this could probably be kept to reasonable levels by relying on interrupts.
Perhaps there are others with more knowledge of such details? I have never worked with ethernet at a low level.
 

Online obiwanjacobi

  • Frequent Contributor
  • **
  • Posts: 988
  • Country: nl
  • What's this yippee-yayoh pin you talk about!?
    • Marctronix Blog
Re: Microcontroller mesh network
« Reply #14 on: January 25, 2015, 07:55:06 am »
My thoughts:

I would think of local buses for the connected nodes and off-loading communications to something very cheap (in $ and power consumption). This would allow the node to pass on data without the main processor waking up - saving energy.
I would also think of a protocol that allows adding nodes dynamically without having to manage the ID's or node-clusters manually (with switches or alike) - you would go insane if this gets too large. Again look to existing network protocols /switches for inspiration.

[2c]
Marc
Arduino Template Library | Zalt Z80 Computer
Wrong code should not compile!
 

Offline zapta

  • Super Contributor
  • ***
  • Posts: 6189
  • Country: us
Re: Microcontroller mesh network
« Reply #15 on: January 25, 2015, 09:32:31 am »
This project reminds the Transputer, used to be hot at the time.
 

Offline Mechanical Menace

  • Super Contributor
  • ***
  • Posts: 1288
  • Country: gb
Re: Microcontroller mesh network
« Reply #16 on: January 25, 2015, 11:34:44 am »
Might be helpful if the OP would tell us what they're trying to build.

It is something of a thought experiment that I am considering taking further and actually seeing if it would work in practice. The idea is that you can have many cheap (think a few dollars) modules that have some processing capacity and can connect to other modules to create a highly parallel computing framework. It would probably use something like the MapReduce approach to allow problems to be distributed over the nodes and solved using the combined power of all nodes. The idea would be that you can keep adding modules (preferably on the fly while the system is running) to increase the processing power of the system. Nodes would share power and data connections with each of their neighbors.

I think you'll end up needing a lot more power per node just to manage the network and distribute jobs, let alone actually get any work done. Just for the compute node OS I think you'll soon run out of ram and flash on the MCUs. Or maybe some sort of layout where you have more powerful control nodes that can support a full OS manage up to so many compute nodes would be a decent compromise?
« Last Edit: January 25, 2015, 11:39:12 am by Mechanical Menace »
Second sexiest ugly bloke on the forum.
"Don't believe every quote you read on the internet, because I totally didn't say that."
~Albert Einstein
 

Offline sirhaggisTopic starter

  • Regular Contributor
  • *
  • Posts: 54
Re: Microcontroller mesh network
« Reply #17 on: January 25, 2015, 12:05:19 pm »
Quote
I think you'll end up needing a lot more power per node just to manage the network and distribute jobs, let alone actually get any work done. Just for the compute node OS I think you'll soon run out of ram and flash on the MCUs. Or maybe some sort of layout where you have more powerful control nodes that can support a full OS manage up to so many compute nodes would be a decent compromise?
You may well be right, but that is part of the challenge :)

I do imagine that there could be nodes with different degrees of capability and could take on more of the burden of administrating the network - I am using cortex MCUs so in theory incorporating anything from an M0 to an M5 would be possible without too much extra work. There are probably also ways of arranging the network so that there doesn't have to be a great deal of routing overhead for each module - for example if a module knows its position in space relative to the other modules and the location of the destination module you could use a simple path finding algorithm that doesn't require it to keep track of nodes other than its immediate neighbors. A similar approach could be used to distribute jobs (in theory). I think using some kind of external communication support circuit could lessen the burden of waking up controllers to deal with some of the lower level details of communication. What that circuit could look like is still a bit of a mystery!
 

Offline Mechanical Menace

  • Super Contributor
  • ***
  • Posts: 1288
  • Country: gb
Re: Microcontroller mesh network
« Reply #18 on: January 25, 2015, 12:26:59 pm »
Is there anything like an ARM equivalent of a QUICC processor? I know the 68360 one not only the had the onboard processor that can deal with most networking on it's own but also exposed the 68020's (I think) data and address lines to the pins so you could add more RAM if needs be. Something of that sort of idea could be ideal.

When it comes to the network topolgy and protocols you've also got to decide what sort of work you'll be doing and whether a constant MIPS or "jobs per month" is more important to you.
Second sexiest ugly bloke on the forum.
"Don't believe every quote you read on the internet, because I totally didn't say that."
~Albert Einstein
 

Offline Richard Crowley

  • Super Contributor
  • ***
  • Posts: 4317
  • Country: us
  • KJ7YLK
Re: Microcontroller mesh network
« Reply #19 on: January 25, 2015, 03:08:41 pm »
Is there anything like an ARM equivalent of a QUICC processor? I know the 68360 one not only the had the onboard processor that can deal with most networking on it's own but also exposed the 68020's (I think) data and address lines to the pins so you could add more RAM if needs be. Something of that sort of idea could be ideal.

When it comes to the network topolgy and protocols you've also got to decide what sort of work you'll be doing and whether a constant MIPS or "jobs per month" is more important to you.

Horsepower does NOT appear to be a requirement here.
The main issue appears to be low-cost distributed I/O.
And the second issue appears to be integration and programming.

One downside of a "mesh" of self-contained micro-controllers is programming them properly to interact with all the other nodes.
It almost makes me think it would be better to have a single central processor that considers all the input parameters,
and makes the decisions and sends out the data to operate the system.  It doesn't take much processing power to do that.

If I were doing this, my first guess would be a second-hand PC as a central processor,
something easy to control, and with the capability of presenting a nice UI on the screen,
and some industry-standard interface protocol like I2C or maybe RS-485,
and using some standard wiring format like UTP/RJ-45 (i.e. common network cabling)

There is quite probably little I/O boards out there already available that implement this kind of thing.

This is not all that different from most industrial control systems where they have been doing this for decades.
And with systems with serious consequences if things aren't done properly.

« Last Edit: January 25, 2015, 03:11:02 pm by Richard Crowley »
 

Offline Mechanical Menace

  • Super Contributor
  • ***
  • Posts: 1288
  • Country: gb
Re: Microcontroller mesh network
« Reply #20 on: January 25, 2015, 03:49:07 pm »
Horsepower does NOT appear to be a requirement here.
The main issue appears to be low-cost distributed I/O.
And the second issue appears to be integration and programming.

Yeah, just properly read stuff like I should before posting that lol. Sorry folks.
Second sexiest ugly bloke on the forum.
"Don't believe every quote you read on the internet, because I totally didn't say that."
~Albert Einstein
 

Online tszaboo

  • Super Contributor
  • ***
  • Posts: 7282
  • Country: nl
  • Current job: ATEX product design
Re: Microcontroller mesh network
« Reply #21 on: January 25, 2015, 04:01:09 pm »
Quote
I think you'll end up needing a lot more power per node just to manage the network and distribute jobs, let alone actually get any work done. Just for the compute node OS I think you'll soon run out of ram and flash on the MCUs. Or maybe some sort of layout where you have more powerful control nodes that can support a full OS manage up to so many compute nodes would be a decent compromise?
You may well be right, but that is part of the challenge :)

I do imagine that there could be nodes with different degrees of capability and could take on more of the burden of administrating the network - I am using cortex MCUs so in theory incorporating anything from an M0 to an M5 would be possible without too much extra work. There are probably also ways of arranging the network so that there doesn't have to be a great deal of routing overhead for each module - for example if a module knows its position in space relative to the other modules and the location of the destination module you could use a simple path finding algorithm that doesn't require it to keep track of nodes other than its immediate neighbors. A similar approach could be used to distribute jobs (in theory). I think using some kind of external communication support circuit could lessen the burden of waking up controllers to deal with some of the lower level details of communication. What that circuit could look like is still a bit of a mystery!
I was thinking about something similar, I wanted an extendable LED game of life simulator. That is communication in 8 directions, but with repeating, it is possible to reduce it to 4.
I've found that cheap microcontroller with existing identical communication interface is not an easy task, then the Atmel SAM D20/21 came out. They have 6 SERCOM, flexible serial communication modules. That seemed to be perfect for my purposes, you should take a look. Set it up as SPI or USART, point to point, and it is done nice and simple. Hope you are fine with the limitations.
 

Offline edavid

  • Super Contributor
  • ***
  • Posts: 3375
  • Country: us
Re: Microcontroller mesh network
« Reply #22 on: January 26, 2015, 01:24:36 am »
It is something of a thought experiment that I am considering taking further and actually seeing if it would work in practice. The idea is that you can have many cheap (think a few dollars) modules that have some processing capacity and can connect to other modules to create a highly parallel computing framework. It would probably use something like the MapReduce approach to allow problems to be distributed over the nodes and solved using the combined power of all nodes. The idea would be that you can keep adding modules (preferably on the fly while the system is running) to increase the processing power of the system. Nodes would share power and data connections with each of their neighbors.

Why?  This is an inefficient way to build a distributed computer.  Using such cheap nodes reduces the cost effectiveness of the overall system.
You might as well buy a uniprocessor that's as fast as all the nodes put together.
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 26682
  • Country: nl
    • NCT Developments
Re: Microcontroller mesh network
« Reply #23 on: January 26, 2015, 11:04:31 am »
On further thought however, it would seem that I would need one ethernet transceiver for each neighbour connection, which means six of these per node. I'm looking into the possibility of ethernet multiplexing, but I'm not sure if that would actually work or not.
If each node has short connections I strongly advise to use 1 microcontroller per node. Distributed systems are a pain in the ass to get working correctly due to all the possible error conditions.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline splin

  • Frequent Contributor
  • **
  • Posts: 999
  • Country: gb
Re: Microcontroller mesh network
« Reply #24 on: January 26, 2015, 02:29:15 pm »
But what application code would you run on each node? If its fixed then fine, each node would be programmed from the outset with the problem solving code, with the problem data being distributed across the network to be processed.

But if you want this creation to be useable for different problems you'll presumably need to distribute new code to every node. Most low end Cortex devices have very limited RAM, so unless the application code is very small, each node will have to write the code into its Flash memory and run it from there. Certainly possible but perhaps rather slow to reprogram the whole network and you may also run into problems due to the limited number of rewrites of the Flash.

Splin
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf