Required protocols

[Mtech1]

I'm currently trying to understand the necessary protocols for a server-client program for any given scenario, mainly asked for educational purposes.

In this scenario, we have a PC, a router, a microcontroller, an Ethernet module, and a temperature sensor.

Our goal is to send temperature data to a server using HTTP. The PC will be running the server program, while the microcontroller will run the client program. The temperature sensor operates on the I2C protocol, and suppose I've already written a client program that reads the temperature using I2C.

Based on my understanding, I believe that in addition to the I2C protocol, we also need to implement IP and TCP and HTTP protocols for communication.

Please let me know if this setup makes sense or if there are other protocols I should consider.

Thank you

[nctnico]

Do yourself a favour and use a Wiznet chip for the ethernet + TCP/IP communication part. HTTP sits on top of TCP/IP

[Nominal Animal]

Using raw UDP/IP instead of TCP/IP and HTTP between the server and the microcontroller for polling the sensor or sensors, saves you both resources and implementation effort.

UDP/IP makes most sense for polled sensors, because there is no connection per se, and each message (or datagram) is independent.  (The datagram header part contains both the sender and recipient IP addresses and ports, and these are accessible to both the microcontroller and the server-side program running under a fully-features operating system.)

UDP and IP stack implementation is much simpler than TCP/IP on the microcontroller.
Each UDP datagram must have at least one byte of data payload, because some implementations do not handle zero-data datagrams correctly.  In general, up to 556 bytes of UDP data (payload) can be supplied in each datagram without having to worry about IP fragmentation.

The downside of UDP/IP compared to TCP/IP is that there is no reliability guarantees, and routers may simply drop UDP datagrams if they're overburdened.  This rarely occurs in local area networks, though, and is easily handled by the server side re-sending the request if there is no response within a specified time limit, for example 15 seconds.  Again, for polled stateless sensors, UDP/IP just makes more sense.

The idea is that the datagram sent to the microcontroller is a query, perhaps identifying the types or sensors or units of measurement, and the datagram the microcontroller sends to the sender of the query is the corresponding response.

The content and the format of the query and response messages is something one should spend effort to design correctly, to allow expansion and for example multiple sensors and sensor types on a single microcontroller.

The above would therefore use UDP/IP for communications between the server and the microcontroller, and I2C or Dallas 1-wire for the communications between the microcontroller and the temperature sensor.

For cryptographic security, external chips like Microchip ATECC608B (cost less than USD $1 even in singles at vendors like Mouser and Digikey) can be used to implement proper encryption with the key (for example, one half of a key pair) stored on the chip.

As to the server software running under a fully-featured OS, it would make more sense for it to poll a configured set of sensors, archiving their readings, and exposing access to the archived (and current) data to a FastCGI/WSGI/etc. application serving HTTP requests under a HTTP(S) server like Apache or Nginx.
A common option is to use an SQL database to store the sensor readings.  Then, the sensor server and the HTTP application would use SQL to a database service to access the sensor readings, and the application would use FastCGI/WSGI/etc. to talk to the HTTP(S) server (although this detail is often well implemented in libraries), with the HTTP(S) server configured to handle TLS security and authentication and access controls.  Those are the protocols needed.  Additional formats and languages are needed by the application to present the data as graphs: HTML for the page itself with CSS for fine-tuning visuals, and Javascript (if the graphs are generated on the client side as recommended since even phones have ample capabilities for this) or SVG (for generating precise vector-graphic scalable graphs on the server side or embedded in the HTML itself).  Such applications are easiest to write in Python, PHP, or Ruby.

[Mtech1]

Sure, I will consider the chip that you suggested, but at this time, I'm just trying to understand the basics that are needed before diving into any coding

[Mtech1]

Thank you for the detailed explanation. Just to confirm, if we have multiple clients, each with their unique IP addresses, and they want to to a server,  Do we need IP protocol or include TCP. Is that the correct understanding?

[nctnico]

You need to understand what kind of server and what protocol is needed. Go talk to the people who are going to write the software which runs on the server. It depends on their skill set what kind of protocol is best to use in terms of engineering time & abilities of people involved.

Likely the easiest is to implement HTTP get requests that convey the information to a webserver. If this information needs to go over internet or an otherwise insecure network, then the least intrusive way is to use off-the-shelve VPN boxes that create an encrypted pipe for the communication.

But first: try to find overlap between the skill sets of the people who are going to work on this project versus the project's requirements (don't forget to ask about security requirements).

[Siwastaja]

Yes, coordinate it well with the server people. Sensible options include raw UDP datagrams, raw TCP socket, or MQTT over TCP if you think you benefit from the existing MQTT tools or TLS authentication offered by the MQTT broker.

[Mtech1]

You need to understand what kind of server and what protocol is needed. Go talk to the people who are going to write the software which runs on the server. It depends on their skill set what kind of protocol is best to use in terms of engineering time & abilities of people involved.

let's considering security aside for now
 assume the server supports TCP/IP, WebSocket, MQTT, and HTTP. 


If you follow the bottom-to-top approach, and since you already have code to read temperature using I2C  which sequences of the protocols would you  implement ? I think IP, TCP and HTTP would sequence needs to implement to achieve goal.

 If my understanding is correct and I can ask next doubt

[nctnico]

If your end goal is to make HTTP requests to a server you'll need to have the following networking protocols implemented:

Base ethernet protocols:
- IPv4 and/or IPv6
- ARP

Then your transport & management protocols:
- TCP/IP
- UDP
- ICMP

Then high level protocols like:
- DNS
- DHCP (if you want to support dynamic IP addresses)
- HTTP client

However, do not try to add security as an afterthought. You can't add security afterwards as good security will very likely affect the way you have to implement your entire system. Security is not just encrypting data but also having a means to limit access to a system and detecting somebody is tampering with the system.

[Nominal Animal]

if we have multiple clients, each with their unique IP addresses, and they want to to a server,  Do we need IP protocol or include TCP.

With TCP/IP or UDP/IP, you can have multiple servers and multiple clients.

I do believe that HTTP is the wrong protocol to interface to the sensor microcontroller, however.  WebSocket I could accept, if the intent is to have any user interface directly to the sensor via their browser.

With raw UDP datagrams, the high-level loop in the microcontroller waits for an UDP datagram to arrive.  When one does, it checks the data payload to see if it is a valid request, and if it is, copies the source address and port from that datagram to a new, outgoing UDP datagram, and fills its data with the sensor reading(s), and sends it. With unsecured communications (on an assumed protected LAN), this is the simplest to implement.

When an external crypto support chip like ATECC608B is added –– I mention this, because you can find lots of examples of how to use one in the Arduino environment; some boards even include it (or the previous model, ATECC508) by default –– you'll want to add queues, so that the crypto chip can work while the microcontroller is doing something useful.  If you draw a state machine diagram, separating things like "request decryption", "request verification", "response construction", "response encryption", and "response sending", you'll see how you can keep both the MCU and the crypto IC working at the same time, whenever there is more than one UDP datagram pending.

The difference between TCP and UDP –– TCP/IP meaning TCP connection over IP networking, and UDP/IP meaning UDP datagrams over IP network –– is that UDP is a connectionless protocol, throwing datagrams around that may be dropped along the way during congestion; whereas TCP is a connection-oriented protocol that involves handshaking and transparently handling retransmissions, so that it looks like a full-duplex serial connection to the programmer.
A TCP/IP stack requires a lot more resources than an UDP/IP stack does, but there things like the Wiznet chips nctnico mentioned that implement almost all of that internally, and your microcontroller only handles the datagrams for UDP, or new connections and data and closing connections for TCP.  Because cryptography involves the data only, really, these ethernet chips are "fully compatible" with crypto chips, because the two operations are really quite separate.

[Siwastaja]

Two high-level design choices which affect everything else:

1) Need security or not? Decide it early on, instead of trying to add security as afterthought as nctnico says. Security is partially choosing and understanding correct libraries, ciphers and configuring them, but also a full process which goes deep into manufacturing, how keys and certificates are managed and so on.

2) Need direct connections to the device, or does everything always go through server? If the end users (and yourself, in admin mode etc.) are fine doing everything through server, then you can make the device-server interface really, REALLY simple (e.g., raw UDP packets or raw TCP socket with simple binary protocol, preferably packed C structs just to annoy nctnico ) and leave the API / UI / UX concerns completely on the server/client pair which is totally separate from the microcontroller and the server-MCU interface.

On the other hand, if your users want to access the device "directly", then you have to listen to your users and implement more stuff on the microcontroller side.

[tellurium]

OP,

Consider https://github.com/cesanta/mongoose/
It does what you want.

Wiznet example for SAMD21 (which you can repeat on pretty much any micro) https://mongoose.ws/documentation/tutorials/arduino/w5500-http/
Also, MQTT counterpart: https://github.com/cesanta/mongoose/tree/master/examples/arduino/w5500-mqtt

Disclaimer - I work for the company that develops it.

[Mtech1]

I see it's important to decide TCP or UDP for any application

From a bottom-to-top approach, we should select TCP or UDP.

I have one use case in my mind. Consider we want to develop a system that should control home lights, TV, fan, from Android  APP as well as send live video to a remote server.

What would be your preferred choice, TCP or UDP?

[HwAoRrDk]

Likely the easiest is to implement HTTP get requests that convey the information to a webserver.

I would suggest not using GET requests, but instead use POST requests. That is the proper HTTP method for submitting data to the server.

Then your transport & management protocols:
- TCP/IP
- UDP
- ICMP

Then high level protocols like:
- DNS
- DHCP (if you want to support dynamic IP addresses)
- HTTP client

Some supplementary notes with regard to what nctnico said:

ICMP: While not strictly required for the scenario where you are simply sending data to a local server on the same LAN, support for this is advisable if you want to be able to handle exceptional conditions when communicating with other networks (i.e. through a router). For example, if a router doesn't support the MTU size you're sending, and the IP packet has the Don't Fragment (DF) flag set, it'll send you a Fragmentation Needed ICMP message. There are also other things like a router sending Destination Unreachable if it can't forward your packets. ICMP messages are also used for 'ping' and 'traceroute' diagnostics, which you may want to implement as troubleshooting features.

DNS: If you're only ever going to address the destination server by IP address, then this is obviously unnecessary. But if you plan to address the server by hostname (e.g. 'foo.bar.local'), then you need to implement DNS name resolution via a nameserver.

One thing I would suggest is to study and become familiar with the OSI network layer model. You'll find things like IP referred to as 'layer 3', TCP as 'layer 4', or HTTP as 'layer 7'; the layer model is what the numbers are referring to.

[Nominal Animal]

Here is an example diagram of how one might use a relatively low-powered microcontroller with an external crypto chip like ATECC608B and external TCP/IP and/or UDP/IP stack as implemented by e.g. some Wiznet chips, collected to an SQL database on a server where a web server-side application under a HTTP server like Apache or Nginx provides a view to the sensor readings to human users via browsers:
               Microcontroller                                    Server
     ┌───────────────────────────────┐         ┌────────────────────────────────────────────┐
     
     sensor ───A─── MCU ───C─── Wiznet ═══D═══ sensorservice    serverapp ───F─── HTTP server ═══G═══ Web clients
                     │                                    │      │
                     B                                    E      E
                     │                                    │      │
                   Crypto                               SQL Database

• A: Typically I²C or Dallas 1-wire to a temperature sensor; SPI bus can also be used.
• B: ATECC608B uses I²C or a single-wire connection to a microcontroller.
• C: Typically SPI is used with Wiznet chips, although I suppose UART and I²C variants also exist.
• D: Typically TCP/IP or UDP/IP.
• E: SQL connection, handled typically by an interface library to hide the details.
• F: Typically FastCGI (or for Python, WSGI).  Details often handled by a standard library interface.
• G: Typically TLS-encrypted HTTP over TCP/IP.  Encryption can be omitted, but is not recommended, as current browsers can even complain.  Virtual private servers can use Lets Encrypt certificates; local and development environments can use self-signed certificates you import directly to the browser.

Note that there can be many servers, clients, and microcontrollers on the same network, with many concurrent D and G connections.

Similarly, a single MCU may have a number of different sensors; typical I²C and Dallas 1-wire buses typically support several sensors on the same chain (but for I²C, you do need sensors whose addresses you can modify so that each address on the same bus is unique).  There are also I²C expanders like TCA9548A that allow you to split a single I²C port on an MCU to 8 separate ones (of which only one is active at the same time), allowing you to connect up to 8 sensors having the same I²C address.  (Of course, TCA9548A and similar chips can also be chained, giving you more alternate buses.)

Others have suggested other alternatives; I find that good.  There is no single right way of doing this.  The one I've described is the minimal one I can think of without compromises, that's all.  Using TCP/IP and TLS, and even existing libraries like Mongoose et al. can be a better starting point in practical terms; I only ask you to remember that those are not the minimum requirement, just perhaps an easier or more robust starting point.
(There are too many people claiming you need X to do Y because Z all over, without really understanding the exact requirements, nor how they translate to hardware.  I'm sure somebody will barge in to this discussion and claim that ARM processors and SBCs are not sufficient for servers, since they cannot run Windows, or something.)

In particular, there are also cheap small SBCs with Ethernet and I²C buses on GPIOs supported in Linux you could use, by simply connecting the temperature sensor(s) to the SBC directly, and it acting as the server.  A Rock Pi S, for example, has a nice Rockchip RK3308 SoC and three I²C and two SPI buses, so you could easily run a small Apache/Nginx installation even with the server application directly polling the sensors.  You wouldn't need a microcontroller at all, then.

[nctnico]

I do believe that HTTP is the wrong protocol to interface to the sensor microcontroller

The problem is that anything else will need some kind of dedicated protocol implementation. From my experience with these kind of projects: web developers are a dime-a-dozen where finding somebody able to implement a dedicated UDP or TCP/IP protocol will be much harder (in addition to defining a protocol). At the microcontroller side it doesn't add / remove much in terms of complexity whether you implement an HTTP get request or a proprietary protocol.

[Siwastaja]

I do believe that HTTP is the wrong protocol to interface to the sensor microcontroller

The problem is that anything else will need some kind of dedicated protocol implementation.

Wat? Last time I looked, HTTP did absolute nothing to simplify or standardize sensor networking. You have to implement "some kind of dedicated protocol" on top of HTTP, plus do everything else as required by HTTP.

[nctnico]

I do believe that HTTP is the wrong protocol to interface to the sensor microcontroller

The problem is that anything else will need some kind of dedicated protocol implementation.

Wat? Last time I looked, HTTP did absolute nothing to simplify or standardize sensor networking. You have to implement "some kind of dedicated protocol" on top of HTTP, plus do everything else as required by HTTP.

With HTTP you can do a get request like htttp://1.2.3.4/my_sensor.php?value=12336
In the PHP script the value is available in a variable and can be put into a database for retrieval. It doesn't come much more standarised or simpler.

[Siwastaja]

I do believe that HTTP is the wrong protocol to interface to the sensor microcontroller

The problem is that anything else will need some kind of dedicated protocol implementation.

Wat? Last time I looked, HTTP did absolute nothing to simplify or standardize sensor networking. You have to implement "some kind of dedicated protocol" on top of HTTP, plus do everything else as required by HTTP.

With HTTP you can do a get request like htttp://1.2.3.4/my_sensor.php?value=12336
In the PHP script the value is available in a variable and can be put into a database for retrieval. It doesn't come much more standarised or simpler.

Definitely useful. I thought we were discussing the interface between server and MCU, my bad. This is why, as I said, the OP needs to decide whether or not it is required/useful to directly communicate human<->MCU, or if it is always human<->server<->device. In latter case, device does not need to implement HTTP but can be much simpler, say something as simple as just transmitting the sensor value every five seconds to a predefined UDP port of the server, or keep a raw TCP socket open and output comma-separated decimal numbers (this time I won't suggest packed C struct).

[nctnico]

I do believe that HTTP is the wrong protocol to interface to the sensor microcontroller

The problem is that anything else will need some kind of dedicated protocol implementation.

Wat? Last time I looked, HTTP did absolute nothing to simplify or standardize sensor networking. You have to implement "some kind of dedicated protocol" on top of HTTP, plus do everything else as required by HTTP.

With HTTP you can do a get request like htttp://1.2.3.4/my_sensor.php?value=12336
In the PHP script the value is available in a variable and can be put into a database for retrieval. It doesn't come much more standarised or simpler.

Definitely useful. I thought we were discussing the interface between server and MCU, my bad. This is why, as I said, the OP needs to decide whether or not it is required/useful to directly communicate human<->MCU, or if it is always human<->server<->device. In latter case, device does not need to implement HTTP but can be much simpler, say something as simple as just transmitting the sensor value every five seconds to a predefined UDP port of the server, or keep a raw TCP socket open and output comma-separated decimal numbers (this time I won't suggest packed C struct).

No, the device (MCU) makes http requests to the server. Almost every tcp/ip stack for MCUs I've seen comes with an HTTP client example which shows how to implement this. https://docs.wiznet.io/Product/Open-Source-Hardware/http_client

Again, if you are going to create your own proprietary UDP or whatever protocol, you are adding unnecessary complexity for the people implementing the server side. Also think about tooling & debugging for that protocol. You can use any browser to test the server side by making GET requests manually (or scripted if you like using command line tools). The webserver will logs the requests nicely for you (if configured to do so) so you can see all the requests and look for problems (manually or automated through scripting). If you choose POST requests, then the browser likes to see some kind of form to POST. This form then needs to be updated as well when the fields change. Using a GET request avoids all this extra work.

[abeyer]

With HTTP you can do a get request like htttp://1.2.3.4/my_sensor.php?value=12336
In the PHP script the value is available in a variable and can be put into a database for retrieval. It doesn't come much more standarised or simpler.

This will almost certainly drop and/or duplicate data points under some circumstances. Use the POST, Luke.

[nctnico]

With HTTP you can do a get request like htttp://1.2.3.4/my_sensor.php?value=12336
In the PHP script the value is available in a variable and can be put into a database for retrieval. It doesn't come much more standarised or simpler.

This will almost certainly drop and/or duplicate data points under some circumstances. Use the POST, Luke.

A simple sequence number does wonders.

[tellurium]

I just can't get rid of the feeling that this is just another one AI bot discussion

[Siwastaja]

I would strongly recommend using XML over HTTP, with XML parser hand-written in assembly for the target microcontroller. Only this way you get acceptable performance. Incorporating the PPP protocol into the XML payload, using base64 encoding for endianness swap increases throughput and data security. Remember though that TLS1.3 is outdated; use SSL2.0 instead.

[krho]

I would strongly recommend using XML over HTTP, with XML parser hand-written in assembly for the target microcontroller. Only this way you get acceptable performance. Incorporating the PPP protocol into the XML payload, using base64 encoding for endianness swap increases throughput and data security. Remember though that TLS1.3 is outdated; use SSL2.0 instead.

Another vote to use MQTT