What is the best protocol to use for IoT device with Big Data? (FTP, HTTP, MQTT)

[StuUK]

Implement SFTP and your own endpoint in whichever cloud service you wish to use and avoid Cloud Implementation specifics... Simple, straightforward, reliable and secure...

[xani]

From mentioned ones:

HTTP(S) would probably be easiest for anyone to write integration with. If you used some simple URL scheme like https://example.com/apploger/device-id/can/data.log. It is ok for transferring large files, but no way to resume it so you might want to limit it to something reasonable (so say one file size is ~minute or two of transfer time) in case of device being on wonky connection

MQTT makes sense if you want to send it to multiple places at once but it would be harder to write as MQTT is primitive as hell and writing server part would be harder too. Probably better to stick to HTTP because whoever wants, can do HTTP-MQTT GW

SCP/SFTP is dedicated to file transfers so if you only want to "put files in a place", it has good security and very secure ssh key-based authentication. But there is not much in a way of application integration so the question is if you want to push files for archive purpose or to parse it in flight. AFAIK there is a builtin ability to resume upload too

I'd go with HTTP if you want to analyse data in some app near-realtime, but for just archiving purposes SCP/SFTP is just as good if not better choice (as you don't *have* to write app, just use some sftp server)

[Martin F]

Thanks again all of you!

I think we're aiming for HTTP/HTTPS. Based on some digging, we've considered https://www.minio.io/ as an option for the file server piece - though if some of you have experiences or recommended alternatives, let us know!

Martin

[nctnico]

Implement SFTP and your own endpoint in whichever cloud service you wish to use and avoid Cloud Implementation specifics... Simple, straightforward, reliable and secure...

Not really. The S in SFTP, HTTPS, etc stands for SSL. SSL is a relatively complex protocol and there have been several succesful attacks on SSL in the past so it needs regular patching.

[krho]

@Martin F: I have already given you a solution, tus. There is tusd that's written in golang. adapting it to support the https and serve the files to someone authenticated shouldn't be more than a 2 days work even for a person who has never written in golang.

IMO someone at your company could do it or I can do it as a freelance for $

[jbb]

Because you will be sucking up quite a lot of data, I highly recommend the use of encryption and authentication. After all, lots from the CAN bus could scrape up a lot of information which might be  sensitive. Unfortunately this is a complicated topic, and i’m not an expert (or a lawyer).

Here are some things  I can say:
- I applaud your goal of putting a server on the client site, or even letting them run their own. But it can be difficult to manage this (especially when you need to update stuff.) How about you offer a pre-configured site server box?
- SSL is now out of date and should be replaced by TLS 1.2
- SSL / TLS (and I suspect SFTP and TUS) have a bunch of ciphersuite options. Some of these options are weak (e.g. RC4) and should be turned off on both the server and client sides. Please read the configuration documentation carefully.
- passwords and encryption keys should be a) updateable and b) unique per device
- firmware should be updateable and cryptographically signed