AWS S3 - mutual TLS 1.2 authentication for e.g. 100+ IoT devices?

[Martin F]

Hi all,

We're working on our new data logger, which will be sending data from an SD card to a server using HTTPS (TLS 1.2).
Currently, we're aiming to have it link up with an AWS S3 server as well as a Minio server as case examples.

Log files will be transferred to a single bucket by potentially hundreds of individual data loggers in the field - each customer will set up their own server solution, so we just want to ensure that our customers can set this up right.

In this regard, one critical aspect for us is to be able to manage individual user credentials across each logger.
Specifically, should one logger be compromised, we want to be able to "remove" the access from that logger to the server - without having to make changes to the other loggers.

We're a bit in doubt of whether/how this is possible in AWS S3:
- Can we simply create a new user for each logger and then remove a compromised logger under users?
- Is there a limit to the #users per bucket in AWS S3?

For Minio, it seems only one user per bucket is possible - anyone knows if this is correct?

Thanks a ton for your inputs on this!

Martin

[jeremy]

How much data are you actually logging? Hundreds of data loggers performing one small request per second is a trivial amount of data. I’ve not heard of someone using S3 buckets like this before, where every device has its own user.

On security, if you are using HTTPS most of your problems have been solved. If you really want to ensure data is authorised, you should sign the payloads with a ECC-DSA with a certificate that is unique to a particular device. Then you send this signature with the data over HTTPS.

But all of these problems are already solved by AWS IoT, so perhaps you should look into that?

[Martin F]

Hi again, thanks for your inputs!

Log files may be 5-10 mb per file, transferred periodically (eg one per 20 min per logger) or in bursts if a logger has been out of WiFi range for a period.

We were of the impression that AWS IoT was more for cases with smaller payloads. Also, we’d like a solution concept that is fairly “generic” so that it could work on AWS S3, a Minio server, or some other cloud serve aka Google Cloud. Not sure if this is possible, but I’d assume relying on AWS IoT methodology would be limiting to clients that want to use something else than AWS for their setup.

[jeremy]

Ok, you are right, that is too big for AWS IoT I think.

Is the system microcontroller or Linux based?

[sokoloff]

Currently, we're aiming to have it link up with an AWS S3 server as well as a Minio server as case examples.

Log files will be transferred to a single bucket by potentially hundreds of individual data loggers in the field - each customer will set up their own server solution, so we just want to ensure that our customers can set this up right.

In this regard, one critical aspect for us is to be able to manage individual user credentials across each logger.
Specifically, should one logger be compromised, we want to be able to "remove" the access from that logger to the server - without having to make changes to the other loggers.

We're a bit in doubt of whether/how this is possible in AWS S3:
- Can we simply create a new user for each logger and then remove a compromised logger under users?
- Is there a limit to the #users per bucket in AWS S3?

I would use AWS Cognito for user management. At your volume, I believe it would free forever, or a nickel per user per month if you're using a generic SAML integration.

Then, use Cognito access control to your S3 bucket (or buckets) as in this example. There is no practical limit and certainly no limit in the three digits number of users.

[Martin F]

Hi again,

Thanks!

Just to clarify: We'll not be hosting the server and it won't necessarily be via AWS S3.
Some customers may want to set up their own dedicated server, in which case we want to recommend using e.g. Minio for this.

In other words, we need our solution to be server-independent - and the mutual authentication solution should ideally work for both Minio and AWS, Google Cloud etc.

One solution we've considered is NGINX, but it seems a bit cumbersome to set up on AWS. But maybe we're overkilling it and there is a simpler solution.

Any tips are welcome!

[jeremy]

Hi Martin,

It sounds like you are on the right track (at least in my opinion). For a super simple approach, you could use FTP over SSL aka FTPS? There are tons of servers available for basically every operating system. It would be easy for you or a client to set up in their own infrastructure (Windows even supports this out of the box I think). Lots of FTP servers also have a GUI, particularly on Windows, so it would be easy for an IT guy to add or delete users. You can also integrate them into existing LDAP/Active Directory authentication systems. I'm not particularly fond of Windows, but lots of businesses use it, so might as well try to fit into their processes as much as possible 

I would be more concerned about how you handle transfer interruptions, etc that leave you with corrupted data. TCP will make sure that your packets are not corrupted in transit, but it can't guarantee that the packets will make it to the endpoint if the wifi drops out. In my experience, 5-10MB per chunk is a lot of data to transfer over a potentially lossy connection. You will need some way to checksum the data on the server to see if it is valid, and I fear that will require writing some custom software. It doesn't appear that you can reliably get a hash out of S3 for a file on the server (well, you can get a "hash" via the ETag, but the algorithm used isn't always something that you can compute on both ends). There is a HASH command extension to FTP, and it does have experimental support in filezilla, but it's hard to tell how many servers actually support it.

There are many options for forwarding the data from FTP to S3/Google Cloud/etc if you really need that, for example: https://www.thorntech.com/products/sftpgateway/

With FTP you would also avoid vendor lock-in.

PS not to nitpick, but I think the more idiomatic expression is something along the lines of "But maybe this solution is overkill, and there is a simpler one available". I haven't heard "overkilling" being used before

[sokoloff]

You will need some way to checksum the data on the server to see if it is valid, and I fear that will require writing some custom software. It doesn't appear that you can reliably get a hash out of S3 for a file on the server (well, you can get a "hash" via the ETag, but the algorithm used isn't always something that you can compute on both ends).

It would be trivial to write a Lambda function that read the S3 object, hashed/checksummed it with whatever function you needed, and returned that value (and/or decorated the object with the hash as a property on the object). Lambda for a use case like this is extremely cheap. You could also invoke Lambda to auto-process the object in S3 after upload if you needed to kick it off after the PUT finishes.

If you MD5 checksum the file on the client, you can pass that MD5 on the [optional] Content-MD5 header of the S3 PUT operation and S3 will MD5 checksum the file upon completion and return an error on the PUT if it doesn't match. (@jeremy is 100% correct that the ETAG is not always the MD5 hash of the object and so you must not rely on the happenstance that it frequently is for single-part uploads.)

Lots of options, most all of them easy and cheap.

[jeremy]

If you MD5 checksum the file on the client, you can pass that MD5 on the [optional] Content-MD5 header of the S3 PUT operation and S3 will MD5 checksum the file upon completion and return an error on the PUT if it doesn't match.

That's very neat, I didn't know that. Thank you for sharing it.