AWS S3 - good guides for getting started from scratch?

[Martin F]

Hi all,

We're producing a data logger and we're going to gradually start looking more into AWS S3 as some of our clients wish to use this.

We are totally from scratch when it comes to setting up Cloud servers - hence I wanted to hear if any of you got good tutorials, videos, articles etc. you can recommend to get from "zero to hero" in this topic?

The aim is to get a basic guidance for setting up an account, creating an S3 bucket, getting started with key management and understanding basics of objects, tags etc in their API - e.g. via Ruby.

Any suggestions would be super helpful!

Thanks,
Martin

[sokoloff]

AWS documentation is absolutely world-class, IMO.

https://aws.amazon.com/getting-started/tutorials/

IAM is a little hard to wrap your head around, and is a place where I often go search out examples/templates.
S3 is dead-easy.

Sign up for an account. (They give you a free tier for a lot of services for 12 months. Light, experimenting may end up costing you $0/mo.)
Make a bucket manually.
Setup an IAM role. Give it s3:* for all you care right now [you need to lock it down later, maybe, but with a new IAM role, you're good to go]
Manually upload a file. https://aws.amazon.com/getting-started/tutorials/backup-files-to-amazon-s3/
Look at the S3 examples, which include Ruby sample code: https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/s3-examples.html

Really, you want to just get started doing it. The AWS documentation is excellent and is therefore primary, IMO.

[rob.manderson]

AWS documentation is absolutely world-class, IMO.

https://aws.amazon.com/getting-started/tutorials/

IAM is a little hard to wrap your head around, and is a place where I often go search out examples/templates.
S3 is dead-easy.

Sign up for an account. (They give you a free tier for a lot of services for 12 months. Light, experimenting may end up costing you $0/mo.)
Make a bucket manually.
Setup an IAM role. Give it s3:* for all you care right now [you need to lock it down later, maybe, but with a new IAM role, you're good to go]
Manually upload a file. https://aws.amazon.com/getting-started/tutorials/backup-files-to-amazon-s3/
Look at the S3 examples, which include Ruby sample code: https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/s3-examples.html

Really, you want to just get started doing it. The AWS documentation is excellent and is therefore primary, IMO.

Totally agree.  I did a document archive service for our software product back in 2012 using AWS S3 and the S3 part went together in about a day.  The remaining 2 months were spent making sure we didn't delete the document from the database until it was actually *in* S3, making sure we could cope with S3 being down and testing the thing to death.  In the 6 years since we've had two outages of S3 and maybe a dozen cases where S3 closes the connection.

There seems to be a limit on document size that I don't recall seeing in the docs (I haven't looked at them in 6 years).  S3 deliberately throttles upload speeds (there has to be a reason why they're so damn cheap!  ) and there seems to be a 10 minute timeout (if the streaming doesn't finish within 10 minutes S3 will close the connection).  This is absolutely repeatable.  We had a 196 Meg document that would error out every single time.  We've successfully uploaded 100 Meg documents but fortunately there are relatively few of those in our database.

[Martin F]

Great input, thanks!

We're developing our data logger to push via HTTPS data log files to an S3 type file server with varying frequency / size.
Expect the data to be between 0.5 MB per file and 20 MB per file, with highest upload frequency being per second.

As part of this, we're looking into the security aspect of connecting up to hundreds of IoT type "nodes" to a network, incl. the implications for setting up the security and key management in an S3 context (e.g. AWS, but we're also looking at Minio to use the S3 API, but with potentially another server than AWS).

Any good primers for IoT security management in an S3 context?

Thanks!