3G/4G IoT module with 'continuous upload' disconnects after a few days

[Martin F]

We are developing an IoT with 3G/4G connectivity. It uploads a small log file containing CSV data via 3G/4G. We use a u-blox LARA LTE module to perform the upload. The upload is done to an AWS S3 bucket. The SIM card we use is a regular locally sourced data SIM with 50 GB/month.

The IoT device uploads data without issues for 3 days, i.e. consistently uploading a new log file every 10 seconds to S3.

However, after ~3 days it suddenly stops connecting to the S3 server at all and appears to be making no attempts at uploading (as seen from the S3 server logs at least). After power cycling the device, however, it starts uploading again.

Our hypothesis is that this may be linked to some form of "best practice" or "cellular data usage" guidelines that may require devices to periodically disconnect from the network - in our case, the device will be continuously connected in the above setting. However, we're unable to find general evidence/guidelines for this.

Any experience with this?

[Halcyon]

I'm not an expert in the way cellular networks are designed, but I would expect there to be some kind of limitation (enforced or otherwise).

If the interruption is occurring at a regular interval, have you considered perhaps the IP address lease is expiring? Do you see various different IP addresses in the logs throughout that 3 days, or just one? Does it change after a reboot?

Could there be something else occurring at the client's end causing it to lock up? Buffer filling up, service crashing, etc...?

Without some more technical details and log files to go over, it's really just a guessing game.

[AndyBeez]

I agree, without log files the issue could be anything from an IP pool refresh or subnet change on the tower, a buffer overflow (segfault) on the IoT device, AWS server config not liking persistent connections or, even 'critters'. How do you upload? Push with FTP, http post, SCP, or pull with an always open port at the AWS end? Are you using a DynamicDNS service? A quick fix might be a reboot script on the IoT device that runs every 24hrs.

EDIT: You mention nothing about the terrain or the EMC environment the IoT device is operating in. I might suspect the problem is the cell tower is hanging up your connection as it assumes you are a cell phone that has been accidentally connected for too long. We don't know? Is the IoT device still running or has it crashed? The quick solution for network loss is to have the IoT device examine the HTTP response code from the upload attempt. If it is anything but OK/200, hang up the modem and reconnect. Remembering to buffer the data needing to be sent. 

[mendip_discovery]

From the device side are you doing any checks to ensure the data is uploaded and what to do when it stops uploading?

[AlbertL]

I used to have a remote installation with a PLC that pushed around 1 KB of data every 5 seconds via HTTP Post to a regular commercial shared hosting server.  It used a cell router with the cheapest T-Mobile 4G data plan: 2 GB/mo for $10.  It was actually an "unlimited" plan in that, after using up the 2 GB allowance at 4G speed, it reverts to (I think) 2G speed, which was still more than enough for my needs.  Never had a problem with the carrier.  Once I tried changing the update interval to 1 second, but the hosting service didn't like that and blacklisted the IP address, so I just went back to the original 5-second interval.  Someday I might try batching the data to see if it's the volume or the hit rate that's causing a problem.

[tom66]

Are you using a commercial SIM that permits commercial/IoT usage or just a regular consumer SIM?

Many consumer networks impose restrictions on the data that travels on their networks and they perform traffic shaping, it could just be that they've decided there is too much data going towards one endpoint and they are terminating the connection as part of their ToS/abuse policy.

Yes it's crap but these kinds of policies are very common with consumer SIM providers and it's one way they keep the prices low.

Moving around the S3 address periodically e.g. move from EU-West-1 to EU-West-2 (if possible) may help.  There should be no fundamental reason you can't open and close the connection several times.  Closing a TCP connection does require a tiny bit of data but unlikely to impact you significantly if you're only doing it every 10 seconds.

[amyk]

How much data volume are you transmitting?

[m98]

I've had a lot of bad experience with u-blox cellular modules. First, get a firmware update, as they usually ship with what feels like a broken alpha. I would say rebooting them at least once per day is a good stability measure in general, unless you really need to save power and can't afford the network reconnect.
Second, what are you doing exactly? How is your modem configured? Logfiles would also help.
The guidelines you're looking for are those: https://www.gsma.com/iot/wp-content/uploads/2016/11/TS.34_v4.0-1.pdf

[VK3DRB]

Modify the code to that if there is no connection or packets sent are not acknowledged, then reconnect to the 4G network or reboot.
If the data rate is low, consider LTE-M. Better coverage, much cheaper, and more robust.
You might want to get a Nordic eval board and see if you have similar issues.