Author Topic: Instability in FTP Push from data logger to Linux server - ideas?  (Read 2161 times)

0 Members and 1 Guest are viewing this topic.

Offline Martin FTopic starter

  • Regular Contributor
  • *
  • Posts: 149
  • Country: dk
Hi all,

We've got a client who uses our data logger to record data from a battery to an SD card.
The data logger then sends the data in chunks of e.g. 20 MB per file to an FTP server (unsecured), utilizing a W-04 FlashAir from Toshiba.
To provide internet access, the client uses a cellular network (3G hotspot).

At 20 MB, a file should be transferred roughly once per hour, which works without problems in our test setup.
In a few cases, the logger may time out in our test setting, but we've added a "recovery" that reboots the FlashAir, which gets it up and running again.

However, at our client setup the stability is far worse - and at 20 MB file size, we often encounter timeouts, i.e. something goes wrong during transfer.
This leads to extended periods without data transfer, which is a problem for our client.
We've tried with smaller file sizes at the client (e.g. 1 MB), which seems to be more stable - but is too small for practicality purposes in their use case.


Any of you have good ideas for what we can try and test/change in the client setup to identify the root cause of the issue, while retaining 20-30 MB file sizes?

Thanks!
 

Offline Psi

  • Super Contributor
  • ***
  • Posts: 9889
  • Country: nz
Re: Instability in FTP Push from data logger to Linux server - ideas?
« Reply #1 on: May 25, 2018, 12:07:01 pm »
ive never used a W-04 FlashAir but its not something i would expect to be very reliable in an industrial setting.

Also, how does the W-04 FlashAir attach to your data logger, maybe the metal SDcard slot  and/or the data logger box is blocking/reflecting some RF and introducing a very unstable wifi link. One where anything moving nearby can cause the link to drop.

You can get like 70cm SDcard extenders, maybe one would help get the FlashAir away from the data logger and into free air where it can get better wifi reception

Another thing to look at is FTP 'resume' mode, that way if the connection drops you dont have to restart the whole 20MB.   Getting a long period of uninterrupted transfer maybe asking a bit much for a 3G hotspot and wifi SDcard.

Another thing, Wifi needs a lot of peak power, and sd cards are small so there is a limit to how much internal power rail capacitance they can have on board.
Maybe the cards not getting stable power from the data logger. (Im guessing its intended to power SD cards, not wifi transmitters). So it might be a good idea to add some extra capacitors across the power rail on the SD card socket.
« Last Edit: May 25, 2018, 12:15:25 pm by Psi »
Greek letter 'Psi' (not Pounds per Square Inch)
 

Offline Rerouter

  • Super Contributor
  • ***
  • Posts: 4694
  • Country: au
  • Question Everything... Except This Statement
Re: Instability in FTP Push from data logger to Linux server - ideas?
« Reply #2 on: May 25, 2018, 12:37:04 pm »
I would suggest begin by ruling out the 3G hotspot, walk in with a phone, setup a hotspot with the same credentials and see if it improves,

Some hotspots i have seen are quite aggressive on pruning clients who are not active for a while, equally some allow mounting external antennas, If its reception the faster bodge may just be to fit a better antenna to the thing.

 

Offline Martin FTopic starter

  • Regular Contributor
  • *
  • Posts: 149
  • Country: dk
Re: Instability in FTP Push from data logger to Linux server - ideas?
« Reply #3 on: May 25, 2018, 04:33:13 pm »
Thanks a lot ! Any ideas on whether settings on the server may be part of this type of behavior?
 

Offline Psi

  • Super Contributor
  • ***
  • Posts: 9889
  • Country: nz
Re: Instability in FTP Push from data logger to Linux server - ideas?
« Reply #4 on: May 26, 2018, 12:54:10 pm »
Maybe a faulty NIC in the server? When they fail they can sometimes continue to function but do some really weird things.

Also maybe check your FTP link is using active FTP rather than passive.
« Last Edit: May 26, 2018, 01:00:21 pm by Psi »
Greek letter 'Psi' (not Pounds per Square Inch)
 

Offline sokoloff

  • Super Contributor
  • ***
  • Posts: 1799
  • Country: us
Re: Instability in FTP Push from data logger to Linux server - ideas?
« Reply #5 on: May 26, 2018, 01:10:04 pm »
What's different between your test setup and your client's deployment? Look at the wifi (distance, AP type, RSSI, other sources of interference, DHCP settings, etc) and 3G hotspot.

It's almost surely something at the client that's different. Does your test setup work over the same 3G hotspot?
Another thing to look at is FTP 'resume' mode, that way if the connection drops you dont have to restart the whole 20MB.   Getting a long period of uninterrupted transfer maybe asking a bit much for a 3G hotspot and wifi SDcard.
Yes. If you need perfection on a 20MB file transfer over unstable wifi over a 3G hotspot, you're going to have a bad time of it. At least with restartable transfer, if you get 5 MB at a time, you eventually finish.

Without any disrespect to Psi on your second question, who was just answering your followup question, I'd encourage you strongly to look at the client side of things. "When you hear hoofbeats, think horses, not zebras." Wifi/3G differences are the horses. Faulty NIC is a zebra.
 

Offline IanMacdonald

  • Frequent Contributor
  • **
  • Posts: 943
  • Country: gb
    • IWR Consultancy
Re: Instability in FTP Push from data logger to Linux server - ideas?
« Reply #6 on: May 27, 2018, 07:04:13 am »
Problems with FTP are almost always down to port allocations. There are two modes, passive and active, and rather than go into a lengthy explanation here I'll refer you to the Filezilla documentation: https://wiki.filezilla-project.org/Network_Configuration
 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2803
  • Country: nz
Re: Instability in FTP Push from data logger to Linux server - ideas?
« Reply #7 on: May 27, 2018, 07:29:08 am »
Router NAT (Network Address Translation) timeouts on the idle control channel during the data transfers can also be an issue (but doesn't sound likely in this case.

If the connection always breaks a 'n' seconds after transfer starts (but faster transfer usually succeed) this could be your cause.
Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 

Offline Martin FTopic starter

  • Regular Contributor
  • *
  • Posts: 149
  • Country: dk
Re: Instability in FTP Push from data logger to Linux server - ideas?
« Reply #8 on: May 27, 2018, 09:31:01 pm »
Thanks all for the inputs and tips!

We've got 6-12 loggers running in a test setup now to identify the error.
In the below link we have the FTP log in a google sheet:
https://docs.google.com/spreadsheets/d/1KWBoUGJMa7sF1b8WsRUJASdiT3deM59KnjbQAXG6uKo/edit?usp=sharing

It's really strange:

Our loggers generate ~30 MB files every 2 hours and send them to the FTP server.
For most of our transfers, we don't get any issues in our setup and files are transferred within 5-15 seconds, no problem.

For 2 transfers, we do encounter a timeout that appears after 10 minutes. After this, the logger would usually not resume transfers later on, but we use a "recovery" mechanism in which we turn off the power for the FlashAir (via the logger itself) for a short period to simulate/force a reboot of the card.

After this, the "timed out" file is transferred upon the next file split as per our recovery specification (cf. the colored lines).

However, the third time an issue occurs there is no resuming.

Apparently our "recovery" mechanism fails, maybe because the card does not fully turn off - or something else plays in.

It's a super long shot and I don't expect anyone to have suggestions here - just wanted to share for update purposes.

Should any of you have ideas/suggestions for how to test/identify the issue, it'd be helpful.
Note that the data logger is not easily able to get any info from the FlashAir as Toshiba probably never intended fully for this kind of use case.

Martin
 

Offline jazz

  • Contributor
  • Posts: 32
Re: Instability in FTP Push from data logger to Linux server - ideas?
« Reply #9 on: May 28, 2018, 01:53:56 am »
Are you running a lua script on the SD card itself to do the ftp transfers? Maybe their ftp function just doesnt handle bad connections very well and stalls the transfer. I guess ideally, you'd want an ftp client with more control, but it doesn't look like you could implement that on the card. Fetching the data from the card using a http client with a proper resume/retry strategy is probably not an option, because you couldn't secure the access to the card's http server, right?

Instead of cutting power to the card, couldn't you use the card's watchdog timer to restart the script, if the ftp function doesn't return after an expected maximum transfer time? Or does the watchdog timer not work that way (doesn't seem well documented)?
 

Offline Martin FTopic starter

  • Regular Contributor
  • *
  • Posts: 149
  • Country: dk
Re: Instability in FTP Push from data logger to Linux server - ideas?
« Reply #10 on: May 28, 2018, 05:32:44 am »
Hi Jazz, the watchdog idea is very good - but agree it’s really hard to find any info on it so far
 

Offline Psi

  • Super Contributor
  • ***
  • Posts: 9889
  • Country: nz
Re: Instability in FTP Push from data logger to Linux server - ideas?
« Reply #11 on: May 29, 2018, 12:36:47 pm »
i would definitely be looking at the cards power rail to make sure its not just dipping too low sometimes under the transmit current and locking up.

What powers your loggers, do they run on a battery or mains from a dc adaptor?
Maybe try testing them running from a very low voltage.

If using AC-DC adapters are they switchmode? or just a transformer.
If they're transformers maybe the mains voltage is different at your test location verses their location and they're loggers are all running at a lower DC voltage and that's having issues keeping the wifi cards stable
« Last Edit: May 29, 2018, 12:40:20 pm by Psi »
Greek letter 'Psi' (not Pounds per Square Inch)
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf