Author Topic: W5500 ethernet... how in the hell it sends 10GB per month?!  (Read 2269 times)

0 Members and 1 Guest are viewing this topic.

Offline honeybadger

  • Contributor
  • Posts: 42
  • Country: cz
W5500 ethernet... how in the hell it sends 10GB per month?!
« on: October 23, 2020, 09:58:20 pm »
Hi,
I have a "smart home" (I hate that term |O) project which needs internet connectivity.

It uses a common W5500 Ethernet controller and it works just fine.
Every 5 seconds it send ~500 characters long text string message to remote server through HTTP POST.
This is roughly 250MB of useful data per month.

Surprise Motherf*cker! It eats through 2GB data plan in a week. It needs roughly 10GB per month!

How is this possible? Where is the 9,75GB per month coming from? Is the HTTP POST really that data hungry with small frequent payload data?

Is there anything I can do to lower the data usage?
* I can not make the data string smaller.
* I need the 5s refresh rate. OK maybe not 5s, 10s would be also fine but no more.
* I can not cache the data and send a big package every a minute or so - not enough memory.
* It works with a standard Arduino Ethernet lib.
* It does not sends the same data 100x, nothing like that. I do not touch the W5500 between the 5s period.
 

Offline magic

  • Super Contributor
  • ***
  • Posts: 2700
  • Country: pl
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #1 on: October 23, 2020, 10:05:31 pm »
tcpdump ;)
 

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 2275
  • Country: us
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #2 on: October 23, 2020, 10:21:41 pm »
Have you watched with wireshark to see what is actually happening?  Are you sure there isn't something else on the network using the bandwidth?  Maybe your ISP is counting wrong?

Overhead shouldn't be more than 4x for this scenario.  Assuming you are making a new HTTP connection for every request there will be ~8 total TCP packets with under 500 bytes total headers/framing.  In addition you will have the 500 bytes of data, a few hundred bytes in HTTP headers, and a few hundred bytes in a minimal response.

Is the arduino ethernet library using multi-byte SPI transfers?  I have never used it but I have used the wiznet ioLibrary.  It has 4 functions you need to implement: read/write byte (or word?  I don't remember) and read/write buffer.  If you don't provide an implementation for the latter, it falls back to doing single byte SPI transactions.  It is possible that the wiznet chip handles this poorly by doing a whole bunch of 1 byte packets.  That would result in the sort of behavior you are seeing.
 

Offline honeybadger

  • Contributor
  • Posts: 42
  • Country: cz
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #3 on: October 23, 2020, 10:32:04 pm »
Colleague will help me with the wireshark next week. I am definitely not a network guy.

There is not anything else stealing the bandwidth. The "thing" with W5500 is connected directly to 4G router. I guess the mobile provider is counting the data correctly.

Not sure about the multi-byte SPI transfer. How can I find it out?
 

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 2275
  • Country: us
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #4 on: October 23, 2020, 10:40:45 pm »
With an oscilloscope or logic analyzer on the SPI bus.

tcpdump/wireshark aren't hard to use.  You don't need to be a networking expert or anything.  If you write applications that communicate over a network then you should consider that essential knowledge.
 
The following users thanked this post: hans, nctnico, Yansi

Online ataradov

  • Super Contributor
  • ***
  • Posts: 6989
  • Country: us
    • Personal site
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #5 on: October 24, 2020, 12:05:11 am »
This was a very common problem with GPRS modems in the past. Wireless providers would have billing setup in 64 Kbyte minimum, so even if you send 2 bytes per hour. you will still get billed for 64 K.

This got solved with introduction of data-only SIM cards and corresponding billing plans.

I'm not saying it is the case here, but definitely could be.

tcpdump/wireshark will tell for sure.
Alex
 
The following users thanked this post: Psi, Someone

Offline free_electron

  • Super Contributor
  • ***
  • Posts: 7569
  • Country: us
    • SiliconValleyGarage
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #6 on: October 24, 2020, 12:22:43 am »
a single ethernet frame is what these days ? 64bytes to1518 bytes. make sure you use smallest frame possible and not waste 1518 bytes to send 3 bits ...
Professional Electron Wrangler.
Any comments, or points of view expressed, are my own and not endorsed , induced or compensated by my employer(s).
 

Offline NiHaoMike

  • Super Contributor
  • ***
  • Posts: 6730
  • Country: us
  • "Don't turn it on - Take it apart!"
    • Facebook Page
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #7 on: October 24, 2020, 01:20:03 am »
Every 5 seconds it send ~500 characters long text string message to remote server through HTTP POST.
Have you considered LoRa?
Cryptocurrency has taught me to love math and at the same time be baffled by it.

Cryptocurrency lesson 0: Altcoins and Bitcoin are not the same thing.
 

Online girishv

  • Regular Contributor
  • *
  • Posts: 52
  • Country: in
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #8 on: October 24, 2020, 02:59:04 am »
It uses a common W5500 Ethernet controller and it works just fine.
Every 5 seconds it send ~500 characters long text string message to remote server through HTTP POST.
This is roughly 250MB of useful data per month.
If you can make changes on server side to send delivery receipts and handle the delivery errors on firmware level, you can consider using UDP.

Furthermore, if you are sending repetitive data, you may ignore the delivery errors.
 

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 2275
  • Country: us
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #9 on: October 24, 2020, 03:34:28 am »
It uses a common W5500 Ethernet controller and it works just fine.
Every 5 seconds it send ~500 characters long text string message to remote server through HTTP POST.
This is roughly 250MB of useful data per month.
If you can make changes on server side to send delivery receipts and handle the delivery errors on firmware level, you can consider using UDP.

Furthermore, if you are sending repetitive data, you may ignore the delivery errors.

Do not do that.  Unless you are a networking wizard (which the OP has assured us they are not) you should just pretend UDP doesn't exist.  That goes doubly so if you are talking about routing over the public internet like it appears the OP is doing.

HTTP is a perfectly fine protocol for this application, something is just going wrong.  Randomly changing to a different protocol without figure out what is going on is a waste of time, especially one with the possibility to create a whole host of other problems.

The only sensible course of action is to use a packet sniffer to see what is actually happening.  Either your networking library/offload chip is doing something silly or your ISP is counting data wrong.
 

Offline jbb

  • Frequent Contributor
  • **
  • Posts: 891
  • Country: nz
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #10 on: October 24, 2020, 07:32:40 am »
+1 for wireshark
I wouldn’t be surprised if the service provider is rounding up per billing period; it’s a common issue.
 
The following users thanked this post: cdev

Offline HB9EVI

  • Frequent Contributor
  • **
  • Posts: 481
  • Country: ch
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #11 on: October 24, 2020, 08:39:35 am »
don't mistake the MTU (maximum transfer unit) which is 1500bytes for the effective packet size which can always be smaller than the MTU, depending on the payload.

very likely the 'problem' is the manner the mobile operator counts the data being transmitted, so even if the payload is only a few bytes every 5min, the operator maybe charges for 5 or 10kbytes minimum each time, that's still common practice for certain data plans.

my experience with the W5500 is, that it doesn't send anything what the code in the mcu behind isn't telling it to send.
 

Offline Syntax Error

  • Frequent Contributor
  • **
  • Posts: 370
  • Country: england
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #12 on: October 24, 2020, 09:08:43 am »
I agree, that's one huge payload but not uncommon for IoT. But every five seconds, suggests a web socket.

Two cyber security questions: where exactly is this 'smart' device calling home (a remote server in China) and, how do you control your smart device/s? With the manufacturer's app I assume?

Definately 'Shark' your network. Pehaps post the captured packets here (sanitised off course).
Now at Defcon Tier 3
 

Offline honeybadger

  • Contributor
  • Posts: 42
  • Country: cz
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #13 on: October 24, 2020, 04:13:17 pm »
Hi,
the device is connected to the client's server. It is not in China. The device is not controlled remotely.

Data is somehow repetitive. It doe not matter if one/two "packets" are missing. It sends bunch of temperatures readings, some other readings and short status message.

LoRa is not possible. When my client installs it he just needs to plug the Ethernet cable and forget about that.

I will paste sanitized Wireshark output next week. Thank you all for suggestions  :-+
 

Offline hli

  • Regular Contributor
  • *
  • Posts: 214
  • Country: de
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #14 on: October 24, 2020, 10:35:32 pm »
Every 5 seconds it send ~500 characters long text string message to remote server through HTTP POST.
This is roughly 250MB of useful data per month.
...
How is this possible? Where is the 9,75GB per month coming from? Is the HTTP POST really that data hungry with small frequent payload data?
tcpdump / wireshark will help you. What you might see:
  • each request could do a DNS query (not sure whether the W5500 has a cache for that). These are not free
  • a POST request contains, at least, the hostname of the server, and the URL which the request goes to. These are (together with the header names) easily hundred bytes
  • depending on how your request looks like exactly, there might be additional headers (a.g. for authentication)
  • and then the server sends a response - which in turn contains at least some headers, and maybe an additional payload
These all count as traffic. Absent any other infoprmation my guess would that the answer of the server is large.
Using WebSockets might help, or just pushing to a MQTT server (which seems to the natural answer to your described problem).
 

Offline amyk

  • Super Contributor
  • ***
  • Posts: 6956
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #15 on: October 25, 2020, 01:33:10 am »
10GB per month is a constant stream of slightly under 4KB/s, whereas you are expecting closer to 100B/s.

Are you actually using HTTPS (HTTP over TLS/SSL)? A single handshake, without sending any application data, can be a few KB for that already.

Otherwise, check the fine print of the service provision agreement. Internet service is not truly symmetrical a lot of the time, "2GB data plan" may be referring to download only and upload data is billed at a different (and higher) rate.

On the topic of recommending things the OP might not be able to do: use a binary protocol for smallest size and best efficiency.
 

Online blueskull

  • Supporter
  • ****
  • Posts: 14199
  • Country: cn
  • Power Electronics Guy
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #16 on: October 25, 2020, 02:49:28 am »
Are you actually using HTTPS (HTTP over TLS/SSL)? A single handshake, without sending any application data, can be a few KB for that already.

This is my excuse of reinventing the wheels. I'd rather use raw TCP protocol and construct my own HTTP headers on my own, so that I can sleep sound at night.
I might move to a more robust, industrial grade framework for production, but for prototyping, I use things that are either very, very validated, or just cook up my own whenever possible to rule out framework mishaps.
 

Offline bson

  • Supporter
  • ****
  • Posts: 1764
  • Country: us
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #17 on: October 25, 2020, 03:23:06 am »
Since you send something every 5 seconds, make sure the library you use keeps the connection open or you can run into all sorts of other problems as well.  More specifically you can run out of ports, or if the library doesn't fully implement TIME_WAIT it can end up trying to use IP address tuples in TIME_WAIT on the server.  Then have to try a gazillion ports until it finds one available - and this will be neither reliable or data efficient.
« Last Edit: October 25, 2020, 03:25:24 am by bson »
 

Online girishv

  • Regular Contributor
  • *
  • Posts: 52
  • Country: in
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #18 on: October 25, 2020, 05:34:17 am »
Data is somehow repetitive. It doe not matter if one/two "packets" are missing. It sends bunch of temperatures readings, some other readings and short status message.
UDP is time tested for use cases like this. As you can afford losing few packets here and there, you should consider using UDP.
 

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 2275
  • Country: us
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #19 on: October 25, 2020, 05:36:35 am »
Are you actually using HTTPS (HTTP over TLS/SSL)? A single handshake, without sending any application data, can be a few KB for that already.

I don't think the arduino ethernet shield library supports HTTPs or TLS.  The wiznet w5500 certainly doesn't support encryption offload.  This seems pretty unlikely.  Wireshark packet capture will show for sure...
 

Offline m k

  • Regular Contributor
  • *
  • Posts: 106
  • Country: fi
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #20 on: October 25, 2020, 08:12:24 pm »
The "thing" with W5500 is connected directly to 4G router.

Routers have data counters, can you check them?

I'm with amyk, your small data plan can be very asymmetric.
 

Offline honeybadger

  • Contributor
  • Posts: 42
  • Country: cz
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #21 on: October 26, 2020, 11:46:47 pm »
edit:

Hi,
first off - there is no HTTPS nor TLS.

second... Wireshark... interesting.

There is something rotten with the "F()" macro!
Everything sent with "F()" (for example client.print(F("POST /")); ) is sent char by char.
Everything sent without (for example client.print(url_address); ) is sent as a whole.

Will test and report.
« Last Edit: October 27, 2020, 12:03:49 am by honeybadger »
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 20121
  • Country: nl
    • NCT Developments
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #22 on: October 27, 2020, 12:48:41 am »
Do not do that.  Unless you are a networking wizard (which the OP has assured us they are not) you should just pretend UDP doesn't exist.  That goes doubly so if you are talking about routing over the public internet like it appears the OP is doing.
Nonsense. UDP is perfect for repeating status updates. Especially if it is a one way status message. All media streaming protocols use UDP because a missing packet is not relevant by the time the receiver finds out it is missing. Especially for realtime communication TCP can be a nightmare. If a packet gets lost there is an incremental time-out period during which there is no communication at all. A few missing TCP packets can jack the time up to 10s of seconds during which there will be no status updates.
« Last Edit: October 27, 2020, 12:57:29 am by nctnico »
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline jbb

  • Frequent Contributor
  • **
  • Posts: 891
  • Country: nz
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #23 on: October 27, 2020, 03:57:41 am »
Everything sent with "F()" ... is sent char by char.

Yowza! So much overhead! I suggest you have a look inside the putc function handler (as buried in the F(...) macro).

If it’s short, could you try pre-cooking the string with and then sending that whole buffer off in one go?
 
The following users thanked this post: splin

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 2275
  • Country: us
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #24 on: October 27, 2020, 04:34:23 am »
There is something rotten with the "F()" macro!
Everything sent with "F()" (for example client.print(F("POST /")); ) is sent char by char.
Everything sent without (for example client.print(url_address); ) is sent as a whole.

I don't remember the details of how it works, but the F macro arranges for string literals to be in program memory instead of RAM, then triggers (via C++ function overloading) an alternate version of string functions that know how to access program memory.  I think the typical implementations read a byte into memory at a time, then call the regular print command with a single byte string.  If this is done slowly the wiznet chip will not coalesce the writes into fewer packets.

The most straighforward fix assuming you don't have enough RAM to ditch all F macros is is a wrapper for client.print() that users strncpy_P() to copy into a temporary variable of a moderate size (say 32 bytes) and then calls client.print() on the buffer.  If the fixed string is more than 32 bytes you would still break it up into separate calls but this would reduce overhead dramatically.

If you have enough memory available, you could assemble your entire request in a single array and then send it in one go with a single call to client.print().  This would be the most efficient if you have the RAM available but require a bit more restructuring of your code.

I think there are ways to do this by mucking about with the wiznet registers.  I think you can pause sending, take as long as you want to load data into the wiznet onboard RAM, and then tell it to start sending again.  This would also be efficient, and would allow you to avoid filling up the AVR ram as you would be doing the assembly on the wiznet chip.  However, this sounds like an enormous pain in the ass to get working properly with the arduino ethernet library.

FWIW, I would consider this a pretty big bug in the arduino ethernet library.  They really shouldn't be falling back on single byte writes.  It doesn't matter for something like a serial port which is what the F() macro is designed for, but it is not really OK for a network interface. 
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf