Author Topic: W5500 ethernet... how in the hell it sends 10GB per month?!  (Read 6428 times)

0 Members and 1 Guest are viewing this topic.

Offline honeybadgerTopic starter

  • Contributor
  • Posts: 42
  • Country: cz
W5500 ethernet... how in the hell it sends 10GB per month?!
« on: October 23, 2020, 09:58:20 pm »
Hi,
I have a "smart home" (I hate that term |O) project which needs internet connectivity.

It uses a common W5500 Ethernet controller and it works just fine.
Every 5 seconds it send ~500 characters long text string message to remote server through HTTP POST.
This is roughly 250MB of useful data per month.

Surprise Motherf*cker! It eats through 2GB data plan in a week. It needs roughly 10GB per month!

How is this possible? Where is the 9,75GB per month coming from? Is the HTTP POST really that data hungry with small frequent payload data?

Is there anything I can do to lower the data usage?
* I can not make the data string smaller.
* I need the 5s refresh rate. OK maybe not 5s, 10s would be also fine but no more.
* I can not cache the data and send a big package every a minute or so - not enough memory.
* It works with a standard Arduino Ethernet lib.
* It does not sends the same data 100x, nothing like that. I do not touch the W5500 between the 5s period.
 

Online magic

  • Super Contributor
  • ***
  • Posts: 6743
  • Country: pl
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #1 on: October 23, 2020, 10:05:31 pm »
tcpdump ;)
 

Online ejeffrey

  • Super Contributor
  • ***
  • Posts: 3709
  • Country: us
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #2 on: October 23, 2020, 10:21:41 pm »
Have you watched with wireshark to see what is actually happening?  Are you sure there isn't something else on the network using the bandwidth?  Maybe your ISP is counting wrong?

Overhead shouldn't be more than 4x for this scenario.  Assuming you are making a new HTTP connection for every request there will be ~8 total TCP packets with under 500 bytes total headers/framing.  In addition you will have the 500 bytes of data, a few hundred bytes in HTTP headers, and a few hundred bytes in a minimal response.

Is the arduino ethernet library using multi-byte SPI transfers?  I have never used it but I have used the wiznet ioLibrary.  It has 4 functions you need to implement: read/write byte (or word?  I don't remember) and read/write buffer.  If you don't provide an implementation for the latter, it falls back to doing single byte SPI transactions.  It is possible that the wiznet chip handles this poorly by doing a whole bunch of 1 byte packets.  That would result in the sort of behavior you are seeing.
 

Offline honeybadgerTopic starter

  • Contributor
  • Posts: 42
  • Country: cz
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #3 on: October 23, 2020, 10:32:04 pm »
Colleague will help me with the wireshark next week. I am definitely not a network guy.

There is not anything else stealing the bandwidth. The "thing" with W5500 is connected directly to 4G router. I guess the mobile provider is counting the data correctly.

Not sure about the multi-byte SPI transfer. How can I find it out?
 

Online ejeffrey

  • Super Contributor
  • ***
  • Posts: 3709
  • Country: us
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #4 on: October 23, 2020, 10:40:45 pm »
With an oscilloscope or logic analyzer on the SPI bus.

tcpdump/wireshark aren't hard to use.  You don't need to be a networking expert or anything.  If you write applications that communicate over a network then you should consider that essential knowledge.
 
The following users thanked this post: hans, nctnico, Yansi

Online ataradov

  • Super Contributor
  • ***
  • Posts: 11234
  • Country: us
    • Personal site
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #5 on: October 24, 2020, 12:05:11 am »
This was a very common problem with GPRS modems in the past. Wireless providers would have billing setup in 64 Kbyte minimum, so even if you send 2 bytes per hour. you will still get billed for 64 K.

This got solved with introduction of data-only SIM cards and corresponding billing plans.

I'm not saying it is the case here, but definitely could be.

tcpdump/wireshark will tell for sure.
Alex
 
The following users thanked this post: Psi, Someone

Offline free_electron

  • Super Contributor
  • ***
  • Posts: 8517
  • Country: us
    • SiliconValleyGarage
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #6 on: October 24, 2020, 12:22:43 am »
a single ethernet frame is what these days ? 64bytes to1518 bytes. make sure you use smallest frame possible and not waste 1518 bytes to send 3 bits ...
Professional Electron Wrangler.
Any comments, or points of view expressed, are my own and not endorsed , induced or compensated by my employer(s).
 

Online NiHaoMike

  • Super Contributor
  • ***
  • Posts: 9003
  • Country: us
  • "Don't turn it on - Take it apart!"
    • Facebook Page
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #7 on: October 24, 2020, 01:20:03 am »
Every 5 seconds it send ~500 characters long text string message to remote server through HTTP POST.
Have you considered LoRa?
Cryptocurrency has taught me to love math and at the same time be baffled by it.

Cryptocurrency lesson 0: Altcoins and Bitcoin are not the same thing.
 

Offline girishv

  • Regular Contributor
  • *
  • Posts: 114
  • Country: in
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #8 on: October 24, 2020, 02:59:04 am »
It uses a common W5500 Ethernet controller and it works just fine.
Every 5 seconds it send ~500 characters long text string message to remote server through HTTP POST.
This is roughly 250MB of useful data per month.
If you can make changes on server side to send delivery receipts and handle the delivery errors on firmware level, you can consider using UDP.

Furthermore, if you are sending repetitive data, you may ignore the delivery errors.
 

Online ejeffrey

  • Super Contributor
  • ***
  • Posts: 3709
  • Country: us
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #9 on: October 24, 2020, 03:34:28 am »
It uses a common W5500 Ethernet controller and it works just fine.
Every 5 seconds it send ~500 characters long text string message to remote server through HTTP POST.
This is roughly 250MB of useful data per month.
If you can make changes on server side to send delivery receipts and handle the delivery errors on firmware level, you can consider using UDP.

Furthermore, if you are sending repetitive data, you may ignore the delivery errors.

Do not do that.  Unless you are a networking wizard (which the OP has assured us they are not) you should just pretend UDP doesn't exist.  That goes doubly so if you are talking about routing over the public internet like it appears the OP is doing.

HTTP is a perfectly fine protocol for this application, something is just going wrong.  Randomly changing to a different protocol without figure out what is going on is a waste of time, especially one with the possibility to create a whole host of other problems.

The only sensible course of action is to use a packet sniffer to see what is actually happening.  Either your networking library/offload chip is doing something silly or your ISP is counting data wrong.
 

Online jbb

  • Super Contributor
  • ***
  • Posts: 1135
  • Country: nz
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #10 on: October 24, 2020, 07:32:40 am »
+1 for wireshark
I wouldn’t be surprised if the service provider is rounding up per billing period; it’s a common issue.
 
The following users thanked this post: cdev

Offline HB9EVI

  • Frequent Contributor
  • **
  • Posts: 722
  • Country: ch
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #11 on: October 24, 2020, 08:39:35 am »
don't mistake the MTU (maximum transfer unit) which is 1500bytes for the effective packet size which can always be smaller than the MTU, depending on the payload.

very likely the 'problem' is the manner the mobile operator counts the data being transmitted, so even if the payload is only a few bytes every 5min, the operator maybe charges for 5 or 10kbytes minimum each time, that's still common practice for certain data plans.

my experience with the W5500 is, that it doesn't send anything what the code in the mcu behind isn't telling it to send.
 

Offline Syntax Error

  • Frequent Contributor
  • **
  • Posts: 584
  • Country: gb
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #12 on: October 24, 2020, 09:08:43 am »
I agree, that's one huge payload but not uncommon for IoT. But every five seconds, suggests a web socket.

Two cyber security questions: where exactly is this 'smart' device calling home (a remote server in China) and, how do you control your smart device/s? With the manufacturer's app I assume?

Definately 'Shark' your network. Pehaps post the captured packets here (sanitised off course).
 

Offline honeybadgerTopic starter

  • Contributor
  • Posts: 42
  • Country: cz
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #13 on: October 24, 2020, 04:13:17 pm »
Hi,
the device is connected to the client's server. It is not in China. The device is not controlled remotely.

Data is somehow repetitive. It doe not matter if one/two "packets" are missing. It sends bunch of temperatures readings, some other readings and short status message.

LoRa is not possible. When my client installs it he just needs to plug the Ethernet cable and forget about that.

I will paste sanitized Wireshark output next week. Thank you all for suggestions  :-+
 

Offline hli

  • Frequent Contributor
  • **
  • Posts: 255
  • Country: de
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #14 on: October 24, 2020, 10:35:32 pm »
Every 5 seconds it send ~500 characters long text string message to remote server through HTTP POST.
This is roughly 250MB of useful data per month.
...
How is this possible? Where is the 9,75GB per month coming from? Is the HTTP POST really that data hungry with small frequent payload data?
tcpdump / wireshark will help you. What you might see:
  • each request could do a DNS query (not sure whether the W5500 has a cache for that). These are not free
  • a POST request contains, at least, the hostname of the server, and the URL which the request goes to. These are (together with the header names) easily hundred bytes
  • depending on how your request looks like exactly, there might be additional headers (a.g. for authentication)
  • and then the server sends a response - which in turn contains at least some headers, and maybe an additional payload
These all count as traffic. Absent any other infoprmation my guess would that the answer of the server is large.
Using WebSockets might help, or just pushing to a MQTT server (which seems to the natural answer to your described problem).
 

Online amyk

  • Super Contributor
  • ***
  • Posts: 8258
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #15 on: October 25, 2020, 01:33:10 am »
10GB per month is a constant stream of slightly under 4KB/s, whereas you are expecting closer to 100B/s.

Are you actually using HTTPS (HTTP over TLS/SSL)? A single handshake, without sending any application data, can be a few KB for that already.

Otherwise, check the fine print of the service provision agreement. Internet service is not truly symmetrical a lot of the time, "2GB data plan" may be referring to download only and upload data is billed at a different (and higher) rate.

On the topic of recommending things the OP might not be able to do: use a binary protocol for smallest size and best efficiency.
 

Offline bson

  • Supporter
  • ****
  • Posts: 2269
  • Country: us
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #16 on: October 25, 2020, 03:23:06 am »
Since you send something every 5 seconds, make sure the library you use keeps the connection open or you can run into all sorts of other problems as well.  More specifically you can run out of ports, or if the library doesn't fully implement TIME_WAIT it can end up trying to use IP address tuples in TIME_WAIT on the server.  Then have to try a gazillion ports until it finds one available - and this will be neither reliable or data efficient.
« Last Edit: October 25, 2020, 03:25:24 am by bson »
 

Offline girishv

  • Regular Contributor
  • *
  • Posts: 114
  • Country: in
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #17 on: October 25, 2020, 05:34:17 am »
Data is somehow repetitive. It doe not matter if one/two "packets" are missing. It sends bunch of temperatures readings, some other readings and short status message.
UDP is time tested for use cases like this. As you can afford losing few packets here and there, you should consider using UDP.
 

Online ejeffrey

  • Super Contributor
  • ***
  • Posts: 3709
  • Country: us
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #18 on: October 25, 2020, 05:36:35 am »
Are you actually using HTTPS (HTTP over TLS/SSL)? A single handshake, without sending any application data, can be a few KB for that already.

I don't think the arduino ethernet shield library supports HTTPs or TLS.  The wiznet w5500 certainly doesn't support encryption offload.  This seems pretty unlikely.  Wireshark packet capture will show for sure...
 

Offline m k

  • Super Contributor
  • ***
  • Posts: 1990
  • Country: fi
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #19 on: October 25, 2020, 08:12:24 pm »
The "thing" with W5500 is connected directly to 4G router.

Routers have data counters, can you check them?

I'm with amyk, your small data plan can be very asymmetric.
Advance-Aneng-Appa-AVO-Data Tech-Fluke-General Radio-H. W. Sullivan-Heathkit-HP-Kaise-Kyoritsu-Leeds & Northrup-Mastech-REO-Simpson-Sinclair-Tektronix-Triplett-YFE
(plus lesser brands from the work shop of the world)
 

Offline honeybadgerTopic starter

  • Contributor
  • Posts: 42
  • Country: cz
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #20 on: October 26, 2020, 11:46:47 pm »
edit:

Hi,
first off - there is no HTTPS nor TLS.

second... Wireshark... interesting.

There is something rotten with the "F()" macro!
Everything sent with "F()" (for example client.print(F("POST /")); ) is sent char by char.
Everything sent without (for example client.print(url_address); ) is sent as a whole.

Will test and report.
« Last Edit: October 27, 2020, 12:03:49 am by honeybadger »
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 26868
  • Country: nl
    • NCT Developments
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #21 on: October 27, 2020, 12:48:41 am »
Do not do that.  Unless you are a networking wizard (which the OP has assured us they are not) you should just pretend UDP doesn't exist.  That goes doubly so if you are talking about routing over the public internet like it appears the OP is doing.
Nonsense. UDP is perfect for repeating status updates. Especially if it is a one way status message. All media streaming protocols use UDP because a missing packet is not relevant by the time the receiver finds out it is missing. Especially for realtime communication TCP can be a nightmare. If a packet gets lost there is an incremental time-out period during which there is no communication at all. A few missing TCP packets can jack the time up to 10s of seconds during which there will be no status updates.
« Last Edit: October 27, 2020, 12:57:29 am by nctnico »
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Online jbb

  • Super Contributor
  • ***
  • Posts: 1135
  • Country: nz
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #22 on: October 27, 2020, 03:57:41 am »
Everything sent with "F()" ... is sent char by char.

Yowza! So much overhead! I suggest you have a look inside the putc function handler (as buried in the F(...) macro).

If it’s short, could you try pre-cooking the string with and then sending that whole buffer off in one go?
 
The following users thanked this post: splin

Online ejeffrey

  • Super Contributor
  • ***
  • Posts: 3709
  • Country: us
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #23 on: October 27, 2020, 04:34:23 am »
There is something rotten with the "F()" macro!
Everything sent with "F()" (for example client.print(F("POST /")); ) is sent char by char.
Everything sent without (for example client.print(url_address); ) is sent as a whole.

I don't remember the details of how it works, but the F macro arranges for string literals to be in program memory instead of RAM, then triggers (via C++ function overloading) an alternate version of string functions that know how to access program memory.  I think the typical implementations read a byte into memory at a time, then call the regular print command with a single byte string.  If this is done slowly the wiznet chip will not coalesce the writes into fewer packets.

The most straighforward fix assuming you don't have enough RAM to ditch all F macros is is a wrapper for client.print() that users strncpy_P() to copy into a temporary variable of a moderate size (say 32 bytes) and then calls client.print() on the buffer.  If the fixed string is more than 32 bytes you would still break it up into separate calls but this would reduce overhead dramatically.

If you have enough memory available, you could assemble your entire request in a single array and then send it in one go with a single call to client.print().  This would be the most efficient if you have the RAM available but require a bit more restructuring of your code.

I think there are ways to do this by mucking about with the wiznet registers.  I think you can pause sending, take as long as you want to load data into the wiznet onboard RAM, and then tell it to start sending again.  This would also be efficient, and would allow you to avoid filling up the AVR ram as you would be doing the assembly on the wiznet chip.  However, this sounds like an enormous pain in the ass to get working properly with the arduino ethernet library.

FWIW, I would consider this a pretty big bug in the arduino ethernet library.  They really shouldn't be falling back on single byte writes.  It doesn't matter for something like a serial port which is what the F() macro is designed for, but it is not really OK for a network interface. 
 

Offline Fredderic

  • Regular Contributor
  • *
  • Posts: 68
  • Country: au
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #24 on: October 27, 2020, 07:32:07 am »
There is something rotten with the "F()" macro!
Everything sent with "F()" (for example client.print(F("POST /")); ) is sent char by char.
Everything sent without (for example client.print(url_address); ) is sent as a whole.

The F() macro as pointed out, stores the string in PROGMEM…  Problem there — at least on the processors I've used where it's typically a thing — is that PROGMEM is 16-bit, not the usual 8-bit byte addressable.  And the F() macro doesn't re-pack the string, so you end up with something like "H\0e\0l\0l\0o\0\0\0" actually being stored — it has no way of knowing exactly what you plan to do with that string, so it sacrifices a little memory for flexibility and ease of access (also, it forms a highly optimisable loop, so it should end up being pretty darned quick).  Plus, you tend to have 8x or 16x as much PROGMEM as regular RAM, so it really usually doesn't actually matter for the kinds of tasks Arduino is aimed at (which is not professional high-end development squeezing out every last drop of available resources).

But, therefore, you end up with a loop that reads a word (16-bit), writes the lower byte (8-bit), reads the next word, writes the lower byte, etc.  Such a loop if it's just doing a memory to memory copy shouldn't actually be any slower once the compiler does it's optimisations, but in the case of writing to a network device … that could well be interpreted by the networking layer as "I want this [1-byte] string content written immediately" (as in, write the character and then do a flush).  If that's what's happening, it's probably a case of the Arduino core again, doesn't know what you're doing with that data, so it's just using the standard stream "write a byte" functionality.  Which for memory-to-memory is fine, it usually optimises away nice and tight.  But if the networking library has a more packet mentality, then it might assume it's being told to send a 1-byte "packet", and goes ahead and does exactly what it's being told to do.  I'm really not sure what happens within the networking stuff…  This part is mostly just a guess.

In any case, that's also why the Arduino typically copies hard-coded strings into RAM at start-up — it does a single mass PROGMEM to RAM copy loop at the beginning, transferring the entire chunk of string space (not necessarily only strings, either) into regular RAM in one go, so it's all nice and contiguous how you'd expect it to be, and other parts of the code using those strings can then just go ahead and use it like usual (pretty sure RAM is generally a little faster, too).  Also, packing the strings would require it to know that it even needs to do that (obviously, not all processors do), and have special handling in the compiler (could probably be handled with some super-fancy core library work in modern compilers, but even then what happens if you tell your compiler to use a prior standard — do you expect it to somehow divide up your code between two different compilers?), etc.  So it doesn't.

Do keep in mind Arduino also really isn't targeted at high-end professional embedded systems developers — don't get me wrong, it's awesome, I love it myself, and most of the time it just works (which is more than I can say for some other compilers…), and doesn't cost thousands of dollars.  But when you're trying to squeeze that last bit of space or performance out, it doesn't always make the most helpful choices, and can tend to get in the way…  Things like digitalWrite() checking that the pin is in the right mode first… every single time!  I really would like a "I know what I'm doing" version…

So yeah, perhaps as suggested already, copy the strings into a RAM buffer, and send from there.  Unfortunately, sometimes the networking library will then be super helpful by copying that into yet another buffer — especially if the data is then transmitted by an ISR.  In one such case, a networking library expected you to give it a fully formed packet, which it copied into it's private buffer leaving room for the header and footer, then after patching them up, proceeded to dump the buffer to the networking device by ISR.  The internal function would then block waiting for the last one to finish, before accepting another write — but since you had to build up each packet as a whole first, it was usually well and truly done by then, and would block on you if it wasn't (you could also check if it was done yet).

Anyhow, I went in and added a stream-based writer directly into the internal buffer starting at the appropriate location (and limiting it to the appropriate length), and split the "send the packet" function to exclude the buffer copy, with the old function now just doing the buffer copy and then calling new library internal function to patch up the header and footer, and begin the sending.  Sounds like more work than it actually was, and it all worked beautifully…

But, that was a choice most likely made by the library developer to make it nice and easy for "Arduino type people" to use correctly, and as I say, they explicitly didn't support stream writing specifically to force you to buffer the packet yourself, giving the library time to finish sending the last one in the background, as well as a providing the library with a nice definitive "the data is ready now" signal.  In your case, some mixed signals between the Arduino core and the networking library could conceivably be causing individual characters to be interpreted as entire packets…  perhaps…  In which case buffering the data yourself will indeed fix that — without going to the lengths that I did.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf