Author Topic: W5500 ethernet... how in the hell it sends 10GB per month?!  (Read 14707 times)

0 Members and 1 Guest are viewing this topic.

Offline Miyuki

  • Frequent Contributor
  • **
  • Posts: 910
  • Country: cz
    • Me on youtube
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #25 on: October 27, 2020, 01:48:45 pm »
W5500 Have a 32kB buffer to easy compose all data into packets, even when MCU working on smaller chunks as Arduino/AVR used here have less RAM
So the only problem here is the library or a bad way of using it
 

Offline cgroen

  • Supporter
  • ****
  • Posts: 645
  • Country: dk
    • Carstens personal web
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #26 on: October 27, 2020, 02:08:09 pm »
It uses a common W5500 Ethernet controller and it works just fine.
Every 5 seconds it send ~500 characters long text string message to remote server through HTTP POST.
This is roughly 250MB of useful data per month.
If you can make changes on server side to send delivery receipts and handle the delivery errors on firmware level, you can consider using UDP.

Furthermore, if you are sending repetitive data, you may ignore the delivery errors.

Do not do that.  Unless you are a networking wizard (which the OP has assured us they are not) you should just pretend UDP doesn't exist.  That goes doubly so if you are talking about routing over the public internet like it appears the OP is doing.

.
.


Totally wrong. UDP is perfect for cases like this. In my work, we send UDP packages from 100.000+ tracking devices over GSM networks all over the world without any problems. On "important data" the servers sends a "ACK" package back to the device (which then removes the message from a circular buffer), the non-important stuff is just sent out. When the device sends a packet (f.ex the "ACK") to a server, the server can return data to the device within several seconds regardless the device is on a "local network" at the GSM providers. Just remember that out-of-order is possible for UDP packets. Everything else is a piece of cake
 

Offline hli

  • Frequent Contributor
  • **
  • Posts: 274
  • Country: de
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #27 on: October 27, 2020, 02:27:52 pm »
Is the TCP_NODELAY option set on the W5500? If yes, each byte is send immediately. (look up "Nagle's algorithm" for some detail). Disabling it might help.
 

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 4456
  • Country: us
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #28 on: October 27, 2020, 03:01:29 pm »
Is the TCP_NODELAY option set on the W5500? If yes, each byte is send immediately. (look up "Nagle's algorithm" for some detail). Disabling it might help.

Maybe.  I'm not actually sure the W5500 supports Nagles algorithm and it might not do as much as you hope if the code is slow and the network fast.  The way I know to do this on the w5500 is to set the port to idle while you write data the set the state to TCP_SEND and let it process the whole chunk.  But this functionality might not be exposed in the arduino libraries that prioritize simplicity over performance or functionality.
 

Offline amyk

  • Super Contributor
  • ***
  • Posts: 8928
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #29 on: October 28, 2020, 12:20:36 am »
This whole thread is a great "why abstraction is bad" example. It wastes the resources of the hardware, and of the humans who have to deal with the mess it creates.
 
The following users thanked this post: nuclearcat

Offline hans

  • Super Contributor
  • ***
  • Posts: 1857
  • Country: 00
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #30 on: October 28, 2020, 08:44:15 am »
I disagree. The whole point of writing an abstraction is to check if the output and operation is acceptable. An integration or unit test could have captured that.

Otherwise we all better go back to programming Assembly. C is too bloody complex to abstract CPU registers like as if they are variables that move around all by themselves!

Using Arduino libraries on the other hand in production... ouch. Finding high quality libraries is far from trivial
 

Offline pardo-bsso

  • Frequent Contributor
  • **
  • Posts: 259
  • Country: ar
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #31 on: November 04, 2020, 12:03:41 am »
Hey... OP,

did you solve your issue?
 

Offline honeybadgerTopic starter

  • Contributor
  • Posts: 42
  • Country: cz
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #32 on: November 10, 2020, 10:48:48 pm »
Hi,
quick update, sorry for not replying I don't have time to do anything.

client.print(F("Host: "));   - sends every character in separate packet
client.print("Host: ");   - sends one packet, BUT each client.print() sends separate packet
client.println("Host: ");  - same as previous BUT it sends additional packet for newline characters

Original Arduino Ethernet lib is f*cking useless :( Will try another lib when I have time.
 
The following users thanked this post: kripton2035, pardo-bsso

Offline jbb

  • Super Contributor
  • ***
  • Posts: 1301
  • Country: nz
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #33 on: November 10, 2020, 11:50:34 pm »
If you call client.print(“First\nSecond\n”) does it do that as a single packet?

If you’ve got enough RAM you would make a buffer, fill it with snprintf() and then call client.print(buffer)
 

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 4456
  • Country: us
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #34 on: November 11, 2020, 12:46:24 am »
Hi,
quick update, sorry for not replying I don't have time to do anything.

client.print(F("Host: "));   - sends every character in separate packet
client.print("Host: ");   - sends one packet, BUT each client.print() sends separate packet
client.println("Host: ");  - same as previous BUT it sends additional packet for newline characters

Original Arduino Ethernet lib is f*cking useless :( Will try another lib when I have time.

That's definitely a crappy library, but presumably designed for memory efficiency and simplicity above anything else including performance.  If you have enough SRAM available just snprintf() into a buffer and client.print() it in a single go, or at least in fairly large chunks.  That will be easier than re-building an ethernet library from scratch.
 
The following users thanked this post: nctnico

Offline Fredderic

  • Regular Contributor
  • *
  • Posts: 69
  • Country: au
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #35 on: November 11, 2020, 04:31:31 pm »
presumably designed for memory efficiency and simplicity above anything else including performance.

That's exactly what it is.  Considering the kind of tasks Arduino is targeted at; even a slow ųC is usually ample fast enough , the Ethernet library already burns a chunk of memory, and a single Ethernet packet itself can be quite a bit larger than the total RAM available on many of the ųC's being used… what size buffer do you expect them to allocate for you, so that you can plod along your merry way and don't have to think about what you're doing?  As it is, one of my projects has to use an older version of the standard library because the hundred bytes or so extra bytes used by the newer ones is too much.

And further, considering an Arduino generally isn't sending over SSL, you really probably shouldn't be using it anywhere except a LAN anyhow, and those Ethernet chips are likely slow enough you don't have a hope of saturating the link either.  So who cares if it sends every character as a separate packet, if it saves having to burn yet more of your super valuable RAM for a buffer?

Basically, it's assuming you either know what you're doing, or aren't using it in an environment where sending a single character as a packet is going to be a problem.  So, doing it the way it does, leaves the choice up to you:
  • Minimise memory by sending each individual piece of data as a separate packet (no buffering needed).
  • Minimise bandwidth by buffering the data yourself, and sending the whole chunk as a single packet.

The thing with F-strings is…  annoying.  But makes sense; the stream print command isn't buffering, it just takes the data it's given, and sends it as a packet.  And again to avoid buffering, the println() command is basically just two print()'s internally, one for the data, and one for the '\n'.  You simply need to know these things, and code accordingly.

There are also buffer writers right there in the system that will collect your prints for you into a buffer you provide (or use String, which does it on the heap)…  This stuff all exists already.  It all makes a lot of perfectly good sense when you stop and think it through — though, not that that's ever stopped anything from going horribly wrong.  Point is, criticising it before you understand it…   |O

Though I'd have thought the networking chip has it's own outbound buffer…  but probably also has some tight limits that would be very hard to express to novice users of the library, so they simply opted for one-packet-per-write, and let you buffer it yourself if needed.

Personally, I hadn't come across that issue because I almost always prefer the C++ << streaming style into locally allocated buffers — it just feels like a more natural style to me, and compiles down to the same or better anyhow.  So I almost always allocate a local buffer, hand it to an instance of a buffer writer, and then when it's all done, just send the completed buffer to wherever it needs to go.  Avoids the whole issue that way.
 

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 4456
  • Country: us
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #36 on: November 11, 2020, 09:54:04 pm »
The chip has an onboard buffer but by default it transmits as soon as it gets data in the buffer.  What you can do is pause the socket, execute multiple buffer writes, and then unpause and let it send the entire buffer in a small number of packets.  But it is hard to implement that transparently and while it could be exposed to the end user is risky.

The other problem is that it is using a simple wrapper to implement the F string version in terms of single character writes.  A better library would provide an F string send function that would handle this properly and do so without requiring a large buffer.

On the other hand I have a pretty dim view of using any microcontroller for networking much less a dinky AVR.  They are just too limited to do it properly even with an offload chip like the wiznet devices.
 

Offline kripton2035

  • Super Contributor
  • ***
  • Posts: 2818
  • Country: fr
    • kripton2035 schematics repository
Re: W5500 ethernet... how in the hell it sends 10GB per month?!
« Reply #37 on: November 11, 2020, 10:22:32 pm »
just curious, does this also happen with the ethernet library of an esp8266 or esp32 using the arduino IDE ?
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf