Anyone here familiar with LWIP?

[peter-h]

My boxes are 10/100 and I do have a hub.

Is there anything in LWIP debug output indicating packet re-transmission due to errors?

What would be a broadcast used for, in the context of an embedded box? What data does LWIP return when it receives a broadcast packet?

[dare]

QoS support in the switch should make no difference.  Its doubtful anything on your network is using QoS.

You log output is jumbled and/or missing lines.  For example, when an IP packet arrives, your log shows:

Code: [Select]

99586: ip4_input: packet accepted on interface st
99587: | 4 | 5 |  0x00 |        40     | (v, hl, tos, len)
99587: |     9821      |000|       0   | (id, flags, offset)
99587: +-------------------------------+
99587: |  192  |  168  |    3  |   64  | (dest)
99587: pbuf_header: old 200050FE new 20005112 (-20)
99587: |    10007      |    49154      | (src port, dest port)
99587: |           0000029846          | (ack no)
99587: |  5 |   |010000|      1917     | (hdrlen, flags (
99587: |    0xc2f3     |         0     | (chksum, urgp)
99588: +-+-+-+-+-+-+-+-+-+-+-+-+-+- tcp_input: flags
99588: State: ESTABLISHED

Instead, when a packet arrives, the LwIP log output should look like this:

Code: [Select]

tcpip_thread: PACKET 0x563fa1ea8860
ip_input: iphdr->dest 0xc801a8c0 netif->ip_addr 0xc801a8c0 (0x1a8c0, 0x1a8c0, 0xc8000000)
ip4_input: packet accepted on interface tp
ip4_input:
IP header:
+-------------------------------+
| 4 | 5 |  0x00 |        60     | (v, hl, tos, len)
+-------------------------------+
|    58354      |010|       0   | (id, flags, offset)
+-------------------------------+
|   64  |    6  |    0xd2af     | (ttl, proto, chksum)
+-------------------------------+
|  192  |  168  |    1  |    1  | (src)
+-------------------------------+
|  192  |  168  |    1  |  200  | (dest)
+-------------------------------+
ip4_input: p->len 60 p->tot_len 60
TCP header:
+-------------------------------+
|    44738      |       80      | (src port, dest port)
+-------------------------------+
|           0151356050          | (seq no)
+-------------------------------+
|           0000000000          | (ack no)
+-------------------------------+
| 10 |   |000010|     64240     | (hdrlen, flags (SYN), win)
+-------------------------------+
|    0x2fcd     |         0     | (chksum, urgp)
+-------------------------------+
tcp_input: packed for LISTENing connection.
TCP connection request 44738 -> 80.

Is your logging mechanism reliable?  In particular, does it queue/block when log output is printed quickly?  If not, you will need to fix that.  Without reliable logging you will not be able to diagnose problems in LwIP.

As others have said, full fidelity packet captures are also a must when diagnosing subtle networking issues.  If you have a linux computer with two ethernet interfaces (e.g. a built-in ethernet port and a USB ethernet dongle) you can configure the computer to act as a bridge between the two network segments (with your embedded devices on either side of the bridge).  Once this is configured, any traffic passing across the bridge be capture by a Wireshark instance running on the computer.

All that said, the problem can likely be diagnosed via the LwIP logs, so get that working first.

What would be a broadcast used for, in the context of an embedded box? What data does LWIP return when it receives a broadcast packet?

There are various uses for broadcast/multicast in typical LANs.  The most common examples are ARP and DHCP for IPv4, and Neighbor Discovery for IPv6.  If your device speaks IPv4 only then it will need to support ARP and possibly DHCP.  All other inbound broadcast/multicast packets will be received but should be ignored by LwIP.

[ejeffrey]

It sounds like you are only transmitting serial port speed over the network.  Ordinary broadcast traffic shouldn't pose any problem at all. 

You say your device are 10/100.... Are any of your devices set to forced full duplex mode?  If so, disable that. Set to auto negotiation or forced half duplex.

[peter-h]

The debugs come out via USB VCP which is very fast - hundreds of k bytes per second. But each line is truncated to 128 bytes. I can change that... I am logging the data with Teraterm, and flow control was disabled to make the debug function non-blocking if there was no terminal on the VCP. I have now enabled flow control and see that the traffic runs a bit slower, as expected.

The ETH port is auto select, or should be. DHCP is currently off; I am using fixed IP. The two IPs are 192.168.3.64 and .68.

Samples:

Random place:

Code: [Select]

36805: State: ESTABLISHED

36781: pbuf_alloc(length=182)

36807: pbuf_alloc: allocated pbuf 20005750

36807: pbuf_alloc(length=182) == 20005750

36806: tcp_fasttmr: delayed ACK

36809: pbuf_alloc(length=20)

36810: pbuf_alloc(length=20) == 20002BD4

36810: tcp_output: sending ACK for 842989

36811: pbuf_header: old 20002C08 new 20002BF4 (20)

36811: ip4_output_if: st0

36812: IP header:

36813: +-------------------------------+

36813: | 4 | 5 |  0x00 |        40     | (v, hl, tos, len)

36814: +-------------------------------+

36815: |       17      |000|       0   | (id, flags, offset)

36815: +-------------------------------+

36816: |  255  |    6  |    0x0000     | (ttl, proto, chksum)

36816: +-------------------------------+

36817: |  192  |  168  |    3  |   64  | (src)

36817: +-------------------------------+

36818: |  192  |  168  |    3  |   68  | (dest)

36818: +-------------------------------+

36819: ip4_output_if: call netif->output()

36820: pbuf_header: old 20002BF4 new 20002BE6 (14)

36820: pbuf_free(20002BD4)

36821: pbuf_free: deallocating 20002BD4

36821: tcpip_thread: PACKET 20006940

36822: pbuf_header: old 20005760 new 2000576E (-14)

36822: ip_input: iphdr->dest 0x4003a8c0 netif->ip_addr 0x4003a8c0 (0x3a8c0, 0x3a8c0, 0x40000000)

36823: ip4_input: packet accepted on interface st

36824: ip4_input:

36825: IP header:

36825: +-------------------------------+

36826: | 4 | 5 |  0x00 |       168     | (v, hl, tos, len)

36826: +-------------------------------+

36827: |     1468      |000|       0   | (id, flags, offset)

36827: +-------------------------------+

36828: |  255  |    6  |    0x2dbf     | (ttl, proto, chksum)

36828: +-------------------------------+

36829: |  192  |  168  |    3  |   68  | (src)

36829: +-------------------------------+

36830: |  192  |  168  |    3  |   64  | (dest)

36830: +-------------------------------+

36831: ip4_input: p->len 168 p->tot_len 168

36831: pbuf_header: old 2000576E new 20005782 (-20)

36832: TCP header:

36833: +-------------------------------+

36833: |    10007      |    49154      | (src port, dest port)

36834: +-------------------------------+

36835: |           0000842989          | (seq no)

36835: +-------------------------------+

36836: |           0000007406          | (ack no)

36836: +-------------------------------+

36837: |  5 |   |011000|      2024     | (hdrlen, flags (
36837: PSH
36838: ACK
36838:

36839: ), win)

36839: +-------------------------------+

36840: |    0x3796     |         0     | (chksum, urgp)

36841: +-------------------------------+

36842: pbuf_header: old 20005782 new 20005796 (-20)

36842: +-+-+-+-+-+-+-+-+-+-+-+-+-+- tcp_input: flags
36843: PSH
36844: ACK
36844:

36845: -+-+-+-+-+-+-+-+-+-+-+-+-+-+

36845: State: ESTABLISHED

36846: tcp_receive: window update 2024

36846: tcp_receive: pcb->rttest 0 rtseq 7150 ackno 7406

36847: tcp_output: nothing to send (00000000)

36847: tcp_output: snd_wnd 2024, cwnd 7479, wnd 2024, seg == NULL, ack 7406

36848: State: ESTABLISHED


One area hopefully captured during a traffic interruption:

Code: [Select]

109346: lwip_recvfrom: top while sock->lastdata=00000000

109347: lwip_recvfrom(2): returning EWOULDBLOCK

109350: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)

109350: lwip_recvfrom: top while sock->lastdata=00000000

109351: lwip_recvfrom(0): returning EWOULDBLOCK

109358: lwip_recvfrom(1, 20010088, 128, 0x0, ..)

109358: lwip_recvfrom: top while sock->lastdata=00000000

109359: pbuf_alloc(length=182)

109360: pbuf_alloc: allocated pbuf 20004A70

109360: pbuf_alloc(length=182) == 20004A70

109361: tcpip_thread: PACKET 200068C8

109361: pbuf_header: old 20004A80 new 20004A8E (-14)

109362: ip_input: iphdr->dest 0x4003a8c0 netif->ip_addr 0x4003a8c0 (0x3a8c0, 0x3a8c0, 0x40000000)

109362: ip4_input: packet accepted on interface st

109363: ip4_input:

109364: IP header:

109364: +-------------------------------+

109365: | 4 | 5 |  0x00 |       168     | (v, hl, tos, len)

109365: +-------------------------------+

109366: |     1968      |000|       0   | (id, flags, offset)

109366: +-------------------------------+

109367: |  255  |    6  |    0x2bcb     | (ttl, proto, chksum)

109367: +-------------------------------+

109368: |  192  |  168  |    3  |   68  | (src)

109368: +-------------------------------+

109369: |  192  |  168  |    3  |   64  | (dest)

109369: +-------------------------------+

109370: ip4_input: p->len 168 p->tot_len 168

109370: pbuf_header: old 20004A8E new 20004AA2 (-20)

109371: TCP header:

109372: +-------------------------------+

109372: |    10007      |    49154      | (src port, dest port)

109373: +-------------------------------+

109374: |           0000888733          | (seq no)

109374: +-------------------------------+

109375: |           0000053510          | (ack no)

109375: +-------------------------------+

109376: |  5 |   |011000|      2432     | (hdrlen, flags (
109376: PSH
109377: ACK
109377:

109378: ), win)

109378: +-------------------------------+

109379: |    0x86ed     |         0     | (chksum, urgp)

109379: +-------------------------------+

109380: pbuf_header: old 20004AA2 new 20004AB6 (-20)

109380: +-+-+-+-+-+-+-+-+-+-+-+-+-+- tcp_input: flags
109381: PSH
109382: ACK
109382:

109383: -+-+-+-+-+-+-+-+-+-+-+-+-+-+

109383: State: ESTABLISHED

109384: tcp_receive: pcb->rttest 0 rtseq 53406 ackno 53510

109384: tcp_output: nothing to send (00000000)

109385: tcp_output: snd_wnd 2432, cwnd 38837, wnd 2432, seg == NULL, ack 53510

109385: State: ESTABLISHED

109363: pbuf_alloc(length=182)

109387: pbuf_alloc: allocated pbuf 200050E0

109387: pbuf_alloc(length=182) == 200050E0

109386: tcp_fasttmr: delayed ACK

109389: pbuf_alloc(length=20)

109389: pbuf_alloc(length=20) == 20002BD4

109390: tcp_output: sending ACK for 888861

109390: pbuf_header: old 20002C08 new 20002BF4 (20)

109391: ip4_output_if: st0

109392: IP header:

109392: +-------------------------------+

109393: | 4 | 5 |  0x00 |        40     | (v, hl, tos, len)

109393: +-------------------------------+

109394: |      610      |000|       0   | (id, flags, offset)

109394: +-------------------------------+

109395: |  255  |    6  |    0x0000     | (ttl, proto, chksum)

109395: +-------------------------------+

109396: |  192  |  168  |    3  |   64  | (src)

109396: +-------------------------------+

109397: |  192  |  168  |    3  |   68  | (dest)

109397: +-------------------------------+

109398: ip4_output_if: call netif->output()

109398: pbuf_header: old 20002BF4 new 20002BE6 (14)

109399: pbuf_free(20002BD4)

109400: pbuf_free: deallocating 20002BD4

109400: tcpip_thread: PACKET 20006940

109401: pbuf_header: old 200050F0 new 200050FE (-14)

109401: ip_input: iphdr->dest 0x4003a8c0 netif->ip_addr 0x4003a8c0 (0x3a8c0, 0x3a8c0, 0x40000000)

109402: ip4_input: packet accepted on interface st

109403: ip4_input:

109404: IP header:

109404: +-------------------------------+

109405: | 4 | 5 |  0x00 |       168     | (v, hl, tos, len)

109405: +-------------------------------+

109406: |     1969      |000|       0   | (id, flags, offset)

109406: +-------------------------------+

109407: |  255  |    6  |    0x2bca     | (ttl, proto, chksum)

109407: +-------------------------------+

109408: |  192  |  168  |    3  |   68  | (src)

109408: +-------------------------------+

109409: |  192  |  168  |    3  |   64  | (dest)

109409: +-------------------------------+

109410: ip4_input: p->len 168 p->tot_len 168

109410: pbuf_header: old 200050FE new 20005112 (-20)

109411: TCP header:

109412: +-------------------------------+

109412: |    10007      |    49154      | (src port, dest port)

109413: +-------------------------------+

109414: |           0000888861          | (seq no)

109415: +-------------------------------+

109416: |           0000053510          | (ack no)

109416: +-------------------------------+

109417: |  5 |   |011000|      2432     | (hdrlen, flags (
109417: PSH
109418: ACK
109418:

109419: ), win)

109419: +-------------------------------+

109420: |    0x0171     |         0     | (chksum, urgp)

109420: +-------------------------------+

109421: pbuf_header: old 20005112 new 20005126 (-20)

109421: +-+-+-+-+-+-+-+-+-+-+-+-+-+- tcp_input: flags
109422: PSH
109423: ACK
109423:

109424: -+-+-+-+-+-+-+-+-+-+-+-+-+-+

109424: State: ESTABLISHED

109425: tcp_receive: pcb->rttest 0 rtseq 53406 ackno 53510

109425: tcp_output: nothing to send (00000000)

109426: tcp_output: snd_wnd 2432, cwnd 38837, wnd 2432, seg == NULL, ack 53510

109426: State: ESTABLISHED

109372: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)

109428: lwip_recvfrom: top while sock->lastdata=00000000

109429: lwip_recvfrom(0): returning EWOULDBLOCK

109359: lwip_recvfrom(1): returning EWOULDBLOCK

109431: lwip_recvfrom(2, 20010088, 128, 0x0, ..)

109431: lwip_recvfrom: top while sock->lastdata=00000000

109432: tcp_recved: received 128 bytes, wnd 2792 (128).

109432: netconn_recv_data: received 20004A70, len=128

109433: lwip_recvfrom: netconn_recv err=0, netbuf=20004A70

109433: lwip_recvfrom: buflen=128 len=128 off=0 sock->lastoffset=0

109434: lwip_recvfrom(2): addr=
109435: 192.168.3.68
109435:  port=10007 len=128

109436: lwip_recvfrom: deleting netbuf=20004A70

109436: pbuf_free(20004A70)

109437: pbuf_free: deallocating 20004A70

109447: lwip_recvfrom(1, 20010088, 128, 0x0, ..)

109447: lwip_recvfrom: top while sock->lastdata=00000000

109447: lwip_recvfrom(1): returning EWOULDBLOCK

109448: lwip_recvfrom(2, 20010088, 128, 0x0, ..)

109448: lwip_recvfrom: top while sock->lastdata=00000000

109449: tcp_recved: received 128 bytes, wnd 2920 (0).

109450: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)

109451: lwip_recvfrom: top while sock->lastdata=00000000

109452: lwip_recvfrom(0): returning EWOULDBLOCK

109450: netconn_recv_data: received 200050E0, len=128

109454: lwip_recvfrom: netconn_recv err=0, netbuf=200050E0

109454: lwip_recvfrom: buflen=128 len=128 off=0 sock->lastoffset=0

109455: lwip_recvfrom(2): addr=
109456: 192.168.3.68
109456:  port=10007 len=128

109457: lwip_recvfrom: deleting netbuf=200050E0

109457: pbuf_free(200050E0)

109458: pbuf_free: deallocating 200050E0

109468: lwip_recvfrom(1, 20010088, 128, 0x0, ..)

109468: lwip_recvfrom: top while sock->lastdata=00000000

109468: lwip_recvfrom(1): returning EWOULDBLOCK

109469: lwip_recvfrom(2, 20010088, 128, 0x0, ..)

109469: lwip_recvfrom: top while sock->lastdata=00000000

109470: lwip_recvfrom(2): returning EWOULDBLOCK

109473: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)

109473: lwip_recvfrom: top while sock->lastdata=00000000

109474: lwip_recvfrom(0): returning EWOULDBLOCK


This bit was captured during a period of a few seconds where traffic was being blocked by something

Code: [Select]

150169: lwip_recvfrom(1): returning EWOULDBLOCK

150170: lwip_recvfrom(2, 20010088, 128, 0x0, ..)

150170: lwip_recvfrom: top while sock->lastdata=00000000

150171: pbuf_alloc(length=182)

150172: pbuf_alloc: allocated pbuf 20004A70

150172: pbuf_alloc(length=182) == 20004A70

150171: tcp_recved: received 128 bytes, wnd 2920 (0).

150173: tcpip_thread: PACKET 20006904

150174: pbuf_header: old 20004A80 new 20004A8E (-14)

150174: ip_input: iphdr->dest 0x4003a8c0 netif->ip_addr 0x4003a8c0 (0x3a8c0, 0x3a8c0, 0x40000000)

150175: ip4_input: packet accepted on interface st

150176: ip4_input:

150177: IP header:

150178: +-------------------------------+

150178: | 4 | 5 |  0x00 |       168     | (v, hl, tos, len)

150179: +-------------------------------+

150180: |     2206      |000|       0   | (id, flags, offset)

150180: +-------------------------------+

150181: |  255  |    6  |    0x2add     | (ttl, proto, chksum)

150181: +-------------------------------+

150182: |  192  |  168  |    3  |   68  | (src)

150182: +-------------------------------+

150183: |  192  |  168  |    3  |   64  | (dest)

150183: +-------------------------------+

150184: ip4_input: p->len 168 p->tot_len 168

150184: pbuf_header: old 20004A8E new 20004AA2 (-20)

150185: TCP header:

150186: +-------------------------------+

150186: |    10007      |    49154      | (src port, dest port)

150187: +-------------------------------+

150188: |           0000907989          | (seq no)

150188: +-------------------------------+

150189: |           0000072406          | (ack no)

150189: +-------------------------------+

150190: |  5 |   |011000|      1536     | (hdrlen, flags (
150190: PSH
150191: ACK
150192:

150192: ), win)

150193: +-------------------------------+

150193: |    0x3dac     |         0     | (chksum, urgp)

150194: +-------------------------------+

150195: pbuf_header: old 20004AA2 new 20004AB6 (-20)

150195: +-+-+-+-+-+-+-+-+-+-+-+-+-+- tcp_input: flags
150196: PSH
150197: ACK
150197:

150198: -+-+-+-+-+-+-+-+-+-+-+-+-+-+

150198: State: ESTABLISHED

150199: tcp_receive: window update 1536

150199: tcp_receive: congestion avoidance cwnd 45404

150200: tcp_receive: ACK for 72406, unacked->seqno 72278:72406

150201: tcp_receive: removing 72278:72406 from pcb->unacked

150202: tcp_receive: queuelen 1 ...
150203: pbuf_free(20002BD4)

150203: pbuf_free: deallocating 20002BD4

150204: 0 (after freeing unacked)

150204: tcp_receive: pcb->rttest 275 rtseq 72278 ackno 72406

150205: tcp_receive: experienced rtt 0 ticks (0 msec).

150206: tcp_receive: RTO 3 (1500 milliseconds)

150207: tcp_output: nothing to send (00000000)

150207: tcp_output: snd_wnd 1536, cwnd 45404, wnd 1536, seg == NULL, ack 72406

150208: State: ESTABLISHED

150209: tcp_fasttmr: delayed ACK

150209: pbuf_alloc(length=20)

150210: pbuf_alloc(length=20) == 20002BD4

150210: tcp_output: sending ACK for 908117

150211: pbuf_header: old 20002C08 new 20002BF4 (20)

150211: ip4_output_if: st0

150212: IP header:

150212: +-------------------------------+

150213: | 4 | 5 |  0x00 |        40     | (v, hl, tos, len)

150213: +-------------------------------+

150214: |      872      |000|       0   | (id, flags, offset)

150214: +-------------------------------+

150215: |  255  |    6  |    0x0000     | (ttl, proto, chksum)

150215: +-------------------------------+

150216: |  192  |  168  |    3  |   64  | (src)

150216: +-------------------------------+

150217: |  192  |  168  |    3  |   68  | (dest)

150217: +-------------------------------+

150218: ip4_output_if: call netif->output()

150219: pbuf_header: old 20002BF4 new 20002BE6 (14)

150219: pbuf_free(20002BD4)

150220: pbuf_free: deallocating 20002BD4

150177: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)

150221: lwip_recvfrom: top while sock->lastdata=00000000

150221: lwip_recvfrom(0): returning EWOULDBLOCK

150173: netconn_recv_data: received 20005750, len=128

150223: lwip_recvfrom: netconn_recv err=0, netbuf=20005750

150224: lwip_recvfrom: buflen=128 len=128 off=0 sock->lastoffset=0

150224: lwip_recvfrom(2): addr=
150225: 192.168.3.68
150225:  port=10007 len=128

150226: lwip_recvfrom: deleting netbuf=20005750

150226: pbuf_free(20005750)

150227: pbuf_free: deallocating 20005750

150227: lwip_send(2, data=20010088, size=104, flags=0x0)

150228: tcp_write(pcb=20008E00, data=20010088, len=104, apiflags=1)

150228: tcp_write: queuelen: 0

150229: pbuf_alloc(length=104)

150230: pbuf_alloc(length=104) == 20002BD4

150230: pbuf_header: old 20002C1C new 20002C08 (20)

150231: tcp_write: queueing 72406:72510

150232: tcp_write: 1 (after enqueued)

150233: tcp_output: snd_wnd 1536, cwnd 45404, wnd 1536, effwnd 104, seq 72406, ack 72406

150233: tcp_output: snd_wnd 1536, cwnd 45404, wnd 1536, effwnd 104, seq 72406, ack 72406, i 0

150234: tcp_output_segment: rtseq 72406

150235: tcp_output_segment: 72406:72510

150236: pbuf_header: old 20002C08 new 20002BF4 (20)

150236: ip4_output_if: st0

150237: IP header:

150238: +-------------------------------+

150238: | 4 | 5 |  0x00 |       144     | (v, hl, tos, len)

150239: +-------------------------------+

150240: |      873      |000|       0   | (id, flags, offset)

150240: +-------------------------------+

150241: |  255  |    6  |    0x0000     | (ttl, proto, chksum)

150241: +-------------------------------+

150242: |  192  |  168  |    3  |   64  | (src)

150242: +-------------------------------+

150243: |  192  |  168  |    3  |   68  | (dest)

150243: +-------------------------------+

150244: ip4_output_if: call netif->output()

150245: pbuf_header: old 20002BF4 new 20002BE6 (14)

150242: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)

150246: lwip_recvfrom: top while sock->lastdata=00000000

150246: lwip_recvfrom(0): returning EWOULDBLOCK

150245: lwip_send(2) err=0 written=104

150255: pbuf_alloc(length=60)

150255: pbuf_alloc: allocated pbuf 20005750

150255: pbuf_alloc(length=60) == 20005750

150256: tcpip_thread: PACKET 20006904

150256: pbuf_header: old 20005760 new 2000576E (-14)

150257: ip_input: iphdr->dest 0x4003a8c0 netif->ip_addr 0x4003a8c0 (0x3a8c0, 0x3a8c0, 0x40000000)

150257: ip4_input: packet accepted on interface st

150258: ip4_input:

150259: IP header:

150260: +-------------------------------+

150260: | 4 | 5 |  0x00 |        40     | (v, hl, tos, len)

150261: +-------------------------------+

150262: |     2207      |000|       0   | (id, flags, offset)

150262: +-------------------------------+

150263: |  255  |    6  |    0x2b5c     | (ttl, proto, chksum)

150263: +-------------------------------+

150264: |  192  |  168  |    3  |   68  | (src)

150264: +-------------------------------+

150265: |  192  |  168  |    3  |   64  | (dest)

150265: +-------------------------------+

150266: ip4_input: p->len 40 p->tot_len 40

150267: pbuf_header: old 2000576E new 20005782 (-20)

150267: TCP header:

150268: +-------------------------------+

150268: |    10007      |    49154      | (src port, dest port)

150269: +-------------------------------+

150270: |           0000908117          | (seq no)

150270: +-------------------------------+

150271: |           0000072510          | (ack no)

150271: +-------------------------------+

150272: |  5 |   |010000|      2920     | (hdrlen, flags (
150272: ACK
150273:

150273: ), win)

150274: +-------------------------------+

150274: |    0x3edc     |         0     | (chksum, urgp)

150275: +-------------------------------+

150276: pbuf_header: old 20005782 new 20005796 (-20)

150276: +-+-+-+-+-+-+-+-+-+-+-+-+-+- tcp_input: flags
150277: ACK
150278:

150279: -+-+-+-+-+-+-+-+-+-+-+-+-+-+

150279: State: ESTABLISHED

150280: tcp_receive: window update 2920

150280: tcp_receive: congestion avoidance cwnd 45450

150281: tcp_receive: ACK for 72510, unacked->seqno 72406:72510

150281: tcp_receive: removing 72406:72510 from pcb->unacked

150282: tcp_receive: queuelen 1 ...
150283: pbuf_free(20002BD4)

150283: pbuf_free: deallocating 20002BD4

150284: 0 (after freeing unacked)

150284: tcp_receive: pcb->rttest 275 rtseq 72406 ackno 72510

150285: tcp_receive: experienced rtt 0 ticks (0 msec).

150286: tcp_receive: RTO 3 (1500 milliseconds)

150287: tcp_output: nothing to send (00000000)

150287: tcp_output: snd_wnd 2920, cwnd 45450, wnd 2920, seg == NULL, ack 72510

150288: State: ESTABLISHED

150289: pbuf_free(20005750)

150289: pbuf_free: deallocating 20005750

150288: pbuf_alloc(length=158)

150291: pbuf_alloc: allocated pbuf 20005750

150291: pbuf_alloc(length=158) == 20005750

150292: tcpip_thread: PACKET 20006904

150292: pbuf_header: old 20005760 new 2000576E (-14)

150293: ip_input: iphdr->dest 0x4003a8c0 netif->ip_addr 0x4003a8c0 (0x3a8c0, 0x3a8c0, 0x40000000)

150293: ip4_input: packet accepted on interface st

150294: ip4_input:

150295: IP header:

150295: +-------------------------------+

150296: | 4 | 5 |  0x00 |       144     | (v, hl, tos, len)

150296: +-------------------------------+

150297: |     2208      |000|       0   | (id, flags, offset)

150297: +-------------------------------+

150298: |  255  |    6  |    0x2af3     | (ttl, proto, chksum)

150298: +-------------------------------+

150299: |  192  |  168  |    3  |   68  | (src)

150299: +-------------------------------+

150300: |  192  |  168  |    3  |   64  | (dest)

150300: +-------------------------------+

150301: ip4_input: p->len 144 p->tot_len 144

150301: pbuf_header: old 2000576E new 20005782 (-20)

150302: TCP header:

150303: +-------------------------------+

150303: |    10007      |    49154      | (src port, dest port)

150304: +-------------------------------+

150305: |           0000908117          | (seq no)

150305: +-------------------------------+

150306: |           0000072510          | (ack no)

150306: +-------------------------------+

150307: |  5 |   |011000|      2920     | (hdrlen, flags (
150307: PSH
150308: ACK
150308:

150309: ), win)

150309: +-------------------------------+

150310: |    0x8617     |         0     | (chksum, urgp)

150310: +-------------------------------+

150311: pbuf_header: old 20005782 new 20005796 (-20)

150311: +-+-+-+-+-+-+-+-+-+-+-+-+-+- tcp_input: flags
150312: PSH
150313: ACK
150313:

150314: -+-+-+-+-+-+-+-+-+-+-+-+-+-+

150314: State: ESTABLISHED

150315: tcp_receive: pcb->rttest 0 rtseq 72406 ackno 72510

150315: tcp_output: nothing to send (00000000)

150316: tcp_output: snd_wnd 2920, cwnd 45450, wnd 2920, seg == NULL, ack 72510

150316: State: ESTABLISHED

150267: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)

150318: lwip_recvfrom: top while sock->lastdata=00000000

150318: lwip_recvfrom(0): returning EWOULDBLOCK

150258: lwip_recvfrom(1, 20010088, 128, 0x0, ..)

150320: lwip_recvfrom: top while sock->lastdata=00000000

150321: lwip_recvfrom(1): returning EWOULDBLOCK

150322: lwip_recvfrom(2, 20010088, 128, 0x0, ..)

150322: lwip_recvfrom: top while sock->lastdata=00000000

150323: tcp_recved: received 128 bytes, wnd 2816 (104).

150323: netconn_recv_data: received 20004A70, len=128

150324: lwip_recvfrom: netconn_recv err=0, netbuf=20004A70

150324: lwip_recvfrom: buflen=128 len=128 off=0 sock->lastoffset=0

150325: lwip_recvfrom(2): addr=
150326: 192.168.3.68
150326:  port=10007 len=128

150327: lwip_recvfrom: deleting netbuf=20004A70

150327: pbuf_free(20004A70)

150328: pbuf_free: deallocating 20004A70

150338: lwip_recvfrom(1, 20010088, 128, 0x0, ..)

150338: lwip_recvfrom: top while sock->lastdata=00000000

150339: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)

150340: lwip_recvfrom: top while sock->lastdata=00000000

150340: lwip_recvfrom(0): returning EWOULDBLOCK

150339: lwip_recvfrom(1): returning EWOULDBLOCK

150342: lwip_recvfrom(2, 20010088, 128, 0x0, ..)

150342: lwip_recvfrom: top while sock->lastdata=00000000

150343: pbuf_alloc(length=20)

150344: pbuf_alloc(length=20) == 20002BD4

150344: tcp_output: sending ACK for 908221

150345: pbuf_header: old 20002C08 new 20002BF4 (20)

150345: ip4_output_if: st0

150346: IP header:

150347: +-------------------------------+

150347: | 4 | 5 |  0x00 |        40     | (v, hl, tos, len)

150348: +-------------------------------+

150349: |      874      |000|       0   | (id, flags, offset)

150349: +-------------------------------+

150350: |  255  |    6  |    0x0000     | (ttl, proto, chksum)

150350: +-------------------------------+

150351: |  192  |  168  |    3  |   64  | (src)

150351: +-------------------------------+

150352: |  192  |  168  |    3  |   68  | (dest)

150352: +-------------------------------+

150353: ip4_output_if: call netif->output()

150354: pbuf_header: old 20002BF4 new 20002BE6 (14)

150354: pbuf_free(20002BD4)

150355: pbuf_free: deallocating 20002BD4

150355: tcp_recved: received 104 bytes, wnd 2920 (0).

150356: netconn_recv_data: received 20005750, len=104

150356: lwip_recvfrom: netconn_recv err=0, netbuf=20005750

150357: lwip_recvfrom: buflen=104 len=128 off=0 sock->lastoffset=0

150358: lwip_recvfrom(2): addr=
150359: 192.168.3.68
150359:  port=10007 len=104

150360: lwip_recvfrom: deleting netbuf=20005750

150360: pbuf_free(20005750)

150361: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)

150362: lwip_recvfrom: top while sock->lastdata=00000000

150362: lwip_recvfrom(0): returning EWOULDBLOCK

150361: pbuf_free: deallocating 20005750

150374: lwip_recvfrom(1, 20010088, 128, 0x0, ..)

150374: lwip_recvfrom: top while sock->lastdata=00000000

150374: lwip_recvfrom(1): returning EWOULDBLOCK

150375: lwip_recvfrom(2, 20010088, 128, 0x0, ..)

150375: lwip_recvfrom: top while sock->lastdata=00000000

150376: lwip_recvfrom(2): returning EWOULDBLOCK

150383: lwip_recvfrom(0, 100053BC, 100, 0x8, ..)

150383: lwip_recvfrom: top while sock->lastdata=00000000

150384: lwip_recvfrom(0): returning EWOULDBLOCK


I noticed e.g.

tcp_receive: congestion avoidance cwnd 45404

which googles to interesting stuff but way above my pay grade. Like this
https://lists.gnu.org/archive/html/lwip-users/2017-08/msg00043.html

My lwipopts.h is below. I spent many days tweaking different parameters and determining experimentally how much RAM each option uses up

Code: [Select]

/**
  ******************************************************************************
  * @file    LwIP/LwIP_HTTP_Server_Netconn_RTOS/Inc/lwipopts.h
  * @author  MCD Application Team
  * @brief   lwIP Options Configuration.
  ******************************************************************************
*
* This sort of explains the memory usage
* [url]https://lwip-users.nongnu.narkive.com/dkzkPa8l/lwip-memory-settings[/url]
* [url]https://www.cnblogs.com/shangdawei/p/3494148.html[/url]
* [url]https://lwip.fandom.com/wiki/Tuning_TCP[/url]
* [url]https://groups.google.com/g/osdeve_mirror_tcpip_lwip/c/lFYJ7Fw0Cxg[/url]
* ST UM1713 document gives an overview of integrating all this.
*
*
*
*
* 7/7/22 PH MEM_SIZE set to 5k (was 10k). Only ~1.5k is used.
* 13/7/22 PH MEMP_MEM_MALLOC=1, MEM_SIZE=16k.
* 14/7/22 PH MEMP_MEM_MALLOC=0; 1 was unreliable. 8 x 512 byte buffers now.
* 6/8/22 PH 4 x MTU buffers for RX. Done partly for EditGetData().
* MEM_SIZE=6k. 5k is used - see .map file for ram_heap address.
* Sizes of various static RAM structures determined experimentally.
*  17/12/22 PH LWIP_TCP_KEEPALIVE=1 (for ethser)
*
*
*
*
*
*
*
  */
#ifndef __LWIPOPTS_H__
#define __LWIPOPTS_H__

/**
 * NO_SYS==1: Provides VERY minimal functionality. Otherwise,
 * use lwIP facilities.
 */
#define NO_SYS                  0

// Flag to make LWIP API thread-safe. The netconn and socket APIs are claimed
// to be thread-safe anyway. The raw API is never thread-safe.
#define LWIP_TCPIP_CORE_LOCKING    1

// This places more objects into the static block defined by MEM_SIZE.
// Uses mem_malloc/mem_free instead of the lwip pool allocator.
// MEM_SIZE now needs to be increased by about 10k.
// It doesn't magically produce extra memory, and causes crashes.
// There is also a performance loss, apparently. AVOID.
#define MEMP_MEM_MALLOC 0


//NC: Need for sending PING messages by keepalive
#define LWIP_RAW 1
#define DEFAULT_RAW_RECVMBOX_SIZE 4

// For ETHSER
#define LWIP_TCP_KEEPALIVE 1

/*-----------------------------------------------------------------------------*/

/* LwIP Stack Parameters (modified compared to initialization value in opt.h) -*/
/* Parameters set in STM32CubeMX LwIP Configuration GUI -*/

/*----- Value in opt.h for LWIP_DNS: 0 -----*/
#define LWIP_DNS 1

/* ---------- Memory options ---------- */
/* MEM_ALIGNMENT: should be set to the alignment of the CPU for which
   lwIP is compiled. 4 byte alignment -> define MEM_ALIGNMENT to 4, 2
   byte alignment -> define MEM_ALIGNMENT to 2. */
#define MEM_ALIGNMENT           4

// MEM_SIZE: the size of the heap memory. This is a statically allocated block. You can find it
// in the .map file as the symbol ram_heap and you can see how much of this RAM gets used.
// If MEMP_MEM_MALLOC=0, this holds just the PBUF_ stuff.
// If MEMP_MEM_MALLOC=1 (which is not reliable) this greatly expands and needs 16k+.
// Empirically this needs to be big enough for at least 4 x PBUF_POOL_BUFSIZE.
// 6k yields a good speed and going to 8k+ makes a minimal improvement. The main
// factor affecting speed is the poll period in ethernetif_input().
// This value also limits the biggest block size sent out by netconn_write. With the
// MEM_SIZE of 6k, the biggest block netconn_write will send out is 4k.
#define MEM_SIZE                (6*1024)

// MEMP_ structures. Their sizes have been determined experimentally, by
// increasing them and seeing free RAM changing.

/* MEMP_NUM_PBUF: the number of memp struct pbufs. If the application
   sends a lot of data out of ROM (or other static memory), this
   should be set high. */
//NC: Increased to 20 for ethser
#define MEMP_NUM_PBUF           20 // each 1 is 20 bytes of static RAM

/* MEMP_NUM_UDP_PCB: the number of UDP protocol control blocks. One
   per active UDP "connection". */
#define MEMP_NUM_UDP_PCB        6 // each 1 is 32 bytes of static RAM

/* MEMP_NUM_TCP_PCB: the number of simultaneously active TCP
   connections. */
//NC: Increased to 20 for ethser
#define MEMP_NUM_TCP_PCB        20 // each 1 is 145 bytes of static RAM

//NC: Have more sockets available. Is set to 4 in opt.h
#define MEMP_NUM_NETCONN    10

/* MEMP_NUM_TCP_PCB_LISTEN: the number of listening TCP
   connections. */
//NC: Increased to 20 for ethser
#define MEMP_NUM_TCP_PCB_LISTEN 20 // each 1 is 28 bytes of static RAM

/* MEMP_NUM_TCP_SEG: the number of simultaneously queued TCP
   segments. */
// Was 8; increased to 16 as it improves ETHSER reliability when running
// HTTP server
#define MEMP_NUM_TCP_SEG        16 // each 1 is 20 bytes of static RAM

/* MEMP_NUM_SYS_TIMEOUT: the number of simulateously active
   timeouts. */
#define MEMP_NUM_SYS_TIMEOUT    10 // each 1 is 16 bytes of static RAM


/* ---------- Pbuf dynamically allocated buffer blocks  ---------- */

// These settings are probably for RECEIVE only and make negligible difference
// to performance, which is dominated by the low_level_input poll period. These
// PBUFs relate directly to the netconn API netbufs etc.

/* PBUF_POOL_SIZE: the number of buffers in the pbuf pool. */
// Statically allocated, so setting this to 2 frees up another 3k of RAM. But any
// value under 4 slows down the data rate a lot.
#define PBUF_POOL_SIZE           4

/* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool. */
// ** It is better to use 4 x MTU than 8 x 512 because in say EditGetData() you are **
// ** much more likely to get the whole file header in the first packet. With 512   **
// ** byte packets the CRLFCRLF marker is only ~64 bytes before the end of the pkt  **
// ** Same issue in UploadGetData where we need to get the whole header in.         **
#define PBUF_POOL_BUFSIZE        1500 + PBUF_LINK_ENCAPSULATION_HLEN + PBUF_LINK_HLEN + PBUF_IP_HLEN + PBUF_TRANSPORT_HLEN

/* --------------------------------------------------------------- */


/* ---------- TCP options ---------- */
#define LWIP_TCP                1
#define TCP_TTL                 255

/* Controls if TCP should queue segments that arrive out of
   order. Define to 0 if your device is low on memory. */
#define TCP_QUEUE_OOSEQ         0

/* TCP Maximum segment size. */
#define TCP_MSS                 (1500 - 40)   /* TCP_MSS = (Ethernet MTU - IP header size - TCP header size) */

/* TCP sender buffer space (bytes). */
#define TCP_SND_BUF             (4*TCP_MSS) // no effect on static RAM

/*  TCP_SND_QUEUELEN: TCP sender buffer space (pbufs). This must be at least
  as much as (2 * TCP_SND_BUF/TCP_MSS) for things to work. */
// Was 2*; increased to 4* as it improves ETHSER reliability when running
// HTTP server
#define TCP_SND_QUEUELEN        (4* TCP_SND_BUF/TCP_MSS) // (2* TCP_SND_BUF/TCP_MSS)

/* TCP advertised receive window. */
// Should be less than PBUF_POOL_SIZE * (PBUF_POOL_BUFSIZE - protocol headers)
#define TCP_WND                 (2*TCP_MSS) // no effect on static RAM


/* ---------- ICMP options ---------- */
#define LWIP_ICMP            1


/* ---------- DHCP options ---------- */
#define LWIP_DHCP               1


/* ---------- UDP options ---------- */
#define LWIP_UDP                1
#define UDP_TTL                 255


/* ---------- Statistics options ---------- */
#define LWIP_STATS 0

/* ---------- link callback options ---------- */
/* LWIP_NETIF_LINK_CALLBACK==1: Support a callback function from an interface
 * whenever the link changes (i.e., link down)
 * 8/2022 this is done from the low_level_input RTOS task.
 */
#define LWIP_NETIF_LINK_CALLBACK        0




/*
   --------------------------------------
   ---------- Checksum options ----------
   --------------------------------------
*/

/*
The STM32F4xx allows computing and verifying the IP, UDP, TCP and ICMP checksums by hardware:
 - To use this feature let the following define uncommented.
 - To disable it and process by CPU comment the  the checksum.
*/
#define CHECKSUM_BY_HARDWARE


#ifdef CHECKSUM_BY_HARDWARE
  /* CHECKSUM_GEN_IP==0: Generate checksums by hardware for outgoing IP packets.*/
  #define CHECKSUM_GEN_IP                 0
  /* CHECKSUM_GEN_UDP==0: Generate checksums by hardware for outgoing UDP packets.*/
  #define CHECKSUM_GEN_UDP                0
  /* CHECKSUM_GEN_TCP==0: Generate checksums by hardware for outgoing TCP packets.*/
  #define CHECKSUM_GEN_TCP                0
  /* CHECKSUM_CHECK_IP==0: Check checksums by hardware for incoming IP packets.*/
  #define CHECKSUM_CHECK_IP               0
  /* CHECKSUM_CHECK_UDP==0: Check checksums by hardware for incoming UDP packets.*/
  #define CHECKSUM_CHECK_UDP              0
  /* CHECKSUM_CHECK_TCP==0: Check checksums by hardware for incoming TCP packets.*/
  #define CHECKSUM_CHECK_TCP              0
  /* CHECKSUM_CHECK_ICMP==0: Check checksums by hardware for incoming ICMP packets.*/ 
  #define CHECKSUM_GEN_ICMP               0
#else
  /* CHECKSUM_GEN_IP==1: Generate checksums in software for outgoing IP packets.*/
  #define CHECKSUM_GEN_IP                 1
  /* CHECKSUM_GEN_UDP==1: Generate checksums in software for outgoing UDP packets.*/
  #define CHECKSUM_GEN_UDP                1
  /* CHECKSUM_GEN_TCP==1: Generate checksums in software for outgoing TCP packets.*/
  #define CHECKSUM_GEN_TCP                1
  /* CHECKSUM_CHECK_IP==1: Check checksums in software for incoming IP packets.*/
  #define CHECKSUM_CHECK_IP               1
  /* CHECKSUM_CHECK_UDP==1: Check checksums in software for incoming UDP packets.*/
  #define CHECKSUM_CHECK_UDP              1
  /* CHECKSUM_CHECK_TCP==1: Check checksums in software for incoming TCP packets.*/
  #define CHECKSUM_CHECK_TCP              1
  /* CHECKSUM_CHECK_ICMP==1: Check checksums by hardware for incoming ICMP packets.*/ 
  #define CHECKSUM_GEN_ICMP               1
#endif


/*
   ----------------------------------------------
   ---------- Sequential layer options ----------
   ----------------------------------------------
*/
/**
 * LWIP_NETCONN==1: Enable Netconn API (require to use api_lib.c)
 */
#define LWIP_NETCONN                    1

/*
   ------------------------------------
   ---------- Socket options ----------
   ------------------------------------
*/
/**
 * LWIP_SOCKET==1: Enable Socket API (require to use sockets.c)
 */
#define LWIP_SOCKET                     1

/*
   ------------------------------------
   ---------- httpd options ----------
   ------------------------------------
*/
/** Set this to 1 to include "fsdata_custom.c" instead of "fsdata.c" for the
 * file system (to prevent changing the file included in CVS) */
#define HTTPD_USE_CUSTOM_FSDATA   0

/*
   ---------------------------------
   ---------- OS options ----------
   ---------------------------------
*/

#define TCPIP_THREAD_NAME              "TCP/IP"
#define TCPIP_THREAD_STACKSIZE          4096
#define TCPIP_MBOX_SIZE                 6
#define DEFAULT_UDP_RECVMBOX_SIZE       6
#define DEFAULT_TCP_RECVMBOX_SIZE       6
#define DEFAULT_ACCEPTMBOX_SIZE         6
#define DEFAULT_THREAD_STACKSIZE        512
#define TCPIP_THREAD_PRIO               osPriorityHigh

#define LWIP_DEBUG 1

#define IP_DEBUG LWIP_DBG_ON
#define DHCP_DEBUG LWIP_DBG_OFF
#define UDP_DEBUG LWIP_DBG_ON
#define SOCKET_DEBUG_LWIP_DBG_ON
//#define ICMP_DEBUG LWIP_DBG_ON|LWIP_DBG_TRACE
//#define NETIF_DEBUG LWIP_DBG_OFF
#define LWIP_DBG_TYPES_ON  (LWIP_DBG_TRACE|LWIP_DBG_STATE)
#define LWIP_SO_RCVTIMEO 1
#define LWIP_NETIF_HOSTNAME 1

#define SO_REUSE 1

// Defining these produces various errors
//#define LWIP_IPV6 1
//#define LWIP_IPV6_DHCP6 1

#endif /* __LWIPOPTS_H__ */

// TODO
#ifdef LWIP_DEBUG

#define MEMP_OVERFLOW_CHECK            ( 1 )
#define MEMP_SANITY_CHECK              ( 1 )

#define MEM_DEBUG        LWIP_DBG_OFF
#define MEMP_DEBUG       LWIP_DBG_OFF
#define PBUF_DEBUG       LWIP_DBG_ON
#define API_LIB_DEBUG    LWIP_DBG_ON
#define API_MSG_DEBUG    LWIP_DBG_ON
#define TCPIP_DEBUG      LWIP_DBG_ON
#define NETIF_DEBUG      LWIP_DBG_ON
#define SOCKETS_DEBUG    LWIP_DBG_ON
#define DEMO_DEBUG       LWIP_DBG_ON
#define IP_DEBUG         LWIP_DBG_ON
#define IP6_DEBUG        LWIP_DBG_ON
#define IP_REASS_DEBUG   LWIP_DBG_ON
#define RAW_DEBUG        LWIP_DBG_ON
#define ICMP_DEBUG       LWIP_DBG_ON
#define UDP_DEBUG        LWIP_DBG_ON
#define TCP_DEBUG        LWIP_DBG_ON
#define TCP_INPUT_DEBUG  LWIP_DBG_ON
#define TCP_OUTPUT_DEBUG LWIP_DBG_ON
#define TCP_RTO_DEBUG    LWIP_DBG_ON
#define TCP_CWND_DEBUG   LWIP_DBG_ON
#define TCP_WND_DEBUG    LWIP_DBG_ON
#define TCP_FR_DEBUG     LWIP_DBG_ON
#define TCP_QLEN_DEBUG   LWIP_DBG_ON
#define TCP_RST_DEBUG    LWIP_DBG_ON
#define PPP_DEBUG        LWIP_DBG_OFF

#define LWIP_DBG_TYPES_ON         (LWIP_DBG_ON|LWIP_DBG_TRACE|LWIP_DBG_STATE|LWIP_DBG_FRESH|LWIP_DBG_HALT)

#endif



The interface between LWIP and the CPU's ETH subsystem is a fairly standard piece of code which has however been subject to extreme controversy on the ST forum because there have been various bugs, then these got fixed and new bugs arrived, etc. I think the code I have (which a huge amount of time was spent on) is good but there could be bottlenecks.

The transmit (low_level_output) i.e. application -> LWIP -> ETH is direct i.e. each packet sent out by LWIP goes straight to the linked list of packets presented to the ETH controller, and this stuff goes out very fast. So fast that any significant buffering is a waste of RAM, I found. I have just 2 buffers in low level ETH.

The receive (low_level_input) i.e. ETH -> LWIP -> application (i.e. calling lwip_read and an error code of -1 together with EWOULDBLOCK means there was no rx data) could be done with ETH interrupts but this has been fraught with problems in the ST 32F4 world, so I am using polling. This limits the packet rate to 1 packet every 10ms (which is still easily fast enough for what I need) but the polling loop is switched to 2ms if any data appears, and stays there until a 10ms data gap. It's a kind of simple optimisation, which also prevents an unintentional "DOS" situation (which with interrupts needs to be handled by limiting the interrupt frequency with a timer, etc). So between 100 and 500 packets per second.

The serial data is 9600 baud or 115200 baud depending on where along the chain. Yes; this is pretty slow.

Sorry for the long post but some say there is not enough detail

[wek]

The receive (low_level_input) i.e. ETH -> LWIP -> application (i.e. calling lwip_read and an error code of -1 together with EWOULDBLOCK means there was no rx data) could be done with ETH interrupts but this has been fraught with problems in the ST 32F4 world, so I am using polling. This limits the packet rate to 1 packet every 10ms (which is still easily fast enough for what I need) but the polling loop is switched to 2ms if any data appears, and stays there until a 10ms data gap. It's a kind of simple optimisation, which also prevents an unintentional "DOS" situation (which with interrupts needs to be handled by limiting the interrupt frequency with a timer, etc). So between 100 and 500 packets per second.

If the network dumps broadcasts at you at a higher rate than 1 packet every 10ms/2ms or whatever, then you'll fail to receive the packets unicast to you.

We have had a "funny" situation where a ***-brand printer suddenly felt the urge to broadcast (okay it's multicast) mDNS/LLMNR queries at a cca 10Hz rate cca 2-3 hours every day. Not only the mcu-based gadgets were confused during those periods.

JW

[peter-h]

Right - and this is partly why I (and the Monday guy who did the original ex Cube MX setup and has been working on this on Mondays for some years) decided to not try to implement interrupt driven RX (and spend many weeks or months debugging that, with "Piranha" on the ST forum being the only person - apart from you - who knows anything about it, in between telling you how stupid you are, and responding to posts 4 weeks after you've made them ) when polling "should" automatically avoid any flooding issues like that, while delivering plenty of speed.

There is a significant difference in these gaps between running on the TP-Link switch and running on the Netgear switch.

Some tests done uploading data via TLS to a well known site was achieving a few hundred kbytes/sec which is way more than is needed.

But maybe this scheme is causing low level congestion? I can't see how but obviously this stuff is so complex and so hard to debug.

To achieve the ST-advertised 95 mbps one obviously need to pull all the tricks out, like sending a fixed data packet using DMA, not going via anything like LWIP. Any real application will be much slower. But I wonder whether something in LWIP is getting into a muddle. I am now running the serial-ETH task in the lowest RTOS priority, so LWIP gets the maximum time to do whatever stuff.

I experimented with blocking RTOS task switching around various bits of code and while nothing helped with this issue, I found that one can do a socket write during this (PortEnterCritical etc), a socket read is blocked until the RTOS has switched to the LWIP task. Perhaps not surprising?

Sounds like speeding up the 2ms poll and seeing if it makes any difference might be worth a try, but then one ends up with a potentially hanging target - in possibly many applications. I tried 1ms and it makes no difference.

If this really is the problem, it would make sense to implement a crude firewall: an RX ETH ISR which does some very simple rule. I don't know if this is possible, given that one doesn't want to simply drop broadcasts. But one could drop all other packets not addressed to one's IP, surely? OTOH a switch should not be delivering these. So if this is the cause, they must be broadcasts.

If the network dumps broadcasts at you at a higher rate than 1 packet every 10ms/2ms or whatever, then you'll fail to receive the packets unicast to you.

What is the solution? Even interrupt driven RX must have the IRQ rate limited somehow.  And then one will miss some good packets. In the polled approach one will miss some good packets.

[ejeffrey]

The receive (low_level_input) i.e. ETH -> LWIP -> application (i.e. calling lwip_read and an error code of -1 together with EWOULDBLOCK means there was no rx data) could be done with ETH interrupts but this has been fraught with problems in the ST 32F4 world, so I am using polling. This limits the packet rate to 1 packet every 10ms (which is still easily fast enough for what I need) but the polling loop is switched to 2ms if any data appears, and stays there until a 10ms data gap. It's a kind of simple optimisation, which also prevents an unintentional "DOS" situation (which with interrupts needs to be handled by limiting the interrupt frequency with a timer, etc). So between 100 and 500 packets per second.

If the network dumps broadcasts at you at a higher rate than 1 packet every 10ms/2ms or whatever, then you'll fail to receive the packets unicast to you.

We have had a "funny" situation where a ***-brand printer suddenly felt the urge to broadcast (okay it's multicast) mDNS/LLMNR queries at a cca 10Hz rate cca 2-3 hours every day. Not only the mcu-based gadgets were confused during those periods.

JW

1 maximum length packet every 2 ms (most broadcast packets are closer to minimum length than maximum) is ~ 6 megabit /s.  Even that level of broadcast traffic would cause zero problem for also transferring <1 Mbps of serial data over a 100 megabit network interface.

The debugs come out via USB VCP which is very fast - hundreds of k bytes per second. But each line is truncated to 128 bytes. I can change that... I am logging the data with Teraterm, and flow control was disabled to make the debug function non-blocking if there was no terminal on the VCP. I have now enabled flow control and see that the traffic runs a bit slower, as expected.

100s of kbyte/second is much faster than the data you expect, but much slower than the network.  Especially since you say you are running in polling mode not interrupt driven, you also need to make sure that your logging calls are non-blocking.  When you transmit a line of text to the console, how long does it take before you can check the network interface? You need to make that time less than the time it takes for the hardware buffers to fill up.  If it takes too long to write the full log message you need to queue the remaining data, go check the network interface, and then continue writing the serial port.

I can't see how but obviously this stuff is so complex and so hard to debug

I strongly recommend using a packet sniffer.  You should be able to diagnose this with the log data, but the packet sniffer is less invasive, and will also show things that never make it to lwip.  You will be able to see if you are getting re-transmits, broadcast packets, and when there is a pause you will be able to see which side transmitted last and figure out which device is failing to respond.  You can also use multiple NICs to sniff in different places on your network to see if e.g., different sides of the hub/switch are seeing different data.

Can you get collision counters from your network interface?  If you are using a hub or a half-duplex connection to a switch you can have collisions, and if something is messed up you can have way more than expected. 

[peter-h]

When you transmit a line of text to the console, how long does it take before you can check the network interface?

It all runs under FreeRTOS, so "concurrently".

On the VCP, there is flow control all the way from the LWIP debug...() function to the Teraterm display and the file logging. This slows down LWIP a bit - probably a factor of two. But the same issues are seen.

I think it is most likely internal to LWIP because if I use a different algorithm for the serial -> ETH transfer, the reliability changes a lot. It could be some LWIP buffer config issue, but despite spending a huge amount of time on this originally I never found info I could understand, or anyone who knew about it. There are some experts and I offered them money to look at my LWIP config, without success.

The funny thing is that loads of products could have this issue and nobody would notice. It's not the sort of detail most people look at.

I strongly recommend using a packet sniffer.

I played with them in the past (Etherreal?) but without a lot of knowledge it was of no use.

Can you get collision counters from your network interface?

According to this
https://www.st.com/resource/en/user_manual/um1725-description-of-stm32f4-hal-and-lowlayer-drivers-stmicroelectronics.pdf
there are collision counter registers.

Also RM0090 contains a vast number of references, again needing detailed understanding of ETH, but no direct registers for collision counts. There may be software counters. I will dig around.

Would a half duplex setting be in the PHY (LAN8742) config? There is no mention of "duplex" in the whole project.

EDIT: the duplex config is indeed in the PHY:

Code: [Select]


/* Section 2: PHY configuration section */

#define LAN8720A_PHY_ADDRESS 0x00
//#define DP83848_PHY_ADDRESS             0x01
/* PHY Reset delay these values are based on a 1 ms Systick interrupt*/
#define PHY_RESET_DELAY                 (0x000000FFU)
/* PHY Configuration delay */
#define PHY_CONFIG_DELAY                (0x00000FFFU)

#define PHY_READ_TO                     (0x0000FFFFU)
#define PHY_WRITE_TO                    (0x0000FFFFU)

/* Section 3: Common PHY Registers */

#define PHY_BCR                         ((uint16_t)0x00)    /*!< Transceiver Basic Control Register   */
#define PHY_BSR                         ((uint16_t)0x01)    /*!< Transceiver Basic Status Register    */
 
#define PHY_RESET                       ((uint16_t)0x8000)  /*!< PHY Reset */
#define PHY_LOOPBACK                    ((uint16_t)0x4000)  /*!< Select loop-back mode */
#define PHY_FULLDUPLEX_100M             ((uint16_t)0x2100)  /*!< Set the full-duplex mode at 100 Mb/s */
#define PHY_HALFDUPLEX_100M             ((uint16_t)0x2000)  /*!< Set the half-duplex mode at 100 Mb/s */
#define PHY_FULLDUPLEX_10M              ((uint16_t)0x0100)  /*!< Set the full-duplex mode at 10 Mb/s  */
#define PHY_HALFDUPLEX_10M              ((uint16_t)0x0000)  /*!< Set the half-duplex mode at 10 Mb/s  */
#define PHY_AUTONEGOTIATION             ((uint16_t)0x1000)  /*!< Enable auto-negotiation function     */
#define PHY_RESTART_AUTONEGOTIATION     ((uint16_t)0x0200)  /*!< Restart auto-negotiation function    */
#define PHY_POWERDOWN                   ((uint16_t)0x0800)  /*!< Select the power down mode           */
#define PHY_ISOLATE                     ((uint16_t)0x0400)  /*!< Isolate PHY from MII                 */

#define PHY_AUTONEGO_COMPLETE           ((uint16_t)0x0020)  /*!< Auto-Negotiation process completed   */
#define PHY_LINKED_STATUS               ((uint16_t)0x0004)  /*!< Valid link established               */
#define PHY_JABBER_DETECTION            ((uint16_t)0x0002)  /*!< Jabber condition detected            */
 
/* Section 4: Extended PHY Registers */

//#define PHY_SR                          ((uint16_t)0x10)    /*!< PHY status register Offset                      */
#define PHY_SR ((uint16_t)31) // LAN8270 PHY SR
#define PHY_MICR                        ((uint16_t)0x11)    /*!< MII Interrupt Control Register                  */
#define PHY_MISR                        ((uint16_t)0x12)    /*!< MII Interrupt Status and Misc. Control Register */
 
#define PHY_LINK_STATUS                 ((uint16_t)0x0001)  /*!< PHY Link mask                                   */
//#define PHY_SPEED_STATUS                ((uint16_t)0x0002)  /*!< PHY Speed mask                                  */
#define PHY_SPEED_STATUS                ((uint16_t)0x0004)  /*!< PHY Speed mask LAN8270 (set if 10M clear if 100M)   */
//#define PHY_DUPLEX_STATUS               ((uint16_t)0x0004)  /*!< PHY Duplex mask                                 */
#define PHY_DUPLEX_STATUS               ((uint16_t)0x0010)  /*!< PHY Duplex mask  LAN8270                          */

#define PHY_MICR_INT_EN                 ((uint16_t)0x0002)  /*!< PHY Enable interrupts                           */
#define PHY_MICR_INT_OE                 ((uint16_t)0x0001)  /*!< PHY Enable output interrupt events              */

#define PHY_MISR_LINK_INT_EN            ((uint16_t)0x0020U)  /*!< Enable Interrupt on change of link status       */
#define PHY_LINK_INTERRUPT              ((uint16_t)0x2000U)  /*!< PHY link status interrupt mask                  */


[ejeffrey]

Wireshark is the "new" (for the last 15 years) name of ethereal, changed due to some legal / trademark ownership issues.

It will set the target interface to view every packet received (rather than only ones that match the MAC address), and display each packet, optionally only those that match a particular filter.

You can see an example of what a capture looks like here: https://www.wireshark.org/docs/wsug_html_chunked/ChUseMainWindowSection.html#ChUseFig01

For each packet you can see the headers and the data, and wireshark dissects most common protocols.  It will even automatically highlight some potential problems in red.  You will be able to see the data packet lengths, when they are acknowledged, if the same data is transmitted multiple times, if there are broadcast requests like ARP if they are responded to or ignored, and more.

Here is a blog post on understanding the phases of a TCP session, using wireshark screen captures at each step: https://packetlife.net/blog/2010/jun/7/understanding-tcp-sequence-acknowledgment-numbers/

[dare]

This bit was captured during a period of a few seconds where traffic was being blocked by something

Regarding this log snippet: You mention the period lasted a few seconds, however the log only covers 215ms.  Is the rest of the log available?

Looking at the given log, there is no obvious sign of communication issues: Over the course of the log, the local system (192.168.3 .64) receives two packets with payload data (128 and 104 bytes) from the peer (192.168.3.68), which it ACKs and passes to the application.  The application accepts this data via multiple calls to lwip_recvfrom().  During this same time, the local application calls lwip_send() once which results in one packet with payload data (104 bytes) being sent to the peer.  An ACK is received from the peer for this packet a short time later.  After this exchange of data there is a short period (20ms) of no activity before the log ends.  During this period, the local system makes no calls to lwip_send() and no packets are received from the peer.

There is no indication of retransmissions in the log, and the TCP send and receive windows are open during the whole exchange.  It is possible that the final ACK from the local host to the peer was lost, and that a retransmisson happened after the end of the log.  However a single lost packet at this point should not have blocked the peer from sending more data, as the local system's window was open.  Without further evidence, I can't speculate why communication halted.

[peter-h]

All your help guys has given me clues and I think I found it.

Code: [Select]

/* Definition of the Ethernet driver buffers size and count */
// 21/7/22 PH increasing both from 2 to 4 makes zero difference to tx speed, and funnily
// enough 1 slightly improves it! But only if the low level poll time is 10ms instead of
// 20ms; then going to 1 halves the speed. Going to 1+1 yields an extra 3k of RAM.
// BUT ETH_RXBUFNB=2 causes multi-second lockups in LWIP due to broadcast packets! 3 fixes
// this, so we use 4 to get a margin.
#define ETH_RX_BUF_SIZE                ETH_MAX_PACKET_SIZE /* buffer size for receive               */
#define ETH_TX_BUF_SIZE                ETH_MAX_PACKET_SIZE /* buffer size for transmit              */
#define ETH_RXBUFNB                    4  // was (4U)       /* Rx buffers of size ETH_RX_BUF_SIZE  */
#define ETH_TXBUFNB                    2  // was (4U)       /* Tx buffers of size ETH_TX_BUF_SIZE  */
The original ST Cube MX values were rx=4 and tx=4.

In my quest to reduce the fairly wild LWIP memory usage I spent a lot of time and - to the extent that little of how LWIP actually works is documented - chopped the usage down by quite a bit, with no apparent performance penalty.

The above ETH buffers are below LWIP; the 32F417 ETH controller is fed via them. I quickly found that on the TX side the data vanishes down the wire so fast that even 1 buffer achieves megabytes/sec, so I went from 4 to 2. On the RX side we are polling (every 10ms, with this dropping to 2ms on any rx data, until 100ms of no data and then it goes back to 10ms) and some more buffering would be expected to be needed. I found 2 is perfectly good enough, though more might be needed on slow WANs etc (4G).

But there is a problem there, if "other packets" arrive.

I waggled a GPIO every time around that 2ms/10ms loop, and while most of the 2ms blocks corresponded with known data, some didn't, and were obviously kicked off by broadcast packets.

And sure enough these mystery packets were kicking off the seconds-long delays in LWIP. They caused some deadlock. Perhaps LWIP is doing something with these but for sure there is no application (on a socket) reading them out.



The red trace is the GPIO waggle.

I increased the ETH_RXBUFNB  back to 4 and it completely solved the problem. 3 also does it, so I went to 4 to get a margin.

I reckon, years ago, somebody at ST who was porting LWIP to the 32F4 found that they needed 4 and didn't document why. Or maybe they just started with 4 and got lucky.

It kind of makes sense that the more "sessions" you want to run, the more buffers you might need, but should LWIP actually lock up? Anyway these are low level buffers - there is a list of pointers which the 32F4's ETH DMA controller picks up. No software involved as such beyond memcpy to/from the buffer(s) (could use DMA) and setting up some pointer. The ETH DMA controller is continually polling those pointers to see if there is anything to send, and similarly it sets them up if something has arrived. There is "something" in LWIP, combined with the way the 32F4 ETH controller drives that buffer list, which doesn't work reliably with just 2 rx buffers. Which is weird since that is a raw physical packet level, nothing to do with sessions, sockets, connections...

I had already tried all the LWIP buffer settings in lwipopts.h and nothing made any difference.

[wek]

> If the network dumps broadcasts at you at a higher rate than 1 packet
> every 10ms/2ms or whatever, then
> *you'll fail to receive the packets unicast to you*.

So, if your buffer count was 2, that's 1 for the spurious broadcast and 1 for the unicast packet you expect. One more broadcast packet before the unicast packet within given poll time, and you've lost the unicast data. Sure, TCP will recover if it's not too often, but that will cost you exactly the delay you observe.

3 or 4 are just arbitrary values "working" in your arbitrary network. A malicious out just runaway device on the network can flood it with short but numerous broadcasts. It's up to you how perfect solution do you want to develop.

This is not to say there may be no other mechanisms how the observed problem can occur.

JW

[peter-h]

that will cost you exactly the delay you observe.

A few seconds? I hope that timer is configurable

3 or 4 are just arbitrary values "working" in your arbitrary network

It can't be that simple, because there is only one RJ45 socket There is only one PHY chip. That and the buffers are the physical layer. TCP/IP can have almost any number of sessions (open sockets). The upper limit on sockets is actually configurable. I have about 5 applications running, 2 running TLS, concurrently. And all this works with 2 buffers.

Even if you had interrupt-driven RX you could still get two packets arriving within the exec time of the ISR. There is no documented issue like that, that the ISR exec time has to be shorter than the shortest possible 100mbps packet.

If the explanation was this simple, ETH would work pretty poorly, because you will always risk congestion of the PHY buffers unless you have enough of them for the worst possible scenario of 100mbps (or 1000mbps), and shortest possible packets. It would be a lot of buffering.

I reckon it is some subtle deadlock, because I was getting the multi-second delays (very rarely) even on the isolated switch, with no broadcasts arriving.

[dare]

It kind of makes sense that the more "sessions" you want to run, the more buffers you might need, but should LWIP actually lock up?

There's no "lock-up" in LwIP.  If the lack of receive buffers at the network interface level is causing the receiver to drop packets, then its up to the sender to discover this and retransmit them.  This takes time, which is resulting in a visible hiccup in throughput.  But at no point is the system unable to make forward progress.  This is standard behavior for TCP  (though tuning can improve things).

(Incidentally, a packet trace in Wireshark would have revealed this scenario rather quickly).

A simple solution to this problem is to apply packet filtering at the network interface level.  It is relatively easy to peek inside the IP header and detect broadcast/multicast packets vs unicast packets.  Using this information you can give preference to unicast packets in the receive queue.  Simply increment a (thread-safe) counter whenever a broadcast packet is queued, and decrement it whenever it is dequeued.  Upon receiving a broadcast packet, if the counter is above a threshold, simply discard the packet rather than queuing it.  The threshold can be as low as 1, since broadcast is inherently unreliable, and all broadcast protocols incorporate some form of retransmission.  I have shipped a number of LwIP-based consumer products that incorporate this strategy to good effect.

[dare]

I reckon it is some subtle deadlock, because I was getting the multi-second delays (very rarely) even on the isolated switch, with no broadcasts arriving.

Just to re-iterate, I have shipped literally millions of consumer IoT devices based on LwIP.  If the system is stumbling or locking up on an isolated network, then there is either a hardware problem or a system configuration problem.  Setup correctly, LwIP is very reliable.

[peter-h]

Thank you, but you are talking in generalities, whereas I have to get specifics to work.

Using this information you can give preference to unicast packets in the receive queue.

Could you post some code, please? AIUI the ETH RX buffer is filled from the dedicated ETH controller which the 32F417 has, so there is no opportunity to intercept that data before that buffer is filled. What CPU are you using?

I mentioned this strategy (a crude "firewall") further back, adding that I never found any implementation of it.

For sure one could implement it in low_level_input. In my case, this is a dedicated RTOS task, which AFAICT copies the data into some "PBUF" queue owned by LWIP. That task could be an ISR instead but again I never found good working code example for that, and an RTOS task is a handy solution because it offers natural DOS protection, while enabling the running of things like link status change detection at the same level.

Perhaps wek will know more detail.

However, as I wrote further back, more buffers will only postpone the vulnerability. AFAICT, unless the ETH hardware and the entire stack above it can handle the physical speed (10Mbytes/sec in this case) the system must be capable of dropping packets, unless it has infinite buffering.

I did a search for TCP/IP retransmission and if running on a fast LAN (trip time ~1ms) the initial retry delay should not be what I am seeing, which is 1-2 secs. So I think those delays are something else - unless LWIP does something else.
This post
https://www.eevblog.com/forum/microcontrollers/lwip-tcp_timer-interval/
suggests LWIP uses 3 seconds but that is a lot longer than I normally see.
This post (whose date shows how old LWIP is)
https://lists.nongnu.org/archive/html/lwip-devel/2007-09/msg00061.html
has some info suggesting 3 seconds too.
But in my project, TCP_TMR_INTERVAL = 250ms.

[wek]

If you think it's something else - and as I've said, it may quite well be - feel free to investigate.

As I've said, I've positively determined broadcasts and described consequent data drop/TCP repair to be cause of "jerks" in my case and I reckon it's quite common in various SOHO LANs. (The associated act of sniffing the LAN was an eye opener, that's why you are recommended to do this by many here.) We then decided that this is something we can/should solve organisationally rather than striving for perfection at our side. As the LANs are an extremely variable environment completely out of our control, and as there may be varied other factors involved (e.g. wireless hops), we chose to defer the problem to the clients ("use a switch with nothing else on it, works as expected? then clean up your LAN, other devices will benefit from this too"). I don't say this is an universally applicable approach, just that there are many ways to skin a cat, some of which don't require messing with lwip/ETH more than usually.

JW

[peter-h]

I see what must be broadcasts (well, whatever gets routed through a switch) roughly every 1-2 seconds.

Inevitably, some of these will end up in the ETH RX buffer list at the same time as a real data packet.

This is OK if there are 3+ RX buffers (each buffer is MTU-size btw) but it was the 2 buffer case which was comprehensively breaking stuff.

I'd like to understand the mechanism for this problem, because those RX buffers are at the PHY level. They all rank equally until they reach LWIP. So two data packets (addressed to the same IP) should have the same effect as 1 data packet and 1 multicast. Is this wrong?

Also I noticed that when I opened up a VPN to another site, more multicast packets appeared - as one would probably expect.

The issue of high speed multicasts making a device grind to a halt, I am happy with. It's not possible to guard against this completely while achieving a decent speed - because even an RX ISR will need to implement a rate limit otherwise a stream of multicasts will totally hang up the box. I don't need dare to come back with some sourcecode because the 32F4 can dump broadcasts internally (but that breaks things totally - see below).

The Q is then: is there a legitimate scenario where we do want broadcasts to reach LWIP? There is a lot online about this. This
https://stackoverflow.com/questions/35130743/how-to-create-a-service-that-sends-receives-udp-broadcasts-on-multiple-interface
suggests that LWIP passes broadcasts back to every open socket.
This suggests that some ST CPUs have a broadcast filter already
https://stackoverflow.com/questions/50132454/stm32-lwip-mcu-load-due-to-broadcast-packet
This seems quite similar to my problem but he fixed it without ever finding out how - could just be a different # of rx buffers (the H7 has/had loads of ETH issues)
https://stackoverflow.com/questions/72323318/lwip-netconn-recv-netbuf-overwritten-from-browser-broadcast-message

Based on this
https://community.st.com/s/question/0D50X0000ALu7XqSQJ/how-configure-lwip-to-receive-udp-broadcast-using-stm32cubemx-
I've done this, in case it helps to dump broadcasts earlier
// Disable broadcasts being received
#define IP_SOF_BROADCAST                1
#define IP_SOF_BROADCAST_RECV           1

Everything still works, and indeed I don't have any LWIP application receiving broadcasts.

I cannot find anything re LWIP itself needing broadcasts for anything.

As usual we've done this before
https://www.eevblog.com/forum/microcontrollers/any-micros-with-eth-and-hardware-incoming-packet-filtering-by-ip/
which led me to BFD of ETH_MACFFR:
Bit 5 BFD: Broadcast frames disable
When this bit is set, the address filters filter all incoming broadcast frames.
When this bit is reset, the address filters pass all received broadcast frames.
and I disabled broadcasts using  using ETH_BROADCASTFRAMESRECEPTION_DISABLE in stm32f4xx_hal_eth.c.

That breaks things completely. Evidently the 32F4 ETH does need to process broadcasts (for network config?) but LWIP is happy to dump them.

So if I was to dump them, they would need to be dumped at the level above the 32F4's ETH controller i.e. in low_level_input, but since that is above the ETH PHY buffers discussed, what would be the point? The real gain of dumping them at the ETH level would be a high level of DOS protection.

[nctnico]

No, you'd need to put a layer between the ethernet driver and LWIP that throws away the broadcast /unicast messages that are not necessary for the device. So the broadcast/unicast messages will still be received but are not fed into LWIP.

[peter-h]

I think that is what I said:

they would need to be dumped at the level above the 32F4's ETH controller i.e. in low_level_input

It would be a neat approach because it could be runtime configurable whereas the lwipopts.h stuff is fixed at compile time (although one could mod all the code which references them...).

$100

Presumably the BFD of ETH_MACFFR solution is usable only for boxes which are addressed by MAC #

Sources for low_level_input:

Code: [Select]


/**
  * @brief Should allocate a pbuf and transfer the bytes of the incoming
  * packet from the interface into the pbuf.
  *
  * @param netif the lwip network interface structure for this ethernetif
  * @return a pbuf filled with the received packet (including MAC header)
  *         NULL on memory error
  */

static struct pbuf * low_level_input(struct netif *netif)
{
  struct pbuf *p = NULL, *q = NULL;
  uint16_t len = 0;
  uint8_t *buffer;
  __IO ETH_DMADescTypeDef *dmarxdesc;
  uint32_t bufferoffset = 0;
  uint32_t payloadoffset = 0;
  uint32_t byteslefttocopy = 0;
  uint32_t i=0;
 
  /* get received frame */

  HAL_StatusTypeDef status = IF_HAL_ETH_GetReceivedFrame(&EthHandle);

  if (status != HAL_OK)
  {
  return NULL; // Return if no RX data
  }
  else
  {
  rxactive=true; // set "seen rx data" flag
  }

  /* Obtain the size of the packet and put it into the "len" variable. */
  len = EthHandle.RxFrameInfos.length;
  buffer = (uint8_t *)EthHandle.RxFrameInfos.buffer;
 
  if (len > 0)
  {
    /* We allocate a pbuf chain of pbufs from the Lwip buffer pool */
    p = pbuf_alloc(PBUF_RAW, len, PBUF_POOL);
  }
 
  if (p != NULL)
  {
    dmarxdesc = EthHandle.RxFrameInfos.FSRxDesc;
    bufferoffset = 0;
   
    for(q = p; q != NULL; q = q->next)
    {
      byteslefttocopy = q->len;
      payloadoffset = 0;
     
      /* Check if the length of bytes to copy in current pbuf is bigger than Rx buffer size */
      while( (byteslefttocopy + bufferoffset) > ETH_RX_BUF_SIZE )
      {
        /* Copy data to pbuf */
        memcpy( (uint8_t*)((uint8_t*)q->payload + payloadoffset), (uint8_t*)((uint8_t*)buffer + bufferoffset), (ETH_RX_BUF_SIZE - bufferoffset));
       
        /* Point to next descriptor */
        dmarxdesc = (ETH_DMADescTypeDef *)(dmarxdesc->Buffer2NextDescAddr);
        buffer = (uint8_t *)(dmarxdesc->Buffer1Addr);
       
        byteslefttocopy = byteslefttocopy - (ETH_RX_BUF_SIZE - bufferoffset);
        payloadoffset = payloadoffset + (ETH_RX_BUF_SIZE - bufferoffset);
        bufferoffset = 0;
      }

      /* Copy remaining data in pbuf */
      memcpy( (uint8_t*)((uint8_t*)q->payload + payloadoffset), (uint8_t*)((uint8_t*)buffer + bufferoffset), byteslefttocopy);
      bufferoffset = bufferoffset + byteslefttocopy;
    }
  }

  /* Release descriptors to DMA */
  /* Point to first descriptor */
  dmarxdesc = EthHandle.RxFrameInfos.FSRxDesc;
  /* Set Own bit in Rx descriptors: gives the buffers back to DMA */
  for (i=0; i< EthHandle.RxFrameInfos.SegCount; i++)
  { 
//__DMB(); - fossil code for the 32F417, apparently.
    dmarxdesc->Status |= ETH_DMARXDESC_OWN;
    dmarxdesc = (ETH_DMADescTypeDef *)(dmarxdesc->Buffer2NextDescAddr);
  }

  /* Clear Segment_Count */
  EthHandle.RxFrameInfos.SegCount =0;
 
  /* When Rx Buffer unavailable flag is set: clear it and resume reception */
  if ((EthHandle.Instance->DMASR & ETH_DMASR_RBUS) != (uint32_t)RESET) 
  {
    /* Clear RBUS ETHERNET DMA flag */
    EthHandle.Instance->DMASR = ETH_DMASR_RBUS;
    /* Resume DMA reception */
    EthHandle.Instance->DMARPDR = 0;
  }
  return p;
}


// Timeout for ethernetif_input "activity" status which drops fast poll to slow poll.
// FreeRTOS timer needs to have a callback, even if this function is empty.
// The other way to check timer status is
// if (xTimerIsTimerActive(logout_timer) == pdTRUE) - gives True if not expired

static void RXactiveTimerCallback( TimerHandle_t rxactive_timer )
{
rx_poll_period=ETH_RX_SLOW_POLL_INTERVAL;
}


/**
  * This function is the ethernetif_input task. It uses the function low_level_input()
  * that handles the actual reception of bytes from the network interface.
  *
  * This is a standalone RTOS task so is a forever loop.
  * Could be done with interrupts but then we have the risk of hanging the unit with fast input
  * (unlikely if ETH is via a switch, but you could have a faulty LAN with lots of
  * broadcasts).
  *
  */

void ethernetif_input( void * argument )
{

struct pbuf *p;
struct netif *netif = (struct netif *) argument;

// Define RX activity timer, for dropping fast poll down to slow poll
TimerHandle_t *rxactive_timer = xTimerCreate("ETH RX active timer", pdMS_TO_TICKS(ETH_SLOW_POLL_DELAY), pdFALSE, NULL, RXactiveTimerCallback);

// Start "rx active" timer
xTimerStart(rxactive_timer, 20); // 20 is just a wait time for timer allocation

do
    {
/*
extern void TopLED(bool state);
TopLED(true);  // waggleGPIO
TopLED(false);
*/

p = low_level_input( netif ); // This sets rxactive=true if it sees data

if (p!=NULL)
{
if (netif->input( p, netif) != ERR_OK )
{
pbuf_free(p);
}
}

if (rxactive)
{
rxactive=false;
// Seen rx data - reload timeout
xTimerReset(rxactive_timer, 20); // Reload "rx active" timeout (with ETH_SLOW_POLL_DELAY)
// and get osDelay below to run fast
rx_poll_period=ETH_RX_FAST_POLL_INTERVAL;
}

// This has a dramatic effect on ETH speed, both ways (TCP/IP acks all packets)
osDelay(rx_poll_period);

    } while(true);

}


[dare]

I cannot find anything re LWIP itself needing broadcasts for anything.

ARP requires reception of broadcast packets.  Without ARP, IPv4 doesn't work.

So if I was to dump them, they would need to be dumped at the level above the 32F4's ETH controller i.e. in low_level_input, but since that is above the ETH PHY buffers discussed, what would be the point? The real gain of dumping them at the ETH level would be a high level of DOS protection.

Every point in the code where you have a queue that is fed/serviced in different contexts is subject to queue overflow/buffer exhaustion whenever the packet arrival rate is faster than the processing rate.  Right now you have two such queues*.  As you've described it, the first queue is fed by the ethernet controller and serviced by your ethernetif_input() task.  The second queue is inside LwIP and is fed by the call to netif->input() in ethernetif_input() and serviced by the LwIP tcpip task itself.

* (Technically there is also a third queue between the LwIP tcpip task and your application task that is calling lwip_recvfrom().  However since your application-level data rate is so low, this queue is unlikely to suffer from congestion.)

Ideally there would only be 1 queue on the input side (more on this later).  But given that you have two queues, both need to be optimized to deal with extraneous input packets.  For the first queue, it seems your only option is to increase the buffer pool size to the point where it is unlikely to be a problem.  So far you have found that 4 buffers works, but some stress testing might reveal that you need more.

For the second queue you should install a filter that blocks extraneous packets from being forwards to the tcpip task.

I don't need dare to come back with some sourcecode

Well, I'm going to give it to you anyway. This function will filter out everything except for ARP packets and IPv4 packets destined to a non-broadcast ether address:

Code: [Select]

static int should_accept_ethernet_packet(const uint8_t * pktData, size_t pktLen)
{
  const struct eth_hdr * pktHdr = (const struct eth_hdr *)pktData;

  /* If the packet is too short, reject it. */
  if (pktLen < sizeof(struct eth_hdr)) {
    return 0;
  }

  /* If the packet is to a broadcast destination... */
  if (eth_addr_cmp(&pktHdr->dest, &ethbroadcast)) {

    /* If the packet is a broadcast ARP packet, accept it. */
    if (ntohs(pktHdr->type) == ETHTYPE_ARP) {
      return 1;
    }
  }

  /* if the packet is to an IPv4 multicast destination, reject it. */
  else if (pktHdr->dest.addr[0] == LL_IP4_MULTICAST_ADDR_0 &&
           pktHdr->dest.addr[1] == LL_IP4_MULTICAST_ADDR_1 &&
           pktHdr->dest.addr[2] == LL_IP4_MULTICAST_ADDR_2) {
    return 0;
  }

  /* if the packet is an IPv4 packet, accept it. */
  else if (ntohs(pktHdr->type) == ETHTYPE_IP) {
    return 1;
  }

  /* Reject all other packets */
  return 0;
}

Place a call to this function early in your low_level_input() function, before you allocate the pbuf.  If the function returns false, simply ignore the packet.  Note that, even with this function in place, you may still need to tune the total number of pbufs.  This is especially true if you have data flowing both ways in the connection and have big TCP window sizes.

Regarding the need for the ethernetif_input() task, I couldn't find anything to suggest that using interrupts for packet reception was unworkable.  Is there a concrete description of the problem you encountered?

[peter-h]

ARP requires reception of broadcast packets.  Without ARP, IPv4 doesn't work.

OK, so the

// Disable broadcasts being received
#define IP_SOF_BROADCAST                1
#define IP_SOF_BROADCAST_RECV           1

is not preventing that, because everything works. It seems that this just prevents broadcasts coming back via the LWIP API.

So far you have found that 4 buffers works, but some stress testing might reveal that you need more.

I found 3 makes it solid so I am running with 4 (which happens to be the original Cube default). I've written it all up in the design doc for the future. BTW in case people wonder why I write these long posts, it is because a) I want to learn stuff and b) I drop the thread URL into the design doc

This function will filter out everything except for ARP packets and IPv4 packets destined to a non-broadcast ether address:

That is truly wonderful of you, but I don't know where to call it. I can see several places where packets can possibly be examined, but how to discard them? For the 32F4 ETH controller to see it "gone" one has to clear some pointer. That is probably all, plus not copy it to LWIP.

Place a call to this function early in your low_level_input() function, before you allocate the pbuf.

You mean inside IF_HAL_ETH_GetReceivedFrame() ?

Code: [Select]



HAL_StatusTypeDef IF_HAL_ETH_GetReceivedFrame(ETH_HandleTypeDef *heth)
{
  uint32_t framelength = 0U;

  /* Check if segment is not owned by DMA */
  /* if (((heth->RxDesc->Status & ETH_DMARXDESC_OWN) == (uint32_t)RESET) && ((heth->RxDesc->Status & ETH_DMARXDESC_LS) != (uint32_t)RESET)) */
  //__DMB();
  if(((heth->RxDesc->Status & ETH_DMARXDESC_OWN) == (uint32_t)RESET))
  {
    /* Check if last segment */
    if(((heth->RxDesc->Status & ETH_DMARXDESC_LS) != (uint32_t)RESET))
    {
      /* increment segment count */
      (heth->RxFrameInfos).SegCount++;

      /* Check if last segment is first segment: one segment contains the frame */
      if ((heth->RxFrameInfos).SegCount == 1U)
      {
        (heth->RxFrameInfos).FSRxDesc =heth->RxDesc;
      }

      heth->RxFrameInfos.LSRxDesc = heth->RxDesc;

      /* Get the Frame Length of the received packet: substruct 4 bytes of the CRC */
      framelength = (((heth->RxDesc)->Status & ETH_DMARXDESC_FL) >> ETH_DMARXDESC_FRAMELENGTHSHIFT) - 4U;
      heth->RxFrameInfos.length = framelength;

      /* Get the address of the buffer start address */
      heth->RxFrameInfos.buffer = ((heth->RxFrameInfos).FSRxDesc)->Buffer1Addr;
      /* point to next descriptor */
      heth->RxDesc = (ETH_DMADescTypeDef*) ((heth->RxDesc)->Buffer2NextDescAddr);

      /* Return function status */
      return HAL_OK;
    }
    /* Check if first segment */
    else if((heth->RxDesc->Status & ETH_DMARXDESC_FS) != (uint32_t)RESET)
    {
      (heth->RxFrameInfos).FSRxDesc = heth->RxDesc;
      (heth->RxFrameInfos).LSRxDesc = NULL;
      (heth->RxFrameInfos).SegCount = 1U;
      /* Point to next descriptor */
      heth->RxDesc = (ETH_DMADescTypeDef*) (heth->RxDesc->Buffer2NextDescAddr);
    }
    /* Check if intermediate segment */
    else
    {
      (heth->RxFrameInfos).SegCount++;
      /* Point to next descriptor */
      heth->RxDesc = (ETH_DMADescTypeDef*) (heth->RxDesc->Buffer2NextDescAddr);
    }
  }

  /* Return function status */
  return HAL_ERROR;
}

which gets called thus:



Not sure where the packet can be found though.

Regarding the need for the ethernetif_input() task, I couldn't find anything to suggest that using interrupts for packet reception was unworkable.  Is there a concrete description of the problem you encountered?

It has been done but there are various issues and I have not found a working code example. Also it isn't trivial because one needs to do rate limiting i.e. disable further ints and then re-enable them in a timer ISR. Another thing, not in the code I posted for brevity, is that we are doing link status change detection (cable plugged in etc) in that RTOS task, and it conveniently avoids issues with code running in a different RTOS context.

Bear in mind all this stuff is totally unsupported. All one finds is code snippets, mostly somebody's work in progress (much of github is WIP) and reports of problems, mostly with no response, or with slagging off of ST

[dare]

OK, so the

// Disable broadcasts being received
#define IP_SOF_BROADCAST                1
#define IP_SOF_BROADCAST_RECV           1

is not preventing that, because everything works. It seems that this just prevents broadcasts coming back via the LWIP API.

Roughly speaking, yes.  These are build flags which enable the support for the SOF_BROADCAST option on raw and UDP PCBs.  Nothing to do with whether the lower parts of the stack accept broadcast packets.

That is truly wonderful of you, but I don't know where to call it. I can see several places where packets can possibly be examined, but how to discard them?
For the 32F4 ETH controller to see it "gone" one has to clear some pointer. That is probably all, plus not copy it to LWIP.

As I mentioned, the place to put it is in low_level_input().  Like so:

Code: [Select]

static struct pbuf * low_level_input(struct netif *netif)
{
  struct pbuf *p = NULL, *q = NULL;
  uint16_t len = 0;
  uint8_t *buffer;
  __IO ETH_DMADescTypeDef *dmarxdesc;
  uint32_t bufferoffset = 0;
  uint32_t payloadoffset = 0;
  uint32_t byteslefttocopy = 0;
  uint32_t i=0;
 
  /* get received frame */

  HAL_StatusTypeDef status = IF_HAL_ETH_GetReceivedFrame(&EthHandle);

  if (status != HAL_OK)
  {
  return NULL; // Return if no RX data
  }
  else
  {
  rxactive=true; // set "seen rx data" flag
  }

  /* Obtain the size of the packet and put it into the "len" variable. */
  len = EthHandle.RxFrameInfos.length;
  buffer = (uint8_t *)EthHandle.RxFrameInfos.buffer;

  /***** IF THE PACKET SHOULD BE ACCEPTED... *****/
  if (should_accept_ethernet_packet(buffer, len))
  {
    /* We allocate a pbuf chain of pbufs from the Lwip buffer pool */
    p = pbuf_alloc(PBUF_RAW, len, PBUF_POOL);
  }
 
  if (p != NULL)
  {
    dmarxdesc = EthHandle.RxFrameInfos.FSRxDesc;
    bufferoffset = 0;
   
    for(q = p; q != NULL; q = q->next)
    {
      byteslefttocopy = q->len;
      payloadoffset = 0;
     
      /* Check if the length of bytes to copy in current pbuf is bigger than Rx buffer size */
      while( (byteslefttocopy + bufferoffset) > ETH_RX_BUF_SIZE )
      {
        /* Copy data to pbuf */
        memcpy( (uint8_t*)((uint8_t*)q->payload + payloadoffset), (uint8_t*)((uint8_t*)buffer + bufferoffset), (ETH_RX_BUF_SIZE - bufferoffset));
       
        /* Point to next descriptor */
        dmarxdesc = (ETH_DMADescTypeDef *)(dmarxdesc->Buffer2NextDescAddr);
        buffer = (uint8_t *)(dmarxdesc->Buffer1Addr);
       
        byteslefttocopy = byteslefttocopy - (ETH_RX_BUF_SIZE - bufferoffset);
        payloadoffset = payloadoffset + (ETH_RX_BUF_SIZE - bufferoffset);
        bufferoffset = 0;
      }

      /* Copy remaining data in pbuf */
      memcpy( (uint8_t*)((uint8_t*)q->payload + payloadoffset), (uint8_t*)((uint8_t*)buffer + bufferoffset), byteslefttocopy);
      bufferoffset = bufferoffset + byteslefttocopy;
    }
  }

  /* Release descriptors to DMA */
  /* Point to first descriptor */
  dmarxdesc = EthHandle.RxFrameInfos.FSRxDesc;
  /* Set Own bit in Rx descriptors: gives the buffers back to DMA */
  for (i=0; i< EthHandle.RxFrameInfos.SegCount; i++)
  {
//__DMB(); - fossil code for the 32F417, apparently.
    dmarxdesc->Status |= ETH_DMARXDESC_OWN;
    dmarxdesc = (ETH_DMADescTypeDef *)(dmarxdesc->Buffer2NextDescAddr);
  }

  /* Clear Segment_Count */
  EthHandle.RxFrameInfos.SegCount =0;
 
  /* When Rx Buffer unavailable flag is set: clear it and resume reception */
  if ((EthHandle.Instance->DMASR & ETH_DMASR_RBUS) != (uint32_t)RESET)
  {
    /* Clear RBUS ETHERNET DMA flag */
    EthHandle.Instance->DMASR = ETH_DMASR_RBUS;
    /* Resume DMA reception */
    EthHandle.Instance->DMARPDR = 0;
  }
  return p;
}


It has been done but there are various issues and I have not found a working code example.

This post contains code that is purported to work: https://lwip-users.nongnu.narkive.com/62tlpY0j/stm32-rbus-receive-buffer-unavailable-bit-set-after-debugger-break#post7.  The code looks roughly sound to me, especially the logic in ethernetif_input() which drains all the packets in the DMA queue before clearing the Rx Buffer unavailable flag and re-enabling reception.

Here is the code from the post in an easier to read form:

Code: [Select]

// from the sample code for the STM32F4x7 ethernet FreeRTOS/lwIP demo.

/**
 * Should allocate a pbuf and transfer the bytes of the incoming
 * packet from the interface into the pbuf.
 *
 * @param netif the lwip network interface structure for this ethernetif
 * @return a pbuf filled with the received packet (including MAC header)
 * NULL on memory error
 */
static struct pbuf *low_level_input(struct netif *netif)
{
    struct pbuf *p, *q;
    u16_t len;
    uint32_t l = 0, i = 0;
    FrameTypeDef frame;
    u8 *buffer;
    __IO ETH_DMADESCTypeDef *DMARxNextDesc;

    p = NULL;

    /* Get received frame */
    frame = ETH_Get_Received_Frame_interrupt();

    if (frame.descriptor && frame.buffer)
    {
        /* check that frame has no error */
        if ((frame.descriptor->Status & ETH_DMARxDesc_ES) == (uint32_t)RESET)
        {

            /* Obtain the size of the packet and put it into the "len"
            variable. */
            len = frame.length;
            buffer = (u8 *)frame.buffer;

            /* We allocate a pbuf chain of pbufs from the pool. */
            p = pbuf_alloc(PBUF_RAW, len, PBUF_POOL);

            /* Copy received frame from ethernet driver buffer to stack
            buffer */
            if (p != NULL)
            {
                for (q = p; q != NULL; q = q->next)
                {
                    memcpy((u8_t *)q->payload, (u8_t *)&buffer[l], q->len);
                    l = l + q->len;
                }
            }
        }

        /* Release descriptors to DMA */
        /* Check if received frame with multiple DMA buffer segments */
        if (DMA_RX_FRAME_infos->Seg_Count > 1)
        {
            DMARxNextDesc = DMA_RX_FRAME_infos->FS_Rx_Desc;
        }
        else
        {
            DMARxNextDesc = frame.descriptor;
        }

        /* Set Own bit in Rx descriptors: gives the buffers back to DMA */
        for (i = 0; i < DMA_RX_FRAME_infos->Seg_Count; i++)
        {
            DMARxNextDesc->Status = ETH_DMARxDesc_OWN;
            DMARxNextDesc = (ETH_DMADESCTypeDef
                                 *)(DMARxNextDesc->Buffer2NextDescAddr);
        }

        /* Clear Segment_Count */
        DMA_RX_FRAME_infos->Seg_Count = 0;
    }
    return p;
}

static void ethernet_watchdog(void)
{
    /* When Rx Buffer unavailable flag is set: clear it and resume
    reception */
    if ((ETH->DMASR & ETH_DMASR_RBUS) != (u32)RESET)
    {
        /* Clear RBUS ETHERNET DMA flag */
        ETH->DMASR = ETH_DMASR_RBUS;

        /* Resume DMA reception. The register doesn't care what you
        write to it. */
        ETH->DMARPDR = 0;
    }
}

void ethernetif_input(void *pvParameters)
{
    struct pbuf *p;

    for (;;)
    {
        if (xSemaphoreTake(s_xSemaphore,
                           emacBLOCK_TIME_WAITING_FOR_INPUT) == pdTRUE)
        {
            while ((p = low_level_input(s_pxNetIf)) != 0)
            {
                if (p != 0)
                {
                    if (ERR_OK != s_pxNetIf->input(p, s_pxNetIf))
                    {
                        pbuf_free(p);
                        p = NULL;
                    }
                }
            }
        }
        ethernet_watchdog();
    }
}


Also it isn't trivial because one needs to do rate limiting i.e. disable further ints and then re-enable them in a timer ISR.

Why?  The interrupt rate should be no faster than the inbound packet rate, which given a 100Mb link and typical packet sizes isn't all that fast.  Plus trains of interrupts are limited by the number of DMA buffers configured.  When all the buffers are full, the interrupts stop.

Another thing, not in the code I posted for brevity, is that we are doing link status change detection (cable plugged in etc) in that RTOS task, and it conveniently avoids issues with code running in a different RTOS context.

That can still happen.   Just put it inside the receive semaphore wait loop (for loop in the above example code) and set the semaphore wait timeout to an appropriate value.

[peter-h]

Many thanks. I will test that later today. Going to a wedding - yawn.

My original polled low level input code did actually loop round until the rx queue was drained (having been checking for data every 10ms) but this was removed because it could be stuck in that loop if stuff was arriving fast. But now that we will be dumping broadcasts, I might put that back in, because at least those packets will not be stuffed into LWIP anymore.

[peter-h]

I don't think the above function is quite right. See test reports below. First, I changed it to a boolean:

Code: [Select]


// Function used to test whether a packet is to be rejected - broadcasts etc
// Returns true if packet is to be accepted.

static bool should_accept_ethernet_packet(const uint8_t * pktData, size_t pktLen)
{
if (g_eth_multi) return true;

const struct eth_hdr * pktHdr = (const struct eth_hdr *)pktData;

/* If the packet is too short, reject it. */
if (pktLen < sizeof(struct eth_hdr))
{
return false;
}

/* If the packet is a broadcast ARP accept it */
if ( eth_addr_cmp(&pktHdr->dest, &ethbroadcast) )
{
if (ntohs(pktHdr->type) == ETHTYPE_ARP)
{
return true;
}
}

// if the packet is to an IPv4 multicast destination, reject it.
if (pktHdr->dest.addr[0] == LL_IP4_MULTICAST_ADDR_0 &&
pktHdr->dest.addr[1] == LL_IP4_MULTICAST_ADDR_1 &&
pktHdr->dest.addr[2] == LL_IP4_MULTICAST_ADDR_2)
{
return false;
}

/* if the packet is an IPv4 packet, accept it. */
if (ntohs(pktHdr->type) == ETHTYPE_IP)
{
return true;
}

/* Reject all other packets WRONG TODO */
return false;
}


Hard to debug because there are so many packets and I don't know the format details. But someone may spot something obvious.

This test is taken at about the right time for where I would expect data packets

Code: [Select]

/* if the packet is an IPv4 packet, accept it. */
else if (ntohs(pktHdr->type) == ETHTYPE_IP) {
TopLED(true); // debug
TopLED(false);
return true;
}
I have made the "multicast dropping" configurable via a config file entry, so it can be changed post-compilation, and curiously dropping non-ARP broadcast packets in low_level_input (as discussed above) does not avoid the need for more RX buffers at the PHY level. I still need 3+ there. So that issue is more subtle - assuming that my multicast dropping above is actually working. And I don't think it is since this code is never taken

Code: [Select]

// if the packet is to an IPv4 multicast destination, reject it.
if (pktHdr->dest.addr[0] == LL_IP4_MULTICAST_ADDR_0 &&
pktHdr->dest.addr[1] == LL_IP4_MULTICAST_ADDR_1 &&
pktHdr->dest.addr[2] == LL_IP4_MULTICAST_ADDR_2)
{

TopLED(true); // debug
TopLED(false);

return false;
}
This bit of code does get taken every few seconds, which seems about right

Code: [Select]


/* If the packet is a broadcast ARP accept it */
if ( eth_addr_cmp(&pktHdr->dest, &ethbroadcast) )
{
if (ntohs(pktHdr->type) == ETHTYPE_ARP)
{
TopLED(true); // todo
TopLED(false);
return true;
}
}
I wondered why not a one-line conditional but then I realised that ntohs() is quite a lot of code so this is a good optimisation.

This bit is not taken during normal running but is taken roughly once around the time DHCP etc is allocated, and it stops startup. I suspect there is some other ("pass") test missing

Code: [Select]


/* Reject all other packets  */
TopLED(true); //
TopLED(false);
return false;  // this needs to be return true for things to work i.e. test bypassed
}
Another interesting data point is that low_level_input executes in 2us if there is no rx data, and 24-28us if there is a packet to be stuffed into LWIP. This puts a bit of perspective on whether an ISR is useful or not. 168MHz 32F417.