Need some advice on transferring data from LVDS to ethernet, around 60 mbit/s

[hamster_nz]

Yes, you would have to use maximum, or nearly maximum, packets but I was thinking end-to-end up through the application layer.  It would be even more difficult if TCP is required.  Maybe it's time for another test.  There's some reason I can't get my UDP rate over 32Mbps.  At 70,000 pps with 58 byte packets, it would seem that all I need to do is increase the packet size.  Yes the pps will drop but probably not as fast as the bps increases.  More testing to do...

For 100BaseT, and maximum 'non-jumbo' UDP payload is 1472 (giving a 1512-byte frame), so with large packets the peah throughput is  12,500,000 / ( 8 + 14 + 20 + 8 + 1472 + 4 + 12) = 12500000/1540 = 8,116 frames per second, or  11,948,051 bytes/sec

However, In my design I'm sending frames with only 16 bytes of user data, 12,500,000 / ( 8 + 14 + 20 + 8 + 16 + 4 + 12) = 152,439 frames per second.

Because of this the OS's is seeing almost 20x the number of interrupts, context switches and so on than would be required to support a link fully saturated with larger packets.

Or to look at it another way, where the PC/laptop would have have over 123 us to process each large packet it now only has 6.56 us (only a few thousand CPU cycles), and it is choking.

However, that is pretty much what I would expect from 'cheap and cheerful' Gigabit NICs. It is most likely engineered to receive around 100k packets per second, which is more than enough to process a fully saturated gigabit link... ...but only if the average frame size is around 1250 bytes or more.

[hamster_nz]

For the rgmii interface, I need to send a logical derivative of tx_en and tx_err on the tx_ctl line on the txglk falling edge.

What is a logical derivative?

tx_err is XORed with tx_en to give 'the logical derivative'.

tx_ctl runs at DDR, and sends tx_en for the first half of a cycle, and tx_err XOR tx_en for the second half.

So in normal use when starting to send a packet it sends sends 00,00,00,00,00,11,11,11,11,11,11,11,11...
when there is errors, it might send 00,00,00,00,00,11,10,11,11,10,11,11,11...

This reduces the number of transitions during normal operation, and therefore reduces power usage (and most likely more importantly) reduces EMI.

[asgard20032]

hamster_nz, how do you debugged/made your implementation work? I am having some trouble with my implementation. When I look on modelsim, everything seem fine, got data in the right order. But when testing it in real life, nothing happen in wireshark. Not sure if it's a clock problem, crc problem, or data alignment with clock problem.

[hamster_nz]

hamster_nz, how do you debugged/made your implementation work? I am having some trouble with my implementation. When I look on modelsim, everything seem fine, got data in the right order. But when testing it in real life, nothing happen in wireshark. Not sure if it's a clock problem, crc problem, or data alignment with clock problem.

Hi,

My test setup is
- Nexys Video Board
- UTP cable
- Gigabit NIC on my laptop.
- Wireshark running under Windows 8 to capture the packets

I have to set the NIC on the laptop to 1Gb/Full Duplex, and then download the design.

Because the packets are not addressed directly to my laptop I don't seen any network bandwidth until I start capturing them.

Under windows you can use "netstat -e" to verify packet counts and look for FCS/checksum issues (bad CRCs increment to the 'error' counter).

Problems I could expect
- My board's PHY is strapped to be 10/100/1000 after reset. Yours may not be, and you may need to configure it through the serial management interface.
- In my HDL design the TX clk is at 90 degrees to the main logic's clock used for the data. Some phys have the option to insert delays on the clocking making 0 degrees the best.
- If it is your own code, the nibble ordering is painful and seems wrong. For example with a MAC of 01:23:45:67:89:AB it should hit the wire in this order 1,0,3,2,5,4,7,6,9,8,B,A.

If you need me to send any simulation traces or CRCs for test data pop me an PM with your email and what would help and I'll do what I can to help, or send me through some simulation traces and I'll take a look.

[asgard20032]

In my implementation, on the falling edge of my clock, I put the data and tx enable in a DDIO interface. The same clock drive the DDIO, so next rising edge, it has the first nibble (nibble also sent in revert order). My transmit clock is phased out by 90 degree. (But not sure if I should do + 90 or -90). Can you confirm my clock scheme is correct?

Didn't know I had to configure the PHY to 10/100/1000. I have the marvell 88e1111, and information about the serial management is so unclear.

[hamster_nz]

Here's what it looks like in the simulator for me... broadcasting from DE:AD:BE:EF:01:12. Should answer a lot of questions.

[asgard20032]

It's alive!!!

[hamster_nz]

It's alive!!!

Sweet!

[asgard20032]

I can see packet from wireshark, no error in packet, but strangely, if i try to intercept packet from python, it just hang there... doing nothing. Have to kill python.

Code: [Select]

class MyUDPHandler(socketserver.BaseRequestHandler):
    """
    # This class works similar to the TCP handler class, except that
    # self.request consists of a pair of data and client socket, and since
    # there is no connection the client address must be given explicitly
    # when sending data back via sendto().
    # """

    def handle(self):
        print("Test")
        data = self.request[0].strip()
        socket = self.request[1]
        print ("{} wrote:".format(self.client_address[0]))
        print (data)

if __name__ == "__main__":
    HOST, PORT = "192.168.25.10", 4096
    server = socketserver.UDPServer((HOST, PORT), MyUDPHandler)
    print ('Starting listening')
    server.serve_forever()

192.168.25.10 is my computer NIC
192.168.25.25 is FPGA
4096 is the port. Packet size is 65 byte worth of data.

Windows 10 64 bit, anaconda python installation.

[hamster_nz]

Does 'netstat -e' show errors? If so that indicates CRC issues.

Code: [Select]

H:\>netstat -e
Interface Statistics

                           Received            Sent

Bytes                     107942320        14045394
Unicast packets              106224           65209
Non-unicast packets          130447            5870
Discards                          0               0
Errors                            0               0
Unknown protocols                 0

H:\>

With smart NICs sometimes bad packets look OK as the CRCs and checksums are verified by the NIC and not the OS... can you post a wireshark of your packet too?

[asgard20032]

no error in netstat

[nctnico]

Python is probably choking on the vast amount of packet. I'd try to slow the rate of the packets down first.

[asgard20032]

Not related to python, since i tried in C# and C++. I finally found the problem. Although wireshark was able to see packet and was saying everything was ok, it was not. The IPV4 checksum is incorrect. I needed to go into wireshark option to enable checksum calculation. Then it flagged all my packet as incorrect. Then wireshark say checksum incorrect, it should be ****. If i copy and past this **** from wireshark into my vhdl, everything work fine.

2 Option now: Always use wireshark for precalculation and copy and past into my code, or correct my checksum algorithm(I tried the same technique as hamsternz for the checksum).

[hamster_nz]

Then wireshark say checksum incorrect, it should be ****. If i copy and past this **** from wireshark into my vhdl, everything work fine.

Grr... checksums.

So playing around with my new-found learning on Ethernet on FPGAs, I'm getting so frustrated with locations of the checksums. Why did they put the checksums at the head of the packet, so you have to delay the data by 1500 cycles or so to allow the checksum in the header to be updated? Grrr

However I've got a working framework built (it has a few glaring omissions that will need to be added later), and am now slowly adding protocol support, hopefully up to a full HTTP Web Server for serving static content at full wire speed - ARP is completed & tested, ICMP is next... on github at https://github.com/hamsternz/FPGA_Webserver

Why? Because I'm stupid and like the challenge - however, it will be one of the few web servers that has no software of firmware involved anywhere, making it unhackable (except for DOS attacks), and it will be nearly fully deterministic (you throw the same data at it, you get nearly the same result)

It could be made fully deterministic if the TX clock was linked to the RX clock, but that would fix the link speed to 1Gb/s...

[nctnico]

For UDP data you don't have to calculate a checksum (leave it 0) and the IPV4 header checksum (which is only calculated over the header) can be pre-calculated and re-used for all packets.

[Scrts]

I've always used Wireshark for network packet analysis and I can say that this tool is the most amazing open source tool I've ever seen. It will also show you CRC problems or packet counter problems. E.g. I had a glitch in my FPGA logic when rolling over 16bit counter of the packet count. It took some time to run 65k packets to see, but wireshark highlighted that quickly.