Author Topic: Complexity of USB stack vs. TCP/IP stack  (Read 8346 times)

0 Members and 1 Guest are viewing this topic.

Offline richardmanTopic starter

  • Frequent Contributor
  • **
  • Posts: 427
  • Country: us
Complexity of USB stack vs. TCP/IP stack
« on: July 15, 2016, 08:52:42 pm »
What are people's thoughts on the complexity of writing a USB stack vs. TCP/IP stack? For USB, let's say the basic HID and CDC class are supported, and the TCP/IP is a full stack more heavy weight than uIP or LWiP. Layered implementations on top of a message passing OS.
// richard http://imagecraft.com/
JumpStart C++ for Cortex (compiler/IDE/debugger): the fastest easiest way to get productive on Cortex-M.
Smart.IO: phone App for embedded systems with no app or wireless coding
 

Offline chickenHeadKnob

  • Super Contributor
  • ***
  • Posts: 1055
  • Country: ca
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #1 on: July 15, 2016, 09:10:25 pm »
Not sure what you are trying to compare is easily comparable. The main reason is that the traditional way to a full tcp/ip stack is to start from the BSD stack, and to use its structure and state machine. That starting point  for TCP/IP changes the design flow compared to usb where I don't think there exists such a universally accepted driver-exemplar.
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 26906
  • Country: nl
    • NCT Developments
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #2 on: July 15, 2016, 09:16:19 pm »
I think a USB stack is easier (but you might just as well fork LUFA). If you think about networking then you immediately enter the world of security and hacking. This would require some way to test a TCP/IP stack for buffer overflows, possible attack scenarios. Also you are likely to want to offer basic protocols like ARP, HTTP, DNS, DHCP, SMTP and each have their specific problem areas where it comes interoperability. Also a BSD style socket interface is easy to interface with.

Anyway, for one of my upcoming projects I'm planning on using a Wiznet chip partly because the whole network stuff will be outside my primary processing core.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline Howardlong

  • Super Contributor
  • ***
  • Posts: 5319
  • Country: gb
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #3 on: July 15, 2016, 09:18:43 pm »
In short, I'd do neither, each is pretty much a life's work before you've even blinked so much as an LED. Leverage someone else's blood, sweat and tears.

Even if you get as far as getting something to "work", how well does it interoperate with other parts of the network, whether USB or Eth etc.?
 
The following users thanked this post: Kilrah

Offline Philfreeze

  • Regular Contributor
  • *
  • Posts: 123
  • Country: ch
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #4 on: July 15, 2016, 09:31:46 pm »
In short, I'd do neither, each is pretty much a life's work before you've even blinked so much as an LED. Leverage someone else's blood, sweat and tears.

Even if you get as far as getting something to "work", how well does it interoperate with other parts of the network, whether USB or Eth etc.?

I don't know about the TCP/IP stack but I assume it is actually really hard but the USB stack is quite easy to implement.
It is just a bus with a bit more complex protocol but as long as you have some USB enabled hardware and you don't have to bit-bang it is quite easy.

As a project I implemented the USB stack (HID only) on a PIC MCU and it took me about a month (around 15-20 days) with testing but I wouldn't exactly say it is an exiting thing to do. It is fine if you want to learn how USB works (this is why I did it) but it really isn't an interesting project. You will just go through the USB protocol specifications and implement all the bit fields, structs, interrupts, timeouts etc. exactly as they are described in the pdf.
 

Offline Engineer_EE_FW_SW

  • Newbie
  • Posts: 6
  • Country: us
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #5 on: July 15, 2016, 09:33:09 pm »
Edit: Now I understand your position based on your update.
Just try to port LWIP or some appropriately designed stack to your platform, if the OS works already and you have network drivers / APIs, it shouldn't be that hard.

USB device class implementations for HID or CDC are almost always 'low level, get it done' code.  They're often intended for high volume consumer electronics where spending money on an extra few kBytes of RAM/FLASH is a big deal to major MCU customers.  USB device class support for HID / CDC is like I2C or SPI or UART in that way -- everyone wants it to work, but nobody wants it to cost much in memory or other costly resources, so implementations aren't so much cross platform portable, layered, and elegant rather they're basic and functional and tend to be obscure and platform specific.

Also mostly no MCU vendor wants their own developed USB device class code to end up being ported and used on someone else's MCU family so cross platform portability isn't a concern except for 3rd party stack / OS vendors that offer solutions for different platforms.

« Last Edit: July 15, 2016, 10:04:15 pm by Engineer_EE_FW_SW »
 

Offline kolbep

  • Frequent Contributor
  • **
  • Posts: 598
  • Country: za
    • ShoutingElectronics.com
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #6 on: July 15, 2016, 09:42:19 pm »
Just stick an ESP8266 Wifi Module on the board.
Then just a few AT Commands and you can read and write to the net, and even have it act as a server...
====================================
www.ShoutingElectronics.com Don't just talk about Electronics, SHOUT ABOUT IT! Electronics Blog Site and Youtube Channel
 

Offline ataradov

  • Super Contributor
  • ***
  • Posts: 11248
  • Country: us
    • Personal site
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #7 on: July 15, 2016, 09:44:00 pm »
USB is easier, just because you don't need to manage tables, expire records and maintain all sorts of timers.
Alex
 

Offline Howardlong

  • Super Contributor
  • ***
  • Posts: 5319
  • Country: gb
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #8 on: July 15, 2016, 09:44:30 pm »
In short, I'd do neither, each is pretty much a life's work before you've even blinked so much as an LED. Leverage someone else's blood, sweat and tears.

Even if you get as far as getting something to "work", how well does it interoperate with other parts of the network, whether USB or Eth etc.?

I don't know about the TCP/IP stack but I assume it is actually really hard but the USB stack is quite easy to implement.
It is just a bus with a bit more complex protocol but as long as you have some USB enabled hardware and you don't have to bit-bang it is quite easy.

As a project I implemented the USB stack (HID only) on a PIC MCU and it took me about a month (around 15-20 days) with testing but I wouldn't exactly say it is an exiting thing to do. It is fine if you want to learn how USB works (this is why I did it) but it really isn't an interesting project. You will just go through the USB protocol specifications and implement all the bit fields, structs, interrupts, timeouts etc. exactly as they are described in the pdf.

Did you hack at the register level or use existing bits of the stack? Plus getting it ratified/certified as kosher isn't trivial.

Either way, I agree, I have better things to do with my life!

Edit: although I highly admire your tenacity, of course.
« Last Edit: July 15, 2016, 09:46:22 pm by Howardlong »
 

Offline richardmanTopic starter

  • Frequent Contributor
  • **
  • Posts: 427
  • Country: us
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #9 on: July 15, 2016, 09:48:01 pm »
Sorry, I should have given more context.

The context is now that we have the OS running, we are writing the stacks to go with our tools to sell to customers.  (By writing, I meant also potentially leveraging open source code, of course we will abide the licensing terms of whatever code we leverage, if any)

They will also go into hardware boards that we are making/selling as well.

Since we are flexible in the order of releases, just gauging which one we should tackle first. One nice thing about TCP/IP is that there are many example implementations and design notes available. Easier to see which features we should implement and why.

The USB stacks situation seems to be less "evolved". Not a whole lot of examples of layered stacks with ease of adding new device classes. most feel like simple "get it done" jobs. I could be wrong of course.
// richard http://imagecraft.com/
JumpStart C++ for Cortex (compiler/IDE/debugger): the fastest easiest way to get productive on Cortex-M.
Smart.IO: phone App for embedded systems with no app or wireless coding
 

Offline ataradov

  • Super Contributor
  • ***
  • Posts: 11248
  • Country: us
    • Personal site
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #10 on: July 15, 2016, 09:49:55 pm »
In that case go with TCP, there is way more readily available stuff and porting existing stacks to new hardware is typically trivial.
Alex
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 26906
  • Country: nl
    • NCT Developments
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #11 on: July 15, 2016, 10:08:36 pm »
The USB stacks situation seems to be less "evolved". Not a whole lot of examples of layered stacks with ease of adding new device classes. most feel like simple "get it done" jobs. I could be wrong of course.
Did you look at LUFA / a.k.a. the USB host/device/OTG stack NXP offers with their controllers? I have used it in a few projects and even though the documentation is lacking (as usual) it seems like a well organised stack with a clear hardware interface and user interface. I don't think it will be difficult to port it to other architectures.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline richardmanTopic starter

  • Frequent Contributor
  • **
  • Posts: 427
  • Country: us
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #12 on: July 15, 2016, 10:11:18 pm »
Did you look at LUFA / a.k.a. the USB host/device/OTG stack NXP offers with their controllers? I have used it in a few projects and even though the documentation is lacking (as usual) it seems like a well organised stack with a clear hardware interface and user interface. I don't think it will be difficult to port it to other architectures.

Yes, indeed, I am in communication with LUFA's author, Dean, on the subject. Leveraging LUFA is definitely a possibility.
// richard http://imagecraft.com/
JumpStart C++ for Cortex (compiler/IDE/debugger): the fastest easiest way to get productive on Cortex-M.
Smart.IO: phone App for embedded systems with no app or wireless coding
 

Offline C

  • Super Contributor
  • ***
  • Posts: 1346
  • Country: us
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #13 on: July 15, 2016, 10:22:53 pm »
which TCP/IP stack?

If you are creating a new OS these days leaving out v6 would not be a great idea.
 

Offline westfw

  • Super Contributor
  • ***
  • Posts: 4199
  • Country: us
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #14 on: July 16, 2016, 04:13:10 am »
A TCP/IP stack is very symetric, very general, and has few "critical" timing requirements.  It's pretty huge and has complications (like security.0
a "USB stack" is NOT symetric at all; it has a device side that is not too hard to implement, and a host side that pretty mind-boggling in the things its expected to perform.  usb hub management, for example.  Keeping track of all 256 (?) nodes that might be connected, and how to poll all of them adequately despite the many operating modes.
A USB device is much easier to write code for than a TCP device, and much of the hard work is done by hardware.
a USB host and a TCP host are probably "close", except the TCP host can cheat and implement a nice slow network interface that avoids inpunging on some of the issues that make USB host complex.
 
The following users thanked this post: oPossum

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 26906
  • Country: nl
    • NCT Developments
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #15 on: July 16, 2016, 02:48:17 pm »
You don't have to implement handling USB hubs in a USB host stack. After all a USB hub is nothing more than a device which multiplexes USB messages. OTOH it should be doable to implement it IF the USB stack has the ability to handle multiple interfaces. IIRC LUFA can deal with multiple host interfaces or is simple to modify so it could be a good starting point.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline LeonV

  • Contributor
  • Posts: 39
  • Country: nz
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #16 on: July 20, 2016, 03:56:45 am »
I have written both before as a proof of concept, from scratch, in PICASM. It was fun. But i would not recommend it.

The USB client stack was a lot easier to get going than TCP/IP, but only because with TCP you need to implement things like ARP and DHCP.

I have looked into making a USB host on a PIC before, but i wasn't going to go there. IP is easier than making a USB host in that way.


But that is from a programming from scratch stand point, you will be using libraries.

If it was a product, i would implement TCP/IP.

Don't forget about security as well, There are a lot of devices that find their way onto the internet over time and are simply not secure.

Slapping an ESP8266 on things is such a simple solution, But i always hated WiFi in products, It always fails. Too susceptible to environmental factors. So so easy though!

And like  C said, IPv6 will become the norm before long, so its essential for consumer products.
Damn forum is making me procrastinate from work!
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 26906
  • Country: nl
    • NCT Developments
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #17 on: July 20, 2016, 07:45:21 am »
And like  C said, IPv6 will become the norm before long, so its essential for consumer products.
They have been saying that for over a decade now but I don't see it happen. The whole idea behind IPv6 is that every device gets it's own unique IP address on the internet. From a security perspective that is a nightmare so that is exactly what you don't want for any device and kind off eliminates the necessity of IPv6. If you look at the IP address of your phone or tablet you'll see it has an IPv4 address in a private range. The providers use NAT routers to allow internet access for the devices but hide them from the big bad outside world. It makes the life of an IoT device a whole lot simpler if it doesn't have to deal with flooding attacks, random messages, etc.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline richardmanTopic starter

  • Frequent Contributor
  • **
  • Posts: 427
  • Country: us
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #18 on: July 20, 2016, 07:55:03 am »
Thanks for all the great discussions and information.

Just an update: I am building LWiP currently. Starting with the latest 2.0 BETA. Most of the code builds now using our compiler. I hope to be able to test the simple functions on the 746 ST-Nucleo some time next week.

Then my idea is to port it to work with message passing protocols, may be dividing the layers into different servers, i.e. IP protocol server, UDP server, TCP server etc., and then add zero-copy buffering so to avoid buffer copying across the layers. The RTOS is preemptive round robin priority with interrupt able to wake up "hibernating" tasks. So in theory, a packet comes in from the ethernet interface would wake up the IP protocol server and so on.

Will see how things work out.
// richard http://imagecraft.com/
JumpStart C++ for Cortex (compiler/IDE/debugger): the fastest easiest way to get productive on Cortex-M.
Smart.IO: phone App for embedded systems with no app or wireless coding
 

Offline C

  • Super Contributor
  • ***
  • Posts: 1346
  • Country: us
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #19 on: July 20, 2016, 07:46:46 pm »
And like  C said, IPv6 will become the norm before long, so its essential for consumer products.
They have been saying that for over a decade now but I don't see it happen. The whole idea behind IPv6 is that every device gets it's own unique IP address on the internet. From a security perspective that is a nightmare so that is exactly what you don't want for any device and kind off eliminates the necessity of IPv6. If you look at the IP address of your phone or tablet you'll see it has an IPv4 address in a private range. The providers use NAT routers to allow internet access for the devices but hide them from the big bad outside world. It makes the life of an IoT device a whole lot simpler if it doesn't have to deal with flooding attacks, random messages, etc.

I would suggest you study up some on NAT

I would say that what you state as security is more a security fail.

You have X devices on internal network of which one or more uses a port to talk to INTERNET.
You had X times 2^16 possible ports.
NAT Reduces X to 1 in most cases.
The array of data that NAT needs to function requires two variables, one 2^32 and one 2^16 with and a port index of 2^16.
With NAT software able to pick the port this array can be reduced in size

A device talking to INTERNET adds entries to this array.

Your large number of possible source ports just became a small range active ports. 

In most cases an active NAT port will accept packets from any address & port from INTERNET side to this active port and forward it to the device.
Some software makes use of this.

You have device security as long as that device does not talk to the INTERNET for anything.
NAT does not prevent a device from talking, that is the firewalls job.

Even this simple NAT described breaks some IP protocols. Trying to improve security via NAT can break more IP protocols with little gain.

There are other versions of NAT that change some things, but the basics is more active ports in a smaller range and this smaller range makes it easer for the outside attacks.

 

Offline timb

  • Super Contributor
  • ***
  • Posts: 2536
  • Country: us
  • Pretentiously Posting Polysyllabic Prose
    • timb.us
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #20 on: July 20, 2016, 10:51:20 pm »
And like  C said, IPv6 will become the norm before long, so its essential for consumer products.
They have been saying that for over a decade now but I don't see it happen. The whole idea behind IPv6 is that every device gets it's own unique IP address on the internet. From a security perspective that is a nightmare so that is exactly what you don't want for any device and kind off eliminates the necessity of IPv6. If you look at the IP address of your phone or tablet you'll see it has an IPv4 address in a private range. The providers use NAT routers to allow internet access for the devices but hide them from the big bad outside world. It makes the life of an IoT device a whole lot simpler if it doesn't have to deal with flooding attacks, random messages, etc.

And it has been slowly rolling out. You could be browsing the web via IPv6 right now and not even know it! Most major data centers around the world are providing IPv6 blocks now, which means most websites are IPv6 ready now.

Over the last 5 years, cell phone carriers have started rolling out IPv6 and, as a result, pretty much every cell phone (here in the US at least) gets an IPv6 address now. (In fact, most cell providers are using NAT for IPv4, because they have nowhere near enough addresses.)

Most home internet connections (DSL and Cable, in North America at least) can also provide an IPv6 address. Though, availability depends largely on your how you have your router configured, the capitulates of your modem and your provider. For example, some providers are still doing 6over4 tunneling, which is atrocious, but at least it allows you to connect to the IPv6 internet with no configuration on your end.

The problem with IPv4 is we ran out of useable address space years ago, and it's getting worse and worse every day. Does every IoT device *need* a public IP address? Well, no, but it's not even about that. Do you think that your *entire household* should have at least *one* public address? Of course, right? But that might not even be the case anymore! ISPs have had to resort to assigning private IPs instead of public ones, using NAT. So essentially you've got groups of a hundred customers sharing the same public IP.

On the surface, that doesn't sound so bad, until you realize that this puts you into a "double NAT" situation: Public IP X.X.X.X > ISP NAT IP 160.x.x.x > LAN NAT IP 192.168.x.x

That makes it very hard for applications that need direct incoming port access to traverse. (Think video conferencing, gaming, etc.)

Anyway, you might be surprised that you're already using IPv6, and don't even know it. I know I was last year!
Any sufficiently advanced technology is indistinguishable from magic; e.g., Cheez Whiz, Hot Dogs and RF.
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 26906
  • Country: nl
    • NCT Developments
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #21 on: July 21, 2016, 07:29:36 am »
Video conferencing and gaming work without needing a fixed IP address. They solved that problem years (>decade) ago because you simply can't rely on people having a fixed IP address. Even IF people have some sort of fixed IP address it can change when they reset their modem or change providers. The only things on internet which need a fixed IP address are servers.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Online gmb42

  • Frequent Contributor
  • **
  • Posts: 294
  • Country: gb
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #22 on: July 21, 2016, 12:51:47 pm »
Video conferencing and gaming work without needing a fixed IP address. They solved that problem years (>decade) ago because you simply can't rely on people having a fixed IP address. Even IF people have some sort of fixed IP address it can change when they reset their modem or change providers. The only things on internet which need a fixed IP address are servers.

You really need to update yourself on IPv6 and not just repeat the NAT mantra.  See RFC 4941 for how devices that don't need a static IP can automatically change it.

My Windows client systems (and the odd Linux VM) have been doing this for the 4 years I've had an IPv6 connection without an issue.  I have a whole /64 block for them to choose from (2^64 addresses), thats 2^32 bigger than the entire IPv4 address space!
« Last Edit: July 22, 2016, 11:05:23 am by gmb42 »
 

Offline timb

  • Super Contributor
  • ***
  • Posts: 2536
  • Country: us
  • Pretentiously Posting Polysyllabic Prose
    • timb.us
Complexity of USB stack vs. TCP/IP stack
« Reply #23 on: July 21, 2016, 01:29:07 pm »
Video conferencing and gaming work without needing a fixed IP address. They solved that problem years (>decade) ago because you simply can't rely on people having a fixed IP address. Even IF people have some sort of fixed IP address it can change when they reset their modem or change providers. The only things on internet which need a fixed IP address are servers.

Well, no, not exactly... First off, are you talking about private IPs (the ones used inside your LAN, like 192.168 and 10.0)? Or public IPs (the one that *should* be globally routable and unique to you).

If you're talking about public IPs, I never said you needed a static IP address. Dynamic works just fine. (Also, no, a fixed (static) IP will never change, even when they reset their modem. That's the entire point! A dynamic IP can change when you reset your modem, or even when the DHCP lease expires.)

The issue I raised has nothing to do with that. The issue is about TCP/UDP ports, and how to correctly route them when you have more than one device using a single public IP. This was solved about a decade ago, with things like NAT-PMP (Network Address Traversal-Port Mapping Protocol) and UPnP (Universal Plug and Play).

First off, what is NAT? Network Address Translation allows multiple devices on a private network, with private IP addresses, to share a *single* public facing IP address (which can be static or dynamic). It works really well when a device on the private side requests something from the public side (browsing a website, for example). NAT keeps track of where the request came from, so it knows which computer on the private side to send the data to when the server responds.

So far so good, right? But what happens when, say, I try to make a direct connection to your public IP, without you requesting anything first? (Say, a BitTorrent client on my LAN trying to connect across the Internet to a BT client running on a machine in your LAN.) Well, it doesn't work! Unless you've gone into your router and manually setup port forwarding, to forward that specific BT port the the private IP of the machine running the client.

That's a huge pain to have to do for every game, P2P client, video chat client, etc.

So, how do we solve that problem? Well, that's what NAT-PMP and UPnP do! They allow any device in your private network to automatically setup temporary port forwarding by just making the appropriate request with your router.

So, that's why we don't have to worry about all this port forwarding stuff anymore. (And why most games and video conferencing clients have "just worked" for the last 10+ years.

So, what was I talking about in my last post? Well, NAT was only designed for private networks (aka your LAN). And NAT is certainly not designed to work behind *another* NAT.

The problem is, since we've run out of IPv4 addresses, ISPs can't get new pools of IPs to assign to their users (remember, each user should have at least *one* public facing IP address, be it statically or dynamically assigned). So, what some ISPs are doing is taking a single public IP and, using NAT, sharing it with 10 to 100 customers. (In that case, the IP assigned to your modem/router is actually a private IP inside their LAN.)

Now, why is that a problem? Well, let's use the BitTorrent example from earlier. So, your BT client uses NAT-PMP to request a specific port be forwarded from your router, which it does. The problem is, your router doesn't have a publicly accessible IP, because you're behind a second level of NAT from your ISP's router, and your BT client can't make a NAT-PMP request to that, therefor the port ultimately can't be forwarded from a public facing IP to your device.

This is called "double NAT" and it's bad. The only way around this is with IPv6, which would allow each device on your LAN to have its own public IP address, allowing us to do away with NAT and port forwarding all together.

Which, by the way, is not at all "insecure" as you suggested. Why? Because the gateway to these devices will still be your own router, which should have a firewall enabled. (Remember, firewalls are for security, NAT isn't.)
« Last Edit: July 21, 2016, 01:33:44 pm by timb »
Any sufficiently advanced technology is indistinguishable from magic; e.g., Cheez Whiz, Hot Dogs and RF.
 

Offline mac.6

  • Regular Contributor
  • *
  • Posts: 225
  • Country: fr
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #24 on: July 21, 2016, 01:38:32 pm »
Based on my work experience, USB stack complexity is not in the software, but in the myriad of exceptions and bugs due to the myraid of bogus/pelicular devices/hosts that are in the field.
On the other hand TCP/IP stack can be software beast but easier to test and with less device issues (except your network device).
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 26906
  • Country: nl
    • NCT Developments
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #25 on: July 21, 2016, 09:05:40 pm »
Video conferencing and gaming work without needing a fixed IP address. They solved that problem years (>decade) ago because you simply can't rely on people having a fixed IP address. Even IF people have some sort of fixed IP address it can change when they reset their modem or change providers. The only things on internet which need a fixed IP address are servers.
Well, no, not exactly... First off, are you talking about private IPs (the ones used inside your LAN, like 192.168 and 10.0)? Or public IPs (the one that *should* be globally routable and unique to you).

If you're talking about public IPs, I never said you needed a static IP address. Dynamic works just fine. (Also, no, a fixed (static) IP will never change, even when they reset their modem. That's the entire point! A dynamic IP can change when you reset your modem, or even when the DHCP lease expires.)

The issue I raised has nothing to do with that. The issue is about TCP/UDP ports, and how to correctly route them when you have more than one device using a single public IP. This was solved about a decade ago, with things like NAT-PMP (Network Address Traversal-Port Mapping Protocol) and UPnP (Universal Plug and Play).

This is called "double NAT" and it's bad. The only way around this is with IPv6, which would allow each device on your LAN to have its own public IP address, allowing us to do away with NAT and port forwarding all together.
AFAIK UPnP has already been abandoned / not recommended for being unsafe. The ISP I'm using has always disabled UPnp in their routers by default and I'm not sure the most recent models even support it. The reality is that you don't need things like NAT-PMP or UPnP. All modern software works without those because most people don't know how to configure their router or are in a (semi) enterprise environment where reconfiguring something on a router takes weeks.

Sure there are ways around it but when making gadgets which just need to work you don't want to rely on people having a publicly routable IP address or certain features enabled in their internet access. And this also leads to where IPv6 support can be low on the list. BTW a big problem with IPV6 is that it uses (may use) the MAC address of the ethernet interface for the IPV6 addres. MAC adresses aren't unique (a very stubborn myth!) and can also be used to identify traffic from a certain device which could make it easier for an attacker to identify (filter) certain traffic.

The same goes for having directly routable IPV6 addresses to devices. With NAT you have to be very lucky to get a response from a device because it has to have an open connection at the time of attack. Without NAT you can attack a device directly on all ports (services it provides to the network) whether they are connected or not. People have been saying NAT is bad but the reality is that it works for billions of devices and performance doesn't seem to be a problem. Maybe NAT is bad from a technical point of view but security wise NAT does offer a first layer of protection by not exposing a device to raw internet directly.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline timb

  • Super Contributor
  • ***
  • Posts: 2536
  • Country: us
  • Pretentiously Posting Polysyllabic Prose
    • timb.us
Complexity of USB stack vs. TCP/IP stack
« Reply #26 on: July 22, 2016, 01:24:27 am »
AFAIK UPnP has already been abandoned / not recommended for being unsafe. The ISP I'm using has always disabled UPnp in their routers by default and I'm not sure the most recent models even support it. The reality is that you don't need things like NAT-PMP or UPnP. All modern software works without those because most people don't know how to configure their router or are in a (semi) enterprise environment where reconfiguring something on a router takes weeks.

You really don't know what you're talking about... While UPnP is an old, mostly obsolete standard, NAT-PMP isn't. The whole reason that modern software "just works" is *because of* protocols like NAT-PMP! The whole point behind that is so that people *don't* have to manually configure their routers...

Sure there are ways around it but when making gadgets which just need to work you don't want to rely on people having a publicly routable IP address or certain features enabled in their internet access. And this also leads to where IPv6 support can be low on the list. BTW a big problem with IPV6 is that it uses (may use) the MAC address of the ethernet interface for the IPV6 addres. MAC adresses aren't unique (a very stubborn myth!) and can also be used to identify traffic from a certain device which could make it easier for an attacker to identify (filter) certain traffic.

For TCP/IP to work as intended, you need a publicly routable IP address. Period. I have no problem with people using NAT on their local networks, but when your ISP is giving you a non-public IP and routing you through a second NAT layer, it breaks all sorts of things. I've seen it first hand. If you don't believe me, head on over to the DSLReports forums and look at some of the threads on the topic. It breaks everything from BitTorrent to FaceTime to Xbox Live!

Also, I don't know where you got the idea that IPV6 uses your MAC address as your IP address, but it doesn't. (Though, if you're assigned a /64, there is an "auto discovery" protocol similar to DHCP that can assign the last few pair of digits based on your MAC address. However, this behavior is up to the particular IPv6 stack you use, and your auto-assigned address can be randomized. iOS and OS X devices do said randomization to prevent any sort of tracking.)

The same goes for having directly routable IPV6 addresses to devices. With NAT you have to be very lucky to get a response from a device because it has to have an open connection at the time of attack. Without NAT you can attack a device directly on all ports (services it provides to the network) whether they are connected or not. People have been saying NAT is bad but the reality is that it works for billions of devices and performance doesn't seem to be a problem. Maybe NAT is bad from a technical point of view but security wise NAT does offer a first layer of protection by not exposing a device to raw internet directly.

This is not at all true. Even with IPv6, you can turn on the Firewall in your router, which should close down all incoming ports. Then, if a device needs that port open for incoming traffic, it can send an outbound packet on that specific port first, essentially "punching through" the firewall. It has the same effect as NAT-PMP.

I never said NAT is bad. It's fine! It's allowed us to stretch IPv4 out for a lot longer than originally thought possible. I have no problem with that... My problem is with an ISP running NAT internally to share a single IPv4 address with multiple customers.

The simple fact is, we can't do that and we've run out of IPv4 space, so, like it or not, we'll all be forced to transition to IPv6 in the next few years, end of story.

TL;DR: NAT = OK, DOUBLE NAT = BAD
« Last Edit: July 22, 2016, 01:26:24 am by timb »
Any sufficiently advanced technology is indistinguishable from magic; e.g., Cheez Whiz, Hot Dogs and RF.
 
The following users thanked this post: Zbig

Offline Zbig

  • Frequent Contributor
  • **
  • Posts: 927
  • Country: pl
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #27 on: July 24, 2016, 09:14:56 pm »
NAT is a dirty kludge and not a security measure. Saying that NAT is good for security is like saying that missing one leg is good for your health as you're not able to run too fast and hurt yourself. And I will keep my publicly routed IP address, nctnico because I do need it, thank you very much. As timb said more than once, firewall is what you want for security, not a bloody NAT. In my Asus router's firmware, IPv6 firewall is enabled by default and is whitelist-based, i.e. all unsolicited incoming traffic is blocked by default unless permitted explicitly.
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 26906
  • Country: nl
    • NCT Developments
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #28 on: July 24, 2016, 10:39:59 pm »
NAT is a dirty kludge and not a security measure. Saying that NAT is good for security is like saying that missing one leg is good for your health as you're not able to run too fast and hurt yourself. And I will keep my publicly routed IP address, nctnico because I do need it, thank you very much. As timb said more than once, firewall is what you want for security, not a bloody NAT. In my Asus router's firmware, IPv6 firewall is enabled by default and is whitelist-based, i.e. all unsolicited incoming traffic is blocked by default unless permitted explicitly.
Like timb you are missing the point completely. Over 20 years of experience with networking, security and IP enabled gadgets have made me very skeptical about security and interoperability. Sure a network 'expert' can configure his router but the average user is completely lost and doesn't want to be bothered with configuring a router. If you look up information about uPnP and NAT-PMP you'll see a comment about both being unsafe because both protocols allow strangers to access/scan the network behind the router/firewall. Also notice the comments about dodgy/incomplete implementations. There is a reason the router I (recently) got from my ISP (one of the biggest telecom operators in the NL!) doesn't support NAT-PMP and has uPNP disabled by default. The router also doesn't support IPv6 so I guess a multi billion dollar company is stupid for not seeing the benefits of IPv6.

I also don't see IPv6 as a solution for the near future. It is too new and a lot of security related issues have to be ironed out. IPv6 actually makes things worse because IPv6 has been naively designed to have all devices connected to a publicly routed address. Rule number one is that you don't want anything accessible from the internet directly. A technical person will instantly Pavlov into 'put a firewall in between and all your problems are solved'. But ask yourself this: say you make an IP enabled gadget which needs internet access and you sell 10.000 pieces. Who is going to configure 10.000 firewalls (routers)? Who is going to sit behind the helpdesk and talk to angry customers with a non-working device because their router needs configuring? Who is going to create step-by-step instructions for each router a customer may have? And what will ordinary users do anyway? That was a rethorical question: they will disable their firewalls alltogether because they really don't want to bother themselves with stuff that doesn't work. They are totally oblivious to the fact they open their internal networks to all worms and malware available. I hope this makes clear that simply putting a firewall in between with a whitelist isn't going to work for a commercially viable device. Whitelisting also requires maintenance because what if the server gets relocated or servers are added (load balancing)? Who is going to inform all the users in a timely manner?

Technical people keep insisting NAT is a dirty kludge but the proof is always missing. After all anecdotal evidence is what it is; if NAT where really that bad it would have been replaced ages ago by something better. Bandaids like uPnP and NAT-PMP have proven to be security nightmares which you don't want. For kicks I tried downloading a torrent (Debian Linux) through a double NAT and it works just fine (as expected). I never hear my kids complain their online games on their various game consoles don't work due to NAT either. The thing is that NAT is the most common way to allow devices from inside the network to access internet without the devices being accessible from the outside (except for devices/processes which expect network traffic) which provides a simple but effective layer of security (note: most routers offer additional protection against other attacks on their internet connected interface). Just like VHS NAT became the defacto standard and NAT has been around for 20 years so interoperability problems are unlikely to happen.

So how do you create an internet connected gadget which works out of the box and doesn't create a service and maintenance nightmare? The answer is simple: design your IP enabled gadget and server software in a way they will work with multiple NAT translations in between. Facebook, Whatsapp, etc can do it... Only worry about IPv6 when customers start asking for it. That will be soon enough (and that point is probably more than a decade away).
« Last Edit: July 24, 2016, 10:42:05 pm by nctnico »
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline timb

  • Super Contributor
  • ***
  • Posts: 2536
  • Country: us
  • Pretentiously Posting Polysyllabic Prose
    • timb.us
Complexity of USB stack vs. TCP/IP stack
« Reply #29 on: July 24, 2016, 11:45:41 pm »
nctnico, you keep digging the hole deeper and deeper... Just admit you're wrong and move on!

You say IPV6 is "too new", which isn't remotely true; it's nearly a *twenty year old* standard now! Every major OS, router and device made in the last 10 years has a working, mature IPV6 stack...

Fun Fact: IPV6 (1998) is older than NAT (1999)! So, if IPV6 is a "new" standard that's "full of security holes and bugs", then so is NAT!

The only way in which NAT-PMP can be exploited is if you *download* and *install* a piece of malware, which then (being on your local network) can setup any port forwarding it wants. Do you know how to stop that? Enable you're firewall on the router. Even with NAT-PMP off, a virus could still setup port forwarding by directly access the router's configuration page. How many people really change the "default" password? If you can change the password, you can turn on the firewall.

The "average user" very much can configure their own router. I once wrote a "how-to" guide explaining how to find, download and flash a specific OpenWRT build onto this particular router. Hundreds of people were able to follow the guide, including a housewife, a plumber and an elderly gentleman. All of whom had zero "expert" knowledge. If they can do that, then there's no excuse why anyone else can't figure out how to configure their router.

The big problem is that most people don't realize they *need* to change any settings. And, though it's getting better, factory or ISP default settings are often very poor from a security point of view. (They would rather the router be insecure than deal with technical support calls from users asking why their Netflix isn't working because the firewall blocked it.)

Not only that, but the firmware on a lot of these ISP provided routers is atrocious, with gaping back doors that hackers could access. At one point, people started writing viruses targeting various insecure routers. They could install it on one router and it would scan the subnet, install itself onto the next router it found and so on. I once knew a guy who had a bot net composed of around 100,000 routers.

Anyway, I'm digressing.

You say that NAT-PMP is a security nightmare, but where's *your* proof? See, without NAT-PMP, you wouldn't be able to use P2P, some video chat protocols, some games, etc. without manually setting up port forwarding. It's an *essential* part of why these services, for the most part, "just work" today.

You also say you tried downloading a torrent through a double-NAT, but how was it setup? Was the second NAT layer your ISP, or did you just plugin two routers back to back? Or use a VM?

You keep trying to say that we're saying NAT is a "dirty kludge" or otherwise bad, but that's not what we're saying at all. NAT is an essential part of the IPV4 Internet, the problem is we're out of IPV4 addresses. NAT was never designed to be run behind another NAT.

So, what I *am* saying, for the third time now, is that an ISP using NAT internally to share a public IP address with multiple users (by assigning their router a private IP, which is in turn shared on their LAN with a second layer of NAT) is a dirty kludge. It breaks all sorts of things. This is a proven fact.

IPV6 has been slowly rolling out for nearly 20 years. It's mature. It's ready. The only thing we're waiting on is the ISPs to get their shit together and start using it.

And, if you don't want the devices on your LAN to have public IPV6 addresses, that's no problem either. Your router can pickup a single public IPV6 address and all the devices on your LAN can use private IPV6 addresses, just like you do today with NAT. *Or* you can use a public IPV6 address and all the devices on your LAN can keep using private IPV4 addresses, simply by using NAT64 or 6to4.

By the way, I spent over 10 years of my life in IT and security (in a professional capacity). At 16 I started a web hosting company which I nurtured into a very successful and profitable business, having tens of thousands of subscribers at its peak. At 23 I sold the company to a large hosting firm and spent the next few years doing consulting. I setup and secured servers for a variety of companies. Not once did any machine I setup get compromised. So, I know a thing or two about this... ;)
« Last Edit: July 24, 2016, 11:51:41 pm by timb »
Any sufficiently advanced technology is indistinguishable from magic; e.g., Cheez Whiz, Hot Dogs and RF.
 

Offline westfw

  • Super Contributor
  • ***
  • Posts: 4199
  • Country: us
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #30 on: July 25, 2016, 12:39:45 am »
TCP/IP also has a long history of published papers and lively discussions and books and classes and so on, all about how to do it right.  Or "well", anyway.  For several different definitions of "well." :-)   As we see :-)

USB has a committee-written 600-page specification (hey!  It's downloadable!  I thought you had to pay before you could look at it!)
 

Offline joeqsmith

  • Super Contributor
  • ***
  • Posts: 11737
  • Country: us
Re: Complexity of USB stack vs. TCP/IP stack
« Reply #31 on: July 25, 2016, 02:09:10 am »
I designed an Ethernet print server that sniffs a Centronics port and routes the data to printer.   All in assembler on an Motorola 6811.   Not something I recommend.   




Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf