C: read() and write().

[hamster_nz]

Who uses read(), write(), open() & close()  functions in C?

Do you:

a) check for EINTR and retry?

b) check for things like short writes, and call write() again?

I do both of these, but some people don't believe it is needed, and some wave it away saying they use and trust SA_RESTART)...

[ataradov]

Never done any of those checks. I just fail on any error. Never had a single problem with that.

[Nominal Animal]

a) check for EINTR and retry?

Depends on whether I'm using timeout (a signal delivered to an userspace handler installed without SA_RESTART).

b) check for things like short writes, and call write() again?

Always.  (Well, there are a couple of exceptions, but those involve specific types of file descriptors with Linux-specific guarantees/semantics.)

For me, this is just the bare minimum, because otherwise you could be losing or garbling data and not know it.  "Never had a problem" is just an indication you don't care, and it never bit you bad enough for you to notice.

The things I've had to argue with people are checking the return value of close() for a delayed write error (or some other kernel-internal filesystem errors); and whether it makes sense to check for malloc()/calloc()/mmap() failures, or just let the process die from segment violation when it dereferences a NULL pointer.  (The latter especially on systems with memory overcommit.)

The purpose for me is that the data my code deals with is important to the user.  If the kernel reports there was something iffy, suspicious, or just abnormal, I believe my ethical responsibility as a software developer is to let the user know about it.  Silently garbling data is evil.

I openly admit this is rather "paranoid" (non-trusting) approach, but this is the way I do my job.  More than once have my programs been the first indication of malfunctioning hardware (although that usually leads to claims my "code is buggy, because everything else works fine" – except that on more careful checking, some data has already been garbled).  Others do it differently.

Smart people do it the way their employer pays them to do it.

[ejeffrey]

Well to check for errors you have to check for a short write. An error that occurs after writing at least one byte will cause a short write.

Whether you have to worry about EINTR or short writes caused by interrupted system calls depends on your application.  If you are writing a library that might be used in a larger program or you don't have control over signal usage or you know you are using signal handlers without SA_RESTART then you need to be prepared for that.

In linux at least, ordinary file writes are not interruptible so if you know you are accessing an ordinary file then short write == error and you don't have to worry about signals. Also, writes to pipes less than the pipe buffer size are guaranteed to be atomic.  Therefore you will never get a short write only an error with no bytes written or success.

[ataradov]

Just to clarify my point. I don't just ignore status codes or short writes. I put asserts on that stuff and let the program fail with a meaningful message. Once it does, I have a good reason to investigate and add the handling into the code.

There are so many ancient error codes and behaviors that no longer happen on modern OSes. You will go crazy for no reason trying to handle all that.

Same with malloc(). I never "handle" NULL result. I just assert.

[radar_macgyver]

My take is there's no 'one size fits all'; I tend towards the paranoid approach when write()ing or read()ing sockets or device fds, but not with files (for the reasons given by ejeffrey - haven't had to write C code outside Linux and embedded). But, as Nominal points out, this can lead to silent data corruption if one isn't careful. For socket and device operations I have some wrapper functions that catch EINTR/short writes and retry them, close() and reconnect the socket for zero-byte writes and syslog() other errors before aborting the program. With files, I check the return value of read() or write() against what I expected to send/receive and if they don't match, syslog() and abort.

[Nominal Animal]

Just to clarify my point. I don't just ignore status codes or short writes. I put asserts on that stuff and let the program fail with a meaningful message.

I'm relieved to hear that; that is definitely sufficient in my opinion.  I consider asserts as one valid way to "handle" errors and unexpected results.  They just need to provide enough information to the user to start reacting and investigating.

[thinkfat]

Who uses read(), write(), open() & close()  functions in C?

Do you:

a) check for EINTR and retry?

b) check for things like short writes, and call write() again?

I do both of these, but some people don't believe it is needed, and some wave it away saying they use and trust SA_RESTART)...

Yes to a), though it rarely happens if you're not using signals.
Yes to b) as well, pretty much mandatory for network i/o, if you use O_NONBLOCK and poll() or select(). Not so much for file i/o.

[thinkfat]

Just to clarify my point. I don't just ignore status codes or short writes. I put asserts on that stuff and let the program fail with a meaningful message. Once it does, I have a good reason to investigate and add the handling into the code.

There are so many ancient error codes and behaviors that no longer happen on modern OSes. You will go crazy for no reason trying to handle all that.

Same with malloc(). I never "handle" NULL result. I just assert.

IMHO, asserts have no place in production code, other than for "did the sky just fall?" type events that cannot be handled sensibly otherwise. There may be occasions where an OOM condition constitutes such an event. An assert for EINTR or a short write - I hope you're kidding. I'd never let that pass.

[Siwastaja]

Short writes and reads just happen all the time. Handling them in your application manually sucks but what else can you do. So you have to write a wrapper function for read/write that does some sane thing (what exactly, depends on your application). Sometimes it's just a blocking loop that tries write() as long as all data has been written.

Sometimes in a real hurry in some throw-away code I just assume short write/read doesn't happen and write an assert so that it fails in a controlled way if my assumption fails. You can save a few minutes of work doing this, in a quick test you usually know the conditions - for example, you open a fresh TCP socket and write 5 bytes, it's extremely unlikely to be a short write - so the assumption may be fairly safe.

Asserts are great whenever you don't have time to write proper error recovery, but want to do better than fail doing completely random things in undebuggable way. Proper error recovery might be a week's job, assert is 10 seconds very well spent. You can't write proper error recovery for every imaginable error, you also can't verify 100% everything, so adding asserts to verify your assumptions is a great idea; if you did your job well, they never trig, but in case of human error they are so much better than failing randomly.

Obviously, asserting that write() wrote the length you asked for is a typical example of assert() used wrong. It will crash, it's only matter of when and where.

[ejeffrey]

Just to clarify my point. I don't just ignore status codes or short writes. I put asserts on that stuff and let the program fail with a meaningful message. Once it does, I have a good reason to investigate and add the handling into the code.

Sure.  If you aren't deliberately using signals to interrupt IO, there is generally no point in "handling" a write failure at the callsite in the sense of trying to remediate the error and retry.  You can either fail immediately, or return a failure status up to the user layer to do something sensible such as  asking the user to free up disk space or reconnecting a socket.  If you *are* using signals like SIGALARM with the intent of interrupting IO, simply retrying a short write() is often not correct.  Instead you would check some status variable and decide whether you should abort or retry the transaction.

IMHO, asserts have no place in production code, other than for "did the sky just fall?" type events that cannot be handled sensibly otherwise. There may be occasions where an OOM condition constitutes such an event. An assert for EINTR or a short write - I hope you're kidding. I'd never let that pass.

Well, you shouldn't use the assert macro since that gets compiled away if NDEBUG is defined.  assert is only intended to be used for internal consistency checks that should never fail, and should not be used for handling errors that can happen.  However, the general idea of aborting when you have an error you don't know how to handle including an IO error is fine depending on the circumstances.  If EINTR is possible then you should handle it, but there are plenty of circumstances where EINTR is not possible and a short write means there was an error.

[ejeffrey]

Short writes and reads just happen all the time. Handling them in your application manually sucks but what else can you do. So you have to write a wrapper function for read/write that does some sane thing (what exactly, depends on your application). Sometimes it's just a blocking loop that tries write() as long as all data has been written.

A blocking loop that retries write is rarely correct.  Short writes do not happen "all the time" -- they only happen in well defined situations.  Namely, a non-blocking file descriptor or socket timeout (if you are using these, obviously you have to handle them properly), invocation of an async signal handler without SA_RESTART (you generally know if you are using this too), or an IO error.  If the first two don't apply, retrying won't actually transfer more data, although it will give you the precise error if you care.

Note that there is an asymmetry between read() and write().  Short *reads* happen all the time on sockets and pipes, and if there is any chance you will be operating one one you need to handle that possibility.  A read on a socket or pipe will always return as soon as there is any data available, the 'n' is the maximum data to read. 

[Siwastaja]

Related comment:

What I really love in embedded microcontroller environment is that by having access to bare metal (and usually at least somewhat decent documentation of it), a lot of hardware accesses simply can't fail. Can't fail meaning, they do not have error conditions. Sure, the sky can fall rendering the device inoperable; cosmic rays can flip a bit, and so on, but a GPIO write operation cannot return any code at all. Having 100% success guaranteed with predictable timing makes writing the application so much easier, because you don't need to deal with error recovery logic for every trivial thing.

This is also why I loathe stupid libraries that add an extra layer on top of such simple hardware and now this library layer may return an error, forcing you to deal with it.

And let's face it, dealing with errors is extremely difficult and time-consuming, people who claim otherwise just don't know what it means and they just don't deal with errors. In embedded control application, printing an error message and quitting isn't an option. Reboot often isn't an option, even if it takes a millisecond. Restarting control logic to a known state in some guaranteed time while guaranteeing important control features do not stop operating is possibly a month of full time job per error source, or sometimes just simply impossible.

Those shit library writers have no idea about writing robust microcontroller applications.

In desktop computing, we have no choice but to have fairly complex drivers with fairly abstract interfaces because of the wide variety of hardware that must be able to used using common language. This greatly limits what the "language" consists of. So it's something super crude like write() and read(), with gazillion possible sources of errors and unexpected timing.

In microcontroller world, you can write, in 10 lines of code, your own read_cpu_temperature() that returns what you want, directly, and cannot fail.

[Siwastaja]

write() may be short simply because some internal kernel buffer can't take more input at that time. As of what size the buffers are, who knows. Can you write a gigabyte to /dev/ttyUSB0? No. Can you assume you can write 100 bytes? Why?

I have personally witnessed the need to try writing in a loop with usb serial devices, with Raspberry Pi's internal /dev/serial0, and with TCP sockets.

In that case, flooding write() in a loop hogs up all CPU. Adding some sleep() is the simplest demonstrable way to solve the issue (and works quite well in a separate thread), but using select() is arguably better.

... and when you select() it, your write() may be still short. So then you need to remember how much you were able to write, update your pointer correctly, and wait for the next invocation.

[thinkfat]

Related comment:

What I really love in embedded microcontroller environment is that by having access to bare metal (and usually at least somewhat decent documentation of it), a lot of hardware accesses simply can't fail. Can't fail meaning, they do not have error conditions. Sure, the sky can fall rendering the device inoperable; cosmic rays can flip a bit, and so on, but a GPIO write operation cannot return any code at all. Having 100% success guaranteed with predictable timing makes writing the application so much easier, because you don't need to deal with error recovery logic for every trivial thing.

This is also why I loathe stupid libraries that add an extra layer on top of such simple hardware and now this library layer may return an error, forcing you to deal with it.

And let's face it, dealing with errors is extremely difficult and time-consuming, people who claim otherwise just don't know what it means and they just don't deal with errors. In embedded control application, printing an error message and quitting isn't an option. Reboot often isn't an option, even if it takes a millisecond. Restarting control logic to a known state in some guaranteed time while guaranteeing important control features do not stop operating is possibly a month of full time job per error source, or sometimes just simply impossible.

Those shit library writers have no idea about writing robust microcontroller applications.

In desktop computing, we have no choice but to have fairly complex drivers with fairly abstract interfaces because of the wide variety of hardware that must be able to used using common language. This greatly limits what the "language" consists of. So it's something super crude like write() and read(), with gazillion possible sources of errors and unexpected timing.

In microcontroller world, you can write, in 10 lines of code, your own read_cpu_temperature() that returns what you want, directly, and cannot fail.

While that's all true for trivial peripherals like a GPIO, already with I2C or SPI you're dealing with external hardware that may behave erratically, and you'll have to deal with it _somehow_, I2C in particular is a quite complex affair with a lot of potential to upset your assumptions. Like, the infamous "clock stretching" that allows for an indefinitely blocked bus if a device falls on its face.

[Siwastaja]

While that's all true for trivial peripherals like a GPIO, already with I2C or SPI you're dealing with external hardware that may behave erratically, and you'll have to deal with it _somehow_, I2C in particular is a quite complex affair with a lot of potential to upset your assumptions. Like, the infamous "clock stretching" that allows for an indefinitely blocked bus if a device falls on its face.

You are right, but the problem gets much worse with I2C or SPI, because used in a typical way, there is just 1-2 well enumerated error sources - for example, receiving NACK - which can be handled in your own code, but using a library then generates 10 more error conditions, and there is no way to know in advance which of the errors will not happen, except to fully study the source code of the library and the documentation of the peripheral; in which case you could have just written bare metal in 1/10th of the time and get exactly what you need.

I2C in itself is a completely unreliable bus, and not only the bus, many I2C slave devices are broken-by-design, so tend to require fairly complex recovery models that also depend on the slave devices, some allow specific transition patterns to reset their FSM's, with others you just need to remember to add the PFET on Vcc. I remember one trivial I2C module ending up almost 1000 lines of code and a month of full-time development just to make it recover from problems of I2C slave FSMs.

But if we had to access said I2C devices through write() and read(), we simply couldn't use them at all. Also, the 1000-line hell I mentioned would have never finished using ST's HAL library. It's simply impossible.

I underline this difference because such way of development sadly isn't possible in the world of general-purpose computers running general-purpose operating systems. In embedded MCU, minimum possible solution is actually doable, and while sometimes really difficult (the 1000-LoC I2C), it's often very simple yet powerful.

[ejeffrey]

write() may be short simply because some internal kernel buffer can't take more input at that time. As of what size the buffers are, who knows. Can you write a gigabyte to /dev/ttyUSB0? No. Can you assume you can write 100 bytes? Why?

No it won't.  File descriptor writes (without O_NONBLOCK) are not limited by the size of kernel buffers.  You can absolutely write a gigabyte to /dev/ttyUSB0 and the call should just block until all the data is written, the device disappears, or an asynchronous signal handler is invoked.

I have personally witnessed the need to try writing in a loop with usb serial devices, with Raspberry Pi's internal /dev/serial0, and with TCP sockets.

In that case, flooding write() in a loop hogs up all CPU. Adding some sleep() is the simplest demonstrable way to solve the issue (and works quite well in a separate thread), but using select() is arguably better.

You are talking about non-blocking sockets.  And in that case, select() (or poll() or epoll()) is not arguably better, it is unarguably the correct way to do it.  If you are going to spin to emulate a blocking write you shouldn't be using non-blocking IO.  If you have to loop like that on blocking sockets and you don't have asynchronous signals (maybe whatever library you used on the Pi installed signal handlers for some reason?) then something is wrong.  If you are doing a blocking write but have the possibility of async signal handlers being invoked, then retrying is correct, but you shouldn't need to insert a sleep, you can retry immediately.

... and when you select() it, your write() may be still short. So then you need to remember how much you were able to write, update your pointer correctly, and wait for the next invocation.

Yes, this is true for non-blocking IO.

[Siwastaja]

Yes, I'm talking about non-blocking sockets or file descriptors opened with O_NONBLOCK, like you would do for /dev/ttyUSB0 for example. I don't know where you did get the idea to limit the discussion to blocking file descriptors only.

Indeed, when I just fopen() a file, I assume write() succeeds with full byte count, unless there is a real error (that doesn't go away by just retrying write with the remaining bytes). In case of short count, I print an error message.

But select() does not have a mechanism to return when some amount of guaranteed buffer space exists. So with select(), as soon as one byte can be written, select() returns and you'll write() to the socket. At that time, it's very well possible you can only write part of your data, so you need to keep track how much you wrote so you know what you'll write next. This is what I have witnessed over and over again so it would be a really unsafe assumption to think you can write any arbitrary amount after select() or poll(), unless the arbitrary amount is 1 byte I guess.

I would like to have a select() which I can tell: return when I can write 42 bytes. This would also save on CPU resources because you could write a larger chunk in one go. Often, it works like this because OS writers have thought about performance, but this is far from guaranteed. The problem also is that your multi-part write() program logic may leave untested because during testing, write() is never short.

The point is, write() and read() seldom result in simple 5-line-of-code solutions. They need logic around them, and all this logic should be rigorously tested, which sadly often doesn't happen in real-world software development.

[Nominal Animal]

(I had to think for quite a while before responding, because this might be seen as nitpicking, but I'll try to express the point clearly.  It is a nuance, but an important one, because it gives a developer the basis to make an informed decision on what they consider reasonable in a given situation.)

write() may be short simply because some internal kernel buffer can't take more input at that time. As of what size the buffers are, who knows. Can you write a gigabyte to /dev/ttyUSB0? No. Can you assume you can write 100 bytes? Why?

No it won't.  File descriptor writes (without O_NONBLOCK) are not limited by the size of kernel buffers.  You can absolutely write a gigabyte to /dev/ttyUSB0 and the call should just block until all the data is written, the device disappears, or an asynchronous signal handler is invoked.

There are a few exceptions, but they are not about kernel buffers per se.

The kernel limits single read and write syscalls to a bit less than two gigabytes (2³¹ bytes).  The reason is complicated, but boils down to minimizing bugs due to unstated assumptions in old filesystem drivers.  (That is, it is difficult to ensure old filesystem drivers work accurately with over 32-bit requests in all cases.)

A file descriptor refers to a file description (an entry in the open file table), which in Linux has an associated set of handlers; a struct file_operations for a file-like object.  Essentially, the write syscall implementation in the kernel (in fs/read_write.c) calls ksys_write() which calls vfs_write(), which maintains the file position associated with the file description, limits the write size to MAX_RW_COUNT (defined in include/linux/fs.h in the Linux kernel, currently exactly one page less than 2 GiB), verifies the target memory buffer (to write data from) is valid, and then calls the filesystem/device specific ->write() handler.  Note that it is up to the filesystem/device specific read/write handlers to honor the O_NONBLOCK flag, too!

Now, because the Linux kernel dictators are pretty big on the principle of least surprise, filesystem drivers should fulfill the entire write (and read).  But, the kernel does not verify they do.  So, the expectation is that the drivers do fulfill the entire request, but there is no enforcement or guarantees.  If a filesystem/device does return a short count when it technically could service the entire operation, the maintainers will accept a clean patch to fix that.  However, do we know that there cannot be any reasonable reasons for a device or filesystem to return a short count?

(I've been considering implementing for USB Serial protocol with pipe/socket semantics, bypassing the TTY layer.  The O_NONBLOCK semantics are clear and easy; it's the blocking syscall semantics that give me a pause: most userspace programmers use the simplest, least amount of code to get things done, and reality rules.  I'd like to return short counts if the device NAKs an USB packet, but will that throw off the simple userspace code?)

So, it boils down to:

Should short reads/writes happen with blocking reads and writes?
No, they shouldn't.

Can short reads/writes happen with blocking reads and writes?
Yes, they can; there is no code in the Linux kernel or the C library that verifies they do not happen; the code that implements them should not return short counts for blocking reads and writes, but technically, they could occur, depending on the kernel driver implementing the object you are reading from/writing to.

[ejeffrey]

Yes, I'm talking about non-blocking sockets or file descriptors opened with O_NONBLOCK, like you would do for /dev/ttyUSB0 for example. I don't know where you did get the idea to limit the discussion to blocking file descriptors only.

Because I considered it implicit in the original question.  The question of whether you have to watch out for short writes on non-blocking file descriptors is kind of trivial.  File descriptors that are generally blocking is the only interesting case.

[DiTBho]

I have recently developed a sATA and SCSI "ram-disk" tester in C and used direct-IO. It needs particular care. No doubt about it 

[gf]

Who uses read(), write(), open() & close()  functions in C?

Do you:

a) check for EINTR and retry?

b) check for things like short writes, and call write() again?

I do both of these, but some people don't believe it is needed, and some wave it away saying they use and trust SA_RESTART)...

unistd.h, open(), close(), read(), write() are not part of the C standard library (https://en.cppreference.com/w/c/header)

For POSIX systems, the semantics are specified in the POSIX standard, e.g.
https://pubs.opengroup.org/onlinepubs/009696699/functions/read.html
https://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html
https://pubs.opengroup.org/onlinepubs/009696699/functions/recv.html (for sockets)
...etc...

I think most issues discussed in this thread are answered by the standard.

For portability between systems, I would try to avoid relying on implementation-defined semantics of a particular OS, which are not guaranteed by the POSIX standard.
(For instance, POSIX does not guarantee that read/write on regular files are uninterruptible, although this is an "old Unix tradition" for local on-disk file systems. And indeed, on network filsystems they can be interruptible, too, e.g. NFS mounted with 'intr' option.)

Things are different when talking to proprietary devices/drivers with specific (non-POSIX) behavior. Then the individual driver semantics need to be considered.

[hamster_nz]

I have recently developed a sATA and SCSI "ram-disk" tester in C and used direct-IO. It needs particular care. No doubt about it 

The big test for this for me was a multithreaded/multiprocess deeply-buffered data mover, to move TBs of data over parallel TCPIP sockets over medium latency, high bandwidth links between Data Centers. Our comms providers were not happy when we could saturate their backbones 

It had lots of signals, used large shared memory segments as buffers, lots of parallelism, lots of throughput.

The one thing surprising thing I did learn is that file pointers (that point to where data will be read and written) are by default shared between parent and child processes when you fork(). Found that out pretty quick!

[thinkfat]

The one thing surprising thing I did learn is that file pointers (that point to where data will be read and written) are by default shared between parent and child processes when you fork(). Found that out pretty quick!

Oh yes, I learned that a while ago, too. Fun fact: those file descriptors (not pointers) are also inherited across an "exec", unless you explicitly open them CLOEXEC. Mighty convenient for inter-process communication, or a gaping security hole waiting to be exploited.

[Siwastaja]

Because I considered it implicit in the original question.  The question of whether you have to watch out for short writes on non-blocking file descriptors is kind of trivial.

Oh! You are expecting people to know what they are doing.

Yet, I have seen code where people expect that a non-blocking write() succeeds with full count without checking. Try opening a fresh TCP socket and write() some hundred bytes. Almost 100% success rate nearly guaranteed. Heck, I have done it myself!

And it may work surprisingly well until...