Interesting. djb describes that problem:
The problem with O_NONBLOCK is that it's too heavy: it hits an entire ofile, not just an fd. For example, suppose I set O_NONBLOCK on a pty on stdout, and another program has the same stdout. That other program then has O_NONBLOCK set too. Many programs can't handle O_NONBLOCK and will break horribly in this situation.
This is the first time I've actually seen it come up.