The current suckiness of terminal technology

Text-mode interfaces rock. It's too bad that the technology surrounding them is so crufty and antiquated.

The vast, vast majority of terminals currently in use are software terminals: xterm, eterm, Terminal.app, putty, etc. And yet, terminal technology is centered around emulating and supporting the interfaces of hardware terminals long since deceased: vt100, vt102, etc.

Say it's 2005 and you are implementing a terminal emulator from scratch. You want to be conscientious and pick a standard terminal type that you can implement robustly. You're in for a rude awakening: the truth is that pretty much every software terminal emulation program in existence is a mish-mash of whatever escape sequences the authors felt like implementing.

For example, Eterm claims to be a vt102 emulator. But it certainly doesn't implement:

Invoke Confidence Test (DECTST)

ESC [ 2 ; 1 y

Power-up test. Terminal resets and performs power-up test.

And why should they? Half the escape sequences for these historical terminals make no sense for software-based terminals. And yet, you're never sure what escape sequences client applications will use.

And for all these obselete escape sequences that no one cares about, we still don't have an escape sequence that will change the text color to an arbitrary RGB value. We're still in 8-color world (not even 8-bit, 8 color), even though 99% of "terminals" out there today are capable of doing better.

Because there is no reasonable standard terminal type for software terminal emulators to support, authors of this software just do whatever works. Many programs put the name of the program in $TERM (for example, xterm and screen), which basically demands that everyone else add that program to their terminfo/termcap. Or they'll reuse existing bastardized term types like xterm-color (Terminal.app does this), even though they almost certainly don't support all the same escape sequences.

The story is just as bad on the client side. Your only hope for writing an interoperable text-based application is to use either terminfo or termcap, which are huge databases of historical terminals that no one owns but that the UNIX community won't let go of. Instead of hard-coding escape sequences in your program, you look up the escape sequences for abstract capabilities (like "move the cursor") and use those. That will work, as long as the user actually set his/her TERM variable correctly, and their terminal type is actually in your termcap/terminfo file.

In practice, most programs give up on that idea and use a library like ncurses to do that work for them. You might hope that ncurses would make your life as a terminal application author simple, but you would be wrong about that. Ncurses has 257 functions in it, most of them with names like mvaddchnstr().

Ncurses isn't horrible, but it's not particularly nice to use, and it doesn't expose the full capabilities of the underlying terminal in some cases. For example, many software terminals support an extension where you can set the foreground or background color to -1 which means "the default color" (ie. the color you see before a program mucks with the colors). This effectively allows 81 color combinations (8 colors plus the "default", so 9 foreground times 9 background colors). But ncurses only supports 64 color pairs.

Then there's the whole mess of pseudo-terminals. If you are like me, you might wonder at first why pseudo-terminals are necessary. Why not just fork a shell and communicate with it over stdin/stdout? Well, the bad news there is that UNIX special-cases terminals. They're not just full-duplex byte streams, they also have to support some special system calls for things like setting the baud rate. And you might think that a change of window size would be delivered to the client program as an escape sequence. But no, it's delivered as a signal (and incidentally, sent from the terminal emulator as an ioctl()).

Of course, you can't send ioctls over a network, so sending a resize from a telnet client to a telnet server is done in a totally different way.

As you can see, it's a big mess. This is unfortunate, because text-mode terminal interfaces can offer a lot, especially to us computer programmers.

Dreaming of Solutions

The problem is pretty clear. Solutions are a bit harder to come by. Here are some ideas.

1. Authors of software terminal emulation programs need a reasonable standard they can target. It should be a very small set of escape sequences -- just the basics of movement and setting character attributes. Realistically, for such a thing to gain any traction at all, it would need to be a subset of either vt102 or more likely xterm. Hopefully, such a thing could be integrated into ESR's canonical termcap/terminfo databases, so that existing clients could communicate with them without sending escape sequences that they can't understand.
2. Authors of text-mode applications could use a library that is a lot slicker than Ncurses. Perhaps it could be built on ncurses for now, with the option to leave all that cruft behind later.
3. There needs to be a standard way of making extensions to the existing set of escape sequences. For example, there should be an escape sequence extension that supports specifying an arbitrary color in RGB. Clients should be able to ask terminals what extensions they support, instead of having a static TERM type to decide what is supported. Terminals should know how to ignore escape sequences they don't implement.
4. Everything should be migrated to use XML.

OK, I was just kidding about that last one. :)

My Related Pet Project

I'm creating a text-mode environment that seeks to integrate all your core text-mode apps (terminal, mail reader, editor) into one uber-scriptable environment. Think Emacs, but object-oriented, and written in Ruby.

It's not ready for prime-time, but it can already multiplex terminal windows (like screen does) well enough to run VIM. It's called silkscreen