15 Feb 2001

Thu, 15 Feb 2001 08:06:58 GMT

In his article of Feb 12, responding to the same article as I have just done, Ryan Muldoon seems to share the notion that we need to be a little more revolutionary in our approach to interface design.

He mentioned Nautilus, a program I've been testing rather extensively. I'm also very excited about the prospect of a GUI made of many useful components accessible from a single graphical shell. It helps "get the interface out of the way" since users don't have to remember what application they need to use to accomplish a given task. And of course, as they say, sometimes the best interface is no interface at all -- which is where autonomous agents enter the picture.

Ryan also mentioned file systems and metadata. For some reason, I have this persistent vision of a filesystem built on a relational database, which perhaps works in concert with a search engine and a traditional file hierarchy (which could be integrated with the database to provide unambiguous but human-readable descriptors). You might want to ask your computer to get "that slideshow from last week for work" or to show "that e-mail from Susan yesterday about the usability testing." Interestingly, Evolution is starting to integrate some of this functionality, for e-mail at least, with the "Virtual Folders" feature. Alas, there will not likely be an NL front-end any time soon.

15 Feb 2001

Thu, 15 Feb 2001 07:46:40 GMT

Regarding the article posted recently on next-generation GUI design...

I've been involved in MIT's small contribution of Gnome development, mostly as a tester and commentator. When I talk to people outside the project (and sometimes inside, as well) I often hear comments like "ah, so it's just like Windows, then?"

On a fundamental level, this is an apt description. Though there is a lot of technical innovation and tremendous leapfrogging of Microsoft's technology underway now, that development is what you could roughly call "incremental." The user interface is fundamentally limited to a keyboard and a pointing device.

PDA interfaces seem like a radical departure from the traditional PC desktop interface. But really, it's just a degenerate case that requires an extensive reworking of the same paradigm. Shift emphasis to the pointer, simplify and modularize interface components so they'll fit on a small device. It doesn't really change the fundamental mode of interaction with the technology - point, tab, nudge, poke.

I'm really happy to see this incremental improvement happening, and it really does take a lot of engineering skill and effort; and it really makes a big difference at the end of the day.

We need something radically different for there to be a real "revolution" in GUI design. Or perhaps it's more of a revolution out of GUIs, and the original author of the article is really speaking of a mini-revolution that might be useful in the existing paradigm.

It seems to be quite obvious what the next step up from GUIs should be - voice and natural language interfaces. Humans have a built-in capacity for extremely high-bandwidth, highly expressive communication using natural language, especially speech. Natural language interfaces have the potential to be much more user-friendly, fluid, and efficient than conventional GUIs. Alas, it's much, much harder to process English commands properly than it is to respond to pokes and prods and taps of labelled buttons.

Alas, most applications demand higher performance from NLP (Natural Language Processing) than is attainable with current production systems. But the technology is improving every day, helped along by advances in hardware performance, by techniques in managing complexity, and by sheer brute force production of a working knowledge of the problem.

This process would, ironically, be helped along a great deal if we had better user interfaces with which to design our NLP systems. Along those lines, I have two particular projects in mind. I'm currently studying NLP at MIT, but I don't have plans for grad school. Maybe I'll end up in a position to actually do serious research in this area, but it's more likely it will be a hobby. Maybe I'm completely in left field and this will take centuries of slow progress to realize, maybe some company will come out with products to do this in the next 20 years, or maybe I'll invent something nifty in my copious free time. In any case, even the simplest of these systems can do fun and amazing things.

1.) Implementing an NL command and control interface.

The sci-fi vision of a computer that responds to natural-language requests - "Computer, what is the time?" or "Computer, play some Bach." - is extremely captivating and represents an obvious paradigm under which to create an extreme intuitive interface.

2.) Implementing an NL programming interface.

It is tempting to try to equate machine languages with natural languages. (When asked what languages I speak, I often tell people I speak Perl fluently.) Of course, they are very different -- but both are extremely powerful. It is certainly possible, for instance to describe, in English, what markup and style I want to use in my HTML document, in addition to actually dictating the content itself. Using a sort of batch processing approach (and the solution to problem #1) one could in a sense translate the description into the desired document. (Of course, a more efficient and user-friendly approach might be to convey the desired markup through an interactive dialog, which adds another layer of complexity.)

But what about arbitrary code in a Turing-complete language? Why can't I write a description of exactly what I want my program to do, and have that "translated" from English into Perl? Assuming this requires a semantic representation of the desired functionality, would this not also make it possible to generate a performance-optimized C version, say? Would this not speed up the software engineering process quite considerably? Would it reduce debugging to a process of refining the program's specification, perhaps in an interactive dialog? A prerequisite to answering those questions would be figuring out at what level of abstraction the English specification would need to be. Can the user merely describe the desired mappings from input to output, or perhaps the screens and buttons the users sees, and some notion of how they interconnect and what functionality they represent. Or would the programmer have to speak in pseudocode, at the level of functions and variables? How do knowledge representation and reasoning interplay with NLP in this application?

Advogato blog for beland

15 Feb 2001

15 Feb 2001