The answer is of course obvious, but in two parts. First, there is the "zeroth law" of usability: know your target audience. vi is indeed completely unusable to non-geeks[1], but the target audience are people capable and willing to learn its interface. Second, a significant aspect of usability is the efficiency of the interface: vi is a blindingly, hugely efficient way of editing data. This is why we all love it so much.
Most people who accuse UI design advocates of wanting to "dumb down" interfaces start off biased from one of these two points. They either confuse efficiency with usability (one is part of the other, and not necessarily the most important part), or they assume that they are the intended audience. And cue massive flamage.
[1] sorry for the term, I know of none better
