Outsourcing administration?
I read an excellent essay
recently which discussed the fallacy that outsourcing programmers was akin
to outsourcing manufacturing capabilities; he convincingly makes the
argument that programming, unlike manufacturing, is an iterative design
process. Just like you'd never see a major automotive manufacturer
outsource it's design team, because design is their core competancy,
outsourcing your programming talent (assuming software is your business)
would similarly cripple your competitive advantage. His discussion got me
thinking about my own chosen profession (systems administration, in case
anyone reading this in the past might have mistaken me for one of those
hippy geek programmers 
). Does his argument apply to systems
administration and management, or do we have a job function that could be
done from an operations manual?
Systems administration consists of a number of well-defined roles, combined with a lot of day-to-day vagueness. The well-defined stuff is obvious:
- Keep the hardware running, and schedule fix/replace work as neccesary.
- Monitor operating system and hardware resource usage, and notify application owners/schedule upgrades as necessary.
- Schedule operating system (and application, if that's in your job description too) patching and regular maintenance.
- Keep up-to-date with the current "state of the art" in systems administration "best practices" and software/hardware revisions.
I could go on a bit longer, but you get the idea. Keep things running, schedule stuff that helps keep it running, and fix stuff when it breaks. Simple, right? So outsourcing sounds, at this point, like it's a good idea. Bring it on: remote facilities and eyes-and-hands services cover 90% of that, and consulting resources can probably manage the remaining work, right?
This is where I bring up that second part of the job: the day-to-day vagueness. It includes things like:
- Help a windows developer become acclimated to programming on UNIX (or, alternatively, commisserate with the poor UNIX programmer forced to write in VBScript all day long). Explain select() to someone whose most complicated previous program has been an interface mockup in MS Access.
- Help project management understand why the great idea they had for streamlining their project plan will actually extend the deadline, because it doesn't take into account an implementation detail that they didn't know about, such as growing SAN capacity or adding switches.
- Serve as a liason between the technology staff and management. Explain that their carefully crafted budget from last year was exceeded by 150%, not because of "out of control geeks", but because management didn't actually ask the geeks how much their projects would cost.
- Write code, because automation gives you more time to write blog entries about how you're automating yourself out of a job.
- Convince a vendor that, yes Virginia, there really is a bug in your product. Provide truss/strace/etc. output as necessary. Step through it with a debugger if you have to, or supply the line number of the "encrypted" perl code they supplied you with that contains the bug.
- Figure out why a particular vendor-supplied operating system patch didn't apply to one machine out of fifty, in fifteen minutes, because that's how long your maintenance window is. Roll back all fifty machines if you can't figure it out.
- Sit in every meeting for a particular project, not because you have anything to contribute to any of them, but because of that one meeting where someone will suggest something utterly impossible, from a technical perspective, and you need to be there to save yourself and your team from the job of making it happen.
I could go on forever with this part. The primary role of a System Administrator, in my mind, is not to do the day-to-day technical work; management is right, all of that can be outsourced. What they can't outsource is the role of someone you can turn to and ask "does this make sense, technically?" Just like automotive design will never be outsourced, systems and network design can't be outsourced without terrible consequences. Envision a case of a large company who has outsourced all of their administrative talent. Who is making the design decisions for the network? Do you outsource that, or does management take on the role? If you outsource it, who represents the designers in new deployments? Who do you turn to when you have questions about how one system interacts with another? How do you fill in gaps in your staff's technical knowledge without technical leaders to turn to?
Make no mistake, though: this argument demands that systems administrators grow into technical leaders. We need to be architects, designers, mentors, programmers, project managers, janitors, and free-thinkers; we need to have a breadth of expertise that a specialist simply cannot bring to the table. Someone who wants to take a class and be "certified" as a "Linux Guru" doesn't have much of a future in this business (or at least, not at this pay grade) because that role can be handed off to anyone who can read an operations manual. Want to keep your fat paycheck? Recognize the areas that can't be outsourced, and excel at them. If there's a "Learn X in 24 Hours" book, it's a pretty good bet that X isn't what you should be exclusively concentrating on.
I started writing this not sure where I'd end up at the end; I
half-expected to end up making the argument that I'm not necessary, but
that would only be half-right. Only those of us who can't provide technical
leadership and handle the "undefinables" of the job are unnecessary. I, for
one, welcome the thinning of the herd. 
Postscript: I wrote this in a coffee shop waiting for a job interview today. I'm writing this addendum on the train ride home afterward, after finding out that I was actually being interviewed for two positions: a "pure" system administration role, where everything is clearly operationally defined and the team has clear goals and deadlines to work in, or a position which was defined as "we do the stuff that falls through the cracks" by one of the senior administrators on the team. I don't think the recruiter had any idea why I had such a huge grin on my face when she told me that, and seemed surprised when I didn't have any difficulty saying which position I felt more attracted to.