Outsourcing administration?
I read an excellent essay
recently which discussed the fallacy that outsourcing programmers was akin
to outsourcing manufacturing capabilities; he convincingly makes the
argument that programming, unlike manufacturing, is an iterative design
process. Just like you'd never see a major automotive manufacturer
outsource it's design team, because design is their core competancy,
outsourcing your programming talent (assuming software is your business)
would similarly cripple your competitive advantage. His discussion got me
thinking about my own chosen profession (systems administration, in case
anyone reading this in the past might have mistaken me for one of those
hippy geek programmers 
). Does his argument apply to systems
administration and management, or do we have a job function that could be
done from an operations manual?
Systems administration consists of a number of well-defined roles, combined
with a lot of day-to-day vagueness. The well-defined stuff is obvious:
- Keep the hardware running, and schedule fix/replace work as
neccesary.
- Monitor operating system and hardware resource usage, and notify
application owners/schedule upgrades as necessary.
- Schedule operating system (and application, if that's in your job
description too) patching and regular maintenance.
- Keep up-to-date with the current "state of the art" in systems
administration "best practices" and software/hardware revisions.
I could go on a bit longer, but you get the idea. Keep things running,
schedule stuff that helps keep it running, and fix stuff when it breaks.
Simple, right? So outsourcing sounds, at this point, like it's a good idea.
Bring it on: remote facilities and eyes-and-hands services cover 90% of
that, and consulting resources can probably manage the remaining work,
right?
This is where I bring up that second part of the job: the day-to-day
vagueness. It includes things like:
- Help a windows developer become acclimated to programming on UNIX (or,
alternatively, commisserate with the poor UNIX programmer forced to write
in VBScript all day long). Explain select() to someone whose most
complicated previous program has been an interface mockup in MS
Access.
- Help project management understand why the great idea they had for
streamlining their project plan will actually extend the deadline, because
it doesn't take into account an implementation detail that they didn't know
about, such as growing SAN capacity or adding switches.
- Serve as a liason between the technology staff and management. Explain
that their carefully crafted budget from last year was exceeded by 150%,
not because of "out of control geeks", but because management didn't
actually ask the geeks how much their projects would cost.
- Write code, because automation gives you more time to write blog
entries about how you're automating yourself out of a job.
- Convince a vendor that, yes Virginia, there really is a bug in your
product. Provide truss/strace/etc. output as necessary. Step through it
with a debugger if you have to, or supply the line number of the
"encrypted" perl code they supplied you
with that contains the bug.
- Figure out why a particular vendor-supplied operating system patch
didn't apply to one machine out of fifty, in fifteen minutes, because
that's how long your maintenance window is. Roll back all fifty machines if
you can't figure it out.
- Sit in every meeting for a particular project, not because you have
anything to contribute to any of them, but because of that one meeting
where someone will suggest something utterly impossible, from a technical
perspective, and you need to be there to save yourself and your team from
the job of making it happen.
I could go on forever with this part. The primary role of a System
Administrator, in my mind, is not to do the day-to-day technical work;
management is right, all of that can be outsourced. What they can't
outsource is the role of someone you can turn to and ask "does this make
sense, technically?" Just like automotive design will never be outsourced,
systems and network design can't be outsourced without
terrible consequences. Envision a case of a large company who has
outsourced all of their administrative talent. Who is making the design
decisions for the network? Do you outsource that, or does management take
on the role? If you outsource it, who represents the designers in new
deployments? Who do you turn to when you have questions about how one
system interacts with another? How do you fill in gaps in your staff's
technical knowledge without technical leaders to turn to?
Make no mistake, though: this argument demands that systems administrators
grow into technical leaders. We need to be architects, designers, mentors,
programmers, project managers, janitors, and free-thinkers; we need to have
a breadth of expertise that a specialist simply cannot bring to the table.
Someone who wants to take a class and be "certified" as a "Linux Guru"
doesn't have much of a future in this business (or at least, not at this
pay grade) because that role can be handed off to anyone who can read an
operations manual. Want to keep your fat paycheck? Recognize the areas that
can't be outsourced, and excel at them. If there's a "Learn X in 24 Hours"
book, it's a pretty good bet that X isn't what you should be exclusively
concentrating on.
I started writing this not sure where I'd end up at the end; I
half-expected to end up making the argument that I'm not necessary, but
that would only be half-right. Only those of us who can't provide technical
leadership and handle the "undefinables" of the job are unnecessary. I, for
one, welcome the thinning of the herd. 
Postscript: I wrote this in a coffee
shop waiting for a job interview today. I'm writing this addendum on
the train ride home afterward, after finding out that I was actually being
interviewed for two positions: a "pure" system administration role, where
everything is clearly operationally defined and the team has clear goals
and deadlines to work in, or a position which was defined as "we do the
stuff that falls through the cracks" by one of the senior administrators on
the team. I don't think the recruiter had any idea why I had such a huge
grin on my face when she told me that, and seemed surprised when I didn't
have any difficulty saying which position I felt more attracted to.
Syndicated 2005-11-22 21:15:00 from esm
). Andy's code has the additional benefit of
handling nested event handlers in a very slick manner.
