The first part of the “What sysadmin things should every programmer know” series bring us to everyone’s favourite topic … as long as they’re not the one on the wrong end of creating it.
<cue ominous music>
Documentation is a funny thing. There’s a real knife edge of usefulness where just a hair to each side of “complete”, “accurate” or “up to date” sits “there’s more info in redacted documents“, “I swear this is written in Swahili” and “does anyone know where the backup tapes are?”
Document everything. If you don’t have one, install an under-the-radar wiki, but make sure you back it up. Start off with collecting facts, and one day, a big picture will form.
Assuming you have no documentation1, you have two problems to immediately solve; what to document and where to store all this newly collected stuff.
“Document everything” (including life and the universe)2 should be the goal for your environment. It’s completely unattainable, impossible and, frankly, insane to even try, yet striving to achieve it pays off in unexpected ways with surprisingly small amounts of effort.
On your side of retaining sanity is this theory’s major flaw; most environments are too big, complex, intricate or old to document from scratch.
So, start simple … collect facts.
I like facts. Facts are easy. They’re small. They don’t take too much effort to find. They’re easy to copy and paste somewhere else.
- Today might be your the descriptions you’ve put on your switch ports (you do keep that updated, right?)
- Tomorrow might be how to flash the BIOS of a server or update the firmware of a RAID card.
- The day after, your backup tape cycle.
- Next week, your network cabling colour code.
You’ve already got all this stuff swimming around in your head … which is your biggest problem.3 If it’s in your head and not anywhere else, it’s unlikely to be in anyone else’s. This means someone either has to work it out each time or they’re asking you before patching a new cable … and I’m sure you’ve got better things to be doing (that new coffee machine’s not going to test itself).
Consider getting this administrivia stuff out as your first priority; something’s better than nothing, and you can always come back and edit for completeness as you’re going along. Even better – it’s a wiki; anyone can help by hitting the “edit” button and adding their own facts or expanding on yours.
My initial promise was, “… one day a big picture will form”. It takes a long time to prime a wiki with enough facts to answer most any question; I prefer to call it a win when you can answer your first question from your new wiki alone, without having to reverse engineer how a system was set up.
After you answer you first question, keep up the documentation and fact collection. It can only get better from there.
If you’re after some recommendations of software, MediaWiki is one of the bigger guys in the game being the basis of Wikipedia. I’ve heard Confluence is nice to use and has some great integration features. Most of my experience has been with Twiki – it has a really simple markup syntax and works well enough, except the search tends towards crud in larger wikis.4 Some well maintained index pages would probably be a decent enough hackaround for this.
Comments are welcome. How do you document your world? What software do you use?
- Documentation quantified by “some” or “incomplete” counts as none in this instance [↩]
- … with apologies to Douglas Adams [↩]
- show int description, BIOS via tftp from service processor’s CLI, use CDROM and video console redirection for RAID card, daily, weekly for 3 months, monthly for 12 months, yearly for 7 years, red: straight through ethernet, yellow: crossover, blue: serial, purple: voice, green: E1, orange fibre: multimode, yellow fibre: single mode — and all that off the top of my head. <sigh> [↩]
- … at least in the versions I’ve used [↩]