Skip to content

What sysadmin things should every programmer know – Documentation – Wiki

The first part of the “What sysadmin things should every programmer know” series bring us to everyone’s favourite topic … as long as they’re not the one on the wrong end of creating it.


<cue ominous music>

Documentation is a funny thing.  There’s a real knife edge of usefulness where just a hair to each side of “complete”, “accurate” or “up to date” sits “there’s more info in redacted documents“, “I swear this is written in Swahili” and “does anyone know where the backup tapes are?”

That said, there are some things that can be done to improve a bad documentation situation.  My documentation-related reply to the original question came in five parts, the first being:

Document everything. If you don’t have one, install an under-the-radar wiki, but make sure you back it up. Start off with collecting facts, and one day, a big picture will form.

Assuming you have no documentation1,  you have two problems to immediately solve; what to document and where to store all this newly collected stuff.

“Document everything” (including life and the universe)2 should be the goal for your environment.  It’s completely unattainable, impossible and, frankly, insane to even try, yet striving to achieve it pays off in unexpected ways with surprisingly small amounts of effort.

On your side of retaining sanity is this theory’s major flaw; most environments are too big, complex, intricate or old to document from scratch.

So, start simple … collect facts.

I like facts.  Facts are easy.  They’re small.  They don’t take too much effort to find.  They’re easy to copy and paste somewhere else.

  • Today might be your the descriptions you’ve put on your switch ports (you do keep that updated, right?)
  • Tomorrow might be how to flash the BIOS of a server or update the firmware of a RAID card.
  • The day after, your backup tape cycle.
  • Next week, your network cabling colour code.

See?  Simple.

You’ve already got all this stuff swimming around in your head … which is your biggest problem.3  If it’s in your head and not anywhere else, it’s unlikely to be in anyone else’s.  This means someone either has to work it out each time or they’re asking you before patching a new cable … and I’m sure you’ve got better things to be doing (that new coffee machine’s not going to test itself).

Consider getting this administrivia stuff out as your first priority; something’s better than nothing, and you can always come back and edit for completeness as you’re going along.  Even better – it’s a wiki; anyone can help by hitting the “edit” button and adding their own facts or expanding on yours.

My initial promise was, “… one day a big picture will form”.  It takes a long time to prime a wiki with enough facts to answer most any question; I prefer to call it a win when you can answer your first question from your new wiki alone, without having to reverse engineer how a system was set up.

After you answer you first question, keep up the documentation and fact collection.  It  can only get better from there.

If you’re after some recommendations of software, MediaWiki is one of the bigger guys in the game being the basis of Wikipedia.  I’ve heard Confluence is nice to use and has some great integration features.  Most of my experience has been with Twiki – it has a really simple markup syntax and works well enough, except the search tends towards crud in larger wikis.4  Some well maintained index pages would probably be a decent enough hackaround for this.

Comments are welcome.  How do you document your world?  What software do you use?

  1. Documentation quantified by “some” or “incomplete” counts as none in this instance []
  2. … with apologies to Douglas Adams []
  3. show int description, BIOS via tftp from service processor’s CLI, use CDROM and video console redirection for RAID card, daily, weekly for 3 months, monthly for 12 months, yearly for 7 years, red: straight through ethernet, yellow: crossover, blue: serial, purple: voice, green: E1, orange fibre: multimode, yellow fibre: single mode — and all that off the top of my head.  <sigh> []
  4. … at least in the versions I’ve used []

Posted in Tech.

What sysadmin things should every programmer know?

I recently joined ServerFault; a new forum for system administrations, built by the same guys that developed Stack Overflow for programmers.

One of the first posts I came across asked:

“As a programmer, we tend to take sysadmins for granted. The few times I’ve been without a good sysadmin have really made me appreciate what you guys do. When we’re venturing into an environment without a sysadmin, what words of wisdom can you offer us?”

I love these sorts of questions.  I enjoy sharing what actually works in the real world and come up with the infrastructure to support best practices.

I have come across all sorts of information and advice for both programmers and sysadmins.  In this series, I’ll go over my response to this question and the background of why I believe this stuff.

Posted in Tech.

ST5K – Adelaide Street Art

I love the Fringe Festival.  It’s the one time of year when my normally sleepy home town comes to life.

This year was a little different than usual.  Aside from seeing my usual batch of comedians and (inadvertently) shooting the creation of a promo for a Fringe Festival show, I came across a event that I’d not heard anything about before; the Format Festival.

Format’s an art festival with a twist; there’s no pomp, no ceremony.  It’s a two week long, artist-run exhibition, displaying their own art and DIY work, with a focus on getting involved.  Of all the scheduled events, the Street Art Art Jam night intrigued me.  Being a photographer-friendly event, I rocked up to see what it was all about.

Given I’m completely useless with a paintbrush, pencil, charcoal, or anything else that may reside in an artist’s box of tricks (hey, I know my limits 🙂 ), an art event was a bit of a strange place for me, yet they had built a truly fantastic environment to be in.  There was around thirty artists over the course of the few hours I was there; all extremely friendly, extremely talented and more than happy for me to shoot their work to show what they could do.  The majority of art being made on the night were stencils, stickups and the astounding freehand spray murals.  Some of the artists have made street art their life’s work (“the joys of a misspent youth” was mentioned more than once); others, such as Sarah, creator of the green face mural, started a few short months prior.

The full shoot from the night has been published on the ST5K gallery.

Cheers to Simon Loffler and Matt Walker for getting Format up and running, to all the fantastic artists I met there, and the ones before me that left their art behind.

Bring on Format 2010!

Posted in Photography.

Ninjas attack Adelaide!

Taken at the Ninja Raid, hosted by the Adelaide Flash Mob.

Posted in Photography.

Did you know?

If you’re a netizen, this will interest you.  8 minutes of fascinating facts, figures and trivia about our world, and the new world — the one without borders.

My favourite comment: “We are living in exponential times…”

Posted in The Internets.

Harsh realities

Harsh realities



Nowhere to go.

Nothing to do but wait.

Posted in Photography.

Prison lock

Prison lock... there is no escape

… there is no escape.

Taken during a trip to the Old Adelaide Gaol.  Full of stories, full of history.

I’d never want to end up there.

Posted in Photography.

A Sysadmin’s advent calendar

This is a fantastic concept for infrastructure/sysadmin type people that was not publicised highly enough.

Sysadvent was an article a day for the advent calendar, containing bunch of really useful information in small enough chunks that doesn’t scare you off, yet has enough meat behind it that you want to add stuff to your “The List“.

If you look after systems, you owe it to yourself to take an hour to read Sysadvent; I’ll add my take and some topics close to my world as time progresses.

Hat’s off to Jordan of semicomplete for running the show.

Posted in Tech.

What do you do when someone else 0wnz your site?

I was recently contacted by by a gentleman having what is colloquially referred to as a Bad Day(tm); someone had managed to get into his e-commerce site and started messing around with the content.

In this case, the page content was modified so that a remotely-hosted javascript file was downloaded and executed on each page view.  This javascript could do pretty much anything; capture username/passwords and form data and send them off site, inject virus-laden images into the page to take over or crash a client application/PC or load a bunch of ads hidden off screen to increase their ad impression rate and therefore get paid higher values … and that’s just three off the top of my head.

If you have anything hosted on the ‘net, I’m sure you can appreciate that’s a pretty bad scanario for most personal sites.  On sites that sell things, use credit cards or have usernames and passwords, it’s not just bad, it’s scary.

So, what do you do when someone else 0wnz your site?

First, clean out the closet.

There’s two main methods of attack; the highly skilled, custom attack, and the the “script kiddie” attack.

The first consists of people trying to actively break software by sending it more data than it can handle (known as a “buffer overflow” attack), sending it bad or malformed data or injecting invalid instructions (which could manifest in an application crash or databases being wiped).  As mentioned, these are generally found by highly skilled people that are capable of leaving little or no trace of their actions.  Depending on their motives, the exploit found will either be used to break into things, or be reported to the vendor so we can all benefit from the security fixes.

The second type of attack requires less skill, and usually involves pre-packaged exploitation tools created by the first kind of attack.  These exploit kits are usually easy to spot as you will see a number of invalid requests in rapid succession in your application logs, trying to get a webserver to do something stupid (like trying to run the Windows command shell on a Unix/Apache webserver).

In either case, preservation of the environment is important if you can learn from it later.  Back it up and put it somewhere safe where it can do no damage, such as within a virtual machine or a development server physically disconnected from the rest of the network.

Second, nuke the entire site from orbit. It’s the only way to be sure.

Yup – you don’t really have a choice here.  You need to consider the entirety of the site compromised and nuke it.  Nuke it all.

PHP files, .Net code, html, images, you name it.

You can’t trust any of it.

Third, we can rebuild it.  We have the technology.

Now, this is where it gets interesting.

If you’re using a content management system of some sort (of which, most e-commerce and blogging software fits in nicely), “rebuilding” a site is pretty simple as most of the site’s content and configuration live inside a database.

Rebuilding involves deploying the latest version of all web software, plugins and utilities, freshly downloaded from the vendor or open source project’s site, and using a new, blank database; the catch to this is you will need to migrate over the content by hand.  To speed things up, you could dump the old database, sanitise it’s contents, and reload to the new location or reload a database dump from before the site compromise occurred (assuming you can nail that down when that happened).  Migrating and sanitising the database does require some skill and knowledge of how the application uses the database; if you don’t know, now’s the time to find make friends with a friendly DBA or group that provides support (commercial or otherwise) for your chosen content management system application.

Note that the break in I described at the top ended up with content being injected via the CMS’ template files; redeploying and re-customising a fresh copy of your template may be a good idea if you’re not familiar with looking through them for strange web requests.

I mentioned nuking the images (and other data-type files, like MP3’s and Word docs); again, if you have a backup from before the compromise, you should be able to use that to help get you back up and running quicker.  If not, the images and the like are probably ok to reuse, however I would recommend virus scanning them as there are exploits in the wild capable of infecting a machine by viewing an image file.  Again, with good backups, you can make it easier.

It’s hard and it takes a lot of work to properly clean up a site.  It is the only safe manner to bring it back up, though.

Fourth, those passwords are no longer yours.  Get yourself some new ones.

Someone else likely knows your password.

They can do things you probably don’t want them doing.

That sucks.

Now you have to think of a bunch of long, good, solid, random passwords start changing any password related to the site as well as anywhere else where you use the same username/password.

Again, it’s the only way to be sure.

Finally, try to make it so it doesn’t happen again.

Work on the least bit principle – if you don’t need a feature, don’t install it.

Work on the least privilege principle – disable permissions until you break the app, then slowly grant more until it works again. 1

Service accounts and sudo are great – service accounts mean you can create application-specific accounts with massive nasty passwords; sudo means you almost never need to know ’em for day-to-day maintenance.

Look into an automated configuration control / file verification system, or hand rolling some form of md5/sha1 digesting of files to look for files that have changed that you don’t know about.

Remember …

… the Interwebs are a scary place.  Do you know where your backups are?

  1. Ideally, you’d start off with no permissions, but that can take time; make you you budget extra time to do it properly.  Remember, most applications don’t need write access to their whole tree; applying write access to their “uploads” directory may be enough. []

Posted in Tech, The Internets.

May all your puddings be flaming…

Flambé puddingTaken during Christmas lunch with family.

This made-from-scratch pudding was certainly the highlight of the meal; the fire made it more awesome 🙂

Posted in Photography.