Documentation.

<cue ominous music>

Documentation is a funny thing.  There’s a real knife edge of usefulness where just a hair to each side of “complete”, “accurate” or “up to date” sits “there’s more info in redacted documents“, “I swear this is written in Swahili” and “does anyone know where the backup tapes are?”

That said, there are some things that can be done to improve a bad documentation situation.  My documentation-related reply to the original question came in five parts, the first being:

Document everything. If you don’t have one, install an under-the-radar wiki, but make sure you back it up. Start off with collecting facts, and one day, a big picture will form.

Assuming you have no documentation¹,  you have two problems to immediately solve; what to document and where to store all this newly collected stuff.

“Document everything” (including life and the universe)² should be the goal for your environment.  It’s completely unattainable, impossible and, frankly, insane to even try, yet striving to achieve it pays off in unexpected ways with surprisingly small amounts of effort.

On your side of retaining sanity is this theory’s major flaw; most environments are too big, complex, intricate or old to document from scratch.

So, start simple … collect facts.

I like facts.  Facts are easy.  They’re small.  They don’t take too much effort to find.  They’re easy to copy and paste somewhere else.

• Today might be your the descriptions you’ve put on your switch ports (you do keep that updated, right?)
• Tomorrow might be how to flash the BIOS of a server or update the firmware of a RAID card.
• The day after, your backup tape cycle.
• Next week, your network cabling colour code.

See?  Simple.

You’ve already got all this stuff swimming around in your head … which is your biggest problem.³  If it’s in your head and not anywhere else, it’s unlikely to be in anyone else’s.  This means someone either has to work it out each time or they’re asking you before patching a new cable … and I’m sure you’ve got better things to be doing (that new coffee machine’s not going to test itself).

Consider getting this administrivia stuff out as your first priority; something’s better than nothing, and you can always come back and edit for completeness as you’re going along.  Even better – it’s a wiki; anyone can help by hitting the “edit” button and adding their own facts or expanding on yours.

My initial promise was, “… one day a big picture will form”.  It takes a long time to prime a wiki with enough facts to answer most any question; I prefer to call it a win when you can answer your first question from your new wiki alone, without having to reverse engineer how a system was set up.

After you answer you first question, keep up the documentation and fact collection.  It  can only get better from there.

If you’re after some recommendations of software, MediaWiki is one of the bigger guys in the game being the basis of Wikipedia.  I’ve heard Confluence is nice to use and has some great integration features.  Most of my experience has been with Twiki – it has a really simple markup syntax and works well enough, except the search tends towards crud in larger wikis.⁴  Some well maintained index pages would probably be a decent enough hackaround for this.

Comments are welcome.  How do you document your world?  What software do you use?

1. Documentation quantified by “some” or “incomplete” counts as none in this instance
2. … with apologies to Douglas Adams
3. show int description, BIOS via tftp from service processor’s CLI, use CDROM and video console redirection for RAID card, daily, weekly for 3 months, monthly for 12 months, yearly for 7 years, red: straight through ethernet, yellow: crossover, blue: serial, purple: voice, green: E1, orange fibre: multimode, yellow fibre: single mode — and all that off the top of my head.  <sigh>
4. … at least in the versions I’ve used

One of the first posts I came across asked:

“As a programmer, we tend to take sysadmins for granted. The few times I’ve been without a good sysadmin have really made me appreciate what you guys do. When we’re venturing into an environment without a sysadmin, what words of wisdom can you offer us?”

I love these sorts of questions.  I enjoy sharing what actually works in the real world and come up with the infrastructure to support best practices.

I have come across all sorts of information and advice for both programmers and sysadmins.  In this series, I’ll go over my response to this question and the background of why I believe this stuff.

Sysadvent was an article a day for the advent calendar, containing bunch of really useful information in small enough chunks that doesn’t scare you off, yet has enough meat behind it that you want to add stuff to your “The List“.

If you look after systems, you owe it to yourself to take an hour to read Sysadvent; I’ll add my take and some topics close to my world as time progresses.

Hat’s off to Jordan of semicomplete for running the show.

In this case, the page content was modified so that a remotely-hosted javascript file was downloaded and executed on each page view.  This javascript could do pretty much anything; capture username/passwords and form data and send them off site, inject virus-laden images into the page to take over or crash a client application/PC or load a bunch of ads hidden off screen to increase their ad impression rate and therefore get paid higher values … and that’s just three off the top of my head.

If you have anything hosted on the ‘net, I’m sure you can appreciate that’s a pretty bad scanario for most personal sites.  On sites that sell things, use credit cards or have usernames and passwords, it’s not just bad, it’s scary.

So, what do you do when someone else 0wnz your site?

First, clean out the closet.

There’s two main methods of attack; the highly skilled, custom attack, and the the “script kiddie” attack.

The first consists of people trying to actively break software by sending it more data than it can handle (known as a “buffer overflow” attack), sending it bad or malformed data or injecting invalid instructions (which could manifest in an application crash or databases being wiped).  As mentioned, these are generally found by highly skilled people that are capable of leaving little or no trace of their actions.  Depending on their motives, the exploit found will either be used to break into things, or be reported to the vendor so we can all benefit from the security fixes.

The second type of attack requires less skill, and usually involves pre-packaged exploitation tools created by the first kind of attack.  These exploit kits are usually easy to spot as you will see a number of invalid requests in rapid succession in your application logs, trying to get a webserver to do something stupid (like trying to run the Windows command shell on a Unix/Apache webserver).

In either case, preservation of the environment is important if you can learn from it later.  Back it up and put it somewhere safe where it can do no damage, such as within a virtual machine or a development server physically disconnected from the rest of the network.

Second, nuke the entire site from orbit. It’s the only way to be sure.

Yup – you don’t really have a choice here.  You need to consider the entirety of the site compromised and nuke it.  Nuke it all.

PHP files, .Net code, html, images, you name it.

You can’t trust any of it.

Third, we can rebuild it.  We have the technology.

Now, this is where it gets interesting.

If you’re using a content management system of some sort (of which, most e-commerce and blogging software fits in nicely), “rebuilding” a site is pretty simple as most of the site’s content and configuration live inside a database.

Rebuilding involves deploying the latest version of all web software, plugins and utilities, freshly downloaded from the vendor or open source project’s site, and using a new, blank database; the catch to this is you will need to migrate over the content by hand.  To speed things up, you could dump the old database, sanitise it’s contents, and reload to the new location or reload a database dump from before the site compromise occurred (assuming you can nail that down when that happened).  Migrating and sanitising the database does require some skill and knowledge of how the application uses the database; if you don’t know, now’s the time to find make friends with a friendly DBA or group that provides support (commercial or otherwise) for your chosen content management system application.

Note that the break in I described at the top ended up with content being injected via the CMS’ template files; redeploying and re-customising a fresh copy of your template may be a good idea if you’re not familiar with looking through them for strange web requests.

I mentioned nuking the images (and other data-type files, like MP3′s and Word docs); again, if you have a backup from before the compromise, you should be able to use that to help get you back up and running quicker.  If not, the images and the like are probably ok to reuse, however I would recommend virus scanning them as there are exploits in the wild capable of infecting a machine by viewing an image file.  Again, with good backups, you can make it easier.

It’s hard and it takes a lot of work to properly clean up a site.  It is the only safe manner to bring it back up, though.

Fourth, those passwords are no longer yours.  Get yourself some new ones.

Someone else likely knows your password.

They can do things you probably don’t want them doing.

That sucks.

Now you have to think of a bunch of long, good, solid, random passwords start changing any password related to the site as well as anywhere else where you use the same username/password.

Again, it’s the only way to be sure.

Finally, try to make it so it doesn’t happen again.

Work on the least bit principle – if you don’t need a feature, don’t install it.

Work on the least privilege principle – disable permissions until you break the app, then slowly grant more until it works again. ¹

Service accounts and sudo are great – service accounts mean you can create application-specific accounts with massive nasty passwords; sudo means you almost never need to know ‘em for day-to-day maintenance.

Look into an automated configuration control / file verification system, or hand rolling some form of md5/sha1 digesting of files to look for files that have changed that you don’t know about.

Remember …

… the Interwebs are a scary place.  Do you know where your backups are?

1. Ideally, you’d start off with no permissions, but that can take time; make you you budget extra time to do it properly.  Remember, most applications don’t need write access to their whole tree; applying write access to their “uploads” directory may be enough.

Potentially losing data in this manner becomes less scary and more of a morbid curiosity …

… especially when you’re aware of exactly what a hard drive head crash is capable of doing.

For those not aware, you’re looking at a storage appliance,² being verbally assaulted.  Each one of the drives consists of spinning platters and heads that “fly” over the platters on a cushion of air produced by the platters.³  Any movement or vibration of the drive can disrupt the cushion of air, causing the heads to either lose their place on the disk or crash into the platters.

In this video, the pressure waves generated by the sound is enough to knock (multiple) drives out of action while they stabilise, find their servo tracking signals and start servicing load again.

This originally came about while engineers were attempting to reproduce an intermittent disk latency problem caused by a missing screw that should have held a disk fast in it’s support tray.  As the problem wasn’t easily reproduced, they introduced some extra vibration to see how close the disks were to exhibiting the same stall-stabilise-resume cycle; it turns out, no more than a scream away

More info on Brendan Gregg’s Fishworks blog at Sun.

This raises a few interesting points:

• The automated diagnostics/monitoring package on the 7400 series appliances, Analytics, is *seriously* awesome if you can use it to (indirectly) detect a loose or missing drive screw by looking for per-disk increases in I/O latency.
• We’re really pushing the boundaries of mechanical stability in disks.  I believe these are 7200RPM SATA disks; if a loud noise can freak ‘em out, I’d hate to think what a similar event would do to 15K FC drives.
• SSD’s are immune to these problems; once the write performance problems have been sorted or worked around (async flushing or similar – dont forget a UPS!), they’re going to become *really* interesting for non-caching/acceleration use.
• The monitoring / diagnostics software is effectively a web interface to DTrace, a non-invasive tracing framework used to gather kernel and application information on production systems safely, without (significantly) impacting performance.  DTrace is a fascinating technology that I really need to learn more about.⁴

… and now for the morbid curiosity.  I have a couple of 12″ subwoofers and a large collection of dance, trance and drum’n'bass music; I wonder what they could do to a JBOD array …

My hat’s off to the Fishworks guys for building an awesome storage platform; I look forward to seeing more of your stuff (and hopefully playing with your kit) soon!

<disclaimer>  No drives were (permanently) hurt in the production of this blog post, although my external drive started whining in sympathy.  As it says in the video, don’t do this.  Especially to my SAN disks.  Ever.  I have a cluebat.  It will be used.  Oh yes.  It will be.  </disclaimer>

[via: Make]

** Update! **

Bryan Cantrill, one the Sun Fishworks guys, found me – thanks for stopping by!  I look forward to seeing what else your group comes up with.

** Update! **

1. Fire, water, power problems, trojan horses, crackers, a lack of suitable sources of caffeine, aliens, zombies and the end of the internets also feature prominently.  The caffeine one, in particular, keeps me up at night.  *twitch*
2. A server with a whole bunch of drives attached providing disk storage to remote machines
3. Photo and a primer here.
4. I’ve put it on The List