almost effortless

These instructions require and assume the following:

• You're running Debian Lenny and you've got root access
• You've got a functioning apache2 installation
• You know the basics of working on the command line (i.e. how to edit files, execute commands, etc.)

If the above is true of your situation, read on to learn how to install Ruby Enterprise, Phusion Passenger and El Dorado from scratch in a sort of "one-off" setting where you've got one server and you want it to run one site.

NB: These instructions don't use git or capistrano. The instructions contained in the El Dorado README describe how to install El Dorado using those tools. Using them makes for an easier and cleaner installation. It also makes for easier scalability, upgrading and patching: I highly recommend using those tools.

1. Resolve Dependencies

The first thing you'll need to do, even before installing RE or PP, is make sure that you've got the development files for the databases that RE and PP applications use:

apt-get install libsqlite3-ruby postgresql-8.3-plruby libmysql-ruby libmysqlclient15-dev postgresql-server-dev-8.3 libsqlite3-dev

If you don't resolve these dependencies now, you'll get a message during the RE installation that prompts you to install gems for mysql, postgres, etc. and then, when you go to install those gems, you'll get an error like this:

ERROR:  Error installing mysql:
	ERROR: Failed to build gem native extension.

So just go ahead and resolve those dependencies in advance.

2. Install Ruby Enterprise

The best practice for this, as far as I know, is to install the current stable release of RE in /opt/. First, download the release you plan to use:

lana:~# cd /opt
lana:/opt# wget http://rubyforge.org/frs/download.php/58677/ruby-enterprise-1.8.6-20090610.tar.gz

Once that's down, untar it and execute the installer script:

lana:/opt# tar -zxvf ruby-enterprise-1.8.6-20090610.tar.gz
[...]
lana:/opt# cd ruby-enterprise-1.8.6-20090610/
lana:/opt/ruby-enterprise-1.8.6-20090610# ./installer

That should run, after a few tappings of ye olde Enter key, to its error-free conclusion. If, during the installation, the installer finds that you're missing software packages, the installer will bail and you'll be given some commands that fill those holes. Resolve those dependencies and finish the installation.

At the end of the installation, you'll be given some syntax that will automatically install PP. You'll use that in the next step.

3. Install Phusion Passenger

Use the automatically generated syntax:

lana:/opt/ruby-enterprise-1.8.6-20090610# /opt/ruby-enterprise-1.8.6-20090610/bin/passenger-install-apache2-module

Again, the installer will bail and prompt you to resolve dependencies if you've got any:

Installation instructions for required software

 * To install Apache 2 development headers:
   Please run apt-get install apache2-prefork-dev as root.

 * To install Apache Portable Runtime (APR) development headers:
   Please run apt-get install libapr1-dev as root.

 * To install Apache Portable Runtime Utility (APU) development headers:
   Please run apt-get install libaprutil1-dev as root.

Resolve dependencies and finish the installation.

Once it's finished, you'll be given some lines to add to your "Apache configuration file". The best file to add these lines to is /etc/apache2/httpd.conf.

Just don't forget that you added them there (as opposed to somewhere else), as you'll need to modify them if you upgrade RE.

You'll also probably want to go ahead and add the following lines while you've got the file open:

PassengerPoolIdleTime 14400
PassengerMaxInstancesPerApp 2

Those lines do exactly what it looks like they do. They're also very sensible settings to start with, as they'll prevent El Dorado from hogging a bunch of system resources, etc. right off the bat.

You can find more information here.

Finally, your /etc/apache2/httpd.conf file should look something like this:

PassengerPoolIdleTime 14400
PassengerMaxInstancesPerApp 2

LoadModule passenger_module /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/passenger-2.2.4/ext/apache2/mod_passenger.so
PassengerRoot /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/passenger-2.2.4
PassengerRuby /opt/ruby-enterprise-1.8.6-20090610/bin/ruby

Once you've made those changes, you're ready to begin installing El Dorado.

When it exits, the PP installer will show you some sample syntax for how to write an apache configuration file for your first application. You can ignore that for now, as we're going to come back to it later.

4. Install El Dorado

First, get the latest release of the software from Trevor's github: http://github.com/trevorturk/eldorado/tree/master

Once you've got the URL of the latest release, switch from root to a less privileged user, make a folder in your home dir for the site, download the latest release of El Dorado to that directory and untar it:

toconnell@lana:~$ mkdir example.com
toconnell@lana:~$ cd example.com
toconnell@lana:~/example.com$ wget wget http://download.github.com/trevorturk-eldorado-a37d0c71e928f605d111d5f48b5786ff613bf676.tar.gz
tar -zxvf trevorturk-eldorado-a37d0c71e928f605d111d5f48b5786ff613bf676.tar.gz

Now, get all of those files out of that big, ugly directory and into the current working directory and ditch those old files:

toconnell@lana:~/example.com$ mv trevorturk-eldorado-a37d0c71e928f605d111d5f48b5786ff613bf676/* .
toconnell@lana:~/example.com$ rm -rf trevorturk-eldorado-a37d0c71e928f605d111d5f48b5786ff613bf676*

Now, follow the instructions in the README and copy the example yml files to the places where the application will look for real, non-example files:

toconnell@lana:~/example.com$ cp config/database.example.yml config/database.yml
toconnell@lana:~/example.com$ cp config/config.example.yml config/config.yml

Now, use your favorite editor to edit the last stanza in config/config.yml so that it matches the information of your site:

production:
  session_key: example_production
  session_secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  # Replace these X's and make this string (at least) 32 random alpha-numerics for good site security
  domain: http://example.com
  mailer: noreply@example.com  

NB: There are "dev" and "test" entries in this default file. If you're not planning on doing anything development related with this installation, you can safely delete those entries.

Once you've edited that file, that's it, so far as the non-git installation is concerned. To get El Dorado up and running, you'll need to do some minor database tasks. Those are covered in the next section.

5. Configure the Database

Since MySQL is deprecated, I'll be using PostgreSQL for the remainder of these instructions.

If you look at config/database.yml, you'll notice that it's essentially a blank template:

development:
  adapter: sqlite3
  database: db/development.sqlite3
  timeout: 5000
  # adapter: mysql
  # database: eldorado_development
  # username:
  # password:
  # host: localhost

test:
  adapter: sqlite3
  database: db/test.sqlite3
  timeout: 5000

production:
  adapter:
  database:
  username:
  password:
  host:

First, edit that file:

production:
  adapter: postgresql
  database: example
  username: example
  password: XXXXXXXXXXXXXXXXXXXX
  host: localhost

NB: Again: once you've added your "production" entries to this file, you can feel free to delete the "test" and "dev" lines, as they do nothing and could cause confusion down the line.

Now, create the database and the user:

toconnell@lana:~/example.com$ sudo su postgres -c "createuser example"
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
toconnell@lana:~/example.com$ sudo su postgres -c "createdb example"

Next, start the postgres monitor as the postgres user and make the a few changes:

toconnell@lana:~/example.com$ sudo su postgres -c psql
Welcome to psql 8.3.7, the PostgreSQL interactive terminal.

postgres=# ALTER USER example PASSWORD 'XXXXXXXXXXXXXXXXXXXX';
ALTER ROLE
postgres=# ALTER DATABASE example OWNER TO example;
ALTER DATABASE

Now, if you've got your Postgres database configured correctly and your new user can access your new postgres database, you're ready to rake the El Dorado production database:

toconnell@lana:~/example.com$ /opt/ruby-enterprise-1.8.6-20090610/bin/rake rake db:schema:load RAILS_ENV=production

Once the database is successfully raked, all you've got to do to finish up is configure Apache.

6. Apache Configuration

The following assumes that you're doing apache the "Debian way".

If this is true, the first thing you'll do is create a symlink in /var/www/ that points at your install directory:

lana:/var/www# ln -s /home/toconnell/example.com/

Next, create a file in /etc/apache2/sites-available with the name of your site and then create a symlink to it in /etc/apache2/sites-enabled.

The file should look something like this:

#
# example.com
#
 
<VirtualHost *:80>
  ServerName example.com
  ServerAlias www.example.com
  ServerAdmin youremail@example.com
  DocumentRoot /home/toconnell/example.com/public
 
  <Directory "/var/www/example.com">
    Options FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
  </Directory>
 
  RewriteEngine On
 
  RewriteCond %{HTTP_HOST} ^www\.example\.com$ [NC]
  RewriteRule ^(.*)$ http://example.com$1 [R=301,L]
 
  RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
  RewriteCond %{SCRIPT_FILENAME} !maintenance.html
  RewriteRule ^.*$ /system/maintenance.html [L]
 
  ErrorLog /var/log/apache2/example_error_log
  CustomLog /var/log/apache2/example_access_log combined
  RewriteLog /var/log/apache2/example_rewrite_log
  RewriteLogLevel 9
 
</VirtualHost>

NB: I've added some apache custom logging. Logs are good.

Once you've got the file in /etc/apache2/sites-available and the symlink in /etc/apache2/sites-enabled that points at that file, you should be ready to restart apache and get rolling:

lana:/etc/logrotate.d# /etc/init.d/apache2 reload

And that, as they say, is that. Once you reload apache, provided that your DNS is set up correctly and you haven't got any system problems beyond the scope of this document, your single instance of El Dorado should be ready for prime time.

Navigate to your site in your browser and create an administrative account: the first user who attempts to login will be the administrator. Once you've got your admin created, you're ready to start tweaking your new El Dorado site's appearance and adding users.

A note on upgrades: if you find you need/want to upgrade an instance of El Dorado that has been installed thus, consult the README. The basic gist is that you're going to want to download/copy the new source/program files over the old ones (while being careful not to erase your user-uploaded files) and then run rake db:migrate RAILS_ENV=production.

File this one under "hacks". Cross-list it under "more basic administrative tasks you can't do on Windows".

Here's the situation: you're the admin of a Windows domain where the Domain Controller is a Linux box serving Samba. Your problem, other than the fact that you're surrounded by Windows users, is that you've got a user who's password you don't know and, for whatever reason, you need to log onto your Windows domain as that user: a simple RUNAS won't cut it this time.

Normally, you'd just nuke his password, change it to "password" (or whatever), log on as him, do your dirty, sinful business, log off, expire his password and then send him an email telling him that his password has been changed to "password" and that he'll be prompted to change it at his next log on.

But what if that wasn't an option? What if you needed to log on to your domain as that user and it was important that he be none the wiser?

Grab the Hashes

First, use your favorite smbldap-type tool to get the current password info on the user you're fixin' to use:

frances:~# smbldap-usershow toconnell
dn: uid=toconnell,ou=Users,dc=domain,dc=com
objectClass: top,person,organizationalPerson,inetOrgPerson,posixAccount,shadowAccount,sambaSamAccount
cn: toconnell
sn: toconnell
givenName: toconnell
uid: toconnell
uidNumber: 1007
[...]
sambaPwdCanChange: 1202398556
sambaPwdMustChange: 9223372036854775807
sambaLMPassword: BE41CD009FF0812C718CCFD7D98A52AA
sambaNTPassword: 9454453CBC8A48DEF442F6B0A10B3EAA
sambaPwdLastSet: 1202398556
userPassword: {SSHA}baSDvXS6C6DSBNkJGyEYplprZ3wslAa/

Copy everything that's bolded and stick it somewhere safe. That dn information is going to be necessary later on, as it contains the ldap tags that you'll need to specify the record you want to modify; those hashes at the bottom are the user's original passwords and, when you want to cover your tracks later on, you'll need that info.

Now that you've got those hashes, you're free to nuke the user's password (again, using your favorite smbldap-type-tool or however else you like to reset passwords), log in as him, do whatever you have to do, and then log out:

frances:~# smbldap-passwd toconnell
Changing UNIX and samba passwords for toconnell
New password:
Retype new password:

Once you're out, you're going to want to set his password back to what it once was. This is where ldapmodify comes into play.

Kerberos

Before you can do that, however, you'll need to get a kerberos ticket. This is because you'll need to be kerberos-authenticated to make your ldap modifications stick. So, first things first, get yourself an admin kerberos ticket:

frances:~# kinit toconnell/admin
Password for toconnell/admin@DOMAIN.COM:
frances:~# 

ldapmodify

A quick glance at the man page for ldapmodify shows that the most convenient way to make changes to an ldap entry is to use the -f flag and an input file. The example in the man page for how to construct the input file is this:

 dn: cn=Modify Me,dc=example,dc=com
           changetype: modify
           replace: mail
           mail: modme@example.com
           -
           add: title
           title: Grand Pooba

So, using the data we got above, we're going to make a similar file containing the original hashes from our target user in order to change his password back to what it used to be:

dn: uid=toconnell,ou=Users,dc=domain,dc=com
changetype: modify
replace: sambaLMPassword
sambaLMPassword: BE41CD009FF0812C718CCFD7D98A52AA
-
replace: sambaNTPassword
sambaNTPassword: 9454453CBC8A48DEF442F6B0A10B3EAA
-
replace: userPassword
userPassword: {SSHA}abSDvXS6C6DSBNkJGyEYplprZ3wslAa/

Remember to include those "-" characters and to give them their own line: if you fail to do that, you'll get mystery errors about unknown types, etc.

Once you've got your file, fire off your changes like this:

ldapmodify -f /path/to/file

And that's all you've got to do. When the original user attempts to log in with his old password, everything will look perfectly normal to him: you never saw his password in plaintext and, as far as he's concerned, none of this ever happened.

Due to recent events involving some computers I administer, I've become very interested in security. Basically, I've dodged enough bullets thanks to little more than beginner's luck and I figure that it's about time to take responsibility for the safety and security of my computers.

Since I was interested in hardening up two Debian boxes running Lenny, I started off by taking a look at the Securing Debian Manual, this very helpful page on LQ and the results of a tiger audit.

Do this to generate a tiger audit of your server:

# aptitude install tiger
tiger -E

The "-E" gets you what's called an "explanation report", which will be useful in helping you understand what can be some fairly cryptic output.

Additionally, the package version of tiger comes with some nice default settings for the main executable and for tigercron, which, as you might imagine, runs some minor scans on a pre-defined schedule.

At any rate, once I had my audit and had picked up a few bright ideas from the SDM, I made a number of changes to all of my web-facing production machines. What follows are some things that you might want to consider doing on your Debian Lenny servers:

Users and Permissions

1. Password Audit: first, I decided to get to know my user accounts a little better. This meant running john (formerly "john the ripper", a password cracker that reads hashed passwords and tries to decipher them) against my /etc/shadow to see who was using dictionary-based passwords and who was using other types of insecure passwords:

   # aptitude install john
   # john /etc/shadow 

   This took a while to run--a little over a day, but I had it niced pretty high--but of the 10 user accounts it cracked, it was good to know which ones were using hilariously insecure passwords and which ones probably weren't going to cracked by your garden variety brute force password cracker.

NB: if you run john against your /etc/shadow and realize that you've got a problem child on your hands, there's always chroot. Here is a really good how-to on chroot-jailing a user.

2. The Prunening: odds are, if you've been living on a system for more than a few months, you've accumulated some users (either from software that you've installed and then removed or by meeting user/developer needs, etc.) that aren't doing anything. One of the basic tenets of server security is having the smallest amount of users with the least amount of access to the smallest number of programs possible.

   In some environments, you've simply got to have a bunch of users in your /etc/passwd. In most situations, however, it makes good sense to just hit all the derelicts with a userdel and only having to worry about angry users not having enough access (rather than having to worry about unauthorized users having too much access).

Keeping Script Kiddies Under Control

In my (admittedly limited) experience, the most trouble you're going to run into from script kiddies are anonymous, unfocused attacks that attempt to gain access to your machine via either a.) the /tmp directory, b.) DoS-based exploints or c.) application attacks like SQL injections, XSS or directory traversal attacks.

Since guarding against application attacks is something that programmers are supposed to be handling, I decided to focus on stopping /tmp abuse and trying to stymie DoS attacks.

1. Mounting /tmp with noexec: in this age of VPSes and shared hosting, it's more often the case than not that you won't get to decide how your machine is partitioned. If, like me, you live on Slicehost and you're running Debian, your partition scheme looks like this:

   lana:/# df -h
   Filesystem            Size  Used Avail Use% Mounted on
   /dev/sda1              19G  9.0G  8.9G  51% /
   tmpfs                 256M     0  256M   0% /lib/init/rw
   udev                  256M   20K  256M   1% /dev

   What this means is that you've got your /tmp directory on your / partition. Which partition is, by necessity, allows files to be executable.

   And that, as I have learned (the hard way), is bad news. What this means is that, you've got a directory on your / partition--the partition where all your apps and data probably live--that is writable/readable by every Li, Ivan and Harry from Taipei to Yaktusk. And if one of the thousands of bots who knocks on your door every month knows how to write to /tmp and you don't catch him right away, it's pretty much game over for your TLD or your IP Address: one day, you'll wake up and find that your info is on every spam list on the Internet because your server has been under remote control via IRC for the last three weeks and now your full time job is trying to get your info off of those lists while planning a full OS re-install.

   Ideally, you would be able /tmp its own partition and mount that partition with noexec. And while it would be optimal, it sometimes isn't an option if you're a part of the VPS set.

   And if you can't control your installation or maybe you just can't take down a production server, what you can do is warn your users/developers that you're about to cause a little temporary chaos (get it? temporary chaos?), move your existing /tmp to some place else and create a small filesystem that you can mount noexec to use as /tmp. On Debian Lenny, that would look approximately like this:

   gonzo:/# mv /tmp /old_tmp
   # dd if=/dev/zero of=/.tmpfs bs=1024 count=1000000
   1000000+0 records in
   1000000+0 records out
   1024000000 bytes (1.0 GB) copied, 29.1302 s, 35.2 MB/s
   gonzo:/# mkfs.ext3 -j .tmpfs
   [...]
   gonzo:/# mount -o loop,noexec,nosuid,rw /.tmpfs /tmp/
   gonzo:/# mv /old_tmp/* /tmp/.

   Et voila! You've got a 1GB "drive" that's mounted noexec at /tmp that's ready to roll out. Any attempts to execute anything on that partition will result in a bad interpreter error. Check it:

   gonzo:/# ls -l tmp/
   total 20
   -rwxr-xr-x 1 root root    37 2009-05-15 14:54 executeMe.py
   drwx------ 2 root root 16384 2009-05-15 14:51 lost+found
   gonzo:/# test/executeMe.py
   bash: test/executeMe.py: /usr/bin/env: bad interpreter: Permission denied

   All you've got to do now is add that mount info to your fstab and you're ready to start sleeping at night again:

   /.tmpfs           /tmp            ext3    loop,nosuid,noexec,rw  0      0

   NB: don't forget that /tmp wants to have the sticky bit turned on (i.e. be chmoded to 1777). Also don't forget to make /var/tmp a symlink that points to /tmp.
   Also: props to Vincent Danen's post on /tmp at TechRepublic for the idea.

2. mod_evasive to Prevent DoS: after a recent DoS experience, I decided to install Apache mod_evasive to reduce the risk of getting flat-lined/broad-sided by DoS/DDoS attacks:

   # aptitude install libapache2-mod-evasive

   The beauty of using packaged software is that that's really all you have to do: apt will copy the files, create the symbolic links and restart apache for you. Nice.

3. sysctl real-time kernel Modifications: Additionally, you might also want to use a slightly obscure command called sysctl (which modifies kernel perameters while the kernel is running, so consider yourself warned) to take a precaution against a DoS tactic called "syn flooding":

   # sysctl -w net.ipv4.tcp_syncookies=1

   This is a sort of controversial measure--apparently it defies some RFC docs for TCP/IP--but setting tcp_syncookies to False has yet to have affected any of my computers.

   There are a number of additional security features you can activate with this command; Google it and prepared to be awed by some of the features of your OS you can control in real-time with sysctl.

For the Tin-foil Hat Crowd

What follows are non-specific countermeasures and settings that, while obscure and probably unlikely to save you from becoming an unwitting member of some Russian bot master's herd, might help you feel more secure in the knowledge that even if someone does get non-root shell access, he's probably not going to be able to do too much damage.

1. Add /usr/bin/mesg n to root's .bashrc File:
   executing mesg n on log in, prevents an admittedly rare exploit through which other users can execute arbitrary code as root by sending messages to his terminal.
2. Modify /etc/inittab to Prevent Non-root Users from Rebooting the System with ctrl+alt+del. In the stock /etc/inititab on Debian Lenny, you've got this line:

   # What to do when CTRL-ALT-DEL is pressed.
   ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -r now

   Which is hardly optimal: this lets pretty much anyone who figures out a way to execute programs in /etc/sbin reboot the system. I changed it to:

   ca:12345:ctrlaltdel:/bin/false

3. SMTPD Settings If you run postfix, you should probably check up on your relay settings and update your external blacklist providers if you haven't done it in a while. Your mail server is the world's first line of defense against everything from phishing/spear-phishing to headline-making super worms:

   smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks,
     reject_unauth_destination,
     reject_unauth_pipelining,
     reject_invalid_hostname,
     reject_rbl_client sbl-xbl.spamhaus.org,
     reject_rbl_client cbl.abuseat.org,
     reject_rbl_client bl.spamcop.net,
     reject_rbl_client zen.spamhaus.org
   smtpd_helo_required = yes
   disable_vrfy_command = yes
   
   smtpd_data_restrictions =
               reject_unauth_pipelining,
               permit
   

   That smtpd_help_required line might not seem like anything special, but I have a script that parses /var/log/mail.log output and, when you do get a spammer that responds to the helo request, a lot of times he'll come back with his actual domain.

   It's fun for research/study/personal amusement reasons, basically.

While some people might say that the above is overkill--that it's just not worth the time and effort to audit and harden at this level--but I'd say that this level of focus on security isn't so much "overkill" as it is "a pretty good start."

Because my thinking is that if you've got the root password, you're probably already the anxious type. And at the very least, being a little bit OCD about security on your all-important, mission-critical application servers might help you feel a little less anxious. Which is definitely worth the effort.

There's a great episode of MASH in which the over-zealous paranoid-schizophrenic Colonel Flagg, in an attempt to coerce the impossibly calm and notoriously imperturbable Dr Stanley Freedman into collusion with one or another of his hare-brained schemes, asks the psychiatrist: "You wanna do your self a favor?"

"Why Not?" replies the doc. "Who deserves one more?"

If you've got a Debian Lenny box out in the wild serving your email with postfix and you're not using spamassassin as a filter, you really ought to consider doing yourself a favor and throwing that spamassassin piece into the mix: it only takes a second, it will increase security for your users-- hardening up your network little bit--and make the world a slightly better place for everyone.

1. If you're OK with letting aptitude resolve your dependencies and manage your packages, all you need to do is install a single package:

   # aptitude install spamassassin

2. Once that's done, crack open /etc/default/spamassassin with your favorite editor and enable it to run as a daemon and update itself automatically:

   # Change to one to enable spamd
   #ENABLED=0
   ENABLED=1
   
   # Cronjob
   # Set to anything but 0 to enable the cron job to automatically update
   # spamassassin's rules on a nightly basis
   #CRON=0
   CRON=1

   (spamd is an old name for spamassassin: you'll notice a lot of the RHEL/CentOS/Fedora boxes out there running spamd.

3. Take a look at /etc/spamassassin/local.cf: there are some fun options that you can uncomment and enable in there. My personal favorite one is:

   rewrite_header Subject *****SPAM*****

   This does what it sounds like it does and rewrites the headers of suspicious emails: you can then easily configure your email client to recognize these headers and filter accordingly. Pretty sweet.

4. Start spamassassin:

   # /etc/init.d/spamassassin start

5. At this point, we're going to edit some postfix conf files, but we need to check on something first. Make sure SA is running, spawning children and listening on the right port:

   # netstat -anp |grep spam
   tcp        0      0 127.0.0.1:783           0.0.0.0:*               LISTEN      11724/spamd.pid
   unix  2      [ ACC ]     STREAM     LISTENING     9096119  1717/master         private/spamassassin
   unix  3      [ ]         STREAM     CONNECTED     9757173  30093/spamd child
   unix  3      [ ]         STREAM     CONNECTED     9757172  11724/spamd.pid  

6. Noting that SA is listening on 783, tack the following on to the bottom of your /etc/postfix/master.cf:

   spamassassin unix -     n       n       -       -       pipe
           user=nobody argv=/usr/bin/spamc -f -e
           /usr/sbin/sendmail -oi -f ${sender} ${recipient}

7. Now find the SMTP/SMTPS lines in your /etc/postfix/master.cf and add the following option:

   -o content_filter=spamassassin

   Assuming you're doing SMTP and SMTPS, you'll have something like this at the top of your /etc/postfix/master.cf:

   smtp      inet  n       -       -       -       -       smtpd     -o content_filter=spamassassin
   smtps     inet  n       -       -       -       -       smtpd     -o content_filter=spamassassin

8. Reload Postfix:

   # postfix reload

And that's it: you're done.

If you can bear in mind that no spam-detection scheme is perfect, my guess is that you'll be pleased with your decision to set up SA: some spam will make it through, of course, but most of makes it through will come a.) as plaintext with escaped characters and b.) a lengthy disclaimer and an itemized spam "score":

Spam detection software, running on the system "molly", has
identified this incoming email as possible spam.  The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email.  If you have any questions, see
the administrator of that system for details.

Content preview:  Having trouble viewing this email? Click here! pharmacy medicine
   cabinet FSA home medical vitamins personal care diet & fitness men's SALE
   Get 80% Discount TODAY: This email was sent to you by drugstore.com. To ensure
   delivery to your inbox (not junk folders), please add drugstore@e.drugstore.com
   to your address book. [...] 

Content analysis details:   (13.5 points, 5.0 required)

 pts rule name              description
---- ---------------------- --------------------------------------------------
 2.0 URIBL_BLACK            Contains an URL listed in the URIBL blacklist
                            [URIs: lewdozed.cn]
 0.5 FH_HELO_EQ_D_D_D_D     Helo is d-d-d-d

...and so on.

So go ahead: do yourself a favor.

Security audits are a necessity.

Unfortunately, due to the constantly changing array of exploits and threats coming at your Internet-facing server from all over the world, there's no best practices manual for a security audit. This is because no one knows exactly what level of openness is appropriate/optimal on a given application server.

But you gotta start somewhere. Because if you're just blithely running an un-hardened *nix server with stock configurations for service apps like Apache and ssh, you're pretty much giving away the store.

What follows is a short how-to dealing with how to get started, security-auditing-wise. It is by no means comprehensive and is only intended to provide a "leg up" for those who feel like they ought to be auditing their servers but aren't sure where to start. I'll be using Debian systems running the current stable release (i.e. Lenny) to demonstrate the techniques, but I'll try to keep things as OS agnostic as possible: most of the packaged software I describe below can be found in your generic, mainstream fedora repo.

The methodology will be to start from the outside and work inward. I recently saw a Linux Journal article that described doing things the opposite way (or at least that's the impression with which I was left), and that doesn't make much sense to me. From where I'm standing, it seems that if you're conducting a security audit, you ought to start out by looking at your server the same way everyone else does.

Accordingly, we'll begin from the outside and from as basic a perspective as possible: scanning ports and checking for known vulnerabilities and obvious mis-configurations. Once we've got a little perspective on how our server looks to the script kiddies and botnets of the world, we'll do some web-server specific scanning to attempt to detect vulnerable plugins, apache mis-configurations and application-level security holes. After that, we'll finish by checking for rootkits and doing some internal auditing.

OpenVAS: Port Probe and All-purpose scan
The first thing to do is to scan your ports.

The odds are good that if you've got a server out there on the Internet (in the DMZ of your intranet or in a hosting company's rack, for example) that it's got a bunch of open ports. You've probably got one or two listening for HTTP requests, one or two listening for SSH requests and so on. What you probably haven't got is a good idea of how those ports look to the world.

So the first part of any security audit is the portscan. In olden times, you'd use a combination of telnet and nmap for this: nmap would handle the port scanning and tell you which ports were listening/open and then you'd use your expert knowledge of various network protocols to use a telnet-like program to check out those ports and see what sort of access and information they were offering to the world.

There are, fortunately, labor-saving apps that will do the scanning and auditing for you. A few months back, you would have been using nessus, as it was the big name in F/OSS auditing. nessus, however, has gone commercial (proprietary and closed) and a new, open project (GPL) called OpenVAS has taken its place.

For anything other than security auditing, using commercial software is probably OK. At the very least it's not always counter-productive to use non-F/OSS for non-security-related tasks. In the case of security-related apps, however, it just doesn't make any sense to take a chance on using software that isn't available for public scrutiny.

At any rate, if you're familiar with the way that nessus works, you'll be happy to know that the big ideas and the general methodology/procedure behind using OpenVAS are essentially the same. If you're unfamilar with programs like nessus and OpenVAS, here's how they work, from an administrator/auditor's perspective:

1. Set up a server
2. Use a client to tell the server to probe the target site
3. View the audit report on the client

At present, if you're using the stock stable/unstable Debian repositories, you haven't got access to the packaged version of the OpenVAS server. That being the case, we're going to go ahead and get a little bit heroic here and do this the Cowboy Way (i.e. from source).

1. Dependencies and Source Files

Make sure that you've got the following packages (some of which the openvas developers list as dependencies, some of which you'll need to compile anything from source) before proceeding:
molluska:/opt/# aptitude install libgnutls-dev libpcap-dev libgpgme11 libgpgme11-dev libglib2.0 libglib2.0-dev build-essential bison

Now that that's handled, there are four "modules" that are required to run an OpenVAS server. The openvas developers say that you've got to install the modules in the following order:

1. openvas-libraries
2. openvas-libnasl
3. openvas-server
4. openvas-plugins

So that's what wer're going to do. I like to do this sort of thing in/opt, but it really doesn't matter where this happens.

Get the files:
molluska:/opt/openvas# wget http://wald.intevation.org/frs/download.php/572/openvas-libraries-2.0.2.tar.gz
molluska:/opt/openvas# wget http://wald.intevation.org/frs/download.php/561/openvas-libnasl-2.0.1.tar.gz
molluska:/opt/openvas# wget http://wald.intevation.org/frs/download.php/562/openvas-server-2.0.1.tar.gz
molluska:/opt/openvas# wget http://wald.intevation.org/frs/download.php/576/openvas-plugins-1.0.6.tar.gz
NB: these URL's are for the version that was current when this was written--no guarantees that they'll be there two hours from now.

2. ./configure && make && make install

Now, we start the compilation process which, thanks to our having resolved the dependencies enumerated above, should go off without a hitch:
molluska:/opt/openvas# tar -zxvf openvas-libraries-2.0.2.tar.gz
[...]
molluska:/opt/openvas# cd openvas-libraries-2.0.2
molluska:/opt/openvas/openvas-libraries-2.0.2# ./configure
[...]
molluska:/opt/openvas/openvas-libraries-2.0.2# make
[...]
molluska:/opt/openvas/openvas-libraries-2.0.2# make install

Once you've successfully installed the openvas libraries, you'll be prompted to modify /etc/ld.so.conf by adding the line "/usr/local/lib" to it and running ldconfig to update your linker. Do that, and then repeat the steps described above (untar, configure, make, make install) in the other three folders to finish installing the OpenVAS modules.

3. Add a User and Generate an SSL Cert

Once you've got everything installed, you'll need to create two things: an OpenVAS user and an SSL certificate. Fortunately, both of these tasks have been nearly fully automated and all you'll have do to get the job done is execute a couple of binaries (which should be on your path, now that you've installed everything according to the above instructions) and follow some on-screen prompts:

molluska:/opt/openvas# openvas-adduser
[...]
molluska:/opt/openvas# openvas-mkcert
[...]

And that's it. Make a note of the paths that the openvas-mkcert program gives you at the end of the certificate creation (as you might need to specify them at some later time; you won't need them again to follow these instructions).

4. Fire it up

Once you've got all your modules installed, your certificate created and your user added, it's time to fire up the server/daemon. The smartest way to do this is to pseudo-daemonize it and tail its output while it loads plugins:molluska:/# nohup openvasd &
[1] 4508
molluska:/# nohup: ignoring input and appending output to `nohup.out'
molluska:/# tail -f nohup.out

Should get you something like this:tail -f nohup.out
Loading the plugins... 714 (out of 10558)

...and so on. Once the plugins are all the way loaded, fire off a quick ps to make sure that the server is running and maybe a quick netstat to make sure you know what port it's listening at, and that's it: you're done with the server side of things and ready to move on to the client and auditing part.

molluska:/opt/openvas# ps aux |grep openvas
root     26129  1.1  0.2  16296    76 ?        S    06:29   1:46 openvasd: waiting for incoming connections
molluska:/opt/openvas# netstat -anp |grep openvas
tcp        0      0 0.0.0.0:9390            0.0.0.0:*               LISTEN      26129/openvasd: wai

Installing the OpenVas client is much easier.

While there is a packaged version of the OpenVAS client app, we're going to install one from source (mostly so our version of the server matches with our version of the client; this is mostly me being OCD, however, and you can probably get away with using the packaged version). To install the client, we'll follow the same steps as above, but on a different machine:

gonzo:/opt/openvas-client# wget http://wald.intevation.org/frs/download.php/575/openvas-client-2.0.3.tar.gz
[...]
gonzo:/opt/openvas-client# tar -zxvf openvas-client-2.0.3.tar.gz
[...]
gonzo:/opt/openvas-client# cd openvas-client-2.0.3
gonzo:/opt/openvas-client/openvas-client-2.0.3# ./configure
[...]
gonzo:/opt/openvas-client/openvas-client-2.0.3# make
[...]
gonzo:/opt/openvas-client/openvas-client-2.0.3# make install

NB: you may, depending on your client system, have to resolve some GTK dependencies and other build/compiler dependencies like the ones listed above: libgtk2.0-dev should solve most of your gtk problems, if you're running Debian Lenny.

Once the GUI client is installed, start it up:

toconnell@gonzo:~$ sudo aptitude install openvas-client
[...]
toconnell@gonzo:/opt/openvas-client/openvas-client-2.0.3/bin$ ./OpenVAS-Client &

Connection Screen, OpenVAS GUI Client

Once your client is connected with your server, you're ready to fill in the blanks and start your first round of tests. This is fairly self-explanatory and, honestly, you wouldn't be reading this if you couldn't figure out simple GUI interfaces: specify your target (i.e. the server you're auditing), make sure that all the plugins are enabled and then click the life preserver to start the "Scan Assistant" and execute the scan. Follow the on-screen prompts: easy as apple pie.

The best thing to do, once your scan starts, is probably to go do something else and come back in a little bit: in my experience these scans can take anywhere from 15 to 45 minutes, depending on your server and your pipe: my server is an old Linksys NSLU2 and my pipe is a consumer-grade Speakeasy residential connection, so I'm used to waiting close to an hour for the scan to finish. Using corporate resources will result in less idle time.

OpenVAS scan results report

Once the scan is done, you're treated to a report view. This is what we've been after all long. In it, you'll see a full run-down of what ports on your server are open and what open ports are listening for what. Additionally, you'll be treated to helpful recommendations about how to close security holes. And while closing those holes is beyond the scope of this article, I will say that almost every recommendation I've gotten from an OpenVAS report has been sane, been sensible and lead to a harder server.

nikto: Web-server Specific Auditing
The second thing to do, in order to perform a robust audit of your system, is to hit it with nikto (http://www.cirt.net/nikto2).

nikto, unlike OpenVAS doesn't require a server/client hook-up: just install the client with apt and fire off some tests, writing the output from those tests to plaintext files:

molluska:/# aptitude install nikto
molluska:/# nikto -h newathens.org -p 80 -output nikto_na80 && nikto -h newathens.org -p 443 -output nikto_na443

You'll get helpful output that points you towards an obvious solution like this:

+ mod_ssl/2.2.9 appears to be outdated (current is at least 2.8.30) (may depend on server version)

And you'll also get put on notice if you've got paths/folders/files with names that automatic exploiters and scripts tend to look for:

+ OSVDB-3092: GET /login/ : This might be interesting...

...to script kiddies and Chinese botnets.

You'll also get put on notice if you've got too much of your software's installation defaults hanging out in the open:

+ OSVDB-3233: GET /icons/README : Apache default file found.

Server-side Checks

chkrootkit
There are a few utilities that allow you to perform quick server-side audits of your security situation. Some of them, like rkhunter will run daily (like logwatch or apticron) and tell you if they've identified any new chinks in your armor. The first one to install and run is chkrootkit.

molly:/# aptitude install chkrootkit
[...]
molly:/# chkrootkit

This is a great place to start your internal audit because it'll tell you if you've picked up any known bugs and whether anything weird, filesystem-wise, appears to be going on with your computer.

The best use for this app is to give you a very quick idea of what sort of shape you're in. If you've got a system littered with suspicious files, odd-looking binaries, etc., you know exactly where to start plugging holes.

rkhunter
While we're on the subject of checking for root kits, let's do rkhunter:

molly:/# aptitude install rkhunter
[...]
molly:/# rkhunter --update
molly:/# rkhunter --check

This gets you a quick check of all your important binaries (to make sure they look like they're supposed to look, i.e. that they haven't been replaced by scripted exploits or an intruder with something that opens a back door) and a quick scan for known exploits of the rootkit variety. You'll also be told whether you're running inetd/xinetd (which tends to open ports in a manner whose security can be less than "ironclad") and other fun facts about potential vulnerabilities.

The best thing to do with this report is think long and hard about what ports/resources/pathways you actually want to make available to the Internet and then start disabling services. Once you've spent some time with that, you're pretty well on your way to having an idea of exactly how hard your server is and how much work you've got to do to keep it safe.

If anyone has any ideas about other utilities or techniques for security auditing, please feel encouraged to share them in the comments.

Generally speaking, I'm a Debian guy.

Sure, I'll mess around on the CentOS box under my desk on the production RHEL servers at work a little bit, but Red Hat is largely terra incognita for me and Debian is where I'm comfortable doing my admin thing and managing packages. I know apt. I am comfortable with apt. And while I wouldn't describe using yum as something that makes me uncomfortable, when I do have to use it, I find myself spending more time Googling and forum-searching than I'd like.

I'm beginning to accept that this is no one's fault but my own.

And so the purpose of this article, therefore, is not a.) to point fingers, b.) to compare apt to yum or c.) to explain yum from the perspective of someone who is accustomed to doing things the Debian way. The Internet is littered with stuff like that like the intersection of Paradise and Tropicana are littered with advertisements for escorts.

In this post you'll find some novice-level trouble-shooting tips, reminders and pointers for the casual yum user that are intended to help reduce the occurrence of forehead-slaps and to decrease the amount of time spent tailing logs and Googling obscure error messages.

1. Preemptive Troubleshooting.

It's not in the documentation, but I have noticed that a lot of dependency issues and version consistency problems are resolved by tossing off the following yum command and then trying again:

 # yum clean packages

It has become my general practice to do this before I do anything else. It's a nice preemptive step.

I've noticed that it's generally not the advice of performance-minded (read: impatient) admins to do the more scorched-Earth yum clean all, as this empties caches, dbcaches (i.e. sqlite files) and can causes longer check-update times.

2. filelists.xml.gz Download Times out.

Let's say you're doing a yum update or a yum upgrade and you get some output like this:

filelists.xml.gz          100% |=========================| 1.4 MB    00:01
filelists.xml.gz          100% |=========================| 1.3 MB    01:48
http://apt.sw.be/redhat/el5/en/i386/dag/repodata/filelists.xml.gz: [Errno 4] Socket Error: timed out
Trying other mirror.

There are good odds, especially if you're using non-standard repositories, that you copied/pasted some text into your yum.conf from somewhere out there on the Internets. If you did, there are even better odds that the text you copied includes something about using GPG to authenticate the repo. If you've got lines like that, you'll need the repository's key.

Generally speaking, you can navigate to a repository's http site and find the URL for their public key. Once you've got that, all it takes to import it is one of these:

# rpm --import http://dag.wieers.com/packages/RPM-GPG-KEY.dag.txt

3. Know your repositories

You can save a lot of backtracking/head-scratching time if, before searching for a package on a machine you don't visit that often, you toss off a quick yum repolist. This handy feature will spit out the names and statuses of all of the repositories in all the files in your /etc/yum.repos.d/ directory and prevent you from doing that thing where you don't realize that you've only got the default CentOS repositories enabled but can't seem to figure out why the eff your yum search for htop just turned up a big goose-egg.

4. Automatic Notifications

If, like me, you're coming at yum from a Debian perspective, one of the first things you'll do when you start administering an RPM-based system is to install the apticron-equivalent known as yum-updatesd (# yum install yum-updatesd.noarch). Something you might forget, however, is that the default behavior of yum-updatesd is to not send emails.

Don't forget to edit /etc/yum/yum-updatesd.conf such that

mit_via = dbus

looks like

mit_via = email

or you won't get those all-important package update emails.

And that's about all that's coming to mind right now.

If anyone else can think of some things that you consistently forget--and then suddenly remember, 20 minutes later--to do when you're working with yum, feel free to leave a comment.

This blog is going to end with a question that I have been unable, after countless minutes of Googling, to answer satisfactorily. It will start, however, with some givens.

I already know that if you want to alter the default settings for all profiles that will be created by a given Firefox installation in the future, you add the line for the preference that you want to effect those profiles to the file FIREFOX_ROOT/defaults/profile/prefs.js.

Similarly, I already also know that if you want to push a preference to all currently existing users on the machine, you add the line for that preference to FIREFOX_ROOT/deftauls/pref/firefox.js.

The caveat there, of course, is that if the user of the profile have already changed a preference in his personal prefs.js (i.e. the one in HOMEDIR/.mozilla/firefox/RANDOMALPHANUMERICS.USERNAME/prefs.js) and it conflicts with your preference in the (global) firefox.js, then you (the admin) are SOL, because the program will defer to the user's personal prefs.js file.

Which brings us to the question: is there a way (short of writing a script to parse individual user's personal prefs.js files and modify them as needed) to push a preference to all users of a given Firefox installation?

Full disclosure: I'm posing this question for two reasons. The first reason is that I'm sort of passive-aggressive with Firefox: ours is a very love-hate relationship. The second reason is that I honestly don't think that what I'm describing--i.e. adding a preference to one, "master" preferences file that effects all users of a given installation, regardless of their personal prefs.js file--can be done.

Am I missing something? Maybe even something truly forehead-slap-worthy that's at the top of all the documentation? Or is this a real limitation of the program?

I was recently doing some server hardening on the computer that serves my email when, upon attempting to check my admin account, I got the following completely unhelpful, vanilla "encryption protocol" error from Thunderbird:

Thunderbird can't connect securely to because the site uses a security protocol which isn't enabled.

So, naturally, I whipped out T-bird's about:config, grepped the list for "ssl" and made sure that most of the contemporary ciphers were enabled (i.e. set to true). And once I was satisfied that things were on the up and up with my client, I decided to go have a look at the server.

Tailing mail.log, I noted this sort of thing happening over and over:

Mar 10 13:06:31 lana postfix/postfix-script[18701]: starting the Postfix mail system
Mar 10 13:06:31 lana postfix/master[18702]: daemon started -- version 2.5.5, configuration /etc/postfix
Mar 10 13:07:27 lana imapd-ssl: couriertls: connect: error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher

This message sort of blew my mind: I was essentially being told that my client, a Debian (Lenny) workstation running Thunderbird, shared no openssl ciphers with my email server, a Debian (Lenny) box out in the wild.

Not being the sort to ignore log warnings, I decided to verify. From the client:

gonzo:/# openssl ciphers
DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:AES256-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:DES-CBC3-SHA:DES-CBC3-MD5:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:AES128-SHA:RC2-CBC-MD5:RC4-SHA:RC4-MD5:RC4-MD5:EDH-RSA-DES-CBC-SHA:EDH-DSS-DES-CBC-SHA:DES-CBC-SHA:DES-CBC-MD5:EXP-EDH-RSA-DES-CBC-SHA:EXP-EDH-DSS-DES-CBC-SHA:EXP-DES-CBC-SHA:EXP-RC2-CBC-MD5:EXP-RC2-CBC-MD5:EXP-RC4-MD5:EXP-RC4-MD5

A fairly robust list. From the server:

lana:/# openssl ciphers
DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:AES256-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:DES-CBC3-SHA:DES-CBC3-MD5:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:AES128-SHA:RC2-CBC-MD5:RC4-SHA:RC4-MD5:RC4-MD5:EDH-RSA-DES-CBC-SHA:EDH-DSS-DES-CBC-SHA:DES-CBC-SHA:DES-CBC-MD5:EXP-EDH-RSA-DES-CBC-SHA:EXP-EDH-DSS-DES-CBC-SHA:EXP-DES-CBC-SHA:EXP-RC2-CBC-MD5:EXP-RC2-CBC-MD5:EXP-RC4-MD5:EXP-RC4-MD5

Same cot-damn list.

"So what gives?"

I Googled around a bit and learned about "s_client", an argument for the openssl tool that lets you debug an SSL exchange. I ran the following on my client:

gonzo:/# openssl s_client -connect mail.XXXXXXX.com:993 -ssl3

It showed me that the port was open, but that there were handshake problems:

26282:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1053:SSL alert number 40
26282:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:530:

From that cryptic output, I decided that it was time to dig into the SSL conf files over on the server.

After a little preliminary troubleshooting--a quick scan of /etc/postfix/main.cf and /etc/postfix/master.cf to check for obvious shenanigans--I decided to have a look at /etc/courier/imapd-ssl and found the source of my new SSL auth problem: I had, in my recent efforts beef up security, managed to overwrite my previous /etc/courier/imapd-ssl with a vanilla version of that conf file that had a big 'ol she in front of the argument that determines exactly which ciphers the IMAP daemon will use to authenticate requests: TLS_PROTOCOL was commented completely out, as was TLS_STARTTLS_PROTOCOL.

I took a quick look at the surrounding comments in the conf file and set both of them to "SSL23":

TLS_PROTOCOL=SSL23
TLS_STARTTLS_PROTOCOL=SSL23

I then reloaded postfix, attempted to connect with my client and, to my dismay, saw this roll up in the mail.log:

Mar 10 13:55:34 lana imapd-ssl: couriertls: connect: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

This message, while cryptic enough to send me off to scratch my head and pore over comments on OsDir and the Ubuntu fora, was one that I eventually figured out. It turns out that those two TLS protocol directives do not want to be identical.

So I chaged the file thus:

TLS_STARTTLS_PROTOCOL=TLS1

And, once I had reloaded the IMAP daemon and postfix, voila: a clean log-in!

Mar 10 13:56:52 lana imapd-ssl: Connection, ip=[::ffff:XX.XXX.XXX.XXX]
Mar 10 13:56:52 lana imapd-ssl: LOGIN, user=admin, ip=[::ffff:XX.XXX.XXX.XXX], port=[42130], protocol=IMAP

A recent LifeHacker article got me interested in anki, a cross-platform Simple Repetition System (SRS) or flash-card app.

anki allows users to create or import modular "decks" of "cards" that they can use to study whatever it is that they want to study. Which is, in and of itself, pretty great: as soon as I learned about this, I got excited. My enthusiasm went on the wane, however, when I discovered that there was no deck for ancient Latin (the casual study of which has long been a hobby of mine).

So I decided to create one by scraping the Latin dictionary at http://classicsunveiled.com/. The details of the scrape and the import of the entries into a Postgres database are relatively uninteresting: there are also hundreds of good tutorial on BeautifulSoup and psycopg2 out there and the world obviously has no use for another one from a self-proclaimed n00b like me.

What the world might find useful, however, is the ancient Latin deck I created. Here's a step-by-step* on how to get it going:

1. Download the .txt file from my personal blog: http://demongin.orghttp://s3.amazonaws.com/almosteffortless/ankiLatin.txt
2. Start anki.
3. Click "File" -> "Import"
4. Click "Choose file..." and browse for ankiLatin.txt.
5. Once you've selected the file, everything should be ready to go: click "Import" (to the right of the "Field mapping" controls), give the program a second to import the deck, select close and get busy.

* NB: these instructions are based on the Debian package version of anki (i.e. 0.9.9.4).

I work mostly on Debian systems (workstations and servers). Recently, during a lull, I decided to throw caution to the wind and go ahead with a reasonably large (<= 1000MB worth of new and upgrade packages) dist upgrade on my personal workstation at the office. Everything went pretty well (as well as could be expected, anyway) and my post-upgrade reboot was mostly uneventful.

OpenVPN, however, did not start. And, not to besmirch the character of its developers or its package maintainers ("it's not you; it's me"), I wasn't exactly surprised. OpenVPN is definitely on my "usual suspects" list. And if you run Debian testing or sid, it should be on yours as well.

And if it's going to be on your "usual suspects" list and be one of the first apps that you expect to fail, you ought to have a trouble-shooting routine. Or at least an idea of where to start.

Generally speaking, when my OpenVPN fails to initialize and connect during boot, I see the following when I tail /var/log/openvpn.log:

Mon Jan 12 11:33:31 2009 Note: Cannot open TUN/TAP dev /dev/net/tun: No such file or directory (errno=2)
Mon Jan 12 11:33:31 2009 Note: Attempting fallback to kernel 2.2 TUN/TAP interface
Mon Jan 12 11:33:31 2009 Cannot allocate TUN/TAP dev dynamically
Mon Jan 12 11:33:31 2009 Exiting

And, honestly, this is the error you're most likely to receive following an kernel upgrade: something will go "kablooie" as the various new modules are initialized, your dev node won't be created and you'll end up without support for the virtual device that will allow you to create your tunnels.

First things first, stop and start OpenVPN with its init script and tail the logs, just to eliminate the possibility that cosmic rays struck your machine during boot and the failure was a fluke:

gonzo:/etc/openvpn# /etc/init.d/openvpn stop
Stopping virtual private network daemon:/etc/init.d/openvpn: line 68: kill: (3047) - No such process
 client.
gonzo:/etc/openvpn# /etc/init.d/openvpn start
Starting virtual private network daemon: client.

If you wind up with the same failure message, it's time to look for modules.Do one of these:

# lsmod |grep tun

If you don't see something like the following, odds are good that your module just plain didn't get initialized:

tun                     8292  1

You'll want to double-check that, of course, by grepping your dmesg. If you don't see something like the following in your dmesg boot log, then it's probably time to take the action recommended below:

gonzo:/etc/openvpn# dmesg |grep tun
[   19.292084] hda: host max PIO4 wanted PIO255(auto-tune) selected PIO4
[   19.292466] hdb: host max PIO4 wanted PIO255(auto-tune) selected PIO4
[   20.700088] hdc: host max PIO4 wanted PIO255(auto-tune) selected PIO4
[  492.481978] tun: Universal TUN/TAP device driver, 1.6
[  492.481989] tun: (C) 1999-2004 Max Krasnyansky 

If this is the case, you might considering adding the string "tun" to your /etc/modules file and trying a manual initialization of the module:

# modprobe tun

If, however, module initialization isn't your problem, you might be experiencing some shenanigans with the population of your /dev directory. If the "tun" module appears to have been initialized successfully, go ahead and ls -l for the node:

# ls -l /dev/net/tun

If this comes back "No such file or directory" your best bet for a quick fix is (assuming that /dev/net already exists) to handle the situation cowboy style and make the dev node yourself:

# mknod /dev/net/tun c 10 200

That should get you a node that looks like this:

crw-r--r-- 1 root root 10, 200 2009-01-12 11:37 /dev/net/tun

And once you've got that, issue a quick stop and start of OpenVPN via init script and tail that /var/log/openvpn.log file one more time; you should be past the tunnel creation process.

...and on to the next catastrophic failure.

/proc is, for anyone interested in what's happening under the hood, an endless source of awesomeness. I recently experienced a bit of /proc awesomeness when sitting down to refactor an old VPN monitoring project using python.

Long story short, OpenVPN (http://openvpn.net/) prints a log called "openvpn-status.log" at fixed intervals. Contained within this log are the MAC addresses and common names of the VPN clients currently connected to the VPN server. The goal of the project was to take that log, swap the MAC addresses for local IP addresses and reprint the log on an intranet webpage.

The old way of doing this involved pinging all of the IP addresses on our subnet within the range of addresses that we had set aside for VPN clients and then scraping the output from # /usr/bin/arp -a to assign MAC addresses to IP addresses. This was alright, but in my program it required making a system (or, more specifically, a subprocess) call. Not optimal.

The new way of doing this is via the new awesomeness: /proc/net/arp. The setup is the same: you ping all the IP addresses in the range alotted for VPN clients to fill the arp cache but then, instead of working with the output from the arp program, you simply create a file-like object /proc/net/arp and work with that much simpler, easy to manipulate output.

To illustrate the difference, allow me to cut and paste both outputs. /usr/bin/arp -a first:

virginia:/var/log/openvpn# arp -v -a
? (192.168.2.89) at  on br0
? (192.168.2.85) at  on br0
? (192.168.2.82) at 00:FF:49:56:76:30 [ether] on br0
? (192.168.2.87) at  on br0
? (192.168.2.83) at 00:FF:2D:53:53:20 [ether] on br0

Now take a gander at what /proc/net/arp spits out:

virginia:/var/log/openvpn# cat /proc/net/arp
IP address       HW type     Flags       HW address            Mask     Device
192.168.2.85     0x1         0x0         00:00:00:00:00:00     *        br0
192.168.2.89     0x1         0x0         00:00:00:00:00:00     *        br0
192.168.2.82     0x1         0x2         00:FF:49:56:76:30     *        br0
192.168.2.87     0x1         0x0         00:00:00:00:00:00     *        br0
192.168.2.83     0x1         0x2         00:FF:2D:53:53:20     *        br0

Basically the same thing, but with /proc/net/arp you've got the option, if you're working with python, of simply creating a file-like object and working with that (instead of making a system call and having to strip off those parentheses):

f = file("/proc/net/arp")

From there, a quick list comprehension will get you all the data you need in a single, easy-to-use list:

output = [line.strip().split() for line in f.readlines()]

Two lines of code and you're ready to throw the data you need into a dictionary and do whatever it is you need to do.

Finally, the awesomeness isn't limited to python: when I'm trying to track down switches or other devices on the network that might not necessarily want to be found, all I've got to do to is rattle off the following bas one-liner and I've got all the data I need to start poking around:

cat /proc/net/arp |awk '{print $1 " " $4}'

So recently, while working with an ancient, hastily written calendar application, it fell to me to pry a decade's worth of data loose from a .mdb file.

And while the original specifications for the project only called for a csv for each of the database's tables, scope began to creep as the client realized that he needed his data to be reformatted and, in some cases, recast. Due to the persnickety nature of this client horrifyingly random nature of his data (some dates were MM/DD/YY while others where a legit timestamp; some commas were escaped, others were not, etc.), I realized that I was going to have to implement an industrial strength solution: something that would scale. So rather than writing a series of one-off python scripts (using the totally kickass csv module), I decided to get the data into a Postgres database and then query this database as necessary.

First, I'll lay out (most of) my program and then I'll do a blow-by-blow, dwelling briefly on certain important or noteworthy parts.

Remember, what we're doing here is grabbing one table at a time and stuffing it into a Postgres database (and maybe doing a little string sanitizing and type casting as we go). Anything more automated might not work, given the totally messed-up nature of the data and of the .mdb format in general.

Here goes:

#!/usr/bin/env python

from cStringIO import StringIO

import csv, os, psycopg2, re, subprocess, sys, time

dbHost = "localhost"
dbUser = "toconnell"
dbPass = "XXXXXXXXXX"
dbName = "toconnell"

# This program is run from the CLI: arguments one and two are the .mdb file
# and the table in that database that we're trying to import
mdbFile = sys.argv[1]
mdbTable = sys.argv[2]

mdbDump = "/usr/bin/mdb-export"
# Dynamically name the table, depending on the day the import is run
tableName = "%s_%s" % (os.path.basename(mdbFile.replace(".","_").lower()),time.strftime("%Y_%m_%d"))

def dumpMDB(mdbFile):
    command = [mdbDump,mdbFile,mdbTable]
    p = subprocess.Popen(command,stdout=subprocess.PIPE)
    mdbData = [line for line in p.stdout.readlines()]

    # Grab the column names
    columns = mdbData.pop(0)

    # Now write a CSV to the buffer
    for line in mdbData:
        tmpFile.write(line)

    # Now use the csv module to make a list where each line is a list
    tmpFile.seek(0,0)
    importData = csv.reader(tmpFile)
    dataList = []
    dataList.extend(importData)
    tmpFile.close()

    return columns,dataList

def createTable(columns):
    # Take this string from the pretend CSV file, break it up and remake it as something like a query
    columnsList = []
    for item in columns.split(","):
        item = item.strip().lower()
        # Also do a little type casting in there (beats having to do it later)
        if item == "date":
            columnsList.append(item + " TIMESTAMP")
        else:
            columnsList.append(item + " TEXT")

    columns = ",".join(columnsList)
    print columns

    # Now connect and do table stuff
    conn = psycopg2.connect("dbname=%s user=%s host=%s password=%s" % (dbName,dbUser,dbHost,dbPass))
    cursor = conn.cursor()

    # Check for previous tables:
    cursor.execute("SELECT * FROM pg_tables WHERE tablename LIKE '%%%s%%'" % tableName)
    results = cursor.fetchone()

    if results != None:
        # Drop previous ones:
        cursor.execute("DROP TABLE %s" % tableName)
    else:
        # Create a new one:
        cursor.execute("CREATE TABLE %s (%s)" % (tableName,columns))
        conn.commit()

def populateTable(columns,query):
    conn = psycopg2.connect("dbname=%s user=%s host=%s password=%s" % (dbName,dbUser,dbHost,dbPass))
    cursor = conn.cursor()
    cursor.execute("INSERT INTO %s(%s) VALUES(%s)" % (tableName,columns,query))
    conn.commit()

def sanitize(query):
    # You can do as many of these as necessary: I've included a single one as an example
    finalList = []
    for item in query:
        item = item.replace("'","\\'")
        finalList.append(item)
    return finalList

if __name__ == "__main__":
    # First, instantiate our pretend cvs file
    tmpFile = StringIO()

    # Now get the data
    columns,dataList = dumpMDB(mdbFile)

    # Now reformat the "columns" string a little bit
    columnsList = [item.strip().lower().replace(" ","") for item in columns.split(",")]
    columns = ",".join(columnsList)

    # Now create a new table (delete the previous one)
    createTable(columns)

    # Finally, insert data, reporting progress on the CLI (better than doing
    # select count(*) every 15 seconds)
    total = len(dataList)
    n = 1
    for item in dataList:
        item = sanitize(item)
        formatQuery = "'" + "','".join(item) + "'"
        populateTable(columns,formatQuery)
        print "Inserting data %s/%s..." % (n,total)
        n += 1
    print "Done."

In my opinion, there are two noteworthy aspects of the above: the use of the csv module and the use of cStringIO to create a csv file in the buffer (instead of simply creating a file and deleting it).

In order to better explain those two aspects of the program, here's that function again, broken into more easily digested pieces and presented with an in-line, blow-by-blow commentary:

def dumpMDB(mdbFile):
    command = [mdbDump,mdbFile,mdbTable]
    p = subprocess.Popen(command,stdout=subprocess.PIPE)
    mdbData = [line for line in p.stdout.readlines()]

Nothing special here: I do some pretty standard subprocess syntax that executes mdb-export (a part of the mdbtools package on Debian) on my file and dumps a table in csv format, one line at a time, into a list. Normally the program would just dump them to stdout: I'm just grabbing them up with subprocess's Popen function.

    # Grab the column names
    columns = mdbData.pop(0)

Now I do a list.pop(0) on this list to get the first line (i.e. the column names) of what I just dumped with mdb-export; using the built-in pop function without an integer gets you the last item in your list (in 2.5, at least).

    # Now write a CSV to the buffer
    for line in mdbData:
        tmpFile.write(line)

    # Now use the csv module to make a list where each line is a list
    tmpFile.seek(0,0)
    importData = csv.reader(tmpFile)
    dataList = []
    dataList.extend(importData)
    tmpFile.close()

    return columns,dataList

This isn't fancy code. I'm essentially creating a CSV file and reading each of its lines into a list. I do this instead of simply using the list I generated with the subprocess call because I'm nervous about splitting the strings that compose that list: as I mentioned above, there is a very real possibility that a given string will contain unescaped commas and other inappropriate characters within it and I don't even want to have to contemplate how to split those strings correctly.

What I want to do is trust the csv module to figure that out for me.

So, in order to make that I happen, I need to give the csv module a CSV file it can read. Rather than writing a file to the filesystem (which no self-respecting sysadmin will do if he can help it), I need to make some cStringIO magic happen. And in order to make that buffer magic happen, I need to do three things:

1. import cStringIO,
2. instantiate the file-like object and
3. do that weird seek to it.

The import (

from cStringIO import StringIO

) is fairly simple; if you've ever used cStringIO for anything, you've done this. The instantiation (

tmpFile = StringIO()

) is also straight out of the documentation.

But once I write all of the lines of my list to the cStringIO buffer, I've got to do

tmpFile.seek(0,0)

in order to then read from that file-like object with python's built-in csv module. There are a number of obscure/arcane reasons for this, but, as I understand it, the seek(0,0) is necessary because, without it, your file-like object is merely a collection of strings and not file-like in the way that the csv module expects it to be file-like: once you do the seek, you're ready to "read" your file-like object with the csv module.

From there, it's a simple matter of using the reader() function of the csv module to get your CSV data from your cStringIO object, manipulating it however you see fit and then looping over it and inserting it in your database.

And then never, ever having to work with (i.e. around) MS Access again.

We're finally (and reluctantly) ditching Windows 2000 for Windows XP Pro at the office. It has fallen upon me to put together a custom unattended (http://unattended.sourceforge.net/) solution.

For those who aren't familiar with the concept of an unattended installation, here's the gist:

1. You're in an environment with mixed hardware that requires all users to have a nearly identical workstation environment.
2. If you had dozens of identical machines, you'd simply get all your software together, install it on a master machine, configure all of your settings, create an image based on that machine, burn an acronis (http://www.acronis.com/)boot disk or two and start cranking out clones; as this is not an option due to your mixed machinery, you've got to actually install all that software and set all those settings.
3. To do this, you use a project like unattended, which automates the installation of the OS, all additional software and allows you to run scripts as necessary during the installation of the OS and the additional softs.

So far, so good.

Problems start to arise when you realize that the unattended project's greatest strength is its greatest weakness: it is extremely flexible and customizable because it does everything in the most "vanilla" manner possible and leaves everything beyond the basics of installing your Microsoft OS and ActiveState perl up to you.

How to actually configure an unattended solution is not, however, what I'm here to discuss. What I've got today are some tips and tricks for automating tasks on Windows XP that I've picked up as I work on my own unattended solution. They concern changing the login style, stopping and disabling services and removing Windows Components.


1.) Automatically disabling the "Welcome Screen"

This, for better or for worse, is a simple registry hack. Create a file called, let's say, winXPfixLogon.reg that looks like this:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon]
"LogonType"=dword:00000000

That "LogonType" dword may not be present by default; adding (or modifying) it will disable the "Welcome Screen" and allow you to use the "classic" logon (i.e. log onto the system with some dignity).

In order to automate this, simply stick a line in a batch file that looks like this:

regedit /s z:\scripts\winXPfixLogon.reg"

and you're ready to roll.


2.) Stopping and/or disabling services

There are two Windows command line tools that you can use to quickly stop and disable a service. "wscsvc" is the name of the hyper-annoying "Security Center" (a.k.a. the red-shield of nagging). To stop it and disable it (i.e. prevent it from starting automatically), all you've got to do is holler thus:

net stop wscsvc
sc config wscsvc start= disabled

Pay special attention to the spacing on that one: the single space after the "start=" is apparently make-or-break.


3.) Automatically removing Windows Components

There are few things as genuinely shitty as the cruft and useless/broken software with which Microsoft has chosen to clog their UI. Much of this bloat cannot be completely removed; Internet Explorer, Outlook Express and Windows Media Player are not going anywhere (unless you plan to avail yourself of the substantial and psychotic registry and filesystem hacks that generally won't pass muster at the office).

Those annoying components can, however, be "disabled" or made invisible to users. With a batch line like this:

SYSOCMGR.EXE /i:%windir%\inf\sysoc.inf /u:z:\install\scripts\disabled_components.txt

and a disabled_components.txt that looks like this:

[components]
freecell=off
hearts=off
minesweeper=off
pinball=off
solitaire=off
spider=off
zonegames=off
vol=off
MSNexplr=off
deskpaper=off
OEaccess=off
IEaccess=off
WMPOCM=off

you can "remove" (i.e. sort of disable) a lot of those annoying "components".

In the above excerpt, most of the names are fairly obvious: the last three are, in order, Outlook Express, Internet Explorer and Windows Media Player. Also in that list are the so-called "Internet Games" (zonegames).

Obviously this is just a light scratching of the surface: what I've laid out above is just the beginning of numerous modifications, deletions and emendations required to make Windows XP a functional operating system (apps like daemontools, cwRsync and certain others that partially un-cripple Windows all come to mind).

Feel free to share your favorite Windows XP mass-administration tricks in the comments. God knows I could use the help.

When I say, "If you don't know the basics of pgrep and pkill, you really need to make it a priority to learn them", what I really mean is, "I just recently learned how to use pgrep and pkill myself and I'm really, really excited about them."

Whether you're automating processes on the server across the Internet or you're trying to wrangle the processes on the client right in front of you, there really is no classier or more efficient way to do it than with these two programs. What follows are two examples of how, using nothing more than some elementary bash, you can quickly retrieve PID numbers, paths to processes and basically micromanage your system into submission.

The first thing to get down with is the l flag. It'll give you the PID as well as the process name. This is useful if you've got processes that are related, but that don't have exactly the same name:

toconnell@kumiko:~$ pgrep -l thunderbird
7425 thunderbird
7433 thunderbird-bin

The f flag is kind of like the industrial strength version of this. When you throw down the f and the l together, you get a PID and an absolute path:

toconnell@kumiko:~$ pgrep -lf thunderbird
7425 /bin/sh /home/toconnell/thunderbird/thunderbird
7428 /bin/sh /home/toconnell/thunderbird/run-mozilla.sh /home/toconnell/thunderbird/thunderbird-bin
7433 /home/toconnell/thunderbird/thunderbird-bin

If you've been paying attention, you'll notice that I got three results from my second command. This is because the f greps against the full path of all running processes. This can be a real life saver when you've got (as above) processes spawning other processes or when you've got to start one process to start another. I frequently find myself doing this with .jar files.

For example, I use tn5250j (an old-timey IBM "green screen" emulator) at work. Nothing against their project, but when I end my tn5250j sessions, I've found that some of the processes that it starts tend not to die completely. I've also found that If I want to look for stranded or undead processes with a simple pgrep, I get nothing. This is because when I start the emulator, my KDE shortcut uses the following syntax:

$ /home/toconnell/jre1.6.0/bin/java -jar /home/toconnell/tn5250j/tn5250j.jar

If I pgrep for java, of course, renegade tn5250j processes will turn up. But so will all my other active java processes:

toconnell@kumiko:~$ pgrep -l java
26100 java
26131 java

If, however, I use the f, and grep against the full path, I see that these are not both tn5250j processes:

toconnell@kumiko:~$ pgrep -lf java
26100 /home/toconnell/jre1.6.0/bin/java -jar /home/toconnell/tn5250j/tn5250j.jar
26131 /usr/bin/java -jar /usr/bin/tightvnc/classes/VncViewer.jar HOST victoria PORT 5900

If I use the f and use "tn5250" as my grep term, I only get the process I want.

Which is really handy if I've got a bunch of these things running around and I want them all dead. pkill will take an f flag and neatly dispose of all of my errant java processes. As a sort of "set it and forget it" method of keeping these things from stacking up, I've created a job in /etc/cron.daily/ that kills all of the stranded processes from yesterday before I even sit down at my desk:

#!/bin/bash
set -e
`which pkill` -f tn5250 > /dev/null 2&>1
exit

But this is only the beginning of the fun. pgrep and pkill are even more invaluable in an emergency, e.g. when you've got a bunch of users (probably system users (e.g. daemons), but maybe human ones too) running around and you only want to kill the processes that one of your users has started.

Let's say that I've got two separate instances of postgres running, one belongs to the user postgres81, the other belongs to postgres83, and the new one happens to be running amok. You're desperate enough to restore order that you're ready to break Rule Number One and kill the postmaster.

The fastest way to put a bullet in your malfunctioning database processes without bothering any of your stable processes?

# pkill -f -u postgres83

Full disclosure: I'm no postgres expert. In fact, I'm pretty much a total noob myself.

Which, ironically, makes me uniquely qualified to write the Postgres tutorial that follows. An old hand is exactly the guy who you don't want to write a guide about common newbie problems because, as the old writers' saw goes, you do your best writing when you write what you know.

And when I set out a few days ago to crank out a backup script for my postgres databases, I knew precious little.

When I began, I had two databases on two servers that were only getting backed up when the whole filesystem got backed up: MySQL is still my bread and butter--it powers most of the applications I support--and while I had a robust enough scheme for making, date-stamping and backing up dumps from my various MySQL databases, I had nothing of the sort in place to protect my Postgres data. And while this was acceptable (due, primarily, to the non-essential nature of the data contained within those databases), it was far from optimal.

I ended up writing my program in python, but what follows will be language-neutral: I plan to mostly talk about things that happen between you (the sysadmin) and your filesystem. I will stop to make some remarks at the end that will be python-specific, but the main points I have to make are more widely applicable in that all you need to know to make use of them is a little bit of bash.

1. Selecting all Databases

Right off the bat, if you're going to be backing up multiple databases on multiple servers, you're going to need a way to automate the query by which you determine which databases you're going to need to dump: every whole-database backup starts with a list of databases.

In MySQL, this is drop-dead easy: a quick "show databases" will get you all the information you need. In Postgres, the pg_database table is going to be your friend. Connect as the postgres user (or someone with equivalent access) and take a look at it: it's chock full of the Good Stuff That Kids Go For.

When you're writing your automated backup, you can retrieve a list of all of your postgres databases thus:

select datname from pg_database;




2. Database "root" user

It is probably an incredibly bad habit that I should not spread to others (but hey: that's what newbies do!), but I like to create a "root" user in my postgres databases. It's an old habit, one of those ones that dies hard, from my MySQL days and, even though I know it's wrong, it makes administering my postgres databases a little bit easier: rather than doing # su postgres -c psql and acting as the built-in postgres user whenever I need to shuffle things around in my database, I just keep a .pgpass (more on that in a second) in my UNIX root's homedir and do a quick # psql whenever I need to get into my database.

It also comes in handy to have "root" user in your database when you're automating backups: you can execute your backup script as UNIX root, execute the necessary database functions without switching users and so on.

In order to create this guy and give him the appropriate permissions, you can toss off the following easy-peasy bash, enter a password and be done with it:

# su postgres -c "createuser -daP root"




3. .pgpass

The final thing that will save you some time when you're whipping up an automated database dumping scheme from scratch is know how to work a .pgpass file. You can (and probably should) read the full documentation here, but what it boils down to is summarized in the following bullet points:

1. The file belongs in the home directory of the UNIX user who will be doing the backup.
3. The file must not be world- or group-readable.
4. The basic syntax is dbhostname:port:database:dbusername:password
5. You can substitute an asterisk (i.e. wildcard) for any of those values except "dbusername" and "password".

So basically, since your automatic backup will probably be executed by your (UNIX) root user, you're going to want to whip up a .pgpass in his homedir with a little bash like this:

# echo "localhost:5432:*:root:xxxxxxxxxx" > .pgpass

Then you're going to modify the permissions on that file like this:

# chmod 0600 ~/.pgpass

And that's that. Your access and permissions are good to go and you're ready to use the scripting language of your choice to set up an automatic backup.

And when you do get around to scripting that backup, you're probably going to want to gzip or bzip your dumps: as most backups end up traveling across some kind of network, it helps to have them be as compressed as possible. The simplest bash one-liner for gzipping up your SQL dumps is probably the one where you throw them all into one big archive like the man page recommends:

cat database1.sql database2.sql | gzip -9 > databaseDump.gz 

But when I wrote my program (in python), I did the actual dump commands with subprocess and created individual archives for my dumps. Here's a snippet:

outputFile = open(dbName+todaysDate+".sql.gz", "wb")
command1 = ["/usr/bin/pg_dump",dbName]
command2 = ["gzip","-9"]
p1 = subprocess.Popen(command1, stdout=subprocess.PIPE)
p2 = subprocess.Popen(command2, stdin=p1.stdout, stdout=outputFile)
p1.wait()
p2.wait()
outputFile.close()

This, as I mentioned, leaves you with a collection of individual archives that you can then stash in some place safe until some programmer's hasty application deploy nukes yesterday's production data and you get to play hero.