Stupid Linux Tricks: Basic Server Hardening (Debian Lenny)

Posted by Timothy O'Connell in General on May 15, 2009

Due to recent events involving some computers I administer, I've become very interested in security. Basically, I've dodged enough bullets thanks to little more than beginner's luck and I figure that it's about time to take responsibility for the safety and security of my computers.

Since I was interested in hardening up two Debian boxes running Lenny, I started off by taking a look at the Securing Debian Manual, this very helpful page on LQ and the results of a tiger audit.

Do this to generate a tiger audit of your server:

# aptitude install tiger
tiger -E

The "-E" gets you what's called an "explanation report", which will be useful in helping you understand what can be some fairly cryptic output.

Additionally, the package version of tiger comes with some nice default settings for the main executable and for tigercron, which, as you might imagine, runs some minor scans on a pre-defined schedule.

At any rate, once I had my audit and had picked up a few bright ideas from the SDM, I made a number of changes to all of my web-facing production machines. What follows are some things that you might want to consider doing on your Debian Lenny servers:

Users and Permissions

  1. Password Audit: first, I decided to get to know my user accounts a little better. This meant running john (formerly "john the ripper", a password cracker that reads hashed passwords and tries to decipher them) against my /etc/shadow to see who was using dictionary-based passwords and who was using other types of insecure passwords:
    # aptitude install john
    # john /etc/shadow 

    This took a while to run--a little over a day, but I had it niced pretty high--but of the 10 user accounts it cracked, it was good to know which ones were using hilariously insecure passwords and which ones probably weren't going to cracked by your garden variety brute force password cracker.

  2. NB: if you run john against your /etc/shadow and realize that you've got a problem child on your hands, there's always chroot. Here is a really good how-to on chroot-jailing a user.

  3. The Prunening: odds are, if you've been living on a system for more than a few months, you've accumulated some users (either from software that you've installed and then removed or by meeting user/developer needs, etc.) that aren't doing anything. One of the basic tenets of server security is having the smallest amount of users with the least amount of access to the smallest number of programs possible.

    In some environments, you've simply got to have a bunch of users in your /etc/passwd. In most situations, however, it makes good sense to just hit all the derelicts with a userdel and only having to worry about angry users not having enough access (rather than having to worry about unauthorized users having too much access).

Keeping Script Kiddies Under Control

In my (admittedly limited) experience, the most trouble you're going to run into from script kiddies are anonymous, unfocused attacks that attempt to gain access to your machine via either a.) the /tmp directory, b.) DoS-based exploints or c.) application attacks like SQL injections, XSS or directory traversal attacks.

Since guarding against application attacks is something that programmers are supposed to be handling, I decided to focus on stopping /tmp abuse and trying to stymie DoS attacks.

  1. Mounting /tmp with noexec: in this age of VPSes and shared hosting, it's more often the case than not that you won't get to decide how your machine is partitioned. If, like me, you live on Slicehost and you're running Debian, your partition scheme looks like this:
    lana:/# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda1              19G  9.0G  8.9G  51% /
    tmpfs                 256M     0  256M   0% /lib/init/rw
    udev                  256M   20K  256M   1% /dev

    What this means is that you've got your /tmp directory on your / partition. Which partition is, by necessity, allows files to be executable.

    And that, as I have learned (the hard way), is bad news. What this means is that, you've got a directory on your / partition--the partition where all your apps and data probably live--that is writable/readable by every Li, Ivan and Harry from Taipei to Yaktusk. And if one of the thousands of bots who knocks on your door every month knows how to write to /tmp and you don't catch him right away, it's pretty much game over for your TLD or your IP Address: one day, you'll wake up and find that your info is on every spam list on the Internet because your server has been under remote control via IRC for the last three weeks and now your full time job is trying to get your info off of those lists while planning a full OS re-install.

    Ideally, you would be able /tmp its own partition and mount that partition with noexec. And while it would be optimal, it sometimes isn't an option if you're a part of the VPS set.

    And if you can't control your installation or maybe you just can't take down a production server, what you can do is warn your users/developers that you're about to cause a little temporary chaos (get it? temporary chaos?), move your existing /tmp to some place else and create a small filesystem that you can mount noexec to use as /tmp. On Debian Lenny, that would look approximately like this:

    gonzo:/# mv /tmp /old_tmp
    # dd if=/dev/zero of=/.tmpfs bs=1024 count=1000000
    1000000+0 records in
    1000000+0 records out
    1024000000 bytes (1.0 GB) copied, 29.1302 s, 35.2 MB/s
    gonzo:/# mkfs.ext3 -j .tmpfs
    [...]
    gonzo:/# mount -o loop,noexec,nosuid,rw /.tmpfs /tmp/
    gonzo:/# mv /old_tmp/* /tmp/.

    Et voila! You've got a 1GB "drive" that's mounted noexec at /tmp that's ready to roll out. Any attempts to execute anything on that partition will result in a bad interpreter error. Check it:

    gonzo:/# ls -l tmp/
    total 20
    -rwxr-xr-x 1 root root    37 2009-05-15 14:54 executeMe.py
    drwx------ 2 root root 16384 2009-05-15 14:51 lost+found
    gonzo:/# test/executeMe.py
    bash: test/executeMe.py: /usr/bin/env: bad interpreter: Permission denied

    All you've got to do now is add that mount info to your fstab and you're ready to start sleeping at night again:

    /.tmpfs           /tmp            ext3    loop,nosuid,noexec,rw  0      0

    NB: don't forget that /tmp wants to have the sticky bit turned on (i.e. be chmoded to 1777). Also don't forget to make /var/tmp a symlink that points to /tmp.
    Also: props to Vincent Danen's post on /tmp at TechRepublic for the idea.

  2. mod_evasive to Prevent DoS: after a recent DoS experience, I decided to install Apache mod_evasive to reduce the risk of getting flat-lined/broad-sided by DoS/DDoS attacks:
    # aptitude install libapache2-mod-evasive

    The beauty of using packaged software is that that's really all you have to do: apt will copy the files, create the symbolic links and restart apache for you. Nice.

  3. sysctl real-time kernel Modifications: Additionally, you might also want to use a slightly obscure command called sysctl (which modifies kernel perameters while the kernel is running, so consider yourself warned) to take a precaution against a DoS tactic called "syn flooding":
    # sysctl -w net.ipv4.tcp_syncookies=1

    This is a sort of controversial measure--apparently it defies some RFC docs for TCP/IP--but setting tcp_syncookies to False has yet to have affected any of my computers.

    There are a number of additional security features you can activate with this command; Google it and prepared to be awed by some of the features of your OS you can control in real-time with sysctl.

For the Tin-foil Hat Crowd

What follows are non-specific countermeasures and settings that, while obscure and probably unlikely to save you from becoming an unwitting member of some Russian bot master's herd, might help you feel more secure in the knowledge that even if someone does get non-root shell access, he's probably not going to be able to do too much damage.

  1. Add /usr/bin/mesg n to root's .bashrc File:
    executing mesg n on log in, prevents an admittedly rare exploit through which other users can execute arbitrary code as root by sending messages to his terminal.
  2. Modify /etc/inittab to Prevent Non-root Users from Rebooting the System with ctrl+alt+del. In the stock /etc/inititab on Debian Lenny, you've got this line:
    # What to do when CTRL-ALT-DEL is pressed.
    ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -r now

    Which is hardly optimal: this lets pretty much anyone who figures out a way to execute programs in /etc/sbin reboot the system. I changed it to:

    ca:12345:ctrlaltdel:/bin/false
  3. SMTPD Settings If you run postfix, you should probably check up on your relay settings and update your external blacklist providers if you haven't done it in a while. Your mail server is the world's first line of defense against everything from phishing/spear-phishing to headline-making super worms:
    smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks,
      reject_unauth_destination,
      reject_unauth_pipelining,
      reject_invalid_hostname,
      reject_rbl_client sbl-xbl.spamhaus.org,
      reject_rbl_client cbl.abuseat.org,
      reject_rbl_client bl.spamcop.net,
      reject_rbl_client zen.spamhaus.org
    smtpd_helo_required = yes
    disable_vrfy_command = yes
    
    smtpd_data_restrictions =
                reject_unauth_pipelining,
                permit
    

    That smtpd_help_required line might not seem like anything special, but I have a script that parses /var/log/mail.log output and, when you do get a spammer that responds to the helo request, a lot of times he'll come back with his actual domain.

    It's fun for research/study/personal amusement reasons, basically.

While some people might say that the above is overkill--that it's just not worth the time and effort to audit and harden at this level--but I'd say that this level of focus on security isn't so much "overkill" as it is "a pretty good start."

Because my thinking is that if you've got the root password, you're probably already the anxious type. And at the very least, being a little bit OCD about security on your all-important, mission-critical application servers might help you feel a little less anxious. Which is definitely worth the effort.

4 Comments

I realized, shortly after I added the bit about

/usr/bin/mesg n

to root’s .bashrc that I was getting mysterious errors from non-interactive processes.

Namely, when I ran rsync via cron, I was getting emails from cron that looked like this:

stdin: is not a tty
stdin: is not a tty
stdin: is not a tty

It didn’t take long for me to put two and two together and realize that this was output from that mesg command: it couldn’t do its thing because it didn’t have a tty on account of the session being a non-interactive one.

In order to silence that particular lamb, I adjusted the mesg command in root’s .bashrc:

if [ ! -f $USER ]
    then /usr/bin/mesg n
fi

It’s sort of ugly, but it gets the job done: if you’re starting an interactive session and you’ve got a $USER variable, then you get your mesg sent to n. If you’re not, then you don’t. Easy-peasy.

[...] almost effortless » Stupid Linux Tricks: Basic Server Hardening (Debian Lenny) – [...]

 vivek

If you mount /tmp with no-exec apt tends to fail since it unzips files in /tmp and runs postinstall from there. Are you sure about this?

 jalfrock

@vivek:

Add the following to your /etc/apt/apt.conf:
DPkg::Pre-Invoke{“mount -o remount,exec /tmp”;};
DPkg::Post-Invoke {“mount -o remount /tmp”;};

(This remounts /tmp exec before running dpkg, then re-remounts it noexec when it’s done.)

Leave a comment

WP_Big_City