Code

Poor Man’s Heroku Backups

Posted by Trevor in Code, Ruby/Rails on September 11, 2009

Here's a quick and dirty way to back up all of your Heroku-powered databases using their Taps gem with one easy command. It works well enough for my simple needs, but any improvements you've got would be most welcome!

Just add the following to your ~/.bash_profile, and you're good to go:

GitHub Protip: Follow other users

Posted by Trevor in Code on June 11, 2009

Inspired by this post, I thought I'd share a tip that helps me get the most out of GitHub.

Don't just follow the projects that you're interested in — follow other users. Here's a list of people that I'm following. They're constantly turning me on to new and interesting projects, because I get to see everything they're working on, and everything they're following.

 

Dig around the users that I follow, check out what they're been up to, and try it out. If you find that your feed becomes a bit much to manage, try subscribing to your personal RSS feed. There's a link on the home page when you're logged in.

Thanks, GitHub. You're the best.

Windows XP Unattended

Posted by Timothy O'Connell in Code on November 05, 2008

We're finally (and reluctantly) ditching Windows 2000 for Windows XP Pro at the office. It has fallen upon me to put together a custom unattended (http://unattended.sourceforge.net/) solution.

For those who aren't familiar with the concept of an unattended installation, here's the gist:

  1. You're in an environment with mixed hardware that requires all users to have a nearly identical workstation environment.
  2. If you had dozens of identical machines, you'd simply get all your software together, install it on a master machine, configure all of your settings, create an image based on that machine, burn an acronis (http://www.acronis.com/)boot disk or two and start cranking out clones; as this is not an option due to your mixed machinery, you've got to actually install all that software and set all those settings.
  3. To do this, you use a project like unattended, which automates the installation of the OS, all additional software and allows you to run scripts as necessary during the installation of the OS and the additional softs.

So far, so good.

Problems start to arise when you realize that the unattended project's greatest strength is its greatest weakness: it is extremely flexible and customizable because it does everything in the most "vanilla" manner possible and leaves everything beyond the basics of installing your Microsoft OS and ActiveState perl up to you.

How to actually configure an unattended solution is not, however, what I'm here to discuss. What I've got today are some tips and tricks for automating tasks on Windows XP that I've picked up as I work on my own unattended solution. They concern changing the login style, stopping and disabling services and removing Windows Components.


1.) Automatically disabling the "Welcome Screen"

This, for better or for worse, is a simple registry hack. Create a file called, let's say, winXPfixLogon.reg that looks like this:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon]
"LogonType"=dword:00000000

That "LogonType" dword may not be present by default; adding (or modifying) it will disable the "Welcome Screen" and allow you to use the "classic" logon (i.e. log onto the system with some dignity).

In order to automate this, simply stick a line in a batch file that looks like this:

regedit /s z:\scripts\winXPfixLogon.reg"

and you're ready to roll.


2.) Stopping and/or disabling services

There are two Windows command line tools that you can use to quickly stop and disable a service. "wscsvc" is the name of the hyper-annoying "Security Center" (a.k.a. the red-shield of nagging). To stop it and disable it (i.e. prevent it from starting automatically), all you've got to do is holler thus:

net stop wscsvc
sc config wscsvc start= disabled

Pay special attention to the spacing on that one: the single space after the "start=" is apparently make-or-break.


3.) Automatically removing Windows Components

There are few things as genuinely shitty as the cruft and useless/broken software with which Microsoft has chosen to clog their UI. Much of this bloat cannot be completely removed; Internet Explorer, Outlook Express and Windows Media Player are not going anywhere (unless you plan to avail yourself of the substantial and psychotic registry and filesystem hacks that generally won't pass muster at the office).

Those annoying components can, however, be "disabled" or made invisible to users. With a batch line like this:

SYSOCMGR.EXE /i:%windir%\inf\sysoc.inf /u:z:\install\scripts\disabled_components.txt

and a disabled_components.txt that looks like this:

[components]
freecell=off
hearts=off
minesweeper=off
pinball=off
solitaire=off
spider=off
zonegames=off
vol=off
MSNexplr=off
deskpaper=off
OEaccess=off
IEaccess=off
WMPOCM=off

you can "remove" (i.e. sort of disable) a lot of those annoying "components".

In the above excerpt, most of the names are fairly obvious: the last three are, in order, Outlook Express, Internet Explorer and Windows Media Player. Also in that list are the so-called "Internet Games" (zonegames).

Obviously this is just a light scratching of the surface: what I've laid out above is just the beginning of numerous modifications, deletions and emendations required to make Windows XP a functional operating system (apps like daemontools, cwRsync and certain others that partially un-cripple Windows all come to mind).

Feel free to share your favorite Windows XP mass-administration tricks in the comments. God knows I could use the help.

PostgreSQL Backup Tips for n00bz

Posted by Timothy O'Connell in Code on October 29, 2008

Full disclosure: I'm no postgres expert. In fact, I'm pretty much a total noob myself.

Which, ironically, makes me uniquely qualified to write the Postgres tutorial that follows. An old hand is exactly the guy who you don't want to write a guide about common newbie problems because, as the old writers' saw goes, you do your best writing when you write what you know.

And when I set out a few days ago to crank out a backup script for my postgres databases, I knew precious little.

When I began, I had two databases on two servers that were only getting backed up when the whole filesystem got backed up: MySQL is still my bread and butter--it powers most of the applications I support--and while I had a robust enough scheme for making, date-stamping and backing up dumps from my various MySQL databases, I had nothing of the sort in place to protect my Postgres data. And while this was acceptable (due, primarily, to the non-essential nature of the data contained within those databases), it was far from optimal.

I ended up writing my program in python, but what follows will be language-neutral: I plan to mostly talk about things that happen between you (the sysadmin) and your filesystem. I will stop to make some remarks at the end that will be python-specific, but the main points I have to make are more widely applicable in that all you need to know to make use of them is a little bit of bash.

  1. Selecting all Databases
  2. Right off the bat, if you're going to be backing up multiple databases on multiple servers, you're going to need a way to automate the query by which you determine which databases you're going to need to dump: every whole-database backup starts with a list of databases.

    In MySQL, this is drop-dead easy: a quick "show databases" will get you all the information you need. In Postgres, the pg_database table is going to be your friend. Connect as the postgres user (or someone with equivalent access) and take a look at it: it's chock full of the Good Stuff That Kids Go For.

    When you're writing your automated backup, you can retrieve a list of all of your postgres databases thus:

    select datname from pg_database;


  3. Database "root" user
  4. It is probably an incredibly bad habit that I should not spread to others (but hey: that's what newbies do!), but I like to create a "root" user in my postgres databases. It's an old habit, one of those ones that dies hard, from my MySQL days and, even though I know it's wrong, it makes administering my postgres databases a little bit easier: rather than doing # su postgres -c psql and acting as the built-in postgres user whenever I need to shuffle things around in my database, I just keep a .pgpass (more on that in a second) in my UNIX root's homedir and do a quick # psql whenever I need to get into my database.

    It also comes in handy to have "root" user in your database when you're automating backups: you can execute your backup script as UNIX root, execute the necessary database functions without switching users and so on.

    In order to create this guy and give him the appropriate permissions, you can toss off the following easy-peasy bash, enter a password and be done with it:

    # su postgres -c "createuser -daP root"


  5. .pgpass
  6. The final thing that will save you some time when you're whipping up an automated database dumping scheme from scratch is know how to work a .pgpass file. You can (and probably should) read the full documentation here, but what it boils down to is summarized in the following bullet points:

    1. The file belongs in the home directory of the UNIX user who will be doing the backup.
    2. The file must not be world- or group-readable.
    3. The basic syntax is dbhostname:port:database:dbusername:password
    4. You can substitute an asterisk (i.e. wildcard) for any of those values except "dbusername" and "password".

    So basically, since your automatic backup will probably be executed by your (UNIX) root user, you're going to want to whip up a .pgpass in his homedir with a little bash like this:

    # echo "localhost:5432:*:root:xxxxxxxxxx" > .pgpass

    Then you're going to modify the permissions on that file like this:

    # chmod 0600 ~/.pgpass

    And that's that. Your access and permissions are good to go and you're ready to use the scripting language of your choice to set up an automatic backup.



And when you do get around to scripting that backup, you're probably going to want to gzip or bzip your dumps: as most backups end up traveling across some kind of network, it helps to have them be as compressed as possible. The simplest bash one-liner for gzipping up your SQL dumps is probably the one where you throw them all into one big archive like the man page recommends:

cat database1.sql database2.sql | gzip -9 > databaseDump.gz 

But when I wrote my program (in python), I did the actual dump commands with subprocess and created individual archives for my dumps. Here's a snippet:

outputFile = open(dbName+todaysDate+".sql.gz", "wb")
command1 = ["/usr/bin/pg_dump",dbName]
command2 = ["gzip","-9"]
p1 = subprocess.Popen(command1, stdout=subprocess.PIPE)
p2 = subprocess.Popen(command2, stdin=p1.stdout, stdout=outputFile)
p1.wait()
p2.wait()
outputFile.close()

This, as I mentioned, leaves you with a collection of individual archives that you can then stash in some place safe until some programmer's hasty application deploy nukes yesterday's production data and you get to play hero.

Stupid Linux Tricks: debug your website like a sysadmin

Posted by Timothy O'Connell in Code on October 21, 2008

I recently had a conversation with a certain programmer friend who, having foolishly overwritten his Firefox 3.0x with an installation of the 3.1 beta, was unable to use any of his plugins. This wasn't a huge problem for him--i.e. he figured he could live without his plugins for a week or two--until he realized that he wasn't going to be able to use LiveHTTPHeaders (http://livehttpheaders.mozdev.org/).

The plugin, which can greatly expedite client-side testing, is not the only way to do that sort of testing. In a pinch, you can view HTTP headers on the command line simply by firing up a terminal session and using wget:

$ wget -S almosteffortless.com -O /dev/null

The majuscule "S" gets you the headers and the "-O /dev/null" dumps the file you're about to retrieve in the bit bucket (rather than into your home directory or where ever). The output you get looks something like this:

toconnell@lana:~/tmp$ wget -S almosteffortless.com
--2008-10-21 13:43:24--  http://tyrannybelle.com/
Resolving tyrannybelle.com... 208.78.99.165
Connecting to tyrannybelle.com|208.78.99.165|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Date: Tue, 21 Oct 2008 18:43:25 GMT
  Server: Apache/2.2.9 (Debian) PHP/5.2.6-5 with Suhosin-Patch mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 OpenSSL/0.9.8g
  X-Powered-By: PHP/5.2.6-5
  Set-Cookie: PHPSESSID=aefe500683f79ed067f91f76d3615d33; path=/
  Expires: Thu, 19 Nov 1981 08:52:00 GMT
  Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
  Pragma: no-cache
  Content-Length: 4932
  Keep-Alive: timeout=15, max=100
  Connection: Keep-Alive
  Content-Type: text/html
Length: 4932 (4.8K) [text/html]

You can see cookie info, cache and pragma values, Apache information and so on. Not too shabby.

I showed this to my pal and, though he was a little impressed, he wasn't entirely convinced that he was going to be able to do all of his client testing using only the CLI: "yes, that's all fine and good", he said, "but I usually need to right-click and 'View Page Source' in order to check on dynamically generated content."

"You're not going to want to hear this," I responded, "but telnet is your friend on this one."

toconnell@lana:~/tmp$ telnet tyrannybelle.com 80 

Will connect you to your website. All you've got to do at that point is manually GET your document. Let's say, for the example, that we just want our index. All you do, once you're connected, is

GET /

telnet will then proceed to make the request, dump the response onto your terminal as html and exit.

Granted, you don't get any syntax highlighting or fancy stuff, but if you're debugging from a mobile device, a computer without windowing software or in any other circumstance you can't use a proper browser, telnet is a great way to eyeball a website.

My friend admitted that this was, in fact, all right in a pinch, but it was plain from the look on his face that he still wasn't sold on ye olde command line as a testing/debugging tool. "What about layout?" he asked.

"Well yeah," I conceded, "there's really no good way to do that without a browser, but if you want to check on your tab order..."

And then, just as I was about to start in lynx, he said, "you know what? Forget it--I'm just going to go ahead and trash this beta install."

Stupid Linux Tricks: cron task @reboot

Posted by Timothy O'Connell in Code on October 15, 2008

A little known fact of cron is that you can use it to execute commands as soon as your machine returns from a reboot. If, for example, I create a file in /etc/cron.d/ called "whatever" that looks like this:

@reboot    root    `which sshfs` user@remotemachine:/mnt/data  /mnt/data

The above command will be executed once the system is fully booted. This is ideal for cases in which depend on a number of services or in which manual intervention is required. One could, for example, create a file called "whatever" in /etc/cron.d/ to remind him that he needed to manually intervene once the boot process was finished and that he was now free to do just that:

@reboot    root    echo "Don't forget the sshfs mount!" | mail -s "[`hostname`] Reboot!" root

There are a handful of other, similarly useful shortcuts that Vixie built into cron but that aren't particularly well known:

    @yearly        Run once a year, "0 0 1 1 *".
    @annually      (same as @yearly)
    @monthly       Run once a month, "0 0 1 * *".
    @weekly        Run once a week, "0 0 * * 0".
    @daily         Run once a day, "0 0 * * *".
    @midnight      (same as @daily)
    @hourly        Run once an hour, "0 * * * *".

(Excerpted from the man page)

Centering a Page with CSS

Posted by Trevor in Code on August 14, 2006

There's a very simple way to center a web-page horizontally, using only a smidgen of CSS:

body {
margin: 0px auto;
width: 760px;
}

k2 Images

Posted by Trevor in Code, Wordpress on June 02, 2006

I love k2 (the Wordpress theme, but I think some of the images are overkill. Luckily, there is an easy way to hide them using CSS. It's easy:

.commentslink, .chronodata, .tagdata,
.comments .commentlist .commentmetadata {
	background: none;
	margin-left: -20px;
}

Make a Div Into a Link

Posted by Trevor in Code on May 25, 2006

It's easy to make a div into a link using a bit of javascript. You can use this technique to make any div "clickable". For example, you might want your "header" div to link to your home page. Here's how:

<div onclick="location.href='http://www.example.com';" style="cursor:pointer;"></div>

If you're using a standard WordPress theme, you can make your header div clickable by doing this:

<div id="header" onclick="location.href='<?php bloginfo('url');?>';" style="cursor:pointer;"></div>

That's it!

Redirect all Traffic with htaccess

Posted by Trevor in Code on December 26, 2005

Moving a site to a new location can mean leaving a lot of broken links behind. However, you can use the "magic" of htaccess to redirect all traffic to your new site. Simply create a new text file in your favorite editor and paste in the following line:

ErrorDocument 404 http://www.example.com/

Of course, you'll have to replace the "example.com" address with whatever page you're trying to redirect the visitor to. Then, all you have to do is save the file as ".htaccess" and upload it to the folder where your site used to be. That's it! You can read more about htaccess if you like, but it's pretty heady stuff, I must say.

Wimp Media Player

Posted by Trevor in Code on November 07, 2005

An oldie, but a goldie. One of my first major undertakings of computer code. Wimp Media Player is a handy little skin for Windows Media Player 7 and above. Our little friend Wimp's sole purpose in life is to make grumpy ol' Windows Media Player move just a little bit quicker. I think Wimp is a fine little piece of work, and I recommend it highly - even to this very day.

Wimp Media Player

Continue reading...

Secure Files

Posted by Trevor in Code, Wordpress on September 22, 2005

Announcing the very first almost effortless plugin: Secure Files. This plugin allows you to upload and download documents that are stored outside of your web document root for security purposes.

This is great for people using plugins like Registered Only, which you can read about here, that secure your Wordpress content (Posts, Pages, etc).

The problem with this is that your loose files arent protected - only the contents of your Wordpress database. So, any images or other documents you've uploaded are easily accessible to those who aren't authenticated via a plugin like Registered Only. This plugin aims to solve this problem.

Click here to Read More »