Config vars and Heroku

Posted by Trevor in El Dorado, Ruby/Rails on June 25, 2009

I don't really care for the suggested approach in the Heroku docs for setting configuration variables locally. I have an open-source project that I'm working to get onto Heroku, so I decided to do a little work to come up with a solution that I prefer. I think this would work well for open source projects, as well as projects with multiple developers.

Here's the basic idea:

You have a config file that contains all of your local configuration variables. It looks a lot like database.yml.

 
# config/config.yml
 
development:
  session_key: example_development
  session_secret: ESl6X3oKM1i1RRrD2QLwUUzz9jr1zxNO
  domain: http://example.com
 
test:
  session_key: example_test
  session_secret: vrwPpJTvwnMVLP1wTSgqigSl7PMI7QcE
  domain: http://example.com
 
production:
  session_key: # any string identifying your app
  session_secret: # a random, secret string at least 32 characters long
  domain: # http://example.com
  mailer: # noreply@example.com
 

You perform a little trickery in environment.rb to prefer the Heroku ENV storage of config vars (in the production environment), but you fall back to your config.yml if the config vars aren't found in ENV (in the development and test environments).

 
# config/environment.rb
 
Rails::Initializer.run do |config|
  require 'yaml'
 
  # support yaml and heroku config vars, preferring ENV for heroku
  CONFIG = (YAML.load_file('config/config.yml')[RAILS_ENV] rescue {}).merge(ENV)
 
  config.action_controller.session = {
    :key => CONFIG['session_key'],
    :secret => CONFIG['session_secret']
  }
end
 

Then, you create a rake task (rake heroku:config) that can be used to send all of the config vars for your production environment up to Heroku. This task can be invoked once to set things up, but can also be run again if you need to make any additions or changes.

 
# lib/tasks/heroku.rake
 
namespace :heroku do
  task :config do
    puts "Reading config/config.yml and sending config vars to Heroku..."
    CONFIG = YAML.load_file('config/config.yml')['production'] rescue {}
    command = "heroku config:add"
    CONFIG.each {|key, val| command << " #{key}=#{val} " if val }
    system command
  end
end
 

This way, you've got all of your config vars stored with the project (.gitignored, of course)...

 
# .gitignore
 
/tmp/**/*
/log/*
*.log
/tmp/restart.txt
/config/config.yml
/config/database.yml
/db/*.sqlite3
 

...and you can easily set what you need on Heroku, like so:

 
$ rake heroku:config
Reading config/config.yml and sending config vars to Heroku...
Adding config vars:
  session_key => example_production
  session_secret => 1WlkMkYYi5611vtF...0ZMS2G3Xl67s4lEIK4sj65
  domain => http://example.com
  mailer => noreply@example.com
Restarting app...done.
 

The result is a pretty nice, I think.

You can see the installation and deployment instructions for my open source project El Dorado if you're curious about the overall flow.

I'd love to get some feedback on this approach, but I really like it so far :)

Install Ruby Enterprise, Phusion Passenger and El Dorado on Debian Lenny

Posted by Timothy O'Connell in El Dorado, Ruby/Rails on June 24, 2009

These instructions require and assume the following:

  • You're running Debian Lenny and you've got root access
  • You've got a functioning apache2 installation
  • You know the basics of working on the command line (i.e. how to edit files, execute commands, etc.)

If the above is true of your situation, read on to learn how to install Ruby Enterprise, Phusion Passenger and El Dorado from scratch in a sort of "one-off" setting where you've got one server and you want it to run one site.

NB: These instructions don't use git or capistrano. The instructions contained in the El Dorado README describe how to install El Dorado using those tools. Using them makes for an easier and cleaner installation. It also makes for easier scalability, upgrading and patching: I highly recommend using those tools.

  1. Resolve Dependencies
  2. The first thing you'll need to do, even before installing RE or PP, is make sure that you've got the development files for the databases that RE and PP applications use:

    apt-get install libsqlite3-ruby postgresql-8.3-plruby libmysql-ruby libmysqlclient15-dev postgresql-server-dev-8.3 libsqlite3-dev

    If you don't resolve these dependencies now, you'll get a message during the RE installation that prompts you to install gems for mysql, postgres, etc. and then, when you go to install those gems, you'll get an error like this:

    ERROR:  Error installing mysql:
    	ERROR: Failed to build gem native extension.

    So just go ahead and resolve those dependencies in advance.

  3. Install Ruby Enterprise
  4. The best practice for this, as far as I know, is to install the current stable release of RE in /opt/. First, download the release you plan to use:

    lana:~# cd /opt
    lana:/opt# wget http://rubyforge.org/frs/download.php/58677/ruby-enterprise-1.8.6-20090610.tar.gz

    Once that's down, untar it and execute the installer script:

    lana:/opt# tar -zxvf ruby-enterprise-1.8.6-20090610.tar.gz
    [...]
    lana:/opt# cd ruby-enterprise-1.8.6-20090610/
    lana:/opt/ruby-enterprise-1.8.6-20090610# ./installer

    That should run, after a few tappings of ye olde Enter key, to its error-free conclusion. If, during the installation, the installer finds that you're missing software packages, the installer will bail and you'll be given some commands that fill those holes. Resolve those dependencies and finish the installation.

    At the end of the installation, you'll be given some syntax that will automatically install PP. You'll use that in the next step.

  5. Install Phusion Passenger
  6. Use the automatically generated syntax:

    lana:/opt/ruby-enterprise-1.8.6-20090610# /opt/ruby-enterprise-1.8.6-20090610/bin/passenger-install-apache2-module

    Again, the installer will bail and prompt you to resolve dependencies if you've got any:

    Installation instructions for required software
    
     * To install Apache 2 development headers:
       Please run apt-get install apache2-prefork-dev as root.
    
     * To install Apache Portable Runtime (APR) development headers:
       Please run apt-get install libapr1-dev as root.
    
     * To install Apache Portable Runtime Utility (APU) development headers:
       Please run apt-get install libaprutil1-dev as root.

    Resolve dependencies and finish the installation.

    Once it's finished, you'll be given some lines to add to your "Apache configuration file". The best file to add these lines to is /etc/apache2/httpd.conf.

    Just don't forget that you added them there (as opposed to somewhere else), as you'll need to modify them if you upgrade RE.

    You'll also probably want to go ahead and add the following lines while you've got the file open:

    PassengerPoolIdleTime 14400
    PassengerMaxInstancesPerApp 2

    Those lines do exactly what it looks like they do. They're also very sensible settings to start with, as they'll prevent El Dorado from hogging a bunch of system resources, etc. right off the bat.

    You can find more information here.

    Finally, your /etc/apache2/httpd.conf file should look something like this:

    PassengerPoolIdleTime 14400
    PassengerMaxInstancesPerApp 2
    
    LoadModule passenger_module /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/passenger-2.2.4/ext/apache2/mod_passenger.so
    PassengerRoot /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/passenger-2.2.4
    PassengerRuby /opt/ruby-enterprise-1.8.6-20090610/bin/ruby

    Once you've made those changes, you're ready to begin installing El Dorado.

    When it exits, the PP installer will show you some sample syntax for how to write an apache configuration file for your first application. You can ignore that for now, as we're going to come back to it later.

  7. Install El Dorado
  8. First, get the latest release of the software from Trevor's github: http://github.com/trevorturk/eldorado/tree/master

    Once you've got the URL of the latest release, switch from root to a less privileged user, make a folder in your home dir for the site, download the latest release of El Dorado to that directory and untar it:

    toconnell@lana:~$ mkdir example.com
    toconnell@lana:~$ cd example.com
    toconnell@lana:~/example.com$ wget wget http://download.github.com/trevorturk-eldorado-a37d0c71e928f605d111d5f48b5786ff613bf676.tar.gz
    tar -zxvf trevorturk-eldorado-a37d0c71e928f605d111d5f48b5786ff613bf676.tar.gz
    

    Now, get all of those files out of that big, ugly directory and into the current working directory and ditch those old files:

    toconnell@lana:~/example.com$ mv trevorturk-eldorado-a37d0c71e928f605d111d5f48b5786ff613bf676/* .
    toconnell@lana:~/example.com$ rm -rf trevorturk-eldorado-a37d0c71e928f605d111d5f48b5786ff613bf676*

    Now, follow the instructions in the README and copy the example yml files to the places where the application will look for real, non-example files:

    toconnell@lana:~/example.com$ cp config/database.example.yml config/database.yml
    toconnell@lana:~/example.com$ cp config/config.example.yml config/config.yml

    Now, use your favorite editor to edit the last stanza in config/config.yml so that it matches the information of your site:

    production:
      session_key: example_production
      session_secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  # Replace these X's and make this string (at least) 32 random alpha-numerics for good site security
      domain: http://example.com
      mailer: noreply@example.com  

    NB: There are "dev" and "test" entries in this default file. If you're not planning on doing anything development related with this installation, you can safely delete those entries.

    Once you've edited that file, that's it, so far as the non-git installation is concerned. To get El Dorado up and running, you'll need to do some minor database tasks. Those are covered in the next section.

  9. Configure the Database
  10. Since MySQL is deprecated, I'll be using PostgreSQL for the remainder of these instructions.

    If you look at config/database.yml, you'll notice that it's essentially a blank template:

    development:
      adapter: sqlite3
      database: db/development.sqlite3
      timeout: 5000
      # adapter: mysql
      # database: eldorado_development
      # username:
      # password:
      # host: localhost
    
    test:
      adapter: sqlite3
      database: db/test.sqlite3
      timeout: 5000
    
    production:
      adapter:
      database:
      username:
      password:
      host:

    First, edit that file:

    production:
      adapter: postgresql
      database: example
      username: example
      password: XXXXXXXXXXXXXXXXXXXX
      host: localhost

    NB: Again: once you've added your "production" entries to this file, you can feel free to delete the "test" and "dev" lines, as they do nothing and could cause confusion down the line.

    Now, create the database and the user:

    toconnell@lana:~/example.com$ sudo su postgres -c "createuser example"
    Shall the new role be a superuser? (y/n) n
    Shall the new role be allowed to create databases? (y/n) n
    Shall the new role be allowed to create more new roles? (y/n) n
    toconnell@lana:~/example.com$ sudo su postgres -c "createdb example"

    Next, start the postgres monitor as the postgres user and make the a few changes:

    toconnell@lana:~/example.com$ sudo su postgres -c psql
    Welcome to psql 8.3.7, the PostgreSQL interactive terminal.
    
    postgres=# ALTER USER example PASSWORD 'XXXXXXXXXXXXXXXXXXXX';
    ALTER ROLE
    postgres=# ALTER DATABASE example OWNER TO example;
    ALTER DATABASE

    Now, if you've got your Postgres database configured correctly and your new user can access your new postgres database, you're ready to rake the El Dorado production database:

    toconnell@lana:~/example.com$ /opt/ruby-enterprise-1.8.6-20090610/bin/rake rake db:schema:load RAILS_ENV=production

    Once the database is successfully raked, all you've got to do to finish up is configure Apache.

  11. Apache Configuration
  12. The following assumes that you're doing apache the "Debian way".

    If this is true, the first thing you'll do is create a symlink in /var/www/ that points at your install directory:

    lana:/var/www# ln -s /home/toconnell/example.com/

    Next, create a file in /etc/apache2/sites-available with the name of your site and then create a symlink to it in /etc/apache2/sites-enabled.

    The file should look something like this:

    #
    # example.com
    #
     
    <VirtualHost *:80>
      ServerName example.com
      ServerAlias www.example.com
      ServerAdmin youremail@example.com
      DocumentRoot /home/toconnell/example.com/public
     
      <Directory "/var/www/example.com">
        Options FollowSymLinks
        AllowOverride None
        Order allow,deny
        Allow from all
      </Directory>
     
      RewriteEngine On
     
      RewriteCond %{HTTP_HOST} ^www\.example\.com$ [NC]
      RewriteRule ^(.*)$ http://example.com$1 [R=301,L]
     
      RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
      RewriteCond %{SCRIPT_FILENAME} !maintenance.html
      RewriteRule ^.*$ /system/maintenance.html [L]
     
      ErrorLog /var/log/apache2/example_error_log
      CustomLog /var/log/apache2/example_access_log combined
      RewriteLog /var/log/apache2/example_rewrite_log
      RewriteLogLevel 9
     
    </VirtualHost>

    NB: I've added some apache custom logging. Logs are good.

    Once you've got the file in /etc/apache2/sites-available and the symlink in /etc/apache2/sites-enabled that points at that file, you should be ready to restart apache and get rolling:

    lana:/etc/logrotate.d# /etc/init.d/apache2 reload

And that, as they say, is that. Once you reload apache, provided that your DNS is set up correctly and you haven't got any system problems beyond the scope of this document, your single instance of El Dorado should be ready for prime time.

Navigate to your site in your browser and create an administrative account: the first user who attempts to login will be the administrator. Once you've got your admin created, you're ready to start tweaking your new El Dorado site's appearance and adding users.

A note on upgrades: if you find you need/want to upgrade an instance of El Dorado that has been installed thus, consult the README. The basic gist is that you're going to want to download/copy the new source/program files over the old ones (while being careful not to erase your user-uploaded files) and then run rake db:migrate RAILS_ENV=production.

Weekly Digest, 6-22-09

Posted by Weekly Digest in Weekly Digest on June 22, 2009

How to speed up gem installs 10x

Answer: Turn off ri and rdoc installation.

Perch

Perch is a really little content management system for when you (or your clients) need to edit content without the hassle of setting up a big CMS.

Installing Ruby on Rails and PostgreSQL on OS X, Third Edition

Over the past few years, I’ve helped you walk through the process of getting Ruby on Rails up and running on Mac OS X. The last version has been getting a lot of comments related to issues with the new Apple Leopard, so I’m going this post will expand on previous installation guides with what’s working for me as of January 2008.

Thoughts on Opera Unite

Opera’s CEO Jon von Tetzchner claims that “Opera Unite now decentralizes and democratizes the cloud." I call bullshit. Opera Unite does indeed rely on a P2P-like network to function, but the big problem is that you must push all your traffic through Opera’s proxy service.

LESS - Leaner CSS

Less is Leaner css. Less extends css by adding: variables, mixins, operations and nested rules. Less uses existing css syntax. This means you can migrate your current .css files to .less in seconds and there is virtually no learning curve.

YC Company Hosting Stats

[Interesting stats and discussion on hosting.]

Rip: a RubyGems Replacement?

This makes package management as simple as passing files between friends. Email me your latest library, and I can run rip install path/to/lib. That’s it — you don’t need spec files, and you don’t need to build anything before your send me your code.

BigTable

BigTable is a fast and extremely large-scale DBMS. However, it departs from the typical convention of a fixed number of columns, instead described by the authors as "a sparse, distributed multi-dimensional sorted map", sharing characteristics of both row-oriented and column-oriented databases. BigTable is designed to scale into the petabyte range across "hundreds or thousands of machines, and to make it easy to add more machines [to] the system and automatically start taking advantage of those resources without any reconfiguration".

Opera Unite reinvents the Web: a Web server on the Web browser

[Very interesting possibilities here. Making it easier for people to serve content on the web can only lead to good things.]

tenderlove's markup_validity

Test for valid markup with test/unit or rspec

Hemlock

Hemlock is an open-source framework that combines the richness of Flash with the scalability of XMPP, facilitating a new class of web applications where multiple users can interact in real time. Games, workspace collaboration, educational tools… The only limit is your imagination.

Rip: A New Package Management System for Ruby

But why a completely new package manager? What's wrong with RubyGems? We asked one of Rip's developers, Chris Wanstrath...

Ruby at ThoughtWorks

ThoughtWorks started using Ruby for production projects in 2006, from then till the end of 2008 we had done 41 ruby projects. In preparation for a talk at QCon I surveyed these projects to examine what lessons we can draw from the experience. I describe our thoughts so far on common questions about Ruby's productivity, speed and maintainability.

[git pull] drm-next

See? All the rules really are pretty simple. There's that somewhat subtle
interaction between "keep your own history clean" and "never try to clean
up _other_ proples histories", but if you follow the rules for pulling,
you'll never have that problem.

GitHub Protip: Follow other users

Posted by Trevor in Code on June 11, 2009

Inspired by this post, I thought I'd share a tip that helps me get the most out of GitHub.

Don't just follow the projects that you're interested in — follow other users. Here's a list of people that I'm following. They're constantly turning me on to new and interesting projects, because I get to see everything they're working on, and everything they're following.

 

Dig around the users that I follow, check out what they're been up to, and try it out. If you find that your feed becomes a bit much to manage, try subscribing to your personal RSS feed. There's a link on the home page when you're logged in.

Thanks, GitHub. You're the best.

Automatically Rotate your Log Files in Development

Posted by Trevor in Ruby/Rails on June 11, 2009

I'm trying to save hard drive space, since I've got this super small (and fast?) SSD hard drive on the way. I noticed that I was using a TON of space to store totally worthless logs for my Rails apps. Now, I know I could set up proper log rotation, but I don't feel like going through the trouble for my local machine.

Here's a quick tip I picked up here that will set your logs to automatically rotate in the test and development environments. Just add the following line to these files:

  • config/development.rb
  • config/test.rb
config.logger = Logger.new(config.log_path, 2, 20.megabytes)

Make sure you've got these in your .gitignore file as well:

/log/*
*.log

That will keep your log files under control, but with plenty of room for digging in if need be.

Speed up your Apache/Passenger Rails app in 2min

Posted by Trevor in Ruby/Rails on June 11, 2009

Here's a quick tip for speeding up your Apache/Passenger powered Rails app. It'll take you about 2 minutes, and I guarantee you'll notice the speed-up.

  • SSH onto your VPS
  • Run the following commands: "a2enmod expires" and "a2enmod deflate"

Now, open up your Apache vhost config for your Rails app. Add the following:

Then, restart Apache by running: "/etc/init.d/apache2 restart"

This will gzip your html, css, and javascript. It'll also add far future expires headers for the appropriate cacheable filetypes. There's no downside, and it only takes a second. Bang for buck.

Edit: Check the comments for some possible downsides... ;)

Weekly Digest, 6-11-09

Posted by Weekly Digest in Weekly Digest on June 11, 2009

In this edition, Timothy moves to Washington DC and Trevor trims down his "watch list" on GitHub and shares many interesting projects with you via his delicious feed.

Trevor's Links

Email. Twice daily. No more, no less.

So, using some motivation from The Four Hour Workweek1, I opted to come back to the studio and change my behavior. That morning, I emailed my entire team and my clients to let them know that I would only be checking my email at 10am and 4pm each day.

How to Build a Popularity Algorithm You can be Proud of

Many web sites allow users to casts vote on items. These visitors' votes are then often used to detect the items' "popularity" and hence rank the rated items accordingly. And when "rank" comes into play things gets tricky...

Online communities, etc.

Anyway, I'm bored on a long bus drive and there's no real moral to the story here, just writing. I will be tuning out of the social networking sites because at the end of the day it's now doing more harm than good in the bigger picture and the experiment seems to have yielded a result. Idiots rule.

Really Simple Rails Log Rotatation

I always used logrotate Linux tool to setup log rotation for my Rails apps which has worked fine although it required finding some external config file and understanding its config options and syntax. [Great tip for development/test environments. Might not be a good idea in production?]

Instapaper bookmarklet, modified to close the current tab

I modified the bookmarklet slightly so that the tab closes immediately, without disturbing the pop-up. This way, saving something for later is one simple action, instead of two.

DeliciousSafari

Use and create Delicious bookmarks from the Safari web browser.

So, about this Shopify Platform

The Shopify platform allows any programmer to create applications that integrate natively with the administration interface or storefront. These applications can be written in any language and communicate with Shopify using our handcrafted REST API. We even provide some amazing rails generators to get started quickly.

Introducing Trample: A Better Load Simulator

Most load sim tools make requests to a static list of urls. They spawn n threads and make requests to the urls on the list in succession, in each thread. Unfortunately, though, if your applicaition makes use of any kind of caching (including your database's internal caching facilities), this kind of load simulation is unrealistic.

TOSBack | The Terms-Of-Service Tracker

TOSBack keeps an eye on 44 website policies. Every time one of them changes, you'll see an update here.

Twitter Blog: Not Playing Ball

We do recognize an opportunity to improve Twitter user experience and clear up confusion beyond simply removing impersonation accounts once alerted. We'll be experimenting with a beta preview of what we're calling Verified Accounts this summer.

cdto

Fast mini application that opens a Terminal.app window cd'd to the front most finder window. This app is designed (including it's icon) to placed in the finder window's toolbar.

Trevor's GitHub Links

quirkey's sammy

Sammy is a tiny javascript framework built on top of jQuery inspired by Ruby's Sinatra.

kabuki's heresy

Heresy is a schema free wrapper around your database, heavily inspired by both CouchDB and FriendFeed.

paulmars's seven_minute_abs

ab testing for rails

binarylogic's searchlogic at v2

Searchlogic has been completely rewritten for v2. It is much simpler and has taken an entirely new approach. To give you an idea, v1 had ~2300 lines of code, v2 has ~350 lines of code.

semanticart's is_paranoid

ActiveRecord 2.3 compatible gem "allowing you to hide and restore records without actually deleting them." Yes, like acts_as_paranoid, only implemented differently...

brynary's webrat

Webrat - Ruby Acceptance Testing for Web applications.

mbleigh's twitter-auth

Standard authentication stack for Rails using Twitter to log in.

courtenay's splam

Simple, pluggable, easily customizable score-based spam filter plugin for Ruby-based applications.

jeremy's ruby-prof

a fast code profiler for Ruby

nakajima's roleful

Generic roles for you and your objects.

37signals's wysihat

A WYSIWYG JavaScript framework

binarylogic's authlogic

A clean, simple, and unobtrusive ruby authentication solution.

joshuaclayton's blueprint-css

A CSS framework that aims to cut down on your CSS development time.

stephencelis's dots

Free progress dots for your scripts. Test::Unit-style.

wycats's merb-extlib

Ruby core extensions library extracted from Merb core.

jodosha's plugin_migrations

Rake tasks for running plugin migrations.

tcocca's acts_as_follower

A Plugin to add "Follow" functionality for models

mojodna's active_queue

A toolkit for queueing tasks and creating worker processes

Weekly Digest, 5-31-09

Posted by Weekly Digest in Weekly Digest on May 31, 2009

"Weekly" <- in scare-quotes

Trevor's Links

Stowe Boyd launches Microsyntax.org

Stowe Boyd launched Microsyntax.org... a number of ideas for making posts on Twitter contain more information than what is superficially presented, and this new effort should create a space in which ideas, research, proposals and experiments can be made and discussed.

Amazon Payments Account Management

Amazon Simple Pay Subscriptions enables you to charge your customers on a recurring basis using a single authorization from the customer. It is for those who offer digital content subscriptions, collect membership dues on a periodic basis, or provide premium services on their website.

7 Great Reasons Not To Take VC Money

Raising venture capital for early stage start-ups seems to be the prevailing path for most entrepreneurs; however, most would-be founders should reconsider.

The importance of stupidity in scientific research

The crucial lesson was that the scope of things I didn't know wasn't merely vast; it was, for all practical purposes, infinite. That realization, instead of being discouraging, was liberating. If our ignorance is infinite, the only possible course of action is to muddle through as best we can.

When to use self in Rails models

When I started with Rails, half the words in my models were self. This wasn’t necessary. Now, when I edit code by other people, I find myself constantly deleting “self” from their code.

The random person test

Why not try to write code that future programmers will thank me for because it was so clear and obvious? Programmer skill should be measured not only in the complexity of the problems that they can solve, but in the clarity of their solutions.

Patience and hard work

There is a gaping chasm between a web app sitting on a server somewhere, and the ingredients of a business. Establishing a brand, getting the right kind of people to listen, and growing your own customer-base doesn’t happen as a by product of really sweet Javascript effects.

Google Wave: What Might Email Look Like If It Were Invented Today

Google wants other providers to adopt Wave - the protocol allows federation between independent Wave clouds. The team hopes that Wave will become as ubiquitous and interoperable as email and instant messaging, not just a Google product.

Ask HN: I'm Tired of Hacking. What Do I Do? Please Advise.

I just can't do it anymore. I hate sitting on my ass all day writing some code. My neck has been hurting for two years for spending so many hours in front of the computer. I kind of have been hating my career for a couple of years now and I have no clue about what I should do.

MacRuby, changing the Ruby ecosystem

MacRuby is an Apple-sponsored, open source, full Ruby implementation on top of Objective-C runtime. In other words, whatever code runs on Ruby 1.9, should/will run on MacRuby. Yes, you read correctly, MacRuby can/will be able to run all your Ruby code.

Mac-friendly Autotest

ZenTest’s autotest is great, but it has one drawback: In order to detect whether you have modified a file, it relies on filesystem polling. In other words it constantly traverses the filesystem and thus causes a lot of CPU and harddrive load.

Include vs Extend in Ruby

Now that we know the difference between an instance method and a class method, let’s cover the difference between include and extend in regards to modules. Include is for adding methods to an instance of a class and extend is for adding class methods. Let’s take a look at a small example.

Class and Instance Methods in Ruby

Class methods can only be called on classes and instance methods can only be called on an instance of a class. It’s simple when you understand it, but I remember being confused when I was learning Ruby. Hope this helps. If I was unclear or incorrect at any point above, let me know. [Nice, easy to follow overview.]

djng—a Django powered microframework

djng is my experiment to see what Django would like without settings.py and with a whole lot more turtles. It’s Yet Another Python Microframework.

Django tip: Caching and two-phased template rendering

We've launched user accounts at EveryBlock and we faced the interesting problem of needing to cache entire pages except for "You're logged in as [username]" bit top page. The solution ended up using is two-phased template rendering.

Assembling Pages Last: Edge Caching, ESI and Rails

[Good overview of ESI pros/cons.]

The open, social web

If I told you that the iPhone was the best example of the success of standards and open source, you’d probably laugh at me, but check it out...

Timothy's Links

Server Monitoring with Cacti + ServerStats | HostingFu

This is kind of cool: if you've got a computer somewhere on your local network and you want the laity to have access to rough stats, all you've got to do is fire this package up, tweak xinetd a little bit, and voila--your boss can look over your shoulder from the comforts of his own office.

Slicehost for Android // Slicehost - VPS Hosting

Trevor pointed me in the direction of this one. It's a neat little app--very minimalist and very Linux-y--that lets you check on your bandwidth, slice stats (e.g. mem/proc/distro name and version) and gives you the option to do a remote /sbin/poweroff or an /sbin/shutdown -h now. Very neat.

Research: Password 'secret question' woefully insecure

Let's get a movement going here: if enough Internet types spread the word that no one in their right mind or who possesses any kind of meaningful credential endorses "secret questions" and that, in fact, the research shows that they make accounts _less_ secure, maybe we can kick up enough dust to get rid of them.

Three Letters

This take on the classic joke has a sysadmin slant; guaranteed to be appreciated by everyone from Exchange rebooters in silk cravats to consolemen who live on the metal.

Samba: change a Windows user’s hashed password. And then change it back.

Posted by Timothy O'Connell in General on May 21, 2009

File this one under "hacks". Cross-list it under "more basic administrative tasks you can't do on Windows".

Here's the situation: you're the admin of a Windows domain where the Domain Controller is a Linux box serving Samba. Your problem, other than the fact that you're surrounded by Windows users, is that you've got a user who's password you don't know and, for whatever reason, you need to log onto your Windows domain as that user: a simple RUNAS won't cut it this time.

Normally, you'd just nuke his password, change it to "password" (or whatever), log on as him, do your dirty, sinful business, log off, expire his password and then send him an email telling him that his password has been changed to "password" and that he'll be prompted to change it at his next log on.

But what if that wasn't an option? What if you needed to log on to your domain as that user and it was important that he be none the wiser?

Grab the Hashes

First, use your favorite smbldap-type tool to get the current password info on the user you're fixin' to use:

frances:~# smbldap-usershow toconnell
dn: uid=toconnell,ou=Users,dc=domain,dc=com
objectClass: top,person,organizationalPerson,inetOrgPerson,posixAccount,shadowAccount,sambaSamAccount
cn: toconnell
sn: toconnell
givenName: toconnell
uid: toconnell
uidNumber: 1007
[...]
sambaPwdCanChange: 1202398556
sambaPwdMustChange: 9223372036854775807
sambaLMPassword: BE41CD009FF0812C718CCFD7D98A52AA
sambaNTPassword: 9454453CBC8A48DEF442F6B0A10B3EAA
sambaPwdLastSet: 1202398556
userPassword: {SSHA}baSDvXS6C6DSBNkJGyEYplprZ3wslAa/

Copy everything that's bolded and stick it somewhere safe. That dn information is going to be necessary later on, as it contains the ldap tags that you'll need to specify the record you want to modify; those hashes at the bottom are the user's original passwords and, when you want to cover your tracks later on, you'll need that info.

Now that you've got those hashes, you're free to nuke the user's password (again, using your favorite smbldap-type-tool or however else you like to reset passwords), log in as him, do whatever you have to do, and then log out:

frances:~# smbldap-passwd toconnell
Changing UNIX and samba passwords for toconnell
New password:
Retype new password:

Once you're out, you're going to want to set his password back to what it once was. This is where ldapmodify comes into play.

Kerberos

Before you can do that, however, you'll need to get a kerberos ticket. This is because you'll need to be kerberos-authenticated to make your ldap modifications stick. So, first things first, get yourself an admin kerberos ticket:

frances:~# kinit toconnell/admin
Password for toconnell/admin@DOMAIN.COM:
frances:~# 

ldapmodify

A quick glance at the man page for ldapmodify shows that the most convenient way to make changes to an ldap entry is to use the -f flag and an input file. The example in the man page for how to construct the input file is this:

 dn: cn=Modify Me,dc=example,dc=com
           changetype: modify
           replace: mail
           mail: modme@example.com
           -
           add: title
           title: Grand Pooba

So, using the data we got above, we're going to make a similar file containing the original hashes from our target user in order to change his password back to what it used to be:

dn: uid=toconnell,ou=Users,dc=domain,dc=com
changetype: modify
replace: sambaLMPassword
sambaLMPassword: BE41CD009FF0812C718CCFD7D98A52AA
-
replace: sambaNTPassword
sambaNTPassword: 9454453CBC8A48DEF442F6B0A10B3EAA
-
replace: userPassword
userPassword: {SSHA}abSDvXS6C6DSBNkJGyEYplprZ3wslAa/

Remember to include those "-" characters and to give them their own line: if you fail to do that, you'll get mystery errors about unknown types, etc.

Once you've got your file, fire off your changes like this:

ldapmodify -f /path/to/file

And that's all you've got to do. When the original user attempts to log in with his old password, everything will look perfectly normal to him: you never saw his password in plaintext and, as far as he's concerned, none of this ever happened.

Weekly Digest, 5-17-09

Posted by Weekly Digest in Weekly Digest on May 17, 2009

Trevor's Links

Interview with Ian Hickson, editor of the HTML 5 specification

You’ve heard it’s coming in 2012. Or maybe 2022. It’s certainly not ready yet, but some parts are already in browsers now so for the standards-savvy developers, the future is worth investigating today. Ian “Hixie” Hickson, editor of the HTML 5 specification, hopes that the spec will go to Last Call Working Draft in October this year.

The Mega RailsConf 2009 Round Up

A week ago, RailsConf 2009 kicked off in Las Vegas. As usual, it didn't fall short on drama, interesting sessions, and inspiration for the 1000+ attendees. This post is an after-event summary and long-term source of links to the best RailsConf 2009 related content found so far.

Nuts & Bolts: Campfire loves Erlang

Erlang definitely isn’t a replacement for Rails, but it is a fantastic addition to our collective toolbox for problems that Rails wasn’t designed to address. It’s always easier to work with the grain than against it, and adding more tools makes that more likely.

Tango Icon Theme Guidelines

The Tango icon theme's goal is to make applications not seem alien on any desktop. A user running a multiplatform application should not have the impression that the look is unpolished and inconsistent with what he or she is used to. While this isn't about merging styles of all desktop systems, we do aim to not be drastically different on each platform.

RightZoom Makes the OS X Maximize Button More Like Windows

Mac OS X only: System utility RightZoom runs in the background and modifies the OS X maximize behavior to fill the whole screen—perfect for readers that recently made the switch to Mac.

Railscasts - Factories not Fixtures

Fixtures are external dependencies which can make tests brittle and difficult to read. In this episode I show a better alternative using factories to generate the needed records. [I prefer Machinist to Factory Girl, but this is a particularly good episode all around.]

db/seeds.rb in Rails

Added db/seeds.rb as a default file for storing seed data for the database. Can be loaded with rake db:seed (or created alongside the db with db:setup). (This is also known as the "Stop Putting Gawd Damn Seed Data In Your Migrations" feature) [DHH]

Timothy's Links

The Security Implications Of Google Native Client

This is a really cool from Matasano about how things like ActiveX and Java work from the perspective of someone trying to execute compiled code from a remote source without giving away the whole store, security-wise. Nice pictures, very informative.

How to Add Date And Time To Your Bash History file -- Debian Admin

This is a neat one-liner for your .bashrc that just might make your .bash_history a little more searchable. Add it to your custom .bashrc lines.

Postfix main.cf analysis

Here's the setup: the one dude pastes his postconf -n and the other dude does through it, telling him what's what. Kind of a cross between a postmortem and an x-ray. Useful to test your postfix knowledge/skills.

SoS Wiki - - Split Screen Vi

If you use vi/vim and you don't do split screen, you are, in the immortal words of whatever Internet meme, doing it wrong. Study up!

Set Gmail as Default Mail Client in Ubuntu :: the How-To Geek

This is a neat little trick for writing a line or two of bash that will allow you to use gmail (via firefox) as your default email client in a gnome environment. It wouldn't take much to adapt the instructions for other desktop environments. (Props to Artie for sending this my way)

Reports: Thief holds Virginia medical data ransom

I guess, technically, that since I'm on the side of the law by virtue of my professional situation, I ought to regard this as terrifying or reprehensible or something. But you gotta admit: something about the idea of a blackhat utterly pwning someone's network to the extent of the pwnage described here is really, really exciting.

Postfix Backup MX eMail Server Anti-Spam Configuration

The English is a little messy on this one, but the conf text is right on. This is a nice little list of basic (yet above and beyond "stock") config options for reducing shenanigans and closing commonly exploited gaps.

Restore a single table from a large MySQL backup

I'm not sure that I understand the ruby syntax completely, but people are passing this link around, so this is my obligatory bump.

Stupid Linux Tricks: Basic Server Hardening (Debian Lenny)

Posted by Timothy O'Connell in General on May 15, 2009

Due to recent events involving some computers I administer, I've become very interested in security. Basically, I've dodged enough bullets thanks to little more than beginner's luck and I figure that it's about time to take responsibility for the safety and security of my computers.

Since I was interested in hardening up two Debian boxes running Lenny, I started off by taking a look at the Securing Debian Manual, this very helpful page on LQ and the results of a tiger audit.

Do this to generate a tiger audit of your server:

# aptitude install tiger
tiger -E

The "-E" gets you what's called an "explanation report", which will be useful in helping you understand what can be some fairly cryptic output.

Additionally, the package version of tiger comes with some nice default settings for the main executable and for tigercron, which, as you might imagine, runs some minor scans on a pre-defined schedule.

At any rate, once I had my audit and had picked up a few bright ideas from the SDM, I made a number of changes to all of my web-facing production machines. What follows are some things that you might want to consider doing on your Debian Lenny servers:

Users and Permissions

  1. Password Audit: first, I decided to get to know my user accounts a little better. This meant running john (formerly "john the ripper", a password cracker that reads hashed passwords and tries to decipher them) against my /etc/shadow to see who was using dictionary-based passwords and who was using other types of insecure passwords:
    # aptitude install john
    # john /etc/shadow 

    This took a while to run--a little over a day, but I had it niced pretty high--but of the 10 user accounts it cracked, it was good to know which ones were using hilariously insecure passwords and which ones probably weren't going to cracked by your garden variety brute force password cracker.

  2. NB: if you run john against your /etc/shadow and realize that you've got a problem child on your hands, there's always chroot. Here is a really good how-to on chroot-jailing a user.

  3. The Prunening: odds are, if you've been living on a system for more than a few months, you've accumulated some users (either from software that you've installed and then removed or by meeting user/developer needs, etc.) that aren't doing anything. One of the basic tenets of server security is having the smallest amount of users with the least amount of access to the smallest number of programs possible.

    In some environments, you've simply got to have a bunch of users in your /etc/passwd. In most situations, however, it makes good sense to just hit all the derelicts with a userdel and only having to worry about angry users not having enough access (rather than having to worry about unauthorized users having too much access).

Keeping Script Kiddies Under Control

In my (admittedly limited) experience, the most trouble you're going to run into from script kiddies are anonymous, unfocused attacks that attempt to gain access to your machine via either a.) the /tmp directory, b.) DoS-based exploints or c.) application attacks like SQL injections, XSS or directory traversal attacks.

Since guarding against application attacks is something that programmers are supposed to be handling, I decided to focus on stopping /tmp abuse and trying to stymie DoS attacks.

  1. Mounting /tmp with noexec: in this age of VPSes and shared hosting, it's more often the case than not that you won't get to decide how your machine is partitioned. If, like me, you live on Slicehost and you're running Debian, your partition scheme looks like this:
    lana:/# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda1              19G  9.0G  8.9G  51% /
    tmpfs                 256M     0  256M   0% /lib/init/rw
    udev                  256M   20K  256M   1% /dev

    What this means is that you've got your /tmp directory on your / partition. Which partition is, by necessity, allows files to be executable.

    And that, as I have learned (the hard way), is bad news. What this means is that, you've got a directory on your / partition--the partition where all your apps and data probably live--that is writable/readable by every Li, Ivan and Harry from Taipei to Yaktusk. And if one of the thousands of bots who knocks on your door every month knows how to write to /tmp and you don't catch him right away, it's pretty much game over for your TLD or your IP Address: one day, you'll wake up and find that your info is on every spam list on the Internet because your server has been under remote control via IRC for the last three weeks and now your full time job is trying to get your info off of those lists while planning a full OS re-install.

    Ideally, you would be able /tmp its own partition and mount that partition with noexec. And while it would be optimal, it sometimes isn't an option if you're a part of the VPS set.

    And if you can't control your installation or maybe you just can't take down a production server, what you can do is warn your users/developers that you're about to cause a little temporary chaos (get it? temporary chaos?), move your existing /tmp to some place else and create a small filesystem that you can mount noexec to use as /tmp. On Debian Lenny, that would look approximately like this:

    gonzo:/# mv /tmp /old_tmp
    # dd if=/dev/zero of=/.tmpfs bs=1024 count=1000000
    1000000+0 records in
    1000000+0 records out
    1024000000 bytes (1.0 GB) copied, 29.1302 s, 35.2 MB/s
    gonzo:/# mkfs.ext3 -j .tmpfs
    [...]
    gonzo:/# mount -o loop,noexec,nosuid,rw /.tmpfs /tmp/
    gonzo:/# mv /old_tmp/* /tmp/.

    Et voila! You've got a 1GB "drive" that's mounted noexec at /tmp that's ready to roll out. Any attempts to execute anything on that partition will result in a bad interpreter error. Check it:

    gonzo:/# ls -l tmp/
    total 20
    -rwxr-xr-x 1 root root    37 2009-05-15 14:54 executeMe.py
    drwx------ 2 root root 16384 2009-05-15 14:51 lost+found
    gonzo:/# test/executeMe.py
    bash: test/executeMe.py: /usr/bin/env: bad interpreter: Permission denied

    All you've got to do now is add that mount info to your fstab and you're ready to start sleeping at night again:

    /.tmpfs           /tmp            ext3    loop,nosuid,noexec,rw  0      0

    NB: don't forget that /tmp wants to have the sticky bit turned on (i.e. be chmoded to 1777). Also don't forget to make /var/tmp a symlink that points to /tmp.
    Also: props to Vincent Danen's post on /tmp at TechRepublic for the idea.

  2. mod_evasive to Prevent DoS: after a recent DoS experience, I decided to install Apache mod_evasive to reduce the risk of getting flat-lined/broad-sided by DoS/DDoS attacks:
    # aptitude install libapache2-mod-evasive

    The beauty of using packaged software is that that's really all you have to do: apt will copy the files, create the symbolic links and restart apache for you. Nice.

  3. sysctl real-time kernel Modifications: Additionally, you might also want to use a slightly obscure command called sysctl (which modifies kernel perameters while the kernel is running, so consider yourself warned) to take a precaution against a DoS tactic called "syn flooding":
    # sysctl -w net.ipv4.tcp_syncookies=1

    This is a sort of controversial measure--apparently it defies some RFC docs for TCP/IP--but setting tcp_syncookies to False has yet to have affected any of my computers.

    There are a number of additional security features you can activate with this command; Google it and prepared to be awed by some of the features of your OS you can control in real-time with sysctl.

For the Tin-foil Hat Crowd

What follows are non-specific countermeasures and settings that, while obscure and probably unlikely to save you from becoming an unwitting member of some Russian bot master's herd, might help you feel more secure in the knowledge that even if someone does get non-root shell access, he's probably not going to be able to do too much damage.

  1. Add /usr/bin/mesg n to root's .bashrc File:
    executing mesg n on log in, prevents an admittedly rare exploit through which other users can execute arbitrary code as root by sending messages to his terminal.
  2. Modify /etc/inittab to Prevent Non-root Users from Rebooting the System with ctrl+alt+del. In the stock /etc/inititab on Debian Lenny, you've got this line:
    # What to do when CTRL-ALT-DEL is pressed.
    ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -r now

    Which is hardly optimal: this lets pretty much anyone who figures out a way to execute programs in /etc/sbin reboot the system. I changed it to:

    ca:12345:ctrlaltdel:/bin/false
  3. SMTPD Settings If you run postfix, you should probably check up on your relay settings and update your external blacklist providers if you haven't done it in a while. Your mail server is the world's first line of defense against everything from phishing/spear-phishing to headline-making super worms:
    smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks,
      reject_unauth_destination,
      reject_unauth_pipelining,
      reject_invalid_hostname,
      reject_rbl_client sbl-xbl.spamhaus.org,
      reject_rbl_client cbl.abuseat.org,
      reject_rbl_client bl.spamcop.net,
      reject_rbl_client zen.spamhaus.org
    smtpd_helo_required = yes
    disable_vrfy_command = yes
    
    smtpd_data_restrictions =
                reject_unauth_pipelining,
                permit
    

    That smtpd_help_required line might not seem like anything special, but I have a script that parses /var/log/mail.log output and, when you do get a spammer that responds to the helo request, a lot of times he'll come back with his actual domain.

    It's fun for research/study/personal amusement reasons, basically.

While some people might say that the above is overkill--that it's just not worth the time and effort to audit and harden at this level--but I'd say that this level of focus on security isn't so much "overkill" as it is "a pretty good start."

Because my thinking is that if you've got the root password, you're probably already the anxious type. And at the very least, being a little bit OCD about security on your all-important, mission-critical application servers might help you feel a little less anxious. Which is definitely worth the effort.

Do This: Spamassassin (Debian, Postfix)

Posted by Timothy O'Connell in General on May 12, 2009

There's a great episode of MASH in which the over-zealous paranoid-schizophrenic Colonel Flagg, in an attempt to coerce the impossibly calm and notoriously imperturbable Dr Stanley Freedman into collusion with one or another of his hare-brained schemes, asks the psychiatrist: "You wanna do your self a favor?"

"Why Not?" replies the doc. "Who deserves one more?"

If you've got a Debian Lenny box out in the wild serving your email with postfix and you're not using spamassassin as a filter, you really ought to consider doing yourself a favor and throwing that spamassassin piece into the mix: it only takes a second, it will increase security for your users-- hardening up your network little bit--and make the world a slightly better place for everyone.

  1. If you're OK with letting aptitude resolve your dependencies and manage your packages, all you need to do is install a single package:
    # aptitude install spamassassin
  2. Once that's done, crack open /etc/default/spamassassin with your favorite editor and enable it to run as a daemon and update itself automatically:
    # Change to one to enable spamd
    #ENABLED=0
    ENABLED=1
    
    # Cronjob
    # Set to anything but 0 to enable the cron job to automatically update
    # spamassassin's rules on a nightly basis
    #CRON=0
    CRON=1

    (spamd is an old name for spamassassin: you'll notice a lot of the RHEL/CentOS/Fedora boxes out there running spamd.

  3. Take a look at /etc/spamassassin/local.cf: there are some fun options that you can uncomment and enable in there. My personal favorite one is:
    rewrite_header Subject *****SPAM*****

    This does what it sounds like it does and rewrites the headers of suspicious emails: you can then easily configure your email client to recognize these headers and filter accordingly. Pretty sweet.

  4. Start spamassassin:
    # /etc/init.d/spamassassin start
  5. At this point, we're going to edit some postfix conf files, but we need to check on something first. Make sure SA is running, spawning children and listening on the right port:
    # netstat -anp |grep spam
    tcp        0      0 127.0.0.1:783           0.0.0.0:*               LISTEN      11724/spamd.pid
    unix  2      [ ACC ]     STREAM     LISTENING     9096119  1717/master         private/spamassassin
    unix  3      [ ]         STREAM     CONNECTED     9757173  30093/spamd child
    unix  3      [ ]         STREAM     CONNECTED     9757172  11724/spamd.pid  
  6. Noting that SA is listening on 783, tack the following on to the bottom of your /etc/postfix/master.cf:
    spamassassin unix -     n       n       -       -       pipe
            user=nobody argv=/usr/bin/spamc -f -e
            /usr/sbin/sendmail -oi -f ${sender} ${recipient}
  7. Now find the SMTP/SMTPS lines in your /etc/postfix/master.cf and add the following option:
    -o content_filter=spamassassin

    Assuming you're doing SMTP and SMTPS, you'll have something like this at the top of your /etc/postfix/master.cf:

    smtp      inet  n       -       -       -       -       smtpd     -o content_filter=spamassassin
    smtps     inet  n       -       -       -       -       smtpd     -o content_filter=spamassassin
  8. Reload Postfix:
    # postfix reload

And that's it: you're done.

If you can bear in mind that no spam-detection scheme is perfect, my guess is that you'll be pleased with your decision to set up SA: some spam will make it through, of course, but most of makes it through will come a.) as plaintext with escaped characters and b.) a lengthy disclaimer and an itemized spam "score":

Spam detection software, running on the system "molly", has
identified this incoming email as possible spam.  The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email.  If you have any questions, see
the administrator of that system for details.

Content preview:  Having trouble viewing this email? Click here! pharmacy medicine
   cabinet FSA home medical vitamins personal care diet & fitness men's SALE
   Get 80% Discount TODAY: This email was sent to you by drugstore.com. To ensure
   delivery to your inbox (not junk folders), please add drugstore@e.drugstore.com
   to your address book. [...] 

Content analysis details:   (13.5 points, 5.0 required)

 pts rule name              description
---- ---------------------- --------------------------------------------------
 2.0 URIBL_BLACK            Contains an URL listed in the URIBL blacklist
                            [URIs: lewdozed.cn]
 0.5 FH_HELO_EQ_D_D_D_D     Helo is d-d-d-d

...and so on.

So go ahead: do yourself a favor.

Weekly Digest, 5-3-09

Posted by Weekly Digest in Weekly Digest on May 03, 2009

Apologies for the 3 week gap in "weekly" posts. I was taking a vacation in Hawaii (pics!) and took a bit of time to enjoy life offline :)

Trevor's Links

Geocities: Lessons So Far

Geocities was once called Beverly Hills Internet. The company was founded in 1994 but it wasn’t until mid-1995 that they publically offered what people now think of as a Geocities trademark: free webpages, or “homesteads”. [An article about the Archive Team trying to save Geocities content before Yahoo takes it down.]

How the OAuth Security Battle Was Won, Open Web Style - ReadWriteWeb

At some point in conversation Hammer-Lahav realized that the problem went far beyond the Twitter implementation. The OAuth protocol had an inherent vulnerability; big companies like Google, Netflix and Yahoo had implemented OAuth and scores of tiny startups had too... OAuth has support, but it doesn't have a centralized authority ready to deal with problems like this. Over the next week a story unfolded as the community moved to deal with the security issue. It's a dramatic story.

Tell me your best worst joke, Reddit.

[Includes such classics as: What's brown and sticky? A stick. --- Why does Snoop carry around an umbrella? Fo Drizzle. --- and, my personal favorite: Two snares and a cymbal fall off a cliff.]

Welcome to the Anti-Pitch

We're sick and tired of hack developers ripping off naive clients. And while I'm completely disgusted by some of the horror-stories I've heard lately, clients keep asking the wrong questions. As real developers, it's our responsibility to make the tough decision to speak the truth. This is an example of what we call the anti-pitch. [Excellent. I'm using this technique next time I'm dealing with potential clients.]

What Twitter Looks Like For Twitter Employees

...hackers sent them screenshots from the site Twitter employees use to manage the microblogging service, admin.twitter.com... [It's amazing to see all of the back-end stuff necessary to run something so "simple" as Twitter.]

Honeypot filter as a Rack middleware

Our site’s suggestion box got hammered by a spambot recently, so I created this simple Rack middleware to protect our app from any requests that include a honeypot field.

Rails Edge: Implement FooController.action(:name)

Rails actions are now Rack endpoints, and can be retrieved via FooController.action(name) and called with an env.

Make your site faster and cheaper to operate in one easy step

Is your web server using using gzip encoding? Surprisingly, many are not. I just wrote a little script to fetch the 30 external links off news.yc and check if they are using gzip encoding. Only 18 were, which means that the other 12 sites are needlessly slow, and also wasting money on bandwidth.

Passenger: Command line done right

What’s really great about Passenger is that the attention to detail doesn’t end at the installer. The Linux process list is a list of programs that are currently running. Usually, programs are shown in this list by their command line name, often an indecipherable mix of letters and numbers. Passenger processes are easy to spot and easy to understand. Human readable names in a machine-centred interface.

Muxtape Pushes Play Again

Muxtape’s stock parts are highly regimented, allowing bands to express themselves with freedom, though not completely freely. Every component is 300 pixels square, and there is virtually zero layout flexibility; you can have whatever arrangement you like, so long as it comes in rows of three. What’s more, for now there are no ‘social’ components to draw upon; no commenting, no friending, no favoriting, etc. The new Muxtape platform is nothing if not regimented.

An Aspirational Twitter

Tweetie is a desktop version of an application of the same name for the iPhone which, in my limited experience, is the first time an application has migrated from the phone to the desktop. As a friend mentioned, “Platform merge in progress!” and he’s right... When I use Tweetie, I’m reminded that a maniacal attention to detail not only makes you want to reach out and touch the digitally untouchable, it describes the familiar as the new, and, most importantly, it speaks of an aspirational future.

adamsanderson's open_gem

Gem Command to easily open a ruby gem with the editor of your choice. [Awesome. See the Issues tab for detail, but you need to set GEM_OPEN_EDITOR to 'mate' in your bash profile despite what the instructions might say.]

Tweetie for Mac

You can download the free version, which is ad-supported, and try it out for as long as you want. [The only Twitter client I've been able to use, aside from Tweetie on the iPhone.]

Benchmarking your Rails tests

The first step to faster tests is knowing what is slow. Fortunately, this is dead simple with the test_benchmark plugin by Tim Connor, and originally built by Geoffrey Groschenbach. Install the plugin, and when you run your tests via Rake, you’ll see handy output showing you the slowest tests, and the slowest test classes.

Twitter Clients Are a UI Design Playground

But perhaps the most important factor that has made Twitter such a rich category for client software is that there is so little friction to switch between apps. There’s nothing to import or export, and zero commitment.

Venture Capital Down 50%. It’s Not Just the Recession, Folks.

There’s a huge difference between what venture capitalists say and what they do. [VC] fell off a cliff in 2001 and 2002 and it’s falling off a cliff now.

A Painful Decision

I can’t reveal details without breaking confidences, but suffice it to say that a significant number of Rails core contributors - with leadership (if that’s the right word) from DHH - apparently feel that being unwelcoming and “edgy” is not just acceptable, but laudable. The difference between their opinions and mine is so severe that I cannot in good conscience remain a public spokesman for Rails. So, effective immediately, I’m resigning my position with the Rails Activists. [I haven't gotten up to speed with the controversy around this issue, but I can say for certain that Mike Gunderloy stepping back from his participation in the Rails community is a real serious bummer.]

Heroku - Commercial Launch

We have over 25,000 apps running on the platform today, and many of our users have been asking for pricing and paid services for some time now. So today we’re pleased to announce that we are officially out of beta and available for commercial use.

ShakeItPhoto Launches

It’s been 3 months in the making and 3 months of waiting for Apple approval, but wait no more… ShakeItPhoto is ready for download at the iTunes App store for the low price of 99 cents. Take a photo and shake it like a polaroid to make it develop!

GitHub Issue Tracker

It gives us great pleasure to announce our integrated issue tracking system! On repository pages you’ll now see an “Issues” tab in the top menu.

Phusion Passenger 2.2.0 w/ Nginx support

After spending weeks on further development and intensive testing, we’ve now come to the point wherein we have the distinct honor to announce Phusion Passenger for Nginx as an addition to the Phusion Passenger server line-up.. Our thanks goes out to Engine Yard for financially sponsoring this first release of Phusion Passenger for Nginx, as well as all the people who have in some way donated in the past for making this release possible in the first place.

Is Open Source Experience Overrated?

Just as commercial software can't possibly exist without customers, perhaps open source experience is only valid if you work on a project that attains some moderate level of critical mass and user base. Remember, shipping isn't enough. Open source or not, if you aren't building software that someone finds useful, if you aren't convincing at least a small audience of programmers that your project is worthwhile enough to join... then what are you really doing?

Rails 2.3.2 upgrade gotchas

With the latest stable release of rails out the door for about a month, we’ve had a chance to upgrade the bulk of the applications we maintain to 2.3.2.1. Here are some “gotchas”, aka issues, aka roadblocks to Strategic Enterprise Adoption that we discovered while upgrading some of them.

Draft: The problem with Project Management tools

While I agree that it’s important to release code, I think pivotal and other similar tools lead to a mindset where releasing code is in itself the unit progress. But, as any successful team will tell you, completed tickets and releases released are horrible units of progress, since unless your customers love every single thing you do (they don’t), your unit of measurement becomes the amount of features and changes deployed.

Clone TinyURL in 40 lines of Ruby code

I wrote Snip with Sinatra then deployed it up to Heroku so this is also a good excuse also to describe Heroku, a truly amazing service for the Ruby programming community. The total number of lines in Snip is actually 43, in a single file named snip.rb. including the view template and layout. [It's amazing what you can accomplish with Sinatra and Heroku.]

ruby gc tuning

In my experience, a typical production Rails app on Ruby 1.8 can recover 20% to 40% of user CPU by applying Stefan Kaes's Railsbench GC patch to the Ruby binary, and using the following environment variables...

Customer driven iteration vs Whiteboard driven iteration

Customer driven iteration takes customer validation rather than released features as its core unit of progress. It assumes that you have not accomplished anything and therefore cannot feel good until your metrics tell you that your market will use and pay for your stuff.

Can the Statusphere Save Journalism?

...the discussion shifted to deep conversation about the future of journalism in the era of socialized media with one simple question, “are newspapers worth saving?” Walt thought for no more than two seconds and assertively replied, “It’s the wrong question to ask. The real question we should ask is if whether or not we can save good journalism.”

Are Blogs Losing Their Authority To The Statusphere?

Attention is engaged at the point of introduction, and for many of us, we’re presented with worthwhile content outside of our RSS readers or favorite bookmarks. Relevant and noteworthy updates are now curated by our peers and trusted or respected contacts in disparate communities that change based on our daily click paths... Retweets (RT) and favorites in Twitter, Likes and comments in FriendFeed and Facebook, posting shortened links that connect friends and followers back to the source post, have changed our behavior and empowered our role in defining the evolution of the connectivity and dissemination of information.

jamis's safe_mass_assignment

ActiveRecord plugin for allowing (careful) mass assignment of protected attributes, separate from values provided via users of your application.

Timothy's Links

Sébastien Wains » Howto : setting up dns2tcp

For the "I can't browse from work" crowd or the "stuck behind the Great Firewall of China" set, there are any number of high-visibility, high-availability solutions: tor, your buddy's apache proxy, etc. For those who want to try an obscurity/security/proxy solution that's a little closer to the metal, there's dns2tcp via ssh which, predictably, sends your encrypted traffic from your computer out of your network as a dns request and returns it the same way: you're secure going out and you're not sending up big, "hey everybody: look at my port 80 requests!" red flags to the secret police or the sysadmin or whomever. Cool stuff.

Securing a Web server

This is a pretty good read: it's got a little too much depth to be considered a crash course, but it's too abstract to be a tutorial or how-to. A nice, mid-level view of best security practices.

Twitter + Stimulus = Conservative Stupidity

Normally I wouldn't bookmark DailyKos--that would be kind of like bookmarking HuffPo or Reddit--but this is a neat little article about social engineering / industrial espionage that involves exploiting confirmation bias among partisans. Short read. Good read.

Lifehacker - Should Comic Sans Be "Banned"? - Fonts

This made me laugh out loud. It may make you laugh out loud as well.

Convert files and data online

Supposedly this is the best online format converter. Handy in a pinch (or if you're tired of your CLI converters screwing the pooch on higher ascii and spitting out comic book character swears in place of kanji).

Testing mail servers with swaks

At first glance, this looks like a "for Dummies" tutorial for a piece of software that is, essentially, "telnet for Dummes". But swak lets you do something that you can't (easily) do with plain, old-fashioned telnet. You can, for instance, set a timeout time, specify authentication types, etc. with a commandline flag or two. Handy if you're troubleshooting that new mail server install or doing some eyeball/ball park benchmarking.

Introduction to Quality Assurance and Metrics

If you're looking for a no-bullshit crash course in QA/QC that has decent depth, look no further.

Fujitsu Develops High-Speed Image-Capture Technology for Palm Vein Biometric Authentication : Akihabara News .com

Palm vein biometric authentication? Seriously? I mean, I guess super-futuristic biometric auth devices that scan _inside_ the body for unique identifiers are kind of cool in an aesthetic sense, but they're certainly not very cool from a security sense: I thought we had agreed as a global society that physical objects, no matter how apparently unique they are, are unsuitable for secure auth because they are, at the end of the day, still just objects. And all objects can be replicated.

Skimmers: Reader Finds Card Skimmer On Bank ATM

First reaction: "wow that's totally awesome--I can't believe someone came up with this." Two seconds later's reaction: "wow, my opinion of the human race just got ratcheted down a peg or two: I can't believe it took us this long to invent the ATM card data skimmer."

The peasant mentality lives on in America

You know, three weeks ago, I had no idea who Matt Taibbi was. Then, courtesy of reddit, I got put on to his write-up of the Meltdown and I've been hooked. This guy hits hard, doesn't pull punches and walks the stylistic tightrope between the unnaturally polite tenor of expose journalism and the warbling catachresis of incendiary blogging.

What happens if I don't pay my taxes?

This is a good article because a.) it's timely and b.) is written from a hacker perspective/mentality. It starts with the question, "what is the nature of the system?" and then wonders about different methods of potentially short-circuiting it or circumventing aspects of it. Kind of makes taxes fun. Almost.

What to do when the root partition is full?

This is a good list of comments to scroll through as it discusses Linux mounting tricks, how to use LVM and, basically, lists reasons why not to panic. And, I don' t know about you, but the fewer reasons I have to panic, the better.

Thanko's Latest 4GB Necktie Camera

Yeah, it's basically just a flat camera and a necktie that's been cut open in the back, but the idea is still totally effinf awesome.

A Short Introduction To Cron Jobs

There are two reasons that introductory level, "how to" type documents for the basics of Linux administration are so ubiquitous: those reasons are that they're useful for experienced users to a.) write and b.) comment upon and they're useful for inexperienced users looking things up. This one is about cron and using crontab. And it's a great example of that.

Audit my Server – A guide to performing a quick and thorough security audit on your web-facing server

Posted by Timothy O'Connell in General on April 29, 2009

Security audits are a necessity.

Unfortunately, due to the constantly changing array of exploits and threats coming at your Internet-facing server from all over the world, there's no best practices manual for a security audit. This is because no one knows exactly what level of openness is appropriate/optimal on a given application server.

But you gotta start somewhere. Because if you're just blithely running an un-hardened *nix server with stock configurations for service apps like Apache and ssh, you're pretty much giving away the store.

What follows is a short how-to dealing with how to get started, security-auditing-wise. It is by no means comprehensive and is only intended to provide a "leg up" for those who feel like they ought to be auditing their servers but aren't sure where to start. I'll be using Debian systems running the current stable release (i.e. Lenny) to demonstrate the techniques, but I'll try to keep things as OS agnostic as possible: most of the packaged software I describe below can be found in your generic, mainstream fedora repo.

The methodology will be to start from the outside and work inward. I recently saw a Linux Journal article that described doing things the opposite way (or at least that's the impression with which I was left), and that doesn't make much sense to me. From where I'm standing, it seems that if you're conducting a security audit, you ought to start out by looking at your server the same way everyone else does.

Accordingly, we'll begin from the outside and from as basic a perspective as possible: scanning ports and checking for known vulnerabilities and obvious mis-configurations. Once we've got a little perspective on how our server looks to the script kiddies and botnets of the world, we'll do some web-server specific scanning to attempt to detect vulnerable plugins, apache mis-configurations and application-level security holes. After that, we'll finish by checking for rootkits and doing some internal auditing.




OpenVAS: Port Probe and All-purpose scan
The first thing to do is to scan your ports.

The odds are good that if you've got a server out there on the Internet (in the DMZ of your intranet or in a hosting company's rack, for example) that it's got a bunch of open ports. You've probably got one or two listening for HTTP requests, one or two listening for SSH requests and so on. What you probably haven't got is a good idea of how those ports look to the world.

So the first part of any security audit is the portscan. In olden times, you'd use a combination of telnet and nmap for this: nmap would handle the port scanning and tell you which ports were listening/open and then you'd use your expert knowledge of various network protocols to use a telnet-like program to check out those ports and see what sort of access and information they were offering to the world.

There are, fortunately, labor-saving apps that will do the scanning and auditing for you. A few months back, you would have been using nessus, as it was the big name in F/OSS auditing. nessus, however, has gone commercial (proprietary and closed) and a new, open project (GPL) called OpenVAS has taken its place.

For anything other than security auditing, using commercial software is probably OK. At the very least it's not always counter-productive to use non-F/OSS for non-security-related tasks. In the case of security-related apps, however, it just doesn't make any sense to take a chance on using software that isn't available for public scrutiny.

At any rate, if you're familiar with the way that nessus works, you'll be happy to know that the big ideas and the general methodology/procedure behind using OpenVAS are essentially the same. If you're unfamilar with programs like nessus and OpenVAS, here's how they work, from an administrator/auditor's perspective:

  1. Set up a server
  2. Use a client to tell the server to probe the target site
  3. View the audit report on the client

At present, if you're using the stock stable/unstable Debian repositories, you haven't got access to the packaged version of the OpenVAS server. That being the case, we're going to go ahead and get a little bit heroic here and do this the Cowboy Way (i.e. from source).

  1. Dependencies and Source Files
  2. Make sure that you've got the following packages (some of which the openvas developers list as dependencies, some of which you'll need to compile anything from source) before proceeding:
    molluska:/opt/# aptitude install libgnutls-dev libpcap-dev libgpgme11 libgpgme11-dev libglib2.0 libglib2.0-dev build-essential bison

    Now that that's handled, there are four "modules" that are required to run an OpenVAS server. The openvas developers say that you've got to install the modules in the following order:

    1. openvas-libraries
    2. openvas-libnasl
    3. openvas-server
    4. openvas-plugins

    So that's what wer're going to do. I like to do this sort of thing in/opt, but it really doesn't matter where this happens.

    Get the files:
    molluska:/opt/openvas# wget http://wald.intevation.org/frs/download.php/572/openvas-libraries-2.0.2.tar.gz
    molluska:/opt/openvas# wget http://wald.intevation.org/frs/download.php/561/openvas-libnasl-2.0.1.tar.gz
    molluska:/opt/openvas# wget http://wald.intevation.org/frs/download.php/562/openvas-server-2.0.1.tar.gz
    molluska:/opt/openvas# wget http://wald.intevation.org/frs/download.php/576/openvas-plugins-1.0.6.tar.gz

    NB: these URL's are for the version that was current when this was written--no guarantees that they'll be there two hours from now.

  3. ./configure && make && make install
  4. Now, we start the compilation process which, thanks to our having resolved the dependencies enumerated above, should go off without a hitch:
    molluska:/opt/openvas# tar -zxvf openvas-libraries-2.0.2.tar.gz
    [...]
    molluska:/opt/openvas# cd openvas-libraries-2.0.2
    molluska:/opt/openvas/openvas-libraries-2.0.2# ./configure
    [...]
    molluska:/opt/openvas/openvas-libraries-2.0.2# make
    [...]
    molluska:/opt/openvas/openvas-libraries-2.0.2# make install

    Once you've successfully installed the openvas libraries, you'll be prompted to modify /etc/ld.so.conf by adding the line "/usr/local/lib" to it and running ldconfig to update your linker. Do that, and then repeat the steps described above (untar, configure, make, make install) in the other three folders to finish installing the OpenVAS modules.

  5. Add a User and Generate an SSL Cert
  6. Once you've got everything installed, you'll need to create two things: an OpenVAS user and an SSL certificate. Fortunately, both of these tasks have been nearly fully automated and all you'll have do to get the job done is execute a couple of binaries (which should be on your path, now that you've installed everything according to the above instructions) and follow some on-screen prompts:

    molluska:/opt/openvas# openvas-adduser
    [...]
    molluska:/opt/openvas# openvas-mkcert
    [...]

    And that's it. Make a note of the paths that the openvas-mkcert program gives you at the end of the certificate creation (as you might need to specify them at some later time; you won't need them again to follow these instructions).

  7. Fire it up
  8. Once you've got all your modules installed, your certificate created and your user added, it's time to fire up the server/daemon. The smartest way to do this is to pseudo-daemonize it and tail its output while it loads plugins:molluska:/# nohup openvasd &
    [1] 4508
    molluska:/# nohup: ignoring input and appending output to `nohup.out'
    molluska:/# tail -f nohup.out

    Should get you something like this:tail -f nohup.out
    Loading the plugins... 714 (out of 10558)

    ...and so on. Once the plugins are all the way loaded, fire off a quick ps to make sure that the server is running and maybe a quick netstat to make sure you know what port it's listening at, and that's it: you're done with the server side of things and ready to move on to the client and auditing part.

    molluska:/opt/openvas# ps aux |grep openvas
    root     26129  1.1  0.2  16296    76 ?        S    06:29   1:46 openvasd: waiting for incoming connections
    molluska:/opt/openvas# netstat -anp |grep openvas
    tcp        0      0 0.0.0.0:9390            0.0.0.0:*               LISTEN      26129/openvasd: wai
    

Installing the OpenVas client is much easier.

While there is a packaged version of the OpenVAS client app, we're going to install one from source (mostly so our version of the server matches with our version of the client; this is mostly me being OCD, however, and you can probably get away with using the packaged version). To install the client, we'll follow the same steps as above, but on a different machine:

gonzo:/opt/openvas-client# wget http://wald.intevation.org/frs/download.php/575/openvas-client-2.0.3.tar.gz
[...]
gonzo:/opt/openvas-client# tar -zxvf openvas-client-2.0.3.tar.gz
[...]
gonzo:/opt/openvas-client# cd openvas-client-2.0.3
gonzo:/opt/openvas-client/openvas-client-2.0.3# ./configure
[...]
gonzo:/opt/openvas-client/openvas-client-2.0.3# make
[...]
gonzo:/opt/openvas-client/openvas-client-2.0.3# make install

NB: you may, depending on your client system, have to resolve some GTK dependencies and other build/compiler dependencies like the ones listed above: libgtk2.0-dev should solve most of your gtk problems, if you're running Debian Lenny.

Once the GUI client is installed, start it up:

toconnell@gonzo:~$ sudo aptitude install openvas-client
[...]
toconnell@gonzo:/opt/openvas-client/openvas-client-2.0.3/bin$ ./OpenVAS-Client &


Once the GUI client is up and running, click the connect icon at the top of the interface (looks like two gray tubes colliding) to open the window that allows you to specify your newly install server's location and settings. Fill in the blanks and connect:

Once your client is connected with your server, you're ready to fill in the blanks and start your first round of tests. This is fairly self-explanatory and, honestly, you wouldn't be reading this if you couldn't figure out simple GUI interfaces: specify your target (i.e. the server you're auditing), make sure that all the plugins are enabled and then click the life preserver to start the "Scan Assistant" and execute the scan. Follow the on-screen prompts: easy as apple pie.

The best thing to do, once your scan starts, is probably to go do something else and come back in a little bit: in my experience these scans can take anywhere from 15 to 45 minutes, depending on your server and your pipe: my server is an old Linksys NSLU2 and my pipe is a consumer-grade Speakeasy residential connection, so I'm used to waiting close to an hour for the scan to finish. Using corporate resources will result in less idle time.

Once the scan is done, you're treated to a report view. This is what we've been after all long. In it, you'll see a full run-down of what ports on your server are open and what open ports are listening for what. Additionally, you'll be treated to helpful recommendations about how to close security holes. And while closing those holes is beyond the scope of this article, I will say that almost every recommendation I've gotten from an OpenVAS report has been sane, been sensible and lead to a harder server.



nikto: Web-server Specific Auditing
The second thing to do, in order to perform a robust audit of your system, is to hit it with nikto (http://www.cirt.net/nikto2).

nikto, unlike OpenVAS doesn't require a server/client hook-up: just install the client with apt and fire off some tests, writing the output from those tests to plaintext files:

molluska:/# aptitude install nikto
molluska:/# nikto -h newathens.org -p 80 -output nikto_na80 && nikto -h newathens.org -p 443 -output nikto_na443

You'll get helpful output that points you towards an obvious solution like this:

+ mod_ssl/2.2.9 appears to be outdated (current is at least 2.8.30) (may depend on server version)

And you'll also get put on notice if you've got paths/folders/files with names that automatic exploiters and scripts tend to look for:

+ OSVDB-3092: GET /login/ : This might be interesting...

...to script kiddies and Chinese botnets.

You'll also get put on notice if you've got too much of your software's installation defaults hanging out in the open:

+ OSVDB-3233: GET /icons/README : Apache default file found.



Server-side Checks

chkrootkit
There are a few utilities that allow you to perform quick server-side audits of your security situation. Some of them, like rkhunter will run daily (like logwatch or apticron) and tell you if they've identified any new chinks in your armor. The first one to install and run is chkrootkit.

molly:/# aptitude install chkrootkit
[...]
molly:/# chkrootkit

This is a great place to start your internal audit because it'll tell you if you've picked up any known bugs and whether anything weird, filesystem-wise, appears to be going on with your computer.

The best use for this app is to give you a very quick idea of what sort of shape you're in. If you've got a system littered with suspicious files, odd-looking binaries, etc., you know exactly where to start plugging holes.

rkhunter
While we're on the subject of checking for root kits, let's do rkhunter:

molly:/# aptitude install rkhunter
[...]
molly:/# rkhunter --update
molly:/# rkhunter --check

This gets you a quick check of all your important binaries (to make sure they look like they're supposed to look, i.e. that they haven't been replaced by scripted exploits or an intruder with something that opens a back door) and a quick scan for known exploits of the rootkit variety. You'll also be told whether you're running inetd/xinetd (which tends to open ports in a manner whose security can be less than "ironclad") and other fun facts about potential vulnerabilities.

The best thing to do with this report is think long and hard about what ports/resources/pathways you actually want to make available to the Internet and then start disabling services. Once you've spent some time with that, you're pretty well on your way to having an idea of exactly how hard your server is and how much work you've got to do to keep it safe.

If anyone has any ideas about other utilities or techniques for security auditing, please feel encouraged to share them in the comments.

YUM for Weekend Warriors

Posted by Timothy O'Connell in General on April 27, 2009

Generally speaking, I'm a Debian guy.

Sure, I'll mess around on the CentOS box under my desk on the production RHEL servers at work a little bit, but Red Hat is largely terra incognita for me and Debian is where I'm comfortable doing my admin thing and managing packages. I know apt. I am comfortable with apt. And while I wouldn't describe using yum as something that makes me uncomfortable, when I do have to use it, I find myself spending more time Googling and forum-searching than I'd like.

I'm beginning to accept that this is no one's fault but my own.

And so the purpose of this article, therefore, is not a.) to point fingers, b.) to compare apt to yum or c.) to explain yum from the perspective of someone who is accustomed to doing things the Debian way. The Internet is littered with stuff like that like the intersection of Paradise and Tropicana are littered with advertisements for escorts.

In this post you'll find some novice-level trouble-shooting tips, reminders and pointers for the casual yum user that are intended to help reduce the occurrence of forehead-slaps and to decrease the amount of time spent tailing logs and Googling obscure error messages.

  1. Preemptive Troubleshooting.
  2. It's not in the documentation, but I have noticed that a lot of dependency issues and version consistency problems are resolved by tossing off the following yum command and then trying again:

     # yum clean packages

    It has become my general practice to do this before I do anything else. It's a nice preemptive step.

    I've noticed that it's generally not the advice of performance-minded (read: impatient) admins to do the more scorched-Earth yum clean all, as this empties caches, dbcaches (i.e. sqlite files) and can causes longer check-update times.

  3. filelists.xml.gz Download Times out.
  4. Let's say you're doing a yum update or a yum upgrade and you get some output like this:

    filelists.xml.gz          100% |=========================| 1.4 MB    00:01
    filelists.xml.gz          100% |=========================| 1.3 MB    01:48
    http://apt.sw.be/redhat/el5/en/i386/dag/repodata/filelists.xml.gz: [Errno 4] Socket Error: timed out
    Trying other mirror.

    There are good odds, especially if you're using non-standard repositories, that you copied/pasted some text into your yum.conf from somewhere out there on the Internets. If you did, there are even better odds that the text you copied includes something about using GPG to authenticate the repo. If you've got lines like that, you'll need the repository's key.

    Generally speaking, you can navigate to a repository's http site and find the URL for their public key. Once you've got that, all it takes to import it is one of these:

    # rpm --import http://dag.wieers.com/packages/RPM-GPG-KEY.dag.txt
  5. Know your repositories
  6. You can save a lot of backtracking/head-scratching time if, before searching for a package on a machine you don't visit that often, you toss off a quick yum repolist. This handy feature will spit out the names and statuses of all of the repositories in all the files in your /etc/yum.repos.d/ directory and prevent you from doing that thing where you don't realize that you've only got the default CentOS repositories enabled but can't seem to figure out why the eff your yum search for htop just turned up a big goose-egg.

  7. Automatic Notifications
  8. If, like me, you're coming at yum from a Debian perspective, one of the first things you'll do when you start administering an RPM-based system is to install the apticron-equivalent known as yum-updatesd (# yum install yum-updatesd.noarch). Something you might forget, however, is that the default behavior of yum-updatesd is to not send emails.

    Don't forget to edit /etc/yum/yum-updatesd.conf such that

    mit_via = dbus

    looks like

    mit_via = email

    or you won't get those all-important package update emails.

And that's about all that's coming to mind right now.

If anyone else can think of some things that you consistently forget--and then suddenly remember, 20 minutes later--to do when you're working with yum, feel free to leave a comment.

Tags: , ,