Archive for June, 2009

Config vars and Heroku

Posted by Trevor in El Dorado, Ruby/Rails on June 25, 2009

I don't really care for the suggested approach in the Heroku docs for setting configuration variables locally. I have an open-source project that I'm working to get onto Heroku, so I decided to do a little work to come up with a solution that I prefer. I think this would work well for open source projects, as well as projects with multiple developers.

Here's the basic idea:

You have a config file that contains all of your local configuration variables. It looks a lot like database.yml.

 
# config/config.yml
 
development:
  session_key: example_development
  session_secret: ESl6X3oKM1i1RRrD2QLwUUzz9jr1zxNO
  domain: http://example.com
 
test:
  session_key: example_test
  session_secret: vrwPpJTvwnMVLP1wTSgqigSl7PMI7QcE
  domain: http://example.com
 
production:
  session_key: # any string identifying your app
  session_secret: # a random, secret string at least 32 characters long
  domain: # http://example.com
  mailer: # noreply@example.com
 

You perform a little trickery in environment.rb to prefer the Heroku ENV storage of config vars (in the production environment), but you fall back to your config.yml if the config vars aren't found in ENV (in the development and test environments).

 
# config/environment.rb
 
Rails::Initializer.run do |config|
  require 'yaml'
 
  # support yaml and heroku config vars, preferring ENV for heroku
  CONFIG = (YAML.load_file('config/config.yml')[RAILS_ENV] rescue {}).merge(ENV)
 
  config.action_controller.session = {
    :key => CONFIG['session_key'],
    :secret => CONFIG['session_secret']
  }
end
 

Then, you create a rake task (rake heroku:config) that can be used to send all of the config vars for your production environment up to Heroku. This task can be invoked once to set things up, but can also be run again if you need to make any additions or changes.

 
# lib/tasks/heroku.rake
 
namespace :heroku do
  task :config do
    puts "Reading config/config.yml and sending config vars to Heroku..."
    CONFIG = YAML.load_file('config/config.yml')['production'] rescue {}
    command = "heroku config:add"
    CONFIG.each {|key, val| command << " #{key}=#{val} " if val }
    system command
  end
end
 

This way, you've got all of your config vars stored with the project (.gitignored, of course)...

 
# .gitignore
 
/tmp/**/*
/log/*
*.log
/tmp/restart.txt
/config/config.yml
/config/database.yml
/db/*.sqlite3
 

...and you can easily set what you need on Heroku, like so:

 
$ rake heroku:config
Reading config/config.yml and sending config vars to Heroku...
Adding config vars:
  session_key => example_production
  session_secret => 1WlkMkYYi5611vtF...0ZMS2G3Xl67s4lEIK4sj65
  domain => http://example.com
  mailer => noreply@example.com
Restarting app...done.
 

The result is a pretty nice, I think.

You can see the installation and deployment instructions for my open source project El Dorado if you're curious about the overall flow.

I'd love to get some feedback on this approach, but I really like it so far :)

Install Ruby Enterprise, Phusion Passenger and El Dorado on Debian Lenny

Posted by Timothy O'Connell in El Dorado, Ruby/Rails on June 24, 2009

These instructions require and assume the following:

  • You're running Debian Lenny and you've got root access
  • You've got a functioning apache2 installation
  • You know the basics of working on the command line (i.e. how to edit files, execute commands, etc.)

If the above is true of your situation, read on to learn how to install Ruby Enterprise, Phusion Passenger and El Dorado from scratch in a sort of "one-off" setting where you've got one server and you want it to run one site.

NB: These instructions don't use git or capistrano. The instructions contained in the El Dorado README describe how to install El Dorado using those tools. Using them makes for an easier and cleaner installation. It also makes for easier scalability, upgrading and patching: I highly recommend using those tools.

  1. Resolve Dependencies
  2. The first thing you'll need to do, even before installing RE or PP, is make sure that you've got the development files for the databases that RE and PP applications use:

    apt-get install libsqlite3-ruby postgresql-8.3-plruby libmysql-ruby libmysqlclient15-dev postgresql-server-dev-8.3 libsqlite3-dev

    If you don't resolve these dependencies now, you'll get a message during the RE installation that prompts you to install gems for mysql, postgres, etc. and then, when you go to install those gems, you'll get an error like this:

    ERROR:  Error installing mysql:
    	ERROR: Failed to build gem native extension.

    So just go ahead and resolve those dependencies in advance.

  3. Install Ruby Enterprise
  4. The best practice for this, as far as I know, is to install the current stable release of RE in /opt/. First, download the release you plan to use:

    lana:~# cd /opt
    lana:/opt# wget http://rubyforge.org/frs/download.php/58677/ruby-enterprise-1.8.6-20090610.tar.gz

    Once that's down, untar it and execute the installer script:

    lana:/opt# tar -zxvf ruby-enterprise-1.8.6-20090610.tar.gz
    [...]
    lana:/opt# cd ruby-enterprise-1.8.6-20090610/
    lana:/opt/ruby-enterprise-1.8.6-20090610# ./installer

    That should run, after a few tappings of ye olde Enter key, to its error-free conclusion. If, during the installation, the installer finds that you're missing software packages, the installer will bail and you'll be given some commands that fill those holes. Resolve those dependencies and finish the installation.

    At the end of the installation, you'll be given some syntax that will automatically install PP. You'll use that in the next step.

  5. Install Phusion Passenger
  6. Use the automatically generated syntax:

    lana:/opt/ruby-enterprise-1.8.6-20090610# /opt/ruby-enterprise-1.8.6-20090610/bin/passenger-install-apache2-module

    Again, the installer will bail and prompt you to resolve dependencies if you've got any:

    Installation instructions for required software
    
     * To install Apache 2 development headers:
       Please run apt-get install apache2-prefork-dev as root.
    
     * To install Apache Portable Runtime (APR) development headers:
       Please run apt-get install libapr1-dev as root.
    
     * To install Apache Portable Runtime Utility (APU) development headers:
       Please run apt-get install libaprutil1-dev as root.

    Resolve dependencies and finish the installation.

    Once it's finished, you'll be given some lines to add to your "Apache configuration file". The best file to add these lines to is /etc/apache2/httpd.conf.

    Just don't forget that you added them there (as opposed to somewhere else), as you'll need to modify them if you upgrade RE.

    You'll also probably want to go ahead and add the following lines while you've got the file open:

    PassengerPoolIdleTime 14400
    PassengerMaxInstancesPerApp 2

    Those lines do exactly what it looks like they do. They're also very sensible settings to start with, as they'll prevent El Dorado from hogging a bunch of system resources, etc. right off the bat.

    You can find more information here.

    Finally, your /etc/apache2/httpd.conf file should look something like this:

    PassengerPoolIdleTime 14400
    PassengerMaxInstancesPerApp 2
    
    LoadModule passenger_module /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/passenger-2.2.4/ext/apache2/mod_passenger.so
    PassengerRoot /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/passenger-2.2.4
    PassengerRuby /opt/ruby-enterprise-1.8.6-20090610/bin/ruby

    Once you've made those changes, you're ready to begin installing El Dorado.

    When it exits, the PP installer will show you some sample syntax for how to write an apache configuration file for your first application. You can ignore that for now, as we're going to come back to it later.

  7. Install El Dorado
  8. First, get the latest release of the software from Trevor's github: http://github.com/trevorturk/eldorado/tree/master

    Once you've got the URL of the latest release, switch from root to a less privileged user, make a folder in your home dir for the site, download the latest release of El Dorado to that directory and untar it:

    toconnell@lana:~$ mkdir example.com
    toconnell@lana:~$ cd example.com
    toconnell@lana:~/example.com$ wget wget http://download.github.com/trevorturk-eldorado-a37d0c71e928f605d111d5f48b5786ff613bf676.tar.gz
    tar -zxvf trevorturk-eldorado-a37d0c71e928f605d111d5f48b5786ff613bf676.tar.gz
    

    Now, get all of those files out of that big, ugly directory and into the current working directory and ditch those old files:

    toconnell@lana:~/example.com$ mv trevorturk-eldorado-a37d0c71e928f605d111d5f48b5786ff613bf676/* .
    toconnell@lana:~/example.com$ rm -rf trevorturk-eldorado-a37d0c71e928f605d111d5f48b5786ff613bf676*

    Now, follow the instructions in the README and copy the example yml files to the places where the application will look for real, non-example files:

    toconnell@lana:~/example.com$ cp config/database.example.yml config/database.yml
    toconnell@lana:~/example.com$ cp config/config.example.yml config/config.yml

    Now, use your favorite editor to edit the last stanza in config/config.yml so that it matches the information of your site:

    production:
      session_key: example_production
      session_secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  # Replace these X's and make this string (at least) 32 random alpha-numerics for good site security
      domain: http://example.com
      mailer: noreply@example.com  

    NB: There are "dev" and "test" entries in this default file. If you're not planning on doing anything development related with this installation, you can safely delete those entries.

    Once you've edited that file, that's it, so far as the non-git installation is concerned. To get El Dorado up and running, you'll need to do some minor database tasks. Those are covered in the next section.

  9. Configure the Database
  10. Since MySQL is deprecated, I'll be using PostgreSQL for the remainder of these instructions.

    If you look at config/database.yml, you'll notice that it's essentially a blank template:

    development:
      adapter: sqlite3
      database: db/development.sqlite3
      timeout: 5000
      # adapter: mysql
      # database: eldorado_development
      # username:
      # password:
      # host: localhost
    
    test:
      adapter: sqlite3
      database: db/test.sqlite3
      timeout: 5000
    
    production:
      adapter:
      database:
      username:
      password:
      host:

    First, edit that file:

    production:
      adapter: postgresql
      database: example
      username: example
      password: XXXXXXXXXXXXXXXXXXXX
      host: localhost

    NB: Again: once you've added your "production" entries to this file, you can feel free to delete the "test" and "dev" lines, as they do nothing and could cause confusion down the line.

    Now, create the database and the user:

    toconnell@lana:~/example.com$ sudo su postgres -c "createuser example"
    Shall the new role be a superuser? (y/n) n
    Shall the new role be allowed to create databases? (y/n) n
    Shall the new role be allowed to create more new roles? (y/n) n
    toconnell@lana:~/example.com$ sudo su postgres -c "createdb example"

    Next, start the postgres monitor as the postgres user and make the a few changes:

    toconnell@lana:~/example.com$ sudo su postgres -c psql
    Welcome to psql 8.3.7, the PostgreSQL interactive terminal.
    
    postgres=# ALTER USER example PASSWORD 'XXXXXXXXXXXXXXXXXXXX';
    ALTER ROLE
    postgres=# ALTER DATABASE example OWNER TO example;
    ALTER DATABASE

    Now, if you've got your Postgres database configured correctly and your new user can access your new postgres database, you're ready to rake the El Dorado production database:

    toconnell@lana:~/example.com$ /opt/ruby-enterprise-1.8.6-20090610/bin/rake rake db:schema:load RAILS_ENV=production

    Once the database is successfully raked, all you've got to do to finish up is configure Apache.

  11. Apache Configuration
  12. The following assumes that you're doing apache the "Debian way".

    If this is true, the first thing you'll do is create a symlink in /var/www/ that points at your install directory:

    lana:/var/www# ln -s /home/toconnell/example.com/

    Next, create a file in /etc/apache2/sites-available with the name of your site and then create a symlink to it in /etc/apache2/sites-enabled.

    The file should look something like this:

    #
    # example.com
    #
     
    <VirtualHost *:80>
      ServerName example.com
      ServerAlias www.example.com
      ServerAdmin youremail@example.com
      DocumentRoot /home/toconnell/example.com/public
     
      <Directory "/var/www/example.com">
        Options FollowSymLinks
        AllowOverride None
        Order allow,deny
        Allow from all
      </Directory>
     
      RewriteEngine On
     
      RewriteCond %{HTTP_HOST} ^www\.example\.com$ [NC]
      RewriteRule ^(.*)$ http://example.com$1 [R=301,L]
     
      RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
      RewriteCond %{SCRIPT_FILENAME} !maintenance.html
      RewriteRule ^.*$ /system/maintenance.html [L]
     
      ErrorLog /var/log/apache2/example_error_log
      CustomLog /var/log/apache2/example_access_log combined
      RewriteLog /var/log/apache2/example_rewrite_log
      RewriteLogLevel 9
     
    </VirtualHost>

    NB: I've added some apache custom logging. Logs are good.

    Once you've got the file in /etc/apache2/sites-available and the symlink in /etc/apache2/sites-enabled that points at that file, you should be ready to restart apache and get rolling:

    lana:/etc/logrotate.d# /etc/init.d/apache2 reload

And that, as they say, is that. Once you reload apache, provided that your DNS is set up correctly and you haven't got any system problems beyond the scope of this document, your single instance of El Dorado should be ready for prime time.

Navigate to your site in your browser and create an administrative account: the first user who attempts to login will be the administrator. Once you've got your admin created, you're ready to start tweaking your new El Dorado site's appearance and adding users.

A note on upgrades: if you find you need/want to upgrade an instance of El Dorado that has been installed thus, consult the README. The basic gist is that you're going to want to download/copy the new source/program files over the old ones (while being careful not to erase your user-uploaded files) and then run rake db:migrate RAILS_ENV=production.

Weekly Digest, 6-22-09

Posted by Weekly Digest in Weekly Digest on June 22, 2009

How to speed up gem installs 10x

Answer: Turn off ri and rdoc installation.

Perch

Perch is a really little content management system for when you (or your clients) need to edit content without the hassle of setting up a big CMS.

Installing Ruby on Rails and PostgreSQL on OS X, Third Edition

Over the past few years, I’ve helped you walk through the process of getting Ruby on Rails up and running on Mac OS X. The last version has been getting a lot of comments related to issues with the new Apple Leopard, so I’m going this post will expand on previous installation guides with what’s working for me as of January 2008.

Thoughts on Opera Unite

Opera’s CEO Jon von Tetzchner claims that “Opera Unite now decentralizes and democratizes the cloud." I call bullshit. Opera Unite does indeed rely on a P2P-like network to function, but the big problem is that you must push all your traffic through Opera’s proxy service.

LESS - Leaner CSS

Less is Leaner css. Less extends css by adding: variables, mixins, operations and nested rules. Less uses existing css syntax. This means you can migrate your current .css files to .less in seconds and there is virtually no learning curve.

YC Company Hosting Stats

[Interesting stats and discussion on hosting.]

Rip: a RubyGems Replacement?

This makes package management as simple as passing files between friends. Email me your latest library, and I can run rip install path/to/lib. That’s it — you don’t need spec files, and you don’t need to build anything before your send me your code.

BigTable

BigTable is a fast and extremely large-scale DBMS. However, it departs from the typical convention of a fixed number of columns, instead described by the authors as "a sparse, distributed multi-dimensional sorted map", sharing characteristics of both row-oriented and column-oriented databases. BigTable is designed to scale into the petabyte range across "hundreds or thousands of machines, and to make it easy to add more machines [to] the system and automatically start taking advantage of those resources without any reconfiguration".

Opera Unite reinvents the Web: a Web server on the Web browser

[Very interesting possibilities here. Making it easier for people to serve content on the web can only lead to good things.]

tenderlove's markup_validity

Test for valid markup with test/unit or rspec

Hemlock

Hemlock is an open-source framework that combines the richness of Flash with the scalability of XMPP, facilitating a new class of web applications where multiple users can interact in real time. Games, workspace collaboration, educational tools… The only limit is your imagination.

Rip: A New Package Management System for Ruby

But why a completely new package manager? What's wrong with RubyGems? We asked one of Rip's developers, Chris Wanstrath...

Ruby at ThoughtWorks

ThoughtWorks started using Ruby for production projects in 2006, from then till the end of 2008 we had done 41 ruby projects. In preparation for a talk at QCon I surveyed these projects to examine what lessons we can draw from the experience. I describe our thoughts so far on common questions about Ruby's productivity, speed and maintainability.

[git pull] drm-next

See? All the rules really are pretty simple. There's that somewhat subtle
interaction between "keep your own history clean" and "never try to clean
up _other_ proples histories", but if you follow the rules for pulling,
you'll never have that problem.

GitHub Protip: Follow other users

Posted by Trevor in Code on June 11, 2009

Inspired by this post, I thought I'd share a tip that helps me get the most out of GitHub.

Don't just follow the projects that you're interested in — follow other users. Here's a list of people that I'm following. They're constantly turning me on to new and interesting projects, because I get to see everything they're working on, and everything they're following.

 

Dig around the users that I follow, check out what they're been up to, and try it out. If you find that your feed becomes a bit much to manage, try subscribing to your personal RSS feed. There's a link on the home page when you're logged in.

Thanks, GitHub. You're the best.

Automatically Rotate your Log Files in Development

Posted by Trevor in Ruby/Rails on June 11, 2009

I'm trying to save hard drive space, since I've got this super small (and fast?) SSD hard drive on the way. I noticed that I was using a TON of space to store totally worthless logs for my Rails apps. Now, I know I could set up proper log rotation, but I don't feel like going through the trouble for my local machine.

Here's a quick tip I picked up here that will set your logs to automatically rotate in the test and development environments. Just add the following line to these files:

  • config/development.rb
  • config/test.rb
config.logger = Logger.new(config.log_path, 2, 20.megabytes)

Make sure you've got these in your .gitignore file as well:

/log/*
*.log

That will keep your log files under control, but with plenty of room for digging in if need be.

Speed up your Apache/Passenger Rails app in 2min

Posted by Trevor in Ruby/Rails on June 11, 2009

Here's a quick tip for speeding up your Apache/Passenger powered Rails app. It'll take you about 2 minutes, and I guarantee you'll notice the speed-up.

  • SSH onto your VPS
  • Run the following commands: "a2enmod expires" and "a2enmod deflate"

Now, open up your Apache vhost config for your Rails app. Add the following:

Then, restart Apache by running: "/etc/init.d/apache2 restart"

This will gzip your html, css, and javascript. It'll also add far future expires headers for the appropriate cacheable filetypes. There's no downside, and it only takes a second. Bang for buck.

Edit: Check the comments for some possible downsides... ;)

Weekly Digest, 6-11-09

Posted by Weekly Digest in Weekly Digest on June 11, 2009

In this edition, Timothy moves to Washington DC and Trevor trims down his "watch list" on GitHub and shares many interesting projects with you via his delicious feed.

Trevor's Links

Email. Twice daily. No more, no less.

So, using some motivation from The Four Hour Workweek1, I opted to come back to the studio and change my behavior. That morning, I emailed my entire team and my clients to let them know that I would only be checking my email at 10am and 4pm each day.

How to Build a Popularity Algorithm You can be Proud of

Many web sites allow users to casts vote on items. These visitors' votes are then often used to detect the items' "popularity" and hence rank the rated items accordingly. And when "rank" comes into play things gets tricky...

Online communities, etc.

Anyway, I'm bored on a long bus drive and there's no real moral to the story here, just writing. I will be tuning out of the social networking sites because at the end of the day it's now doing more harm than good in the bigger picture and the experiment seems to have yielded a result. Idiots rule.

Really Simple Rails Log Rotatation

I always used logrotate Linux tool to setup log rotation for my Rails apps which has worked fine although it required finding some external config file and understanding its config options and syntax. [Great tip for development/test environments. Might not be a good idea in production?]

Instapaper bookmarklet, modified to close the current tab

I modified the bookmarklet slightly so that the tab closes immediately, without disturbing the pop-up. This way, saving something for later is one simple action, instead of two.

DeliciousSafari

Use and create Delicious bookmarks from the Safari web browser.

So, about this Shopify Platform

The Shopify platform allows any programmer to create applications that integrate natively with the administration interface or storefront. These applications can be written in any language and communicate with Shopify using our handcrafted REST API. We even provide some amazing rails generators to get started quickly.

Introducing Trample: A Better Load Simulator

Most load sim tools make requests to a static list of urls. They spawn n threads and make requests to the urls on the list in succession, in each thread. Unfortunately, though, if your applicaition makes use of any kind of caching (including your database's internal caching facilities), this kind of load simulation is unrealistic.

TOSBack | The Terms-Of-Service Tracker

TOSBack keeps an eye on 44 website policies. Every time one of them changes, you'll see an update here.

Twitter Blog: Not Playing Ball

We do recognize an opportunity to improve Twitter user experience and clear up confusion beyond simply removing impersonation accounts once alerted. We'll be experimenting with a beta preview of what we're calling Verified Accounts this summer.

cdto

Fast mini application that opens a Terminal.app window cd'd to the front most finder window. This app is designed (including it's icon) to placed in the finder window's toolbar.

Trevor's GitHub Links

quirkey's sammy

Sammy is a tiny javascript framework built on top of jQuery inspired by Ruby's Sinatra.

kabuki's heresy

Heresy is a schema free wrapper around your database, heavily inspired by both CouchDB and FriendFeed.

paulmars's seven_minute_abs

ab testing for rails

binarylogic's searchlogic at v2

Searchlogic has been completely rewritten for v2. It is much simpler and has taken an entirely new approach. To give you an idea, v1 had ~2300 lines of code, v2 has ~350 lines of code.

semanticart's is_paranoid

ActiveRecord 2.3 compatible gem "allowing you to hide and restore records without actually deleting them." Yes, like acts_as_paranoid, only implemented differently...

brynary's webrat

Webrat - Ruby Acceptance Testing for Web applications.

mbleigh's twitter-auth

Standard authentication stack for Rails using Twitter to log in.

courtenay's splam

Simple, pluggable, easily customizable score-based spam filter plugin for Ruby-based applications.

jeremy's ruby-prof

a fast code profiler for Ruby

nakajima's roleful

Generic roles for you and your objects.

37signals's wysihat

A WYSIWYG JavaScript framework

binarylogic's authlogic

A clean, simple, and unobtrusive ruby authentication solution.

joshuaclayton's blueprint-css

A CSS framework that aims to cut down on your CSS development time.

stephencelis's dots

Free progress dots for your scripts. Test::Unit-style.

wycats's merb-extlib

Ruby core extensions library extracted from Merb core.

jodosha's plugin_migrations

Rake tasks for running plugin migrations.

tcocca's acts_as_follower

A Plugin to add "Follow" functionality for models

mojodna's active_queue

A toolkit for queueing tasks and creating worker processes