*scottstuff*

I’m finally back home after being away for two weeks, first at OSCON and then on vacation. I’m slowly digging my way through a 1,000-message backlog; hopefully I’ll be able to fix the current batch of Typo bugs and release 4.0.1 by Friday.

I’m on vacation this week, traveling in Oregon. I’ll be checking my mail from time to time, but don’t expect a quick response.

A number of people have had a hard time installing Typo from the .gem on hosting providers’ systems where they don’t have the ability to install things as root. So, I sat down with Jim Weirich at OSCON to talk over the best way to handle non-root installs. Apparently RubyGems has support for this, plus or minus a few bugs. Try this:

  $ export GEM_PATH=~/gems
  $ gem install -i ~/gems typo
  $ ~/gems/bin/typo install /some/path

Rubygems 0.9.0 will end up re-installing .gems that are already installed in the system gem directory, but that’s not fatal. It’s just a bit slower then it should be.

Hopefully the next rubygems release will fix this and add a “you aren’t running as root, use -i” warning.

I’ve mostly finished extracting Typo’s installer into its own Rails project and .gem. The installer makes it trivial to build an installable .gem for any Rails project, so the install process looks like this:

$ gem install my-project
$ my-project install /some/path

The installer source lives in Google Code. That includes a mailing list and bug tracker. I’ll upload my .gem to RubyForge later today, along with some documentation. For now, the rdoc is available.

Here’s what you’ll need to do to add the installer to your existing Rails app:

1. Create a .gem that depends on rails-app-installer, rails, and all other .gems that you need to have installed.
2. Add an executable entry to your gemspec. If your app is called my-app, then add executable = ['my-app'].
3. Finally, create bin/my-app, using one of the examples in the rails-app-installer SVN tree as an example.

Here’s a short example bin/my-app:

  #!/usr/bin/env ruby

  require 'rubygems'
  require 'rails-installer'

  class AppInstaller < RailsInstaller
    application_name 'my_app'
    support_location 'my website'
    rails_version '1.1.4'
  end

  # Installer program
  directory = ARGV[1]

  app = AppInstaller.new(directory)
  app.message_proc = Proc.new do |msg|
    STDERR.puts " #{msg}"
  end
  app.execute_command(*ARGV)

That’s all that’s needed–as long as the installer gem is installed, this will give you a full installer that supports installs, upgrades, db backups and restores, and all of the other things that the Typo installer currently provides. Adding application-specific installer subcommands is easy. Here’s the sweep_cache implementation from Typo’s installer:

  class SweepCache < RailsInstaller::Command
    help "Sweep Typo's cache"

    def self.command(installer, *args)
      installer.sweep_cache
    end
  end

That’s all that’s needed to implement the typo sweep_cache /some/path installer command.

Update: The gem is out. gem install rails-app-installer.

For the next version of my installer, I’d love a generic, DB-agnostic way to perform database backups under Rails. Ideally, I’d be able to serialize (and unserialize) an entire DB to YAML. That way the installer could easily perform backups for people before they upgrade, and the backups would be portable between databases. So someone could start out with SQLite and move to PostgreSQL or MySQL.

The problem is that I don’t actually want to have to write this myself, but I’m not having a lot of luck searching for one. It seems like a pretty obvious tool, though, so perhaps I’m just searching wrong. It shouldn’t be very hard to write the serializer, but the hard part will be getting the deserializer right–with Postgres, you’re going to have to play games to get the DB’s internal sequence numbers set right. Other databases will probably have similar (but different) issues.

Any pointers?

As I mentioned in the Typo 4.0.0 announcement, Typo now includes a .gem-based installer that makes it easy to install Typo. Just install the Typo gem (gem install typo) and run the Typo installer (typo install /some/path) to create a new Typo blog in /some/path. The installer will install all of Typo’s files, create a working set of config files, create a SQLite database for you, and start the Mongrel web server on a random TCP port. It’ll also create a set of sample Apache and Lighttpd configuration files to show you how to tie Typo into your existing web server. One warning: this will only work right if you already have SQLite 3 and SWIG installed on your system. If they’re missing, then you’ll get weird warnings and errors. SWIG is particularly strange–if it’s missing, then you’ll get sporadic test failures when trying to use SQLite.

The same installer can also be used for upgrades–if you’ve installed one of the Typo 3.99.x pre-releases, then you can upgrade the same way you installed Typo in the first place–run gem install typo to grab a newer Typo gem, and then typo install /some/path to upgrade. Typo will recognize the existing install, back up the database, shut down the existing Mongrel server, install new files, upgrade the database, and restart Mongrel.

Once Typo is installed you can test it by connecting directly to Mongrel with your web browser; the installer will display the URL for you. Normally, for production use, you’d configure some sort of proxy or load balancer (like Apache’s mod_proxy) in front of Mongrel, so users talk to Apache and Apache talks to Mongrel. The installer creates a number of example configs in the installer/ directory. Once thing to be careful about–you’ll need to make sure that Mongrel and Typo are restarted when your web server reboots. You can start them by running typo start /some/path. You’ll need to talk to your system administrator or hosting provider to learn the best way to start Typo on boot.

Compared to the half-dozen mutually contradictory install guides that existed before, this is a big step forward. However, not everyone wants to (or can) run Typo under Mongrel with SQLite. Some hosting environments make HTTP proxying difficult, while others would rather use a “real” database. So, in the Rails spirt of convention over configuration, I built the installer to use Mongrel and SQLite by default, but you can configure it for your favorite database with a bit of extra work. There are a number of configuration settings that control the installer’s behavior. The typo config /some/path command will show existing variables. You can change them via typo config /some/path var=value.

As of Typo 4.0.0, the installer knows about 6 different configuration variables:

• web-server: which web server technology Typo will use. It defaults to mongrel. Other options are mongrel_cluster and external. If you want to use FastCGI, then set web-server to external.
• threads: if web-server is set to mongrel_cluster, then threads controls how many Mongrel back ends are used.
• port-number: which TCP port Mongrel listens on. This defaults to a random number between 4000 and 5000. The mongrel_cluster server uses one TCP port per thread, starting with port-number and counting up.
• url-prefix: if Mongrel 0.3.13.4 or higher is installed, then url-prefix can be used to move Typo into a subdirectory. If you want to run Typo on http://www.example.com/blog, then you’ll need to set url-prefix to /blog.
• bind-address: which IP address Mongrel binds to.
• database: which database server Typo will use. The default is sqlite. If you change this, then the installer won’t create a SQLite database for Typo or try to back the SQLite database up during upgrades.

So, if you want to use the Typo installer with FastCGI and Mysql, then you’ll want to do this:

  $ typo config /some/path web-server=external database=mysql

You’ll also need to edit database.yml and create your own database. There are schema files in db/schema.*.sql for several different databases. Pick the one that matches your database.

The typo command supports 7 sub-commands:

• install [version] [config=value ...]. Installs or upgrades Typo. You can optionally specify which version to install, if you have multiple Typo .gems installed. You can also use the installer to install directly out of a Subversion checkout by specifying version cwd and running typo from inside of the Subversion directory.
• start. Starts the Mongrel or mongrel_cluster webserver. If Mongrel has been disabled via web-server=external, then this command does nothing.
• run. Just like starts, but runs Mongrel in the foreground when possible.
• stop. Stops Mongrel. Like start, it is ignored if Mongrel has been disabled.
• restart. Stops and restarts Mongrel.
• config [name=[value] ...]. Without parameters, it shows Typo’s current configuration. With parameters, it sets the configuration parameters. If you specify name= without a value, then it clears the variable.
• sweep_cache. Sweeps Typo’s cache. This can be useful for troubleshooting.

That should be all that you need to know to install Typo and keep it running. Any questions?

I love Amtrak. The Seattle-to-Portland run is pretty good, as far as Amtrak goes, but it’s 7:55, and we aren’t actually in Portland yet. My original ticket said we were due at 5:15, and I’d hoped to catch Ted Leung and James Duncan Davidson’s photography tour at 6:00. Apparently there was a brushfire on the tracks or something.

Oh well, at least I had some time to read and write a bit of Typo documentation.

I’d like to announce the release of Typo 4.0.0, the latest version of the most widely-used Ruby-based blogging software. This is the first official release of Typo 4.0, and the product of almost a year’s work by the Typo team. This is a huge upgrade over the previous Typo release, version 2.6.0. You can download it from Rubyforge, or you can use the new Typo .gem and installer.

At least a dozen people have contributed new code to this version of Typo, and dozens more have helped report bugs. The core Typo team has had two new additions since the last major release–Kevin Ballard and Piers Cawley. They’ve both made a huge contribution to this release in many ways. They’ve added all of the useful features, while I’ve mostly specialized in adding bugs.

Here’s a partial list of changes since Typo 2.6:

• A new installer and a Typo .gem file. Run gem install typo and then typo install /some/path to install Typo.

• Text filter plugins, including easy inline Flickr image support and syntax highlighting for code.

• Enhanced feed support. Atom 1.0 and RSS 2.0 are both supported. Atom 0.3 has been removed. Both feed types have better UUIDs. There are also per-tag, -category, and -author feeds. Most pages have their own content-specific feeds available via feed autodiscovery.

• Tags. The ‘keywords’ field in the Typo admin UI (as well as many blog editors) has been commandeered to provide tagging for Typo. Tags are separated by spaces (just like Flickr). If you want to include a space in a tag, then use quotes.

• Improved spam management. There’s a “Feedback” tab in the admin interface that lists all comments and trackbacks so they can be bulk-deleted. In addition, Typo can now use Akismet for spam filtering.

• File uploads. You can now upload images and other content directly from the admin UI.

• Podcast support (experimental).

• Email and/or Jabber notification of new content, including comments and trackbacks.

• Support for posting articles with a future posting date. Pre-posted articles don’t appear on the blog or feeds until their posting date passes.

• A new cache system that automatically times out stale entries. Several types of content, including the Flickr sidebar, will automatically cause the page to be rebuilt every few hours to ensure freshness.

• Better theme support. Some of this was back-ported to Typo 2.6.0.

• A redirect table to help users migrating to Typo. You can enter new URLs into the Redirect table and Typo will look there whenever it doesn’t recognize a URL. So you can move from Movable Type-style permalinks to Typo-style permalinks without losing the perma- in your links.

• Cleaner migrations.

• Rails 1.1 support. Rails 1.1.4 is strongly recommended. Rails 1.0 won’t work at all.

• Improved sidebar support, with a cleaner API and more built-in sidebars.

• Google sitemap support.

• Gravatar support for comments.

• Comment previews.

• Markup help for comments, articles, and pages.

The single most exciting change for me is the new installer. Typo is almost certainly the world’s most widely distributed Rails app, and we’ve found that it’s really hard for people to get all of Typo’s dependencies installed and working the first time. Even worse, our old documentation wasn’t very helpful. I’ve heard from a lot of people who have spent hours getting Typo working, sometimes without success. My personal favorite comment came from a co-worker:

I tried installing typo last night, and the experience was so comically horrible that I was seriously tempted to blog about it, and make the whole world point and laugh at Typo, haw haw haw.

Properly shamed, we built an installer for Typo. If you’re new to Typo, you can install it like this:

  $ gem install typo
  $ typo install /some/directory

This will install Typo in /some/directory, using SQLite and Mongrel by default. As they say, there is no step three. There are a few prerequisites that you’ll have to have on your system before this will work (Ruby, Gems, SQLite 3, and SWIG), but most people should find the installer a lot easier to work with then the traditional installation mechanism. Of course, if you’re happy with your current Typo install, then there’s no need to use the installer–it’s optional. Checking things out from Subversion or downloading .tar or .zip files still works fine. It’s just more work.

I’ll post more details on the installer here soon. I’m planning on extracting it from Typo and bundling it into its own Rubyforge project so other Rails apps can use it. Let me know if you’re interested.

For now, please report Typo bugs on typosphere.org or the Typo mailing list.

Typo 3.99.4 is out. Hopefully this will be the final 3.99.x release, and I’ll be able to release 4.0.0 this weekend.

There are a ton of bugfixes in this release, plus a couple new spam-handling features.

First, Typo now includes Akismet support. Akismet is a blog spam filter implemented as a web service. You’ll need to register with them and get an API key before it’ll work. Once its enabled it’ll work alongside the existing spam blacklist system. If a new comment or trackback fails the spam test then it will be saved in a queue for moderation.

There’s a also new ‘Feedback’ tab in the admin interface that shows all recent comments and trackbacks and allows you to publish, unpublish, and delete comments.

I’m really happy with 3.99.4. Give it a spin. You should be able to run gem install typo once Rubyforge finishes replicating it, or you can download the .gem from my server, or you can download a tarball or zip file from RubyForge.

I’m headed for OSCON in Portland next week, along with about half of my blogroll. I’m heading down to Portland on Sunday via Amtrak’s afternoon run; if anyone else is taking the same train, feel free to stop by and say hi. I’ll be the guy in business class editing Ruby code with TextMate on a 17” PowerBook.

Monday and Tuesday are tutorial days at OSCON; I’m going to be sitting in on Amy’s Javascript tutorial on Monday morning, followed by tutorials on Linux drivers, Haskell, and web testing.

The rest of my schedule is still in flux. Here’s the bits that are nailed down:

• I’ll be in the Google booth from 2:30 to 5:00 Wednesday and Thursday. Free to stop by and say hi.
• Google’s throwing a reception in the booth on Wednesday, from 5:00 until 7:30.
• PDX.rb is hosting FOSCON again this year, starting at 7:30 on Wednesday. Last year’s FOSCON was great, but I don’t know how they’ll compete without _why this year.
• I’d like to catch Greg’s talk at 1:45 on Thursday.

Leave a comment if there are any talks that I really shouldn’t miss.

I just committed the last big chunk of code for Typo 4.0.0: improved spam filtering. We now know about Akismet, and will use them for spam filtering if you enter an Akismet key on Typo’s setttings page. Incoming comments (and trackbacks) that fail the spam check aren’t published unless the blog owner approves them via the handy new ‘feedback’ page.

I really want to get Typo 4.0.0 out before Monday, so please pound on this and let me know how it goes.

One of the dirty little secrets of Typo development is that this blog has been running the same Typo snapshot for almost 9 months. I’ve added a lot of code to Typo, but I haven’t been eating my own dogfood. There are several reasons for this, but the biggest was my threaded comment patch from early last year. It wasn’t merged into the trunk, and over time my blog and the Typo trunk diverged to the point where it wasn’t really practical to merge them anymore.

I finally gave up today and ditched my threaded comment code. I’ll rewrite it for Typo 4.1.

The migration to the current Typo trunk was reasonably painless, but it took most of an hour. I have more content then most users, and I’m running on slower hardware, so an hour is probably the worst-case scenario.

Now that I’m on the trunk, it’ll be a lot easier for me to test out spam filtering techniques here before committing them for general consumption.

Let me know if you see anything broken.

We made a lot of progress this weekend on Typo. We’re down to 9 open tickets, and several of those aren’t really code issues–things like “set up a Typo blog on http://typosphere.org” (which is mostly done, actually). There are a few bugs left, and one or two minor development points, but at this point the thing we really need most is some testing. If you use Typo, then I’d love it if you could either test the latest trunk or the latest .gem release (sudo gem install typo; typo install /some/path) and file bugs.

I think I just killed The Great Typo Memory Leak of 2006 (not to be confused with last year’s great memory leak).

I think this was the leak that has been causing so many problems for Typo users on shared hosting systems lately. I’d been putting off debugging it, because there really aren’t any good tools for Ruby memory leak tracing, and there’s enough magic in Rails that makes it hard to even fully understand the control flow in your rails app, much less when memory is being allocated and freed.

Since Typo 4.0 is nearly ready for release, I was running out of time to fix it. Fortunately, during some benchmarking this morning I discovered something useful:

1. With the cache disabled, my Typo processes stay around 30 MB.
2. With the cache enabled, my Typo processes grew to 70 MB within a couple hundred hits to the same URL.

Since the cache-hit path is about 1/100th as long as the cache-miss path, it’s a lot easier to debug. And since we’re using our own reimplementation of Rails’s Action Cache, I have a decent understanding of the caching code. As it turns out, though, I think the normal action cache would have had the same problem.

The problem was that we have multiple before and after filters, and at least one of them saves state in the before filter and expects the after filter to free it. But suppose our filter order looks like this:

  stateful_before_filter
    action_cache_before_filter
      render a page
    action_cache_after_filter
  stateful_after_filter

The problem is that the action cache’s before filter will cancel the rest of the render chain if it gets a cache hit. So stateful_after_filter will never be called. So, whatever stateful_before_filter saved (by shoving it into a class variable, for instance) is now with us forever.

In general, you should be able to fix this by adjusting the order that you apply your filters. The first call to around_filter goes on the outside and all further calls are nested inside it. However, Typo currently applies a couple filters in a superclass controller, so there’s no easy way to move the caches_action call into the superclass. So, as a workaround, I changed the call to around_filter in caches_action_with_params to be prepend_around_filter instead of around_filter. Now my Typo processes stay around 32 MB even after thousands of cached hits.

Longer-term, I want to clean up our controller hierarchy, because I don’t fully understand why we need multiple layers of stacked controller classes when 95% of Typo is in one controller. But for now the memory leak is fixed and we’re back on track for Typo 4.0 in the next week or so.

There’s a lesson here to other Rails users–be careful nesting filters when you’re using the action cache (or any filter that can terminate the request), because you can’t depend on the after_filter being called. If you save things in the before_filter, expect leaks unless you’re very careful with your filter nesting order.

It looks like Python for Nokia S60 3rd edition phones (like my E61) is now available. You can download it from SourceForge.

I’d really rather see Ruby on my phone, but Python will work. I’ll download and install it after I finish getting Typo 3.99.2 out.