*scottstuff*

Typo 2.5.7 is available

Posted by Scott Laird Wed, 09 Nov 2005 00:22:12 GMT

I just released Typo 2.5.7. This is a relatively minor upgrade over 2.5.6, but it adds two important features:

1. Support for Rails 0.14.x. Due to one little change, Typo 2.5.6 worked fine with Postgres and the new Rails, but MySQL failed horribly. This should now be fixed.
2. Support for overriding views from within themes. The theme contest folks really want this, and it’s been available in the trunk for months, so I backported it.

Posted in Typo | Tags release, rubyonrails, typo | 1 comment

Typo Theme Contest, again

Posted by Scott Laird Tue, 08 Nov 2005 02:53:16 GMT

As a reminder, the Typo Theme Contest is still running. The deadline for entry is November 28th, so there’s still time to start, but you don’t want to wait much longer. The prize pool has grown again–the top prizes are a new 15” PowerBook, a 12” iBook, an iPod Nano, and a year’s free hosting. Although it pales in comparison to a new PowerBook, I’d love to bundle the top couple themes with future Typo releases, so you can count on thousands of Typo users enjoying your work.

Posted in Typo | Tags rubyonrails, themes, typo | 2 comments

Typo moves to Rails 0.14.2

Posted by Scott Laird Fri, 04 Nov 2005 19:06:20 GMT

As of this morning, the Typo trunk requires Rails 0.14.2. I ran rails . and then spent a couple hours cleaning up Rakefile and environment.rb, and then updated all of the tests to work without instantiated fixtures. As advertised, this makes tests run quite a bit quicker; since Typo’s test suite had been flirting with the 5 minute mark on my laptop, anything that we can do to speed up tests is welcome.

Now that we’re safely requiring 0.14.x versions of Rails, I’ve started adding session :off all over the place. My sessions table for scottstuff.net has around 70,000 sessions in it right now, and I doubt that more then 15 or 20 of them were ever useful. We use sessions for user authentication, but that really only matters for admin pages plus a few other cases, like comment posting. In essence, any page that can be cached will never need access to a session. So, I made a little change to the Article controller. Instead of

class ArticlesController < ApplicationController
  caches_page :index, :read, :permalink, :category, 
    :find_by_date, :archives, :view_page, :tag

  ...
end

I’m now using

class ArticlesController < ApplicationController
  cached_pages = [:index, :read, :permalink, :category, 
    :find_by_date, :archives, :view_page, :tag]
  caches_page *cached_pages
  session :off, :only => cached_pages

  ...
end

This is a really common pattern, because pages that can use the page cache really shouldn’t depend on the session in any way. If I was doing this in more then one place, I’d probably want to extract it into a nice little plugin, but I’m just not feeling the motivation right now.

Posted in Typo | Tags ruby, rubyonrails, typo | no comments

The Great Typo Memory Leak

Posted by Scott Laird Mon, 24 Oct 2005 19:30:55 GMT

A number of users complained this weekend that Typo was using way too much memory, with reports of 100+ MB per FastCGI dispatcher. Typo usually uses around 20 MB, and even that’s too much; 100 MB is enough to cause big problems with hosting providers like TextDrive.

The first step that I took was to verify that the problem actually exists outside of TextDrive. I set up a test Apache/FastCGI/Typo server, disabled caching, and then pounded on it using curl:

# while true; do curl http://typo1/ > /dev/null; done

I let that run for a few seconds and watched while my dispatch.fcgi processes grew from 22 MB to 80 MB. I then did a bit of experimenting:

• The main index page leaked
• RSS feeds didn’t leak
• Individual article pages leak
• Static pages, like /pages/about leak
• Error pages even leak

Disabling the layout for a leaking page and then re-testing it showed that the leak followed the layout. Turning layouts back on and removing the sidebar block fixed the leak.

Entertainingly enough, disabling the sidebar from inside of the sidebar infrastructure didn’t fix the leak. The mere act of calling render_component to generate the sidebars seemed to be causing the memory leak. Since Typo is one of the very few users of Rails components, this suggests that render_component may have a leak that no one else has noticed, so I created a new test Rails app with only two files. First, app/controllers/foo_controller.rb:

class FooController < ApplicationController
  def bar
    render_component :layout => false, 
      :controller => 'sidebars/sidebar', :action => 'index'
  end
end

Then components/sidebars/sidebar_controller.rb:

module Sidebars
  class SidebarController < ApplicationController
    def index
      render :text => 'test', :layout => false
    end
  end
end

This is about as minimal as a Rails app can get. Then I set up a FastCGI server running this project, and ran curl against /foo/bar, and watched the process size climb. So the leak is part of Rails, not really part of Typo.

Unfortunately, I’m not sure where the leak is coming from. I read component.rb and made a few small changes, but the leak hasn’t stopped. So I’m going to file this as a Rails bug and see if we can get it fixed before 1.0.

Update: Rails bug 2589.

Update: Thanks to Scott Barron, the bug has been fixed. Users with memory problems should probably install the patch, although a bit of testing would obviously be recommended first. The next release of Rails (either 1.0rc3 or 1.0; I’m not sure what they’re planning) should include this fix.

Posted in Typo | Tags memoryleak, ruby, rubyonrails, typo | 8 comments

Typo 2.5.6 and Rails 1.0

Posted by Scott Laird Thu, 20 Oct 2005 01:37:53 GMT

As far as I can see, Typo 2.5.6 (the most recently released stable version) should work fine with Rails 1.0. I just did a brief round of testing with 1.0rc2, and all of the tests pass. Er, except for one test that had a stupid typo that somehow still worked with Rails 0.13.1; the bug is in the test itself, though, so it’s not worth releasing Typo 2.5.7 just for that. If we ever release Typo 2.5.7, then I’ll make sure that the fixed test is included.

Also, Typo 2.5.6 should work just fine with Ruby 1.8.3, too, as long as you’re using Rails 1.0. I haven’t actually tested this yet, but I’d be surprised if it doesn’t work perfectly.

Surprisingly enough, the current Typo trunk (r683 or so) doesn’t work with Rails 1.0. All of the filtering code is broken; I’ll fix it shortly and check in the fix. Fortunately, the current Typo trunk is pinned to Rails 0.13.1 for now, so it should be safe to upgrade the version of Rails on the box; Typo will just ignore the new Rails for now.

Update: the Typo trunk r685 or later should be compatible with Rails 1.0rc2. I’ll probably break Rails 0.13.1 compatibility soon, so it’s time to upgrade.

Posted in Typo | Tags ruby, rubyonrails, typo | 1 comment

Typo and Ruby 1.8.3

Posted by Scott Laird Wed, 19 Oct 2005 14:51:48 GMT

Just for the record, current versions of Typo (either 2.5.6 or the current Subversion trunk) don’t work with Ruby 1.8.3. There are two problems–the Logger bug that keeps Rails 0.13.1 from working with Ruby 1.8.3 (this is easy to fix), and a second bug that I haven’t read about anywhere else–apparently YAML serialization is broken with Ruby 1.8.3 and Rails 0.13.1. This keeps Typo’s sidebar from working properly.

I’m going to see what it’ll take to get the Typo trunk working with Rails 1.0 (rc1 or rc2, if it’s out today), and then see if it works properly with Ruby 1.8.3. Once that’s done, the trunk will probably shift from 0.13.1-only to 1.0-only.

Update: That was quick. ChrisNolan on IRC pointed out that Rails bug #2304 contains a patch to fix this. The patch is already a part of the current Rails trunk, but you’ll need to patch 0.13.1 manually if you want to use it with Ruby 1.8.3.

Posted in Typo | Tags ruby, rubyonrails, typo | 1 comment

Benchmarking Typo

Posted by Scott Laird Mon, 17 Oct 2005 16:51:13 GMT

I finally had a bit of time to do some Typo benchmarking over the weekend and (as usual) found that my instincts were all wrong.

I was specifically interested in the performance difference between the page cache and the action cache–my guess was that the action cache was a 10x performance hit.

So I set up a test environment under Xen, running Typo r683, PostgreSQL, Apache 2, FastCGI, Ruby 1.8.2, and Rails 0.13.1. I didn’t do any Apache or Postgres tuning–I just ran them out of the box.

Then I ran ab against a snapshot of scottstuff.net from a couple weeks ago. I used the index page for my testing, as it’s a fairly large page and I wanted to give Typo a real workout.

Here’s what I found:

That really wasn’t what I’d expected. The action cache underperformed my expectations by a factor of 20.

I then did a bit of experimentation. I created a new uncached action in my test Typo setup that did nothing but render :text => 'foo', :layout => false, just to see if the caching system was slowing things down. Result? 10 requests/second. Then I created a new Rails project from scratch and added a new controller with the same action, and saw the same results. Still 10 requests/second.

However, in Rails’s logs, it said that it handled the request in 2 ms, and I should be seeing 500 hits/sec for the “foo” page. So something is adding an extra 98 ms to each request. I’m still hunting for this–I don’t know if it’s something Xen-related on my system, an artifact of my Apache config, or what.

I’ve tried upgrading to Rails 0.14.0, but that’s a whole other article.

Conclusions:

1. The page cache is really, really fast.
2. The action cache is a substantial improvement over the uncached case–about 10x on this system–but can’t touch the performance of the page cache.
3. Changing the concurrency settings on ab and/or the number of FastCGI backends in use didn’t make a substantial performance difference. Settings 2-15 gave roughly the same results.
4. My caches_action_with_params is slightly faster then the stock action cache.
5. Moving the action/fragment cache from the FileStore (Typo default) to MemoryStore gives no real performance boost. Moving to the MemCacheStore is a substantial performance hit (~2 hits/sec vs 10 hits/sec).
6. Adding a new uncached action in ArticlesController that simply returns a fixed string (render :layout => false, :text => 'foo') is no faster then the action cache.
7. Moving the new action from the previous step to a Controller of its own doesn’t help.
8. Removing all of the routes except the default :controller/:action/:id route doesn’t help, either.

Frankly, on this hardware, I don’t seem to be able to get more then 10 requests/sec out of Rails no matter what I do. I’m pretty sure that this is a mistake, so I’ll post a followup when I figure out what’s wrong.

Update 1: Another datapoint. Running an example Ruby FCGI on this box gives me 832 hits per second. So whatever the problem is, it’s not fundamental to Ruby FCGI on this box. So I need to look into Rails and see what’s happening.

Update 2: Making some progress. Apparently I screwed up when I tested a new, standalone Rails FCGI app before (forgot to restart FCGI?). This time, I got 130 req/sec, which is a vast improvement over the 10 req/sec that I was seeing before. Most of that speed hit seems to come from using Postgres for session storage. Unfortunately, even after making that change, my null controller is still only getting 40 req/sec with Typo. Transporting the same controller to a blank Rails project gives me 130 req/sec with the same code. Watching strace, it looks like something is forcing Typo to reload the digest/md5 module for every hit. Unfortunately, I can’t figure out how that’s happening–my environment.rb is identical between the two trees, as is my database.yml. I’ll get back to this later; I have other things that I need to finish today.

Posted in Typo | Tags apache, benchmark, caching, rubyonrails, typo | 9 comments

UUIDs for Typo

Posted by Scott Laird Sat, 24 Sep 2005 15:03:48 GMT

One of the small things that we’ve been adding to Typo recently is UUIDs for assorted bits of content. The only real consumer right now is the Atom 1.0 feed code, but they’re stored in the DB and sooner or later we’ll update the RSS feeds to use them too. Every item in a RSS or Atom feed is supposed to have a unique ID that is stable through time and across aggregators. Done properly, that means that readers will be able to recognize when they receive the same content via two different paths–like when you’re subscribed directly to a blog but also receive some of the same articles via *Planet-style aggregators or search feeds.

Unfortunately, Ruby doesn’t come with any code for generating UUIDs. When I started adding UUIDs to Typo, I found an old project on RubyForge that hasn’t been updated in 18 months, but it doesn’t work right on OS X and I didn’t really feel like fixing it. So I cheated and generated a MD5 from a combination of the object state and the clock and then shoved it into the DB. It’s not RFC 4122-compliant, but it works well enough in practice. Since our previous Atom feed code generated utterly non-unique tag: IDs (there’s nothing wrong with tag:, but we weren’t putting nearly enough data into it to actually generate unique IDs), even non-compliant MD5s were a big step in the right direction, and I figured that it’d be easy enough to plug a real UUID library in once once arrived.

It looks like I was right–Bob Aman sent me mail yesterday to tell me that he’s released a new UUID library for Ruby. I spent a couple minutes this morning ripping out my old UUID hack and replacing it with his RFC-compliant implementation, and once I finish running the last round of tests I’ll check it into the tree. Thanks, Bob.

Posted in Typo | Tags typo, uuid | 1 comment

Typo 2.5.6 is out

Posted by Scott Laird Fri, 23 Sep 2005 15:27:42 GMT

I just released Typo 2.5.6. This is a minor bug-fix release, but it’s an annoying bug. Flickr changed their RSS feeds slightly a couple weeks ago and this broke Typo’s Flickr sidebar. This release contains the 1-line fix required to make it work again.

Posted in Typo | Tags flickr, typo | 3 comments

Push me, pull me

Posted by Scott Laird Mon, 05 Sep 2005 01:25:00 GMT

Someone pointed out today that none of the “convert from your old blog system to Typo” converters in the current Typo development tree were working. They all produce articles without any HTML in them. This was caused by my big filter update from a week or so ago; apparently no one has tried to convert directly to a development version of Typo in the last week or two. The problem is that none of the text filters were running. Unfortunately, there’s no easy way to make them run because they need access to a working Rails controller, and there isn’t one available from inside of the converters.

At the same time, Piers Cawley asked for an easy way to rebuild all of the HTML generated by filters on his site–he was doing filter development and he needed to rebuild everything. Unfortunately, the filter design doesn’t make this easy, either.

These two are basically the same problem–the way that we run text filters is kind of painful in the current Typo tree. In Typo 2.5 and earlier, filters were applied at the model level, and nothing outside of the model really needed to worry about them–the filters were automatically applied every time that the article (or comment, or page) body changed. Due to the changes in the dev tree, this just isn’t possible any more, but I’d tried to hack it together by changing the dozen or so actions that changed Articles. It worked, but it was ugly, and it breaks when something like a converter needs to create a new article, because the converter has no way to run the filter.

So I’ve been making a few changes to Typo.

The basic problem is that we’ve been using a “push” model for updating the HTML version of articles, when we should really be using a “pull” model. That is, instead of updating the HTML when the article changes, we should really be generating the HTML when the article is viewed and then caching the HTML so we don’t have to do it more then once per article.

Fortunately, this change was pretty easy to make–I just had to search for every reference to body_html, extended_html, or full_html and change it to a reference to article_html(article). Then I moved the filter calls into article_html(article), saving the generated HTML back into article.body_html.

Once that was done, I could rip out all of the complicated filtering code that I’d had to put in to make the new filters work right, and everything Just Worked. I had to tweak a few tests that expected the HTML to be available in the database immediately after posting new content, but I already had tests that verified that the content viewed right, so it was just a matter of removing code, not really adding new code.

There’s one more change that I’m debating making. From an architectural standpoint, we shouldn’t really be stuffing things back into body_html–we should be using Rails’ fragment cache. Switching to the fragment cache would be trivial, it would only take a couple extra lines in article_html, and then I could rip a bunch of lines in the editor actions, because I could use a sweeper instead of explicit calls to article.body_html = nil.

Unfortunately, if we do that then we’ll end up killing Typo’s performance when it’s running in development mode, because caching is disabled in dev mode. So it’d be cleaner, but probably too slow to be useful. I’ll probably revisit this again before the next Typo release–there are a bunch of performance tweaks that we need to make before the next release; once those are done, we might be able to stand the performance hit.

Posted in Typo | Tags converters, filters, rubyonrails, typo | 3 comments

Typo progress

Posted by Scott Laird Sun, 04 Sep 2005 13:43:29 GMT

We've been making pretty good progress towards the next major release of Typo. Here's a short list of what's went in so far:

• Tags
• File uploads
• Gravatars
• Filter plugins, including:
  • Easy flickr image linking
  • Syntax highlighting
  • Sparklines
  • Auto-generating Amazon affiliate links
• Comment previews
• More powerful themes
• Atom 1.0 support
• Per-category and per-tag RSS and Atom feeds

There are still quite a few features left on our wishlist; some of those will make it into the next release, some won't. I'm starting to feel like we've passed the halfway point on this release cycle, but we don't have any firm plans for the next Typo release yet. Still, if there's anything that people really want to see in the next major Typo release, now would be a great time to speak up.

Posted in Typo | Tags rubyonrails, todo, typo | 1 comment

Introduction to Typo filters

Posted by Scott Laird Wed, 24 Aug 2005 02:13:00 GMT

Although blogs are inherently HTML-based, HTML isn’t really a great format for writting plain-text documents. If nothing else, manually adding <p> and </p> around paragraphs interrupts the flow of writing. Most people would prefer to write in a more user-friendly manner, either via a GUI editor or a light-weight markup language like Markdown, which is then translated to HTML automatically. Very few people really want to write raw HTML blog postings on a daily basis.

Out of the box, Typo 2.5 supports the Markdown and Textile markup languages and the SmartyPants HTML-post processing filter, which adds typographical quotes and dashes to HTML. Adding additional filters is difficult because the filter setup is hard-coded into Typo. One of the new features that I’ve been working to add to Typo is the ability to easily add new text filters via filter plugins, similar to the sidebar plugins in Typo 2.5. At the same time, I’ve also added several new filtering plugins that extend Typo’s abilities in a number of useful ways.

The goal of all of this is to make it easier to write using Typo. I’ve tried to find things that cause me pain and then fix them. I want to make it easy to do common writing tasks without having to fire up an external tool. Admittedly, my definition of “common writing tasks” is probably different from most people’s, but the easy ability to extend Typo’s filtering system will allow people to adapt Typo to their own needs without having a deep understanding of Typo’s internals.

Posted in Typo | Tags filters, programming, ruby, rubyonrails, typo | 6 comments

Typo filters nearing completion

Posted by Scott Laird Sun, 21 Aug 2005 06:19:50 GMT

My little Typo filter project is finally nearing completion. I think I’ve been working on this for almost two weeks now, which makes it the most time-consuming Typo project that I’ve undertaken yet. I’ve added about 400 lines of code and 200 lines of new tests, and changed at least 400 more lines. Typo’s current trunk is only 2600 lines long, so I’ve touched almost 40% of the code.

At this point, almost everything works. I can drop new filters into components/plugins/textfilters and they’re immediately available for use. All of the current filters work (Textile, Markdown, SmartyPants), and I’ve added several new filters as well. There’s still a lot of cleanup left to do, and there are a bunch of corner cases that I need to write tests for, but the core code seems pretty solid, and it’s essentially feature-complete.

Posted in Typo | Tags filters, rubyonrails, typo | no comments

Typo 4.0 development begins

Posted by Scott Laird Fri, 19 Aug 2005 17:43:45 GMT

Typo development slowed down for a while after we released Typo 2.5.0 so we could make sure that we had a stable release for people to use. It’s been a couple weeks now, and Typo 2.5.5 looks good, so we’ve started working on the next release, which will be called Typo 4.0. We’re going to skip Typo 3.0, because it conflicts with Typo3.

The two biggest changes so far are tags and file upload support. Over the next few weeks we’ll add the beginnings of podcast support, unify our RSS and Atom feed code, and (hopefully) add the filter code that I’ve been working on for the last week. Our current wishlist for Typo 4.0 is available in the Typo wiki.

I’m starting to feel good about my filter code. It’s a huge patch–it currently touches 68 files, not including vendor updates. I have three or four things left on my to-do list, and then I’ll spend a bit of time doing audit, cleanup, and testing. Most likely it’ll be ready by early next week.

Posted in Typo | Tags programming, typo | 2 comments

Another day, another filter plan

Posted by Scott Laird Wed, 17 Aug 2005 15:13:38 GMT

I’ve spent a bit of time playing with moving my Typo filter patch to use controller instances for filtering, and it doesn’t look too hideous on the filter side, but I’m going to need a lot of infrastructure changes before I can deploy it.

Unfortunately, the main user of filters right now is Article#set_defaults, which is called whenever an Article is saved. If I make filtering a controller issue, then I’ll have to rip out the filtering in Article (also Comment and Page) and move it to ArticleController. This won’t be a big performance hit, because I can use fragment caching, but it’s very invasive.

So, before I go any further down this road, I need to decide if this is really cleaner then just caching a base URL and using it to manually generate URLs. I was originally against this, but a thread on the Typo mailing list reminded me that we’re going to need a base URL if we want to support multiple blogs in a single Typo install anyway.

Ugh.

Posted in Typo | Tags filters, typo | 8 comments