*scottstuff*

Benchmarking Typo

Posted by Scott Laird Mon, 17 Oct 2005 16:51:13 GMT

I finally had a bit of time to do some Typo benchmarking over the weekend and (as usual) found that my instincts were all wrong.

I was specifically interested in the performance difference between the page cache and the action cache–my guess was that the action cache was a 10x performance hit.

So I set up a test environment under Xen, running Typo r683, PostgreSQL, Apache 2, FastCGI, Ruby 1.8.2, and Rails 0.13.1. I didn’t do any Apache or Postgres tuning–I just ran them out of the box.

Then I ran ab against a snapshot of scottstuff.net from a couple weeks ago. I used the index page for my testing, as it’s a fairly large page and I wanted to give Typo a real workout.

Here’s what I found:

That really wasn’t what I’d expected. The action cache underperformed my expectations by a factor of 20.

I then did a bit of experimentation. I created a new uncached action in my test Typo setup that did nothing but render :text => 'foo', :layout => false, just to see if the caching system was slowing things down. Result? 10 requests/second. Then I created a new Rails project from scratch and added a new controller with the same action, and saw the same results. Still 10 requests/second.

However, in Rails’s logs, it said that it handled the request in 2 ms, and I should be seeing 500 hits/sec for the “foo” page. So something is adding an extra 98 ms to each request. I’m still hunting for this–I don’t know if it’s something Xen-related on my system, an artifact of my Apache config, or what.

I’ve tried upgrading to Rails 0.14.0, but that’s a whole other article.

Conclusions:

1. The page cache is really, really fast.
2. The action cache is a substantial improvement over the uncached case–about 10x on this system–but can’t touch the performance of the page cache.
3. Changing the concurrency settings on ab and/or the number of FastCGI backends in use didn’t make a substantial performance difference. Settings 2-15 gave roughly the same results.
4. My caches_action_with_params is slightly faster then the stock action cache.
5. Moving the action/fragment cache from the FileStore (Typo default) to MemoryStore gives no real performance boost. Moving to the MemCacheStore is a substantial performance hit (~2 hits/sec vs 10 hits/sec).
6. Adding a new uncached action in ArticlesController that simply returns a fixed string (render :layout => false, :text => 'foo') is no faster then the action cache.
7. Moving the new action from the previous step to a Controller of its own doesn’t help.
8. Removing all of the routes except the default :controller/:action/:id route doesn’t help, either.

Frankly, on this hardware, I don’t seem to be able to get more then 10 requests/sec out of Rails no matter what I do. I’m pretty sure that this is a mistake, so I’ll post a followup when I figure out what’s wrong.

Update 1: Another datapoint. Running an example Ruby FCGI on this box gives me 832 hits per second. So whatever the problem is, it’s not fundamental to Ruby FCGI on this box. So I need to look into Rails and see what’s happening.

Update 2: Making some progress. Apparently I screwed up when I tested a new, standalone Rails FCGI app before (forgot to restart FCGI?). This time, I got 130 req/sec, which is a vast improvement over the 10 req/sec that I was seeing before. Most of that speed hit seems to come from using Postgres for session storage. Unfortunately, even after making that change, my null controller is still only getting 40 req/sec with Typo. Transporting the same controller to a blank Rails project gives me 130 req/sec with the same code. Watching strace, it looks like something is forcing Typo to reload the digest/md5 module for every hit. Unfortunately, I can’t figure out how that’s happening–my environment.rb is identical between the two trees, as is my database.yml. I’ll get back to this later; I have other things that I need to finish today.

Posted in Typo | Tags apache, benchmark, caching, rubyonrails, typo | 9 comments

Apache tuning for Rails and FastCGI

Posted by Scott Laird Wed, 20 Jul 2005 08:23:36 GMT

There’s a surprisingly small amount of documentation out there on tuning Apache for optimum Rails performance. Almost everyone mentions the first step (use FastCGI, not regular CGI), but that’s such a huge performance boost that it’s really obvious–waiting 2-3 seconds per hit for Rails to start up is an indicator that you’re doing something wrong.

Once you get past that, there’s not a lot of documentation. There are examples from place to place, but no one seems to discuss what they mean or why they should be used.

Ever since I switched to Typo, I’ve been seeing occasional HTTP 500 errors from Apache, suggesting that Apache was unable to talk to Typo. Looking in the logs shows that Apache was usually in the middle of restarting a FastCGI instance whenever the errors occurred. Digging through the mod_fastcgi shows that FastCGI can work in three different modes with Apache:

1. Static. FastCGI servers are started when Apache is reloaded and remain running.
2. Dynamic. FastCGI server processes are started whenever a FastCGI URL is hit. Excess processes are killed off when there’s no traffic.
3. External. Apache and your FastCGI app talk via TCP sockets.

Dynamic mode is the default, but that’s not a good fit for Rails, because of its slow startup time. Switching to static mode really helps. To do that, I added this line to /etc/apache2/apache.conf on my Debian server:

FastCgiServer /var/web/typo/public/dispatch.fcgi -idle-timeout 120 \
       -initial-env RAILS_ENV=production -processes 2

Notice that I had to list the full path to Rails’s dispatch.fcgi file; on some systems you may be able to get away with only listing public/dispatch.fcgi, but that will almost certainly not work if you’re using virtual hosting.

By default, FastCGI assumes that your server will respond to queries within 30 seconds. I added the -idle-timeout 120 parameter just so I can deal with really slow responses better. Typo’s article admin page currently tries to list all 466 articles on one page, and that can take over 30 seconds to process.

The -processes parameter tells Apache how many FastCGI processes should run for this application. For 95% of users, 1 or 2 will be best. If you get a lot of traffic, then raising this to 3-4x the number of CPUs in your system might get you slightly better performance.

Finally, the -initial-env bit makes sure that Rails runs in production mode, talking to my production DB and not returning error backtraces to the user.

Posted in Typo, Ruby, Web stuff | Tags apache, fastcgi, rails, rubyonrails, sysadmin, tuning | 40 comments

Apache ToS marking?

Posted by Scott Laird Fri, 29 Apr 2005 00:15:59 GMT

I’ve spent a fair bit of effort getting QoS on my home DSL link working right, so VoIP isn’t overwhelmed by downloads or by people hitting my web server. At this point, I’m down to one remaining problem–when Google and friends fire up their web crawlers and find a new directory full of JPEGs, they can slow other HTTP traffic to a crawl.

If I could tell Apache (2.0) to change the IP ToS flags associated with HTTP web crawler traffic, then my network’s QoS config would do the right thing and send user-driven HTTP traffic ahead of web crawler traffic. Unfortunately, I don’t see any obvious way to do this. I’d rather filter based on HTTP User-Agent, not network block, and that means either using a really smart packet filter or having my web server do the work on its own. And, as far as I can see, Apache 2 doesn’t have a ToS-setting module available. Dean Gaudet wrote mod_iptos for Apache 1.3, but it hasn’t been ported to Apache 2, and I’m not very eager to do it myself.

Does anyone have any suggestions?

Posted in Web stuff, Computer Networking | Tags apache, apache2, qos, tos | 1 comment

More Drupal

Posted by Scott Laird Fri, 21 May 2004 10:43:14 GMT

I spent another hour or so looking at Drupal, and I think I’ve solved most of my problems. First, the mod_rewrite issue. I’m not sure if it is simply an apache2-ism or what, but everything started working once I added the DocumentRoot as a prefix in the RewriteCond lines and added an extra / in the RewriteRule line. These lines are from the VirtualHost block of my Apache 2.0.49 config:

RewriteEngine on
RewriteLog /var/log/apache2/rewrite.log
RewriteLogLevel 9
RewriteCond /var/www/testing.scottstuff.net/%{REQUEST_FILENAME} !-f
RewriteCond /var/www/testing.scottstuff.net/%{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php?q=$1 [L,QSA]

I also had to prune the DirectoryIndex settings or Apache would try to feed index.html through the rewrite code and come out really confused:

DirectoryIndex index.php

Once that was done, I could turn on clean URLs and everything worked correctly.

Next, I looked into conversion from Movable Type. There are a pile of scripts floating around out there, but they’re all slightly broken. Several assume that you’re using MySQL, while I’m using Postgres. Others leave out comments. None of them seem to set up Drupal’s path module to do URL rewriting, so old MT URLs won’t work.

So I spent a bit of time and fixed most of the problems. My conversion script is based on one from Tim Allman and Morbus Iff that’s been floating around. It handles comment conversion and adds URI re-writing entries for all blog entries and for each category index. It re-uses MT’s URLs as much as possible, so it’ll probably even do the right thing if you’ve changed URL formats, but don’t quote me on that.

The way the script works is kind of cool–you set it up as a Movable Type index template, and have it export to a file called import.php. Then copy the file to the root of your Drupal tree and then run it via your web browser. The script will execute and import everything that MT gave it into Drupal.

So, at this point, here’s what’s still broken:

• No Markdown filter.
• Trackbacks aren’t converted.
• It needs a better theme.
• I need to write a script or two to integrate the project pages from http://svn.scottstuff.net into Drupal.
• Similarly, it’d be nice to integrate my book event page into Drupal, but I’m not sure how practical that is.

Posted in Blog stuff | Tags apache, blog, drupal, php | 4 comments

Apache 2.0, subversion, WebDAV, and iCal

Posted by Scott Laird Tue, 24 Feb 2004 11:42:21 GMT

Somewhere in the middle of my MPx200 adventure, something ate all of my repeating annual calendar entries, including birthdays and anniversaries. Personally, I blame PocketMac, but it’s really my fault for not keeping a good backup of my calendar.

That’s not to say that I don’t have a backup–I sync iCal with my home web server every few minutes–but there’s no way to track changes or dig up old versions of my calendar. Once entries are deleted, they’re gone for good. What I really want is revision control for my calendar.

And now I have it. Through a feat of astounding geekiness, I’ve finished installing the subversion version control system on my home web server, and my calendar WebDAV share is actually a subversion repository. What that means is that every time iCal exports a new calendar to my home server, it’s transparently checked into subversion in the background. I can get subversion to give me a list of all of the changes that have been made over time, generate diffs between specific versions, and even revert to older versions.

It’s actually pretty easy. Just install Apache 2, and then install Subversion with the ‘mod_dav_svn’ Apache module. Configure up a virtual server and then add these lines to your virtual host config files:

      <Location /xxx>
        Dav svn
        AuthType Basic
        AuthName "Auth Required"
        AuthUserFile /etc/apache/htpasswd
        Require valid-user
        SVNPath /path/to/svn/repository
        SVNAutoversioning on
        AuthzSVNAccessFile /etc/apache2/svn_authz
      </Location>

Now create /path/to/svn/repository with svnadmin, reload apache, and you should have a nice, revision-controlled web-based file server. Most modern desktop OSes can mount WebDAV-enabled web servers just like a more-traditional file server. Both OS X and XP can do it natively, and there should be at least a dozen ways to accomplish it in Linux. This is almost certainly the easiest (cheap) way to get reliable version-controlled storage.

Thanks to Sterling Hughes and Justin Erenkrantz for providing inspiration and documentation.

Posted in Computer System Administration | Tags apache, backups, ical, subversion, webdav | 2 comments

Apache 2.0, here we come

Posted by Scott Laird Tue, 24 Feb 2004 07:57:07 GMT

Later today, I’m going to upgrade scottstuff.net from Apache 1.3 to Apache 2.0. The newer version has a ton of issues, but I really want to run one specific service that is only available for 2.0. So, out with 1.3, and in with 2.0. More details after the upgrade.

Update: Upgrade complete. Let me know if you see anything broken.

Posted in Computer System Administration | Tags apache | no comments