Posted by Scott Laird
Tue, 24 Jul 2007 14:20:00 GMT
I really should have done this nine months ago, but I just don’t have time to devote to my rails-app-installer project, and I’d love it if someone would volunteer to take it over.
This is the installer used in the Typo GEM; it’s designed to make it easy to turn any Rails app into an easily-installed GEM.
The code lives in Google Code; feel free to take a look around.
If you’re interested, then please either send me mail or let me know in person at OSCON this week.
Tags installer, rails, ruby, typo | no comments
Posted by Scott Laird
Thu, 10 Aug 2006 21:11:33 GMT
I just released Typo 4.0.2. This is mostly a security upgrade; it’s designed to work with Rails 1.1.6, but it also includes workarounds for all of the known Rails 1.1.x security bugs, thanks to Piers Cawley.
In addition, we’ve fixed several bugs in the installer. MySQL users should be able to upgrade from 4.0.0 or 4.0.1 without problems now, fixing a problem with the 4.0.1 upgrader.
Upgrade directions:
Repeat however you installed Typo in the first place. If you downloaded the .tgz or .zip files, then you’ll need to download them again and install them over the top of your existing install. Then run rake migrate and restart your Typo processes.
If you’re using the .gem installer, just run gem install typo and then typo install /some/path to upgrade.
Tags rails, ruby, typo | 32 comments
Posted by Scott Laird
Thu, 10 Aug 2006 16:28:07 GMT
Apparently Rails 1.1.5 is still broken. It fixed one attack but left a couple other holes open. Piers has a workaround that should work for any Rails app.
Now would be a great time to fix this, because the exploit is fairly obvious right now.
Tags rails, ruby | 4 comments
Posted by Scott Laird
Thu, 10 Aug 2006 07:22:03 GMT
I just released Typo 4.0.1. By and large, this is a bug-fix release. Please make sure that you use Typo 4.0.1 with Rails 1.1.5, as earlier versions (1.1.4 especially) have substantial security problems.
A number of changes were made to the installer for this release:
- It has been spun off into its own
.gem and lives in its own source repository, so other projects can use it as well.
- It now supports Postgres as well as SQLite3. I’ll write about this soon.
- It performs database-agnostic backups to a
.yml file.
- A number of bugs have been squashed.
If you installed Typo 4.0.0 via the new installer, then upgrading is easy–just gem install typo and then typo install /some/path to upgrade. If you’re using the tar or zip files from Rubyforge, then unpack them over the top of your existing install, run ‘gem install rails’ to make sure that you have Rails 1.1.5, run ‘rake migrate’, and then restart your FastCGI processes.
If you’re installing for the first time, then you’ll probably find the .gem installer to be easier. If you can install things as root (or your hosting provider has already installed the Typo .gem), then do this:
$ sudo gem install typo
$ typo install /some/path
If you don’t have the ability to run gem install as root, then do this:
$ export GEM_PATH=~/gems
$ gem install -i ~/gems typo
$ ~/gems/bin/typo install /some/path
As usual, let me know if you hit problems.
Tags rails, ruby, typo | 10 comments
Posted by Scott Laird
Wed, 09 Aug 2006 21:13:28 GMT
The Rails team just released Rails 1.1.5, a “mandatory security” upgrade. So, if you’re running Rails, now would be a great time to upgrade.
I’m going to release Typo 4.0.1 tonight, including Rails 1.1.5 and a bunch of bug fixes.
Tags rails, ruby, typo | no comments
Posted by Scott Laird
Fri, 28 Jul 2006 19:50:40 GMT
A number of people have had a hard time installing Typo from the .gem on hosting providers’ systems where they don’t have the ability to install things as root. So, I sat down with Jim Weirich at OSCON to talk over the best way to handle non-root installs. Apparently RubyGems has support for this, plus or minus a few bugs. Try this:
$ export GEM_PATH=~/gems
$ gem install -i ~/gems typo
$ ~/gems/bin/typo install /some/path
Rubygems 0.9.0 will end up re-installing .gems that are already installed in the system gem directory, but that’s not fatal. It’s just a bit slower then it should be.
Hopefully the next rubygems release will fix this and add a “you aren’t running as root, use -i” warning.
Tags rails, rubyonrails, typo | 2 comments
Posted by Scott Laird
Fri, 28 Jul 2006 17:53:08 GMT
I’ve mostly finished extracting Typo’s installer into its own Rails project and .gem. The installer makes it trivial to build an installable .gem for any Rails project, so the install process looks like this:
$ gem install my-project
$ my-project install /some/path
The installer source lives in Google Code. That includes a mailing list and bug tracker. I’ll upload my .gem to RubyForge later today, along with some documentation. For now, the rdoc is available.
Here’s what you’ll need to do to add the installer to your existing Rails app:
- Create a
.gem that depends on rails-app-installer, rails, and all other .gems that you need to have installed.
- Add an
executable entry to your gemspec. If your app is called my-app, then add executable = ['my-app'].
- Finally, create
bin/my-app, using one of the examples in the rails-app-installer SVN tree as an example.
Here’s a short example bin/my-app:
#!/usr/bin/env ruby
require 'rubygems'
require 'rails-installer'
class AppInstaller < RailsInstaller
application_name 'my_app'
support_location 'my website'
rails_version '1.1.4'
end
# Installer program
directory = ARGV[1]
app = AppInstaller.new(directory)
app.message_proc = Proc.new do |msg|
STDERR.puts " #{msg}"
end
app.execute_command(*ARGV)
That’s all that’s needed–as long as the installer gem is installed, this will give you a full installer that supports installs, upgrades, db backups and restores, and all of the other things that the Typo installer currently provides. Adding application-specific installer subcommands is easy. Here’s the sweep_cache implementation from Typo’s installer:
class SweepCache < RailsInstaller::Command
help "Sweep Typo's cache"
def self.command(installer, *args)
installer.sweep_cache
end
end
That’s all that’s needed to implement the typo sweep_cache /some/path installer command.
Update: The gem is out. gem install rails-app-installer.
Tags installer, rails, rubyonrails, typo | 1 comment
Posted by Scott Laird
Sat, 03 Sep 2005 08:09:00 GMT
I just uploaded the first version of my schema generator to rubyforge. This is a Rails generator that knows how to take a collection of migration scripts and use them to build up a valid SQL schema file.
You should be able to install it via gem install schema_generator, and run it on any Rails project by running ./script/generate schema from the root of the Rails project. The current release (0.1.0) supports MySQL, PostgreSQL, and SQLite. It will auto-generate a schema for each DB in db/schema.DBTYPE.sql every time it runs, prompting you before overwriting existing files.
For this to work, your Rails migrations must describe your complete database schema. Many projects, like Typo, are older then Rails’s migration support, so their migrations don’t start with a clean slate; instead they describe how to migrate from a specific old version of the DB schema to the current version. In this case, either create a 0_initial_schema migration or to modify the existing migration #1 to create all of the original tables. I just committed an example to Typo’s subversion tree, feel free to use it as an example.
Read more...
Tags database, generators, rails, ruby, rubyonrails, schema
Posted by Scott Laird
Fri, 02 Sep 2005 22:56:00 GMT
My Rails Schema Generator is nearly complete. Here’s a sample run:
$ ./script/generate schema
Found 6 migration classes
Starting migration for AddSidebars
Starting migration for AddCacheTable
Starting migration for AddPages
Starting migration for AddPageTitle
Starting migration for AddTags
Starting migration for AddTextfilters
Adding TextFilters table
Migrations complete.
Tables found: 6
Indexes found: 1
Records found: 8
exists db
overwrite db/schema.postgresql.sql? [Ynaq] y
force db/schema.postgresql.sql
overwrite db/schema.mysql.sql? [Ynaq] y
force db/schema.mysql.sql
overwrite db/schema.sqlite.sql? [Ynaq] y
force db/schema.sqlite.sql
The migration classes that I’m using are copied straight from Typo without modification. I’ve left out all of the migrations that add features to “legacy” tables–tables like articles–since there isn’t a table definition that I can use. That’s my next project–adding a 0_initial_schema migration for Typo. Once that’s complete, I have a bit of code cleanup and then I’ll release my schema generator code to the world. Hopefully that’ll be later today.
Tags database, generators, rails, ruby, rubyonrails, schema, typo | no comments
Posted by Scott Laird
Fri, 02 Sep 2005 02:26:00 GMT
Yesterday was an interesting day–my now-former employer is restructuring “to support changes in direction and a new funding strategy,” so I’m now looking at new modes of employment, as are several former co-workers.
I’m viewing this as more of an opportunity to move on to new things, rather then a setback. I’ve spent quite a bit of time over the past few months thinking about what I wanted to do next, even though I didn’t expect this job to end quite so suddenly.
Here’s what I’ve decided:
- I don’t want to work on long, slow, multi-year projects. The world doesn’t work that way anymore. Agility is the future, and I’m not going to enjoy working for any company that takes years to bring a product to market.
- I want to work on things that I enjoy and challenge me, like Ruby and Rails.
- I want to spend less time commuting.
- I’d like the freedom to follow interesting trends and opportunities as they arise, for my own profit and the profit of those I work for.
With these goals in mind, I’ve decided to start looking for Rails consulting/contract work, although I’m also willing to consider full-time employment in the right environment. Please forgive a bit of shameless self-promotion:
My background is a combination of Unix system administration and programming; I’m very good at getting things to work and keeping them working. I’ve been programming professionally since 1988, I have 3 years of Ruby experience, over 6 months of Rails experience, and 13 years of Linux experience. I’m a major contributor to Typo, a Rails-based blog engine; I can easily provide code samples for those who are interested. I’ve written extensively about Typo, Rails, and Ruby here on my blog, feel free to look around. I was once the senior system administrator in charge of over 700 production Linux, BSD, and Solaris systems in a 24x7 environment; I can handle any level of system administration task that may be required.
Most importantly, I’m used to working with small groups of people to design effective, scalable, and affordable solutions to their problems.
Over the next few days I’ll post more details on this site, including a resume and some details on previous experiences. For now, if you have questions, please feel free to send me email, and I’ll get back to you as soon as possible.
Tags consulting, layoff, rails, ruby, seattle | 1 comment
Posted by Scott Laird
Thu, 01 Sep 2005 08:14:00 GMT
One of the things that has really bugged me with recent Typo development is the pain of maintaining 3 different database schema files (PostgreSQL, MySQL, SQLite) along with a set of Rails DB migration scripts. Every time we add a new table, we have to edit 4 different files, even though all of the information that we need is available in the migration file. Unfortunately, without the static schemas, new users would be adrift, so we’re stuck having to hand-modify each of the static schema files. This violates the DRY principle, causes errors, and irritates developers.
So I figured I’d fix it by writing some code that can take a set of Rails DB migrations, fold, spindle, and mutilate Rails itself, and then spit out a database-specific schema file showing all of the tables, indexes, and seed data provided by the migration files. This includes handling cases where a table is added in migration #4, two new fields are added in migration #6, and one field is deleted in migration #9. There are some corner cases that just can’t be handled, mostly relating to seed data that needs to be migrated to be correct with more recent schemas, but I think I can come close enough to make Typo happy, and probably a lot of other open-source Rails projects.
This turned out to be easier then I expected. I’ve put about 4 hours into it so far, and I can take this migration:
class Test4 < ActiveRecord::Migration
def self.up
create_table :sidebars do |t|
t.column :controller, :string
t.column :active_position, :integer
t.column :active_config, :text
t.column :staged_position, :integer
t.column :staged_config, :text
end
Sidebar.create(:active_position=>0, :controller=>'category')
Sidebar.create(:active_position=>1, :controller=>'static')
Sidebar.create(:active_position=>2, :controller=>'xml')
end
def self.down
drop_table :sidebars
end
end
And then do this:
$ irb
irb(main):001:0> require 'migrate'
=> true
irb(main):002:0> require 'db/migrate/4_test4' # the code above
=> true
irb(main):003:0> Test4.up
=> ...
irb(main):004:0> puts DBMigrator::Database.dump('postgresql')
CREATE TABLE sidebars (id serial primary key, controller character varying(255), active_position integer, active_config text, staged_position integer, staged_config text) ;
BEGIN;
INSERT INTO sidebars ("staged_position", "active_config", "active_position", "controller", "staged_config") VALUES(NULL, NULL, 0, 'category', NULL);
COMMIT;
BEGIN;
INSERT INTO sidebars ("staged_position", "active_config", "active_position", "controller", "staged_config") VALUES(NULL, NULL, 1, 'static', NULL);
COMMIT;
BEGIN;
INSERT INTO sidebars ("staged_position", "active_config", "active_position", "controller", "staged_config") VALUES(NULL, NULL, 2, 'xml', NULL);
COMMIT;
Postgres works now, at least with the 4 or 5 examples that I’ve swiped from Typo’s migrations. SQLite and MySQL are nearly working; I think I just need to fake out a couple classes each and they’ll be up and running. Once that’s done, I’ll bundle this all up into a Rails generator so people can do this:
$ ./script/generate schema postgresql
create db/schema.postgresql.sql
$ ./script/generate schema mysql
create db/schema.mysql.sql
$ ./script/generate schema sqlite
create db/schema.sqlite.sql
Tags database, migrations, rails, ruby, rubyonrails, schema, typo | 8 comments
Posted by Scott Laird
Tue, 09 Aug 2005 04:23:41 GMT
One of the biggest improvements in Typo 2.5 is page caching. By using Rails’s built-in page cache, we can get 100x the performance on many benchmarks without doing more then a few lines of work. This lets us serve high-volume weblogs (like weblog.rubyonrails.com) without requiring heroic measures like clustering.
Unfortunately, there are a number of hidden problems with Rail’s 0.13.1’s page cache implementation. We’ve had to work around a number of them in order to get Typo 2.5 out the door.
Basic page cache usage
Enabling Rails’s page cache is amazingly simple–just add caches_page :actionname to the top of your controller class and the :actionname action will spit out page cache files automatically. A couple small tweaks to Apache’s .htaccess file, and Apache will now serve cached files all on its own without involving Rails. If a client asks for http://blog.example.com/articles/2005/08/08/foo, Apache will first check for a articles/2005/08/08/foo.html file in Typo’s public directory. If that file exists, then it’s sent off to the client without touching Rails at all.
Sweeping
That part of caching is easy. It’s the other end that’s hard: sweeping the cache to remove stale cache entries. Rails provides a simple cache sweeper that can remove specified pages, but that’s not really good enough for us. With Typo, there are a number of events that end up touching a huge number of cached files. Adding a comment, for example, touches the cached article page, but it also changes the comment counter on the main index (if the article is still on the front page), the day, month, and year indexes, some number of category indexes, tag indexes, and potentially paginated versions of all of the above. The code to track these all down was trouble-prone and frequently missed one of the pages that needed to be changed; this led to stale caches. Even worse, some actions, like changing themes, need to invalidate all pages. Rails’s page cache doesn’t keep a list of cached pages, so there’s no clean way to sweep them all.
What we ended up doing was adding a page_caches table to the database and adding hooks to insert a new PageCache entry every time a page was cached. We also added a hook to remove entries from the page cache table whenever a page was manually swept, and then added a PageCache.sweep_all method to flush the entire page cache. For now, we’ve simply ripped out all of our old “smart” sweeping code and force a full sweep of the entire cache whenever anything substantial changes. Sooner or later we’ll start adding smart cache sweeping back in, but for now this works surprisingly well.
Query Parameters and Aliasing
Another shortcoming of Rails’s page cache implementation shows up when you start using query strings. Asking for http://blog.example.com/articles?page=2 ends up handing the ?page=2 parameter to the static .html cache page if it exists instead of calling Rails to ask for page 2. Even worse–if this cached page doesn’t exist, then Rails will generate it and store it for future access, even though it’s the second page of the index, not the first.
Finally, and worst of all, in Typo http://blog.example.com/articles is actually equivalent to http://blog.example.com/, because the article index view is the default index page. This means that the cached page for http://blog.example.com/articles?page=2 is actually /index.html, so anyone visiting page 2 of the article index screws up the front page of the blog. There’s no easy way around this with Rails 0.13.1; for now we’ve had to do work to keep ?page= from paginating anything. There’s one point that we could interrupt the page cache process from inside of Typo, but it doesn’t have any way to see the @request object or any of the query strings.
Long-term, we’re going to need to patch Rails to add a cachable property to @request that gets set to false when there’s a query string present, and also tweak Apache’s rewrite rules to skip static files if a query string is present. That assumes that Apache is even able to do that–every time I read the mod_rewrite documentation I end up with a headache. Since Typo officially supports lighttpd as well as Apache, we’ll need to get both of them to do the right thing, which is far from trivial.
Non 7-bit ASCII URLs and Caching
Finally, Rails screws up cached filenames when the URL has non-ASCII characters. So any URL with accented characters or any non-ASCII script is totally uncachable. At least with Apache and Webrick, Rails sees non-ASCII characters in the URL encoded using the usual %XX URL-encoding scheme. Unfortunately, both servers actually look for unencoded filenames. So Rails writes out the cache file for /foö as public/fo%C3%B6.html (assuming UTF-8 encoding), but Apache actually looks for public/fo<C3><B6>.html (where <C3> is a byte with the value of C3 in hex). This is actually not all that hard to fix–just add a URI::Util.decode to the right place inside of Rails–but it’s not clear what the security implications of this are.
Given all of these problems, I’ve been tempted to try using Rail’s action cache instead of the page cache–the action cache doesn’t let Apache serve the cached files directly, so Typo would have a brief chance to block the cache from handling specific files, and we could approach sweeping from the opposite direction. It’s not clear how big of a speedup the action cache would actually give us, though, compared to the massive win that we get from the page cache. We’d really like to keep using the page cache and fix all of its bugs to its usable by other Rails users.
Posted in Typo, Ruby, Web stuff | Tags cache, page, rails, ruby, rubyonrails, typo | 6 comments
Posted by Scott Laird
Fri, 05 Aug 2005 21:42:57 GMT
I missed this when it first came out, but Why has a pointer to a WebDAV server plugin for Webrick, Ruby’s native web server. I wonder how easy this would be to port out of Webrick and into a generic FCGI controller, so we could use it with Rails? That would let Rails act as a file server, which would be useful for some sorts of user interaction. I wonder if we could make use of it in Typo?
Posted in Typo, Ruby | Tags rails, ruby, rubyonrails, webdav | 1 comment
Posted by Scott Laird
Fri, 05 Aug 2005 15:23:00 GMT
It’s OSCON season, and most of the open-source world has descended on Portland for the week. This includes most of the leaders of the Ruby community, so the Portland Ruby Group held their own “Free OSCON” night at FreeGeek. Four of the people giving Ruby talks at OSCON gave their talks for free at FOSCON, and most of the rest of the Ruby speakers were there lurking in the back of the room, including Dave Thomas and Matz.
I dragged my trusty D60 along and took a few pictures, along with a few notes.
The first speaker was David Heinemeier Hansson, the creator of Ruby on Rails and O’Reilly’s “Hacker of the Year” for 2005. He basically gave us his 15-minute OSCON keynote on Rails, which was both a brief introduction and a marketing talk for Rails. His big theme was “flexibility is overrated”–by reducing the number of ways that you can approach web development, Rails makes it enormously easier to actually get things done.
When David’s talk was complete, our host Phil (Tomson, I presume) presented him with a vintage copy of How to Win Friends and Influence People.
This was deeply cool. Rich Kilmer gave a brief presentation on his current project, ActionStep, a port of OS X’s Cocoa API to Flash. Right now, there are free tools that can create Flash .swf files, but Macromedia’s licensing keeps them from legally using any of Flash’s windowing tools. So Rich decided to write his own windowing toolkit, using NextStep/Cocoa API. They’re over halfway done, and expect to release the first complete version before the end of 2005.
I’m not particularly fond of flash, so I wasn’t paying a lot of attention until Rich started showing off his ideas for Rails integration. He’s building a layer that will glue his Flash front end to a Rails back end, and the demo code that he presented made it look even easier then creating HTML forms for user interaction. I’m not sure that it’ll be something that I’ll ever end up using, but it looked deeply cool. Here are a couple screenshots:
Next up, Glenn Vanderburg gave a talk on metaprogramming in Ruby. He showed how Ruby itself uses metaprogramming to implement things like attr_accessor, and then showed how people have used and extended Ruby over the years. One theme was the continuing development of metaprogramming idioms in Ruby; he showed how things have changed over the years, starting with an X protocol wrapper that someone wrote years ago, through Rich Kilmer’s Java debug protocol system, and up through Rails. Ruby’s metaprogramming ability is one of the things that makes Rails so successful–the ability to extend the language to let you say things like:
class Article < ActiveRecord::Base
has_and_belongs_to_many :categories
has_many :comments
belongs_to :user
...
end
This is one of the things that makes Rails so useful and so much fun to program.
Why
Finally, Why The Lucky Stiff came on stage. Why (or sometimes _why) is sort of the rock star of the Ruby world. His real identity is a closely guarded secret. He’s the author of why’s (poignant) guide to Ruby, which is easily the strangest programming book that I’ve ever seen.
His FOSCON talk was sort of a performance-art interpretation of his book.
We knew that interesting things were afoot when he showed up with a backup band, and we had to stop for a break while they set up on stage.
When he finally got on stage, we were treated to an animated production that skipped and jumped around, occasionally touching on some feature in Ruby and the jumping back off into the unknown. There were shadow puppets:
Once that bit was done, they launched into song. Why had a nice little piece that was essentially the Ruby lexer set to music. “A symbol starts with a colon and is followed by lowercase letters and numbers! A constant is composed of capital letters and underscores!”
This sort of thing went on for a while. He alternated between animated segments on the projector, demonstrations of distributed Ruby programming (with audience participation), singing about Ruby, and utter non-sequiturs.
Why’s presentation ended with the immortal words “stop her, she’s stealing our eigenclasses.”
Thanks to Phil and everyone else from the Portland Ruby group, and all of the presenters for giving us a very memorable and enlightening night. I have a few more pictures on Flickr, if anyone’s interested.
Posted in Ruby | Tags foscon, foscon05, oscon, pdxruby, photography, rails, ruby | 3 comments
Posted by Scott Laird
Wed, 20 Jul 2005 08:23:36 GMT
There’s a surprisingly small amount of documentation out there on tuning Apache for optimum Rails performance. Almost everyone mentions the first step (use FastCGI, not regular CGI), but that’s such a huge performance boost that it’s really obvious–waiting 2-3 seconds per hit for Rails to start up is an indicator that you’re doing something wrong.
Once you get past that, there’s not a lot of documentation. There are examples from place to place, but no one seems to discuss what they mean or why they should be used.
Ever since I switched to Typo, I’ve been seeing occasional HTTP 500 errors from Apache, suggesting that Apache was unable to talk to Typo. Looking in the logs shows that Apache was usually in the middle of restarting a FastCGI instance whenever the errors occurred. Digging through the mod_fastcgi shows that FastCGI can work in three different modes with Apache:
- Static. FastCGI servers are started when Apache is reloaded and remain running.
- Dynamic. FastCGI server processes are started whenever a FastCGI URL is hit. Excess processes are killed off when there’s no traffic.
- External. Apache and your FastCGI app talk via TCP sockets.
Dynamic mode is the default, but that’s not a good fit for Rails, because of its slow startup time. Switching to static mode really helps. To do that, I added this line to /etc/apache2/apache.conf on my Debian server:
FastCgiServer /var/web/typo/public/dispatch.fcgi -idle-timeout 120 \
-initial-env RAILS_ENV=production -processes 2
Notice that I had to list the full path to Rails’s dispatch.fcgi file; on some systems you may be able to get away with only listing public/dispatch.fcgi, but that will almost certainly not work if you’re using virtual hosting.
By default, FastCGI assumes that your server will respond to queries within 30 seconds. I added the -idle-timeout 120 parameter just so I can deal with really slow responses better. Typo’s article admin page currently tries to list all 466 articles on one page, and that can take over 30 seconds to process.
The -processes parameter tells Apache how many FastCGI processes should run for this application. For 95% of users, 1 or 2 will be best. If you get a lot of traffic, then raising this to 3-4x the number of CPUs in your system might get you slightly better performance.
Finally, the -initial-env bit makes sure that Rails runs in production mode, talking to my production DB and not returning error backtraces to the user.
Posted in Typo, Ruby, Web stuff | Tags apache, fastcgi, rails, rubyonrails, sysadmin, tuning | 40 comments
