*scottstuff*

I’ve never been happy with my web stats software for this blog. I’ve been limping along with Awstats for a while, because it’s better then nothing, but only slightly. It’s never been able to tell me what I really want to know–things like “where are the visitors to this specific page coming from” or “what pages are especially popular today”. I’ve poked around at other stats packages, but none of the free ones seem any better (more secure, perhaps, but not much more useful). I’ve always had a nagging sense that I was missing something, but I assumed that if I waited long enough, a generic package would show up that was good enough and would do what I wanted.

Today, I finally figured out what I was missing–I want a report that shows the most popular categories on my blog. I want to know that I’ve had 900 Asterisk hits this month but only 252 PDA hits. Since there’s no way that a generic stats package can know that this page belongs to my “Ruby” category, it’s pretty clear that I’m going to need a stats package that knows about my blog software. And since no stats packages know anything about Typo, I’m going to have to write one.

I’m going to play with this a bit in my spare time over the next week or so.

For now, what I’d really like to see from people is a list of questions that they’re like their blog stats software to be able to answer. I’ll start the list here by giving some of the basics and repeating a couple from above:

• How many hits am I getting per day?
• Where are the visitors coming from?
• What search terms are people using?
• What categories generate the most hits?
• Which pages are getting a larger-then-normal number of hits today?

Feel free to leave more in the comments and I’ll add them to my list.

I spent almost all of my free time this weekend working on Typo. We managed to merge my sidebar code and my new theme code, although both of them changed a lot in the process.

Typo 2.5 should be out sometime this week. It’ll be a huge upgrade from Typo 2.0.6–the current Typo tarball is around 320k, up from 86k for 2.0.6. We’ve added drag-and-drop configurable sidebars, new text-formatting options, new importers, a new theme (and a new theme engine), more RSS feeds, a vastly improved administration UI, support for static non-blog pages, and an awesome new caching system. It’s now a decent competitor for Movable Type and Wordpress.

I upgraded this site to Typo revision 351 last night. It was a bigger pain then I anticipated–there were conflicts with some of my local patches that took a while to find and fix. Now that it’s working, I’m really impresses–Typo now caches the HTML that it generates and gets Apache to serve it up directly without involving Typo at all whenever possible. This gives a huge speed increase, but it’s taken Tobias several revisions to get it all working correctly.

Unfortunately, I found another static caching bug one I put it up on my test site: caching only works when the Typo site lives in the root of a website. I have mine in a subdirectory (http://scottstuff.net/blog/), and that did entertaining things to the caching system. It would work right for the first hit, but once the cache was generated then Apache would throw 400 errors every time the page was hit. That kinda sucked.

Fortunately, the fix wasn’t too hard, and now everything seems to be working perfectly. As usual, let me know if you see any bugs.

I spent half the night working on a monster patch for Typo that totally reworks the way Typo’s sidebar works. Before, all of the items that you see on the right side of this blog were statically coded into the app/views/layout/article.rhtml file. To change the URL for your Flickr images, you had to edit the source by hand. Adding new items involved editing three or four files.

That’s all gone now.

I moved all of the sidebar items into a new component/sidebars/plugins directory, and added a sidebar controller that drives them. It automatically loads all of the sidebars in the plugin directory, so you can install new sidebars by simply dropping the files into place. No more editing source files and then fighting to keep the changes merged. I moved all of the configuration data for each plugin into Typo’s configuration database, so you can edit everything via the admin interface.

Finally, I added another admin block to control the sidebar layout. You can use Rail’s nifty Javascript drag-and-drop support to enable and disable individual sidebar items and to change the order in which they appear. It’s kind of ugly right now, but a bit of CSS will fix that eventually.

I posted my patch to Typo’s bug-tracking database, but it’s really young code, mostly written in the middle of the night, so I wouldn’t recommend using it quite yet. Once I’ve had a chance to clean it up a bit I’ll give it a bit of testing here and then submit a newer patch to trac and hopefully it’ll get merged.

As mentioned before, I’ve been using SVK to manage the changes that I’ve been making to Typo.

One of the items on my SVK to-do list was to figure out how to mirror a local SVK repository onto a public Subversion server. This is useful for a number of reasons:

• It provides a backup if the local SVK tree is lost.
• It allows me to use SVK on my laptop and work against the same SVK tree that I’m managing on my server.
• It allows others to see what I have in my SVK tree.

The SVK documentation doesn’t provide an exact recipe for doing this, but it turns out to be pretty simple. First, make sure that you have a publicly-accessible Subversion server where you have write privileges. Follow the Subversion Book if you need help with this.

Next, tell SVK to mirror a chunk of the Subversion share into your local SVK depot:

$ svk mkdir //mirror
$ svk ls http://svn.scottstuff.net/public/
<follow the prompts and have it mirror it onto //mirror/public>
$ svk sync --all
$ svk mkdir //mirror/public/typo
$ svk smerge --baseless //typo/ //mirror/public/typo -m 'Initial publication'

The final mkdir operation should prompt you for your Subversion username and password; once this is done, you should have write access to the Subversion share for all future operations.

One thing that I probably should have done was use the -Il option to smerge instead of -m. With -Il, SVK commits each local change to the remote server individually, using the local commit message. Without it, all changes get bundled up in one big change, and this isn’t really ideal.

At this point the public Subversion server should have a complete copy of your local Typo tree. To keep it up to date, you’ll need to run something like svk smerge -Il //typo/ //mirror/public/typo from time to time.

I just applied about 30 changesets worth of Typo changes all at once. If anyone sees any problems, please let me know.

Now that I’m up to date again, I can start back in on my big list of Typo changes.

My poor laptop’s hard drive has been filling up with unprocessed photos again, so I took a couple hours this morning to organize things and offload them to my home fileserver. I’ve never been all that happy with my web-based photo gallery, but I haven’t been willing to spend the week or two that it’d take to write something better, and I haven’t found an open-source gallery program that works any better for me then what I have now.

So, I decided to give Flickr another try. Part of this was motivated by the Typo’s Flickr sidebar plugin–it’s the closest thing I can get to photo gallery/blog integration, and that’s something that’s been on my to-do list for around two years.

Since I use iView Media Pro for organizing my photos, I wanted to find something that could automate the process of getting pictures from iView into Flickr. A bit of searching found PictureSync, which isn’t perfect, but it works well enough for now. I can select a block of pictures in iView and drag them to PictureSync’s icon, and it will convert them to sRGB, scale them down, extract metadata from iView to stuff into Flickr tags, and then upload the whole batch. Unfortunately, it seems to have keychain issues that force me to re-create my Flickr upload settings every time I run it, and it’s not all that great at extracting metadata from iView’s “people” field. Still, it’s easier to use PictureSync and Flickr then it was to copy files to my server and re-run my make-album script, and that’s good enough for me.

So, I paid Flickr $25 to upgrade my account to “pro” status, which ups my upload limit from 20 MB to 2 GB and started uploading blocks of pictures. It’s going kind of slowly (550 MHz G4s aren’t all that great at resizing multi-megabyte images), but there’s no way around that for now. Eventually, I’ll probably write a Ruby upload script to work around the problems with PictureSync, and then I’ll be able to do uploads from a faster Linux box, but I’m pretty happy with what I have for now. It’s good enough.

In celebration of getting out of the photo hosting business, here are a few random pictures. First, Sophie looking cute, then my brother and his youngest son, my family watching a walrus at the zoo, and a mountain biker in Whistler.

I’m basically finished with my last block of changes to Typo, and most of them have been merged upstream. At this point, I have most of the features that I care about, but there are still a few things left to do:

• Implement the rest of the Movable Type API, including the DB fields behind all of the useful stuff. Basically, everything that’s available in common blog editors should be available in Typo.
• Optionally link article keywords to Technorati tags.
• Look into turning filters and sidebar items into plugins, where all that’s needed to add them is to drop their files into a directory and then enabling/disabling/reordering them from the admin pages. (sidebars are done: #157)
• Add a Flickr text-formating plugin that will let me say something like [[flickr:scottlaird/24727421 "Some guy on a bicycle"]] and have that turned into a clickable image (with size tags) and a caption. I’m not sure what the right syntax will be for this–I know what Markdown uses, so I can avoid running into it, but I’m not all that familiar with Textile. It might be worth looking at Unicode-only brackets, like «» or 「」. Of course, adding untypable brackets sort of cuts down on the utility of shortcuts like this.
• Add an Amazon text-formatting plugin that allows amazon:<ASIN> URLs and transforms them into links, optionally with affiliate ID attached. This is a lot less complex the the Flickr filter, because it can just search for <a href="amazon:...">.
• Add a better way of mapping stacks of text-formatting filters to names. The Admin UI and the MT API both want symbolic names for filters; right now, this is hard-coded. It’d be nice to make this dynamically managed, but it looks like a total pain.

Done

These used to be up above, but they’re done now.

• Link to author’s email address if provided. (done, #156)
• Add GeoURL support, with a tag in the headers for it and a config option for providing your latitude and longitude. (done, #154)
• Add Flickr config parameters so adding Flickr doesn’t require editing the source. (done, #155)
• Fix the Flickr sidebar so it doesn’t do weird things with portrait images–as it is, the border around the images fits landscape images perfectly but leaves a big gap with portrait images. Alternately, just use the square layout that flickr likes. (done, I’m using square images)

Update: Pretty much everything here is complete as of August 26th. See the Typo Wishlist wiki page for details on where we’re going from here.

I’ve updated this blog again, using my latest Typo patches. It’s currently running Rails 0.13 (less then 2 days old), Typo SVN r280 (the most recent), and every change that I’ve submitted to the Typo Trac system. This includes:

• Per-article RSS feeds, so you can track comments and article changes via RSS.
• Fixed ‘comment’ link on the main index page; it now points to the #comment anchor instead of the head of the article.
• Configurable number of articles included in index pages and RSS feeds.
• Improved comments. This includes comment threading, per-comment subject lines, and optional email addresses.
• Better text filtering, so I can use SmartyPants-style quote fixing and use different filtering styles for comments and articles.

I spent an hour or so this morning making sure that everything for this blog is checked into SVK and then created a couple new trees–one as a testing/staging area for the blog, and another for actual production use.

That gives me 4 branches that I use on a daily basis:

1. //typo/trunk–a copy of Typo’s public Subversion tree.
2. //typo/local–where I do development.
3. //typo/staging–where I stage changes to scottstuff.net.
4. //typo/production–the actual production website.

There are several site-specific changes that live in //typo/staging. This includes changes to the CSS for this site, local graphics, Adsense blocks, links, and eventually things like Flickr integration. They aren’t appropriate for //typo/local, because everything in the local tree is eventually supposed to be merged into the main Typo tree. The production tree has a couple additional changes that only make sense in the production tree–there are a couple small changes to Rails’s public/.htaccess file that need to be made every time I push a change from staging to production. Now that it’s all in SVK, I don’t need to worry about merging this sort of thing anymore–I just run svk smerge //typo/local //typo/staging to push local diffs to the staging area, and then svk smerge //typo/staging //typo/production once I’ve tested things in staging.