Posted by Scott Laird
Fri, 29 Jul 2005 21:37:47 GMT
I’ve never been happy with my web stats software for this blog. I’ve been limping along with Awstats for a while, because it’s better then nothing, but only slightly. It’s never been able to tell me what I really want to know–things like “where are the visitors to this specific page coming from” or “what pages are especially popular today”. I’ve poked around at other stats packages, but none of the free ones seem any better (more secure, perhaps, but not much more useful). I’ve always had a nagging sense that I was missing something, but I assumed that if I waited long enough, a generic package would show up that was good enough and would do what I wanted.
Today, I finally figured out what I was missing–I want a report that shows the most popular categories on my blog. I want to know that I’ve had 900 Asterisk hits this month but only 252 PDA hits. Since there’s no way that a generic stats package can know that this page belongs to my “Ruby” category, it’s pretty clear that I’m going to need a stats package that knows about my blog software. And since no stats packages know anything about Typo, I’m going to have to write one.
I’m going to play with this a bit in my spare time over the next week or so.
For now, what I’d really like to see from people is a list of questions that they’re like their blog stats software to be able to answer. I’ll start the list here by giving some of the basics and repeating a couple from above:
- How many hits am I getting per day?
- Where are the visitors coming from?
- What search terms are people using?
- What categories generate the most hits?
- Which pages are getting a larger-then-normal number of hits today?
Feel free to leave more in the comments and I’ll add them to my list.
Posted in Typo, Blog stuff | Tags accesslog, blog, blogstats, stats, typo | 3 comments
Posted by Scott Laird
Mon, 25 Jul 2005 21:20:00 GMT
I spent almost all of my free time this weekend working on Typo. We managed to merge my sidebar code and my new theme code, although both of them changed a lot in the process.
Typo 2.5 should be out sometime this week. It’ll be a huge upgrade from Typo 2.0.6–the current Typo tarball is around 320k, up from 86k for 2.0.6. We’ve added drag-and-drop configurable sidebars, new text-formatting options, new importers, a new theme (and a new theme engine), more RSS feeds, a vastly improved administration UI, support for static non-blog pages, and an awesome new caching system. It’s now a decent competitor for Movable Type and Wordpress.
Posted in Typo, Blog stuff | no comments
Posted by Scott Laird
Wed, 20 Jul 2005 14:51:07 GMT
I upgraded this site to Typo revision 351 last night. It was a bigger pain then I anticipated–there were conflicts with some of my local patches that took a while to find and fix. Now that it’s working, I’m really impresses–Typo now caches the HTML that it generates and gets Apache to serve it up directly without involving Typo at all whenever possible. This gives a huge speed increase, but it’s taken Tobias several revisions to get it all working correctly.
Unfortunately, I found another static caching bug one I put it up on my test site: caching only works when the Typo site lives in the root of a website. I have mine in a subdirectory (http://scottstuff.net/blog/), and that did entertaining things to the caching system. It would work right for the first hit, but once the cache was generated then Apache would throw 400 errors every time the page was hit. That kinda sucked.
Fortunately, the fix wasn’t too hard, and now everything seems to be working perfectly. As usual, let me know if you see any bugs.
Posted in Typo, Blog stuff | Tags blog, typo | no comments
Posted by Scott Laird
Fri, 15 Jul 2005 15:45:53 GMT
I spent half the night working on a monster patch for Typo that totally reworks the way Typo’s sidebar works. Before, all of the items that you see on the right side of this blog were statically coded into the app/views/layout/article.rhtml file. To change the URL for your Flickr images, you had to edit the source by hand. Adding new items involved editing three or four files.
That’s all gone now.
I moved all of the sidebar items into a new component/sidebars/plugins directory, and added a sidebar controller that drives them. It automatically loads all of the sidebars in the plugin directory, so you can install new sidebars by simply dropping the files into place. No more editing source files and then fighting to keep the changes merged. I moved all of the configuration data for each plugin into Typo’s configuration database, so you can edit everything via the admin interface.
Finally, I added another admin block to control the sidebar layout. You can use Rail’s nifty Javascript drag-and-drop support to enable and disable individual sidebar items and to change the order in which they appear. It’s kind of ugly right now, but a bit of CSS will fix that eventually.
I posted my patch to Typo’s bug-tracking database, but it’s really young code, mostly written in the middle of the night, so I wouldn’t recommend using it quite yet. Once I’ve had a chance to clean it up a bit I’ll give it a bit of testing here and then submit a newer patch to trac and hopefully it’ll get merged.
Posted in Typo, Blog stuff | Tags sidebars, typo | 7 comments
Posted by Scott Laird
Fri, 15 Jul 2005 07:55:38 GMT
As mentioned before, I’ve been using SVK to manage the changes that I’ve been making to Typo.
One of the items on my SVK to-do list was to figure out how to mirror a local SVK repository onto a public Subversion server. This is useful for a number of reasons:
- It provides a backup if the local SVK tree is lost.
- It allows me to use SVK on my laptop and work against the same SVK tree that I’m managing on my server.
- It allows others to see what I have in my SVK tree.
The SVK documentation doesn’t provide an exact recipe for doing this, but it turns out to be pretty simple. First, make sure that you have a publicly-accessible Subversion server where you have write privileges. Follow the Subversion Book if you need help with this.
Next, tell SVK to mirror a chunk of the Subversion share into your local SVK depot:
$ svk mkdir //mirror
$ svk ls http://svn.scottstuff.net/public/
<follow the prompts and have it mirror it onto //mirror/public>
$ svk sync --all
$ svk mkdir //mirror/public/typo
$ svk smerge --baseless //typo/ //mirror/public/typo -m 'Initial publication'
The final mkdir operation should prompt you for your Subversion username and password; once this is done, you should have write access to the Subversion share for all future operations.
One thing that I probably should have done was use the -Il option to smerge instead of -m. With -Il, SVK commits each local change to the remote server individually, using the local commit message. Without it, all changes get bundled up in one big change, and this isn’t really ideal.
At this point the public Subversion server should have a complete copy of your local Typo tree. To keep it up to date, you’ll need to run something like svk smerge -Il //typo/ //mirror/public/typo from time to time.
Posted in Typo, Blog stuff, Computer Programming | Tags mirror, public, smerge, subversion, svk | 5 comments
Posted by Scott Laird
Thu, 14 Jul 2005 04:25:55 GMT
I just applied about 30 changesets worth of Typo changes all at once. If anyone sees any problems, please let me know.
Now that I’m up to date again, I can start back in on my big list of Typo changes.
Posted in Typo, Blog stuff | Tags blog, typo | no comments
Posted by Scott Laird
Mon, 11 Jul 2005 06:53:24 GMT
My poor laptop’s hard drive has been filling up with unprocessed photos again, so I took a couple hours this morning to organize things and offload them to my home fileserver. I’ve never been all that happy with my web-based photo gallery, but I haven’t been willing to spend the week or two that it’d take to write something better, and I haven’t found an open-source gallery program that works any better for me then what I have now.
Sophie looking cute
So, I decided to give Flickr another try. Part of this was motivated by the Typo’s Flickr sidebar plugin–it’s the closest thing I can get to photo gallery/blog integration, and that’s something that’s been on my to-do list for around two years.
Rusty and Colton at Whistler
Since I use iView Media Pro for organizing my photos, I wanted to find something that could automate the process of getting pictures from iView into Flickr. A bit of searching found PictureSync, which isn’t perfect, but it works well enough for now. I can select a block of pictures in iView and drag them to PictureSync’s icon, and it will convert them to sRGB, scale them down, extract metadata from iView to stuff into Flickr tags, and then upload the whole batch. Unfortunately, it seems to have keychain issues that force me to re-create my Flickr upload settings every time I run it, and it’s not all that great at extracting metadata from iView’s “people” field. Still, it’s easier to use PictureSync and Flickr then it was to copy files to my server and re-run my make-album script, and that’s good enough for me.
Extreme cyclist at Whistler
So, I paid Flickr $25 to upgrade my account to “pro” status, which ups my upload limit from 20 MB to 2 GB and started uploading blocks of pictures. It’s going kind of slowly (550 MHz G4s aren’t all that great at resizing multi-megabyte images), but there’s no way around that for now. Eventually, I’ll probably write a Ruby upload script to work around the problems with PictureSync, and then I’ll be able to do uploads from a faster Linux box, but I’m pretty happy with what I have for now. It’s good enough.
A Walrus at the Zoo
In celebration of getting out of the photo hosting business, here are a few random pictures. First, Sophie looking cute, then my brother and his youngest son, my family watching a walrus at the zoo, and a mountain biker in Whistler.
Posted in Photography , Blog stuff | Tags flickr, photography | 6 comments
Posted by Scott Laird
Sun, 10 Jul 2005 21:43:57 GMT
I’m basically finished with my last block of changes to Typo, and most of them have been merged upstream. At this point, I have most of the features that I care about, but there are still a few things left to do:
- Implement the rest of the Movable Type API, including the DB fields behind all of the useful stuff. Basically, everything that’s available in common blog editors should be available in Typo.
- Optionally link article keywords to Technorati tags.
- Look into turning filters and sidebar items into plugins, where all that’s needed to add them is to drop their files into a directory and then enabling/disabling/reordering them from the admin pages. (sidebars are done: #157)
- Add a Flickr text-formating plugin that will let me say something like
[[flickr:scottlaird/24727421 "Some guy on a bicycle"]] and have that turned into a clickable image (with size tags) and a caption. I’m not sure what the right syntax will be for this–I know what Markdown uses, so I can avoid running into it, but I’m not all that familiar with Textile. It might be worth looking at Unicode-only brackets, like «» or 「」. Of course, adding untypable brackets sort of cuts down on the utility of shortcuts like this.
- Add an Amazon text-formatting plugin that allows
amazon:<ASIN> URLs and transforms them into links, optionally with affiliate ID attached. This is a lot less complex the the Flickr filter, because it can just search for <a href="amazon:...">.
- Add a better way of mapping stacks of text-formatting filters to names. The Admin UI and the MT API both want symbolic names for filters; right now, this is hard-coded. It’d be nice to make this dynamically managed, but it looks like a total pain.
Done
These used to be up above, but they’re done now.
- Link to author’s email address if provided. (done, #156)
- Add GeoURL support, with a tag in the headers for it and a config option for providing your latitude and longitude. (done, #154)
- Add Flickr config parameters so adding Flickr doesn’t require editing the source. (done, #155)
- Fix the Flickr sidebar so it doesn’t do weird things with portrait images–as it is, the border around the images fits landscape images perfectly but leaves a big gap with portrait images. Alternately, just use the square layout that flickr likes. (done, I’m using square images)
Update: Pretty much everything here is complete as of August 26th. See the Typo Wishlist wiki page for details on where we’re going from here.
Posted in Blog stuff | Tags blog, ruby, todo, typo | 3 comments
Posted by Scott Laird
Fri, 08 Jul 2005 15:07:29 GMT
I’ve updated this blog again, using my latest Typo patches. It’s currently running Rails 0.13 (less then 2 days old), Typo SVN r280 (the most recent), and every change that I’ve submitted to the Typo Trac system. This includes:
Posted in Typo, Blog stuff | Tags blog, typo | no comments
Posted by Scott Laird
Fri, 08 Jul 2005 14:59:33 GMT
I spent an hour or so this morning making sure that everything for this blog is checked into SVK and then created a couple new trees–one as a testing/staging area for the blog, and another for actual production use.
That gives me 4 branches that I use on a daily basis:
//typo/trunk–a copy of Typo’s public Subversion tree.//typo/local–where I do development.//typo/staging–where I stage changes to scottstuff.net.//typo/production–the actual production website.
There are several site-specific changes that live in //typo/staging. This includes changes to the CSS for this site, local graphics, Adsense blocks, links, and eventually things like Flickr integration. They aren’t appropriate for //typo/local, because everything in the local tree is eventually supposed to be merged into the main Typo tree. The production tree has a couple additional changes that only make sense in the production tree–there are a couple small changes to Rails’s public/.htaccess file that need to be made every time I push a change from staging to production. Now that it’s all in SVK, I don’t need to worry about merging this sort of thing anymore–I just run svk smerge //typo/local //typo/staging to push local diffs to the staging area, and then svk smerge //typo/staging //typo/production once I’ve tested things in staging.
Posted in Typo, Blog stuff | Tags distributeddevelopment, subversion, svk, svn, typo | no comments
Posted by Scott Laird
Thu, 07 Jul 2005 22:31:00 GMT
As mentioned yesterday, I’ve been looking for a good way to do revision-control my changes to Typo. Typo is maintained in a publicly-visible Subversion tree, so it’s easy to see when changes occur, but I don’t have the ability to write to the Typo tree, so I can’t use it to track my own changes. Since I currently have 4 or 5 different patches for Subversion floating around, plus a pile of local modifications (like adding Adsense banners and changing the ‘Link’ section in the sidebar), I really need some form of revision control.
I briefly tried to use Submaster yesterday, but it hasn’t been updated for Subversion 1.2, and I was unable to commit any changes. So I tried using SVK. Unlike Submaster, SVK is available in Debian’s sid tree, so it was trivial to install. After running apt-get install svk, I followed the tutorial and was up and running in a few minutes. It took a few minutes to import all 272 revisions from the Typo Subversion tree, but once that was complete I was able to make my own branch and start making changes within a couple minutes.
Here are the important steps to using SVK:
Initialize your local repository
The most recent SVK docs suggest using a single repository for everything, instead of managing a repository per project, so I’ll document that path. The command needed is:
This will create a new depot in ~/.svk/local. This only needs to happen once, when you first install SVK.
Import your external Subversion repository
Since I want to track Typo’s SVN tree, I needed to tell SVK about it. That’s pretty easy, too:
$ svk mirror svn://leetsoft.com/typo/trunk //typo/trunk
$ svk sync --all
Once those were done, I had a full copy of the Typo SVN tree in my SVK repository’s //typo/trunk directory.
Build a local branch
It’s a lot easier to keep track of changes if you make a local branch and then apply all of your development directly to the branch. This works just like it does in Subversion:
$ svk cp //typo/trunk //typo/local
Check out the local branch
A local working branch isn’t much good if you can’t edit it, so check it out somewhere:
$ svk checkout //typo/local local-typo-changes
This will produce a directory called local-typo-changes in your current working directory, and will fill it will all of the files from the local Typo branch. You can now edit files in this tree and use all of the usual version control commands (svk commit, svk add, svk diff, etc–all of the usual Subversion or CVS commands will work with SVK as well).
Producing patches for external consumption
So, let’s say that you’ve hacked a couple nice new features into your local SVK tree. In my case, I added several different features, committing the changes after each change, but I sometimes I found bugs in features after I committed changes to other features. For instance, my threaded-comment patch was modified in revisions 229, 232, and 246. I wanted to roll up all three of those changes into a single testable patch, but not include any of the other changes that were in my tree, like per-article RSS comment feeds.
This turns out to be very easy. First, I created a subdirectory for patchsets within SVK. This isn’t necessary, but I felt like a bit of organization would be useful:
$ svk mkdir //typo/patchsets
Then I created a new threaded-comments branch, starting with the official Typo SVN tree:
$ svk cp //typo/trunk //typo/patchsets/commentthread
Next, I merged the comment thread changesets from my local tree into the commentthread tree:
$ svk merge -c 229 //typo/local //typo/patchsets/commentthread -l
$ svk merge -c 232 //typo/local //typo/patchsets/commentthread -l
$ svk merge -c 246 //typo/local //typo/patchsets/commentthread -l
The -l uses the previous commit message as part of the new merge commit message.
Finally, I checked out the commentthread tree and tested it:
$ svk checkout //typo/patchsets/commentthread commentthread
$ cd commentthread
$ ruby script/server
This way, I had a completely clean tree, using only the comment thread changes, and I was able to verify that everything worked correctly. As it turned out, I’d missed a couple small changes that had snuck into my main tree as part of other, non-commentthread commits, so if I’d submitted this patch without testing, it wouldn’t have worked for people. So, I made the changes, tested things, checked the changes in, and then produced a patch:
$ xemacs <files>
$ ruby script/server
$ svk commit -m 'Fixed stuff'
$ svk diff //typo/trunk //typo/patchsets/commentthread
Now I can go back to working on my main branch. When I fix bugs in the comment thread code in my local branch, I just need to merge the new commit into the commentthread branch and re-diff. Assuming that revision 270 is comment-thread related:
$ cd commentthread
$ svk merge -c 270 -m 'Merge commit 270 from local tree'
$ svk update
$ ruby script/server
$ svk diff //typo/trunk //typo/patchsets/commentthread
One new patch, tested and ready for submission.
Merging with upstream changes
So, what happens when the upstream SVN tree changes? Well, that’s easy enough. First, re-sync your local copy of the trunk:
$ svk sync //typo/trunk
$ svk smerge //typo/trunk //typo/patchsets/articlerss
<follow the directions>
$ cd <working directory>
$ svk update
<verify and test>
$ svk diff //typo/trunk //typo/patchsets/articlerss
I my case, one of my patches was accepted upstream with minor changes. Once I accepted the merge into my patchset tree, I wanted to promote the changes into my main working tree. I did it this way because it should be easier to merge in this order. Probably. Anyway:
$ svk smerge //typo/patchsets/articlerss //typo/local
<follow the directions>
$ ruby script/server
<test>
At this point, my working tree is synced back up with the mainline.
All in all, SVK isn’t much harder to use then plain Subversion, and the ability to import and modify complete trees is fantastic. My next couple goals are:
Figure out how to export my local changes to a publically-visible SVN server, like the one at svn.scottstuff.net.
Figure out how to manage a SVK share on my development web server and on my laptop. This shouldn’t be hard, but I’m still not fully up to speed with SVK.
Posted in Typo, Blog stuff, Computer Programming | Tags distributeddevelopment, subversion, svk, svn, typo | 11 comments
Posted by Scott Laird
Thu, 07 Jul 2005 02:19:00 GMT
I can’t believe I missed that: Typo doesn’t support extended blog entries, where the main index page only shows a subset of the whole article. Yet another patch to write. Sigh.
Of course, a side-effect of this is that the 5 blog posts that I’ve written with extended entries didn’t transfer correctly from Movable Type. Ugh.
Posted in Typo, Blog stuff | Tags typo | no comments
Posted by Scott Laird
Wed, 06 Jul 2005 15:36:00 GMT
I just upgraded this blog. It was running the latest released version of Typo (version 2.0.6). It’s now running a development version of Typo from Typo’s Subversion tree, plus the patches that I posted here earlier.
One upshot of this is that threaded comments should now work. Feel free to play with them here. If you have any problems with them, then please send me mail. Thanks.
One other change is that comments are now formatted with Markdown, so it’s easy to use bold, italics, lists, and links. See the Markdown page for details; eventually I’ll add a blurb to the comment form explaining the simple stuff.
Posted in Typo, Blog stuff | Tags blog, markdown, typo | 2 comments
Posted by Scott Laird
Wed, 06 Jul 2005 14:10:15 GMT
I work up early today, so I made a few more changes to Typo. In addition to the earlier threaded comments patch, I’ve added:
- Sort category lists by name. By default, Typo displays category lists in a more or less random order. This patch causes them to be sorted by name.
- Enhanced text filtering. This adds support for SmartyPants-style filtering with RubyPants, fixes a cross-site-scripting vulnerability, adds the ability to stack filters, and splits comment formating away from article formatting, so you can use different defaults for each.
All three patches are against revision 262 from the Typo SVN repository. For convenience, I’ve generated a combined patch with all three sets of changes, plus a couple other minor fixes.
Posted in Typo, Blog stuff | Tags rails, ruby, typo | no comments
Posted by Scott Laird
Wed, 06 Jul 2005 04:19:03 GMT
I finally finished my patches to add threaded comments to Typo. I haven’t updated this site yet, but my test server is up and running with the threaded code. The same code also adds subject lines for comments and keeps track of commenters email addresses (if they choose to submit them).
The Ruby side of threaded comments was really easy, but getting all of the Javascript to allow threaded comment editing completely in-place using Ajax was a royal pain in the neck. It’s been years since I last touched Javascript, so I was forced to do a lot of googling. It would have been a lot easier if I’d just created a new page for comment editing and then bounced the user between that and the comment display page, but it wouldn’t have been as cool.
Next up, I’m going to add support for RubyPants, so I can get smart quotes and dashes working correctly again. Typo’s current code for text filtering assumes that you only want to apply a single filter to each item; I’m not sure if I’ll just create a ‘markdown+rubypants’ filter or add support for stacking filters. I can think of a couple other filters that would be useful, but I don’t know that I really want to create a management interface for filter stacking.
After that, I’m going to add the first bit of comment authentication. I’d like to have my comments automatically tagged so I can apply visual effects to the comments–see Mike Davidson’s blog for a nice example of the effect that I’d like to achieve. In the short run, my goal is to add an optional user_id field to the comment field and then link that to the users table. Then I’ll add a css_class field to the users table and use it when generating comments.
Once that’s done, I’ll start in on modifying the CSS files that come with Typo. Right now, this blog looks identical to every other Typo blog out there, and I really don’t care for that.
Posted in Typo, Blog stuff | Tags comments, typo | no comments
