Although blogs are inherently HTML-based, HTML isn’t really a great format for writting plain-text documents. If nothing else, manually adding <p> and </p> around paragraphs interrupts the flow of writing. Most people would prefer to write in a more user-friendly manner, either via a GUI editor or a light-weight markup language like Markdown, which is then translated to HTML automatically. Very few people really want to write raw HTML blog postings on a daily basis.

Out of the box, Typo 2.5 supports the Markdown and Textile markup languages and the SmartyPants HTML-post processing filter, which adds typographical quotes and dashes to HTML. Adding additional filters is difficult because the filter setup is hard-coded into Typo. One of the new features that I’ve been working to add to Typo is the ability to easily add new text filters via filter plugins, similar to the sidebar plugins in Typo 2.5. At the same time, I’ve also added several new filtering plugins that extend Typo’s abilities in a number of useful ways.

The goal of all of this is to make it easier to write using Typo. I’ve tried to find things that cause me pain and then fix them. I want to make it easy to do common writing tasks without having to fire up an external tool. Admittedly, my definition of “common writing tasks” is probably different from most people’s, but the easy ability to extend Typo’s filtering system will allow people to adapt Typo to their own needs without having a deep understanding of Typo’s internals.### Inside Typo Filters

The new filter code supports three different types of filter plugins:

1. Markup filters, like Textile and Markdown
2. Macro filters
3. Post-processing filters, like SmartyPants

Markup filters convert from a specific markup language into XHTML. You generally only want to use one markup language per article.

Macro filters convert certain Typo-specific macro tags into longer HTML sequences. These will be explained below.

Post-processing filters convert valid HTML into valid (but possibly enhanced) HTML.

Typo’s filtering system allows the user to create filter sets that use one markup filter and any mixture of post-processing filters. Macro filters are always enabled; they’re difficult to trigger accidentally and this greatly simplifies the filter management user interface.

Using Typo Filters

Typo 2.5 came with 5 hard-coded filter sets:

• No filtering
• Textile
• Markdown
• SmartyPants
• Markdown with SmartyPants

The new filtering code comes with the same filters defined. If one of these fits your needs perfectly, then you can continue using it unchanged. If you need to make changes, Typo’s admin system now includes a “Text Filters” tab that lets you edit these filter sets and create new ones.

Each text filter defined in the admin interface has a drop-down box for the markup language used (currently None, Markup, or Textile) and check boxes for each available post-processing filter.

Macro filters

Macro filters convert certain Typo-specific tags to longer HTML sequences. The new filter code comes with three macro filter plugins:

• <typo:code>: displays formatted code snippets, optionally with syntax highlighting and line numbering.
• <typo:flickr>: produces an image tag linked to an image on Flickr, optionally with a caption.
• <typo:sparkline>: displays a Sparkline–Tufte’s name for a small in-line chart.

All macro filters use <typo:NAME>-style tags. The <typo:NAME> tag is then replaced by the output of the macro filter during the filtering process. For example, the Flickr macro filter would replace this:

<figure class="wp-caption alignnone">
<a data-flickr-embed="true" data-header="false" data-footer="false"
  data-context="false"
  href="https://www.flickr.com/photos/scottlaird/31366117/in/dateposted-public"
  title="">
  <img src="https://farm8.staticflickr.com/" width="640" height="480" alt="" />
  </a><script async src="//embedr.flickr.com/assets/client-code.js" charset="utf-8"></script>
  <figcaption class="wp-caption-text">[31366117]</figcaption>
</figure>

with

<div style=\"float:left\" class=\"flickrplugin\">
  <a href=\"http://www.flickr.com/photo_zoom.gne?id=31366117&size=sq\">
    <img src=\"http://photos23.flickr.com/31366117_b1a791d68e_s.jpg\" width=\"75\" height=\"75\" alt=\"Matz\" title=\"Matz\"/>
  </a>
  <p class=\"caption\" style=\"width:75px\">
      This is Matz, Ruby's creator
  </p>
</div>

Notice that the <typo:flickr> line is a lot less typing.

The other macro tags work similarly. Here’s a brief example of the code plugin in action:

```ruby
  class Foo
    def bar
      "abcde"
    end
  end
</ typo:code>

The end result is basically the same as <pre>...</pre>, except that the text in the middle gets Ruby-specific syntax highlighting and all HTML is escaped.

Documentation enhancements

Each filter plugin has the opprotunity to define a self.help_text method that returns a help string. The admin interface currently has a button to show the help text for each filter; in the near future we’ll extend this to the content and comment editing pages as well. This way users will be able to see text formatting help that’s specific to the exact filter configuration in use.

Writing filters

Basic filters are pretty simple. Here’s a minimal markup filter, for example:

class Plugins::Textfilters::TextileController < TextFilterPlugin::Markup
  def self.display_name
    "Textile"
  end

  def self.description
    'Textile markup language'
  end

  def filtertext
    text = params[:text]
    render :text => RedCloth.new(text).to_html
  end
end

This is about as basic as it can be–it doesn’t include any help text, but it’s a fully functional text filter. Drop this into components/plugins/textfilters/textile_controller.rb, and Typo will automatically gain the ability to use Textile formatting.

To create markup filters, your filter class needs to be a subclass of TextFilterPlugin::Markup. Post-processing filters are essentially the same, except they’re subclasses of TextFilterPlugin::PostProcess.

Macro filters are slightly different. First, there are two different macro classes, TextFilterPlugin::MacroPre and TextFilterPlugin::MacroPost–one runs before markup filters, and the other runs after. Second, macro filters don’t define a filtertext method; instead they define a macrofilter method that looks like this:

def macrofilter(attrib,params,text="")
  data = text.to_s.split(/\s+/).join(',')

  if(attrib['data'])
    data = attrib.delete('data').to_s.split.join(',')
  end

  url = url_for(
    {:controller => '/textfilter', 
     :action => 'public_action', 
     :filter => 'sparkline',
     :public_action => 'plot', 
     :data => data}.update(attrib))
  "<img src=\"#{url}\"/>"
end

The attrib parameter is a hash of all attributes to the <typo:macroname> tag, params contains filter-wide parameters (see below), and text is the text between <typo:macro>...</typo:macro> tags, if any.

Filters are controllers, and they have access to all of the usual ActiveController methods, like url_for and friends. By default, none of the actions in plugins are visible to the public, so you don’t have to worry about someone feeding http://blog.example.com/plugins/textfilters/foo/exploit_me into their web browser and running code inside of your plugin. In some cases, though, you want to have certain methods in your plugin be accessible via URL. For instance, your plugin might need to use Ajax for something, or it might need to produce images, like the Sparkline plugin does.

To accomplish this, use plugin_public_action, like this:

class Plugins::Textfilters::SparklineController < TextFilterPlugin::MacroPost
  plugin_public_action :plot
  def plot
    ...
  end
end

This will connect http://blog.example.com/plugins/textfilters/sparkline/plot to SparklineController#plot. If you need to use views, then create a controllers/plugins/textfilters/<plugin> directory and put your views in there.

Plugin parameters

Some filter plugins need more information then they can easily collect when filtering each article. For instance, think about a hypothetical WikiWords auto-linking filter that turned WikiWords into links to a Wiki somewhere. If it’s going to link words, then it’ll need to know which wiki to link them to. That’s where filter parameters come in. Each filter plugin can have a default_config method like this:

def self.default_config
  {"wiki-link" => {
    :default => "", 
    :description => "Wiki URL to link WikiWords to",
    :help => "The WikiWords plugin links..."}}
end

Typo collects all of the default_config items from all enabled plugins and presents them to the user in the Text Filter admin area. If the WikiWords filter was installed, then each filter set would have an editing box labeled “Wiki URL to link WikiWords to”.

Using filters from inside of Typo

In Typo 2.5, filters were called via the HtmlEngine.transform library method. Unfortunately, this had to change with the new plugin system, because several plugins need to be called from a Controller context so they can use views and helpers like url_for.

Unfortunately, this means that it’s no longer possible to call filters directly from Models–they have to be called from Controllers so that they have the right context available. Fortunately, the code wasn’t too hard to convert, even though there was a lot of it.

To use filter plugins from inside of a controller, just call filter_text, like this:

filter_text('text to be filtered',[:markdown, :macropost, :smartypants])

This is rather low-level. To use whole filter sets, use this:

filter_text_by_name('more text to be filtered','markdown')

This will look up the filter set named ‘markdown’ in the text_filters table and apply it to the text more text to be filtered.

Any time that Article#body (or any of the similar models, like Comment and Page) changes, the controller must manually call filter_text_by_name This happens around 10 times in the current Typo tree.

Update: The API for filters changed somewhat around r685; the programming examples given here are a bit out of date now. I’ll write a “writing filters” document once the interface is stable.