mykzilla

This blog has moved

2010-03-05T23:03:00.001-08:00

This blog is now located at http://mykzilla.blogspot.com/.
You will be automatically redirected in 30 seconds, or you may click here.

For feed subscribers, please update your feed subscriptions to
http://mykzilla.blogspot.com/feeds/posts/default.

The Skinny on Raindrop's Mailing List Extensions

2009-11-17T17:26:00.000-08:00

Raindrop is an exploration of messaging innovation that strives to intelligently assist people in managing their flood of incoming messages. And mailing lists are a common source of messages you need to manage. So, with assistance from the Raindrop hackers, I wrote extensions that make it easier to deal with messages from mailing lists.

Their goal is to soothe two particular pain points when dealing with mailing lists: grouping their messages together by list and unsubscribing from them once you're no longer interested in their subject matter.

This post explains how the extensions do this; touches on some aspects of Raindrop's message processing and data storage models; and speculates about possible future directions for the extensions.

Raindrop Extensibility

Raindrop is being built with the explicit goal of being broadly and deeply extensible, and it includes a number of APIs for adding and modifying functionality. The mailing list enhancements comprise two related extensions, one in the backend and one in the user interface.

The backend extension plugs into Raindrop's incoming message processor, intercepting incoming email messages and extracting info about the mailing lists to which they belong. It also handles much of the work of unsubscribing from a list.

The frontend extension plugs into Raindrop's Inflow application, modifying its interface to show you the most recent mailing list messages at a glance, group mailing list conversations together by list, and provide a button you can press to easily unsubscribe from a mailing list.

Message Processing and Data Storage

Before getting into how the extensions work, it's useful to know a bit about how Raindrop processes and stores messages.

Raindrop stores information using CouchDB, a document-centric database whose principal unit of information storage and retrieval is the document (the equivalent of a record in SQL databases). Documents are just JSON blobs that can contain arbitrary name -> value pairs (unlike SQL records, which can only contain values for predeclared columns).

To distinguish between different kinds of documents, Raindrop assigns each a schema (similar to a table in SQL parlance) that describes (and may one day constrain) its properties. The rd.msg.email schema is the primary schema representing an email message, while the rd.mailing-list is the schema representing a mailing list, and the rd.msg.email.mailing-list is a simple schema that associates messages with their lists.

(In an SQL database, rd.msg.email and rd.mailing-list would be tables whose rows represent email messages and mailing lists, while rd.msg.email.mailing-list would be a table whose rows map one to the other.)

Note that there's a many-to-one relationship between messages and lists, since messages belong to a single list, although lists contain many messages, so rd.msg.email.mailing-list isn't strictly necessary. Its list-id property (which identifies the list to which the message belongs) could simply be a property of rd.msg.email docs (or, in SQL terms, a foreign key in the rd.msg.email table).

But putting it into its own document has several advantages. First, it improves robustness, as it reduces the possibility of conflicts between extensions and core code writing to the same documents.

It also improves write performance, as it's faster to add a document than to modify an existing one (although index generation and read performance can be an issue).

Finally, it improves extensibility, because it makes it possible to write an extension that extends the backend mailing list extension.

That's because Raindrop's incoming message processing model allows extensions to observe the creation of any kind of document, including those created by other extensions.

So just as the mailing list extension observes the creation of rd.msg.email documents, another extension can observe the creation of rd.msg.email.mailing-list documents and process them further in some useful way. If the mailing list extension simply modified the original document instead of creating its own, that would require some additional and more complicated API.

The Backend Extension

The primary function of the backend extension is to examine every incoming message and dress the ones from mailing lists with some additional structured information that the frontend can use to organize them.

Backend extensions are accompanied by a JSON manifest that tells Raindrop what kinds of incoming documents it wants to intercept. The mailing list extension's manifest registers it as an observer of incoming rd.msg.email documents, which get created when Raindrop retrieves an email message:

"schemas" : {
  "rd.ext.workqueue" : {
      "source_schemas" : ["rd.msg.email"],
...

The extension itself is a Python script with a handler function that gets passed the rd.msg.email document and looks to see if it contains a List-ID header (or, in certain cases, another identifier) identifying the mailing list from which the message comes:

def handler(message):
  ...
  if 'list-id' in message['headers']:
      # Extract the ID and name of the mailing list from the list-id header.
      # Some mailing lists give only the ID, but others (Google Groups,
      # Mailman) provide both using the format 'NAME <id>', so we extract them
      # separately if we detect that format.
      list_id = message['headers']['list-id'][0]
  ...

If it doesn't find a list identifier, it simply returns, and Raindrop continues processing the message:

if not list_id:
    logger.debug("NO LIST ID; ignoring message %s", message_id)
    return

Otherwise, it calls Raindrop's emit_schema function to create an rd.msg.email.mailing-list document linking the message document to an rd.mailing-list document representing the mailing list:

emit_schema('rd.msg.email.mailing-list', { 'list_id': list_id })

In this function call, rd.msg.email.mailing-list is the type of document to create, while { 'list_id': list_id } is the document itself, written as Python that will get serialized to JSON.

A document created inside a backend extension like this automatically gets a reference to the document the extension is processing (i.e. the rd.msg.email document), so the only thing it has to explicitly include is a reference to the list document, in the form of a list_id property whose value is the list identifier.

The extension also checks if there's an rd.mailing-list document in the database for the mailing list itself, and if not, it creates one, populating it with information from the message's List-* headers, like how to unsubscribe from the list. Otherwise, it updates the existing mailing list document if the message's List-* headers contain updates.

The Frontend Extension

The frontend extension uses the information extracted by the backend to help users manage mailing lists in the Inflow application.

It adds a widget to the Home view that shows you the last few messages from your lists at the bottom of the page, so you can keep an eye on those messages without having to give them your full attention:

It adds a list of your mailing lists to the Organizer widget:

And when you click on the name of a list, it shows you its conversations in the conversation pane:

In traditional mail clients, users who want to break out their list messages into separate buckets like this typically have to create a folder for each list to contain its messages and then a filter for each list to move incoming list messages into the appropriate folders. The extension does this for you automatically!

Finally, while viewing list conversations, if the extension knows how to unsubscribe you from the list, it displays an Unsubscribe button:

Pressing the button (and then confirming your decision) unsubscribes you from the list. You don't have to do anything else, like remembering your username/password for some web page, sending an email, or confirming your request with the list admin. The extensions handle all those details for you so you don't have to know about them!

List Unsubscription

In case you do want to know the details, however, it goes like this...

First, the frontend extension sends a message to the list's admin address requesting unsubscription, with a certain command (like "unsubscribe") in the subject or body of the message (lists often specify exactly what command to send in the mailto: link they include in the List-Unsubscribe header):

From: Jan Reilly 
To: wasbigtalk-admin@example.com
Subject: unsubscribe

Then the server responds with a message requesting confirmation of the request, often putting a unique token into the Subject or Reply-To header to track the request:

From: wasbigtalk-admin@example.com
To: jan@example.com
Subject: please confirm unsubscribe from wasbigtalk (4bc3b7e439fd)

Hello jan@example.com,

We have received a request to unsubscribe you from wasbigtalk.
Please confirm this request to unsubscribe by replying to this email.
...

Then the backend extension responds with a message confirming the request that includes the unique token:

From: jan@example.com
To: wasbigtalk-admin@example.com
Subject: Re: please confirm unsubscribe from wasbigtalk (4bc3b7e439fd)

Finally, the server responds with a message confirming that the subscriber has, indeed, been unsubscribed:

From: wasbigtalk-admin@example.com
To: jan@example.com
Subject: you have been unsubscribed from wasbigtalk

Hello jan@example.com,

Your unsubscription from wasbigtalk was successful.
...

At this point, the backend extension marks the list unsubscribed in the database, and the frontend extension marks it unsubscribed in the user interface.

This process matches the way much mailing list server software works, although there are daemons in the details, so the extensions have to be programmed to support each server individually.

Currently, they know how to handle Google Groups and Mailman lists. Majordomo2 (used by the Bugzilla and OpenBSD projects, among others) is not supported, because it doesn't send List-* headers (alhough supposedly it can be configured to do so). The W3C's list server is not yet supported, although it does send List-* headers, and support should be fairly easy to add.

Note that some of the processing the extension does is (locale-dependent) "screen"-scraping, as Google Groups and Mailman don't consistently identify the list ID and message type in some of their correspondence. In the long run, hopefully server software will improve in that regard. Perhaps someone can spearhead an effort to make it so?

The Future

The extensions' current features fit in well with Raindrop's goal of helping people better handle their flood of incoming messages. But there is surely much more they could do to help in this regard.

Besides general improvements to reliability and robustness--like support for additional list servers and handling of localized admin messages--they could let you resubscribe to a mailing list from which you've unsubscribed. And perhaps they could automatically fetch the messages you missed while you were away. Or even retrieve the entire archive of a list to which you're subscribed, so you can browse the archive in Raindrop!

What bugs you about mailing lists? And how might Raindrop's mailing list extensions make them easier (and even funner) to use?

Building/Releasing Personas

2009-11-04T15:58:00.001-08:00

Want to know how a popular extension like Personas gets built and released? Neither do I! Yet I know anyway. And I've written it down for your edification! So check it out.

Makefile for building/distributing extensions

2009-10-12T19:32:00.001-07:00

I've written a Makefile to help build and distribute Personas and Snowl. It automates the process of generating manifests; packaging the extension into a JAR and a XPI; and, for a "development" distribution channel, autogenerating successive version numbers and rsyncing the builds and update.rdf file to a distribution server.

The file makes it easy for me to update the latest development build of Personas after checking in a fix via:

make build channel=dev make package channel=dev make publish

For release builds, it's:

make build channel=rel make package channel=rel [manually upload to AMO]

And when I just want to test locally, I can build and install to a local profile:

make build make install profile=~/dev-profile firefox -profile ~/dev-profile

Or I can package the build and install via Firefox's Add-ons Manager:

make build make package [manually install via Firefox's Add-ons Manager]

Because I reuse it for multiple projects, project-specific variables are defined in a separate client.mk file. If you have similar needs and would find it useful, you are welcome to it!

Just grab the Makefile, see client.mk for the variables you should define, and check out the Personas source directory for the general layout that the Makefile handles as well as the install.rdf.in, chrome.manifest.in, and update.rdf.in templates from which their respective files are built.

Notes:

It's originally based on Weave's top-level Makefile, although it doesn't currently support the binary components that Weave's Makefile builds.
It assumes a directory structure in which content/, locale/, and skin/ directories are at the top level of the source directory (not within a chrome/ subdirectory).
It currently builds in the source directory rather than in an object directory, which means it can leave old install.rdf, chrome.manifest, and chrome.jar files around, so sometimes I need to make clean && make build to get rid of the ones I previously built (f.e. for a release build whose chrome files are archived) and generate new ones for continued development.
For Personas, I use a defaults/preferences/prefs.js.in template to generate prefs that contain the Mercurial revision ID (so I can identify the version of code in a development build with which someone is reporting problems) and the distribution channel (which the extension uses to suppress the "You've been updated!" page for the "dev" channel, since that would get annoying fast).

Personas for Email Messages

2009-08-29T07:54:00.000-07:00

Here's a thought (I had in the middle of a dream about brightly garbed superheroes): use Personas to style email messages in Thunderbird! It'd be a simple way to easily add some style to your messages to friends and family, highlighting your current mood, your perennial interests, or the subject of the message.

A ~~dramatization~~ mockup:

There are some tricky issues, like how to select a background color for the message that blends in attractively with the bottom of the header image. And it's not clear if the footer image is useful in this case.

But overall the idea seems quite promising. What do you think? And, if you're a Mozilla extension hacker (or would like to become one), would you be interested in prototyping it?

The Problems With Pagination

2009-04-23T08:15:00.000-07:00

Many web sites and applications paginate their content. Often this makes them harder to use.

Partly it's because site-provided pagination interacts poorly with the browser's built-in scrolling, because the content on any given web page usually takes up more than one vertical screen of space, and you have to jump back and forth between site pagination and browser scrolling, hunting down their controls each time you do so.

Sites compound this problem by moving their pagination controls around, so you can't leave your mouse over the Next Page button and assume you can simply click the mouse button repeatedly to jump to subsequent pages.

And pagination control click targets are small, which makes them hard to hit. Page-specific targets are often a single number (1, 2, 3), while the Next Page target is sometimes a single character (>).

Finally, pagination confounds searching through content using a browser's Find in Page feature, since it requires you to switch back and forth between searching for the content on the current page and navigating to the next one until you find the text you're looking for (and beyond, if you're looking for multiple occurrences of it).

And it similarly confounds any other activity in which you want to engage with all the content in an application, like when I wanted to turn off tracebacks/pingbacks for all 56 posts in a Wordpress blog, which is a six step process, and I had to repeat those six steps four times each because Wordpress's list of posts only lets me see 15 at a time.

Nevertheless, there are legitimate uses for pagination:

sites whose content is computationally intensive to generate (like search engines) use it to reduce load and cost;
those with large quantities of content that strains server, client, and network resources to display all at once (search engines again, the Personas gallery) use it to improve responsiveness;
those whose content cannot usefully be browsed in its entirety much of the time (search engines a third time, messaging applications) use it to reduce the cognitive overload of providing all their content at once;
and those with an ad-based revenue model (you guessed it, search engines, but also online news and many other sites) use it to show more ads.

For those kinds of sites, pagination makes sense, although doing it right still matters, like giving its controls large click targets and putting them in the same place on every page.

And, for content that spills across multiple screens, including the pagination controls at both the top and bottom of the content, the two scroll positions at which users often find themselves when they want to change pages; or positioning them absolutely, so they're always visible; or only loading a screenful of content at a time, so users never have to both page and scroll; or using endless/infinite scrolling.

Are there even better approaches, and are there ways for the web platform to support them so that web sites don't have to roll their own? Perhaps browsers and servers could collaborate to provide content in screenfuls with integrated scrolling and paging controls that play well together and don't change from site to site.

What do you think?

syntaxes for iterating arrays in JavaScript

2009-04-16T18:17:00.001-07:00

JavaScript lets you iterate arrays using the for each...in statement:

for each (var item in [1, 2, 3]) alert(item);

JavaScript 1.6 added the Array.forEach method:

[1, 2, 3].forEach(function(item) { alert(item) });

I've always preferred Perl's statement modifiers, though, for the popular English-like order of their clauses ("do this for each of those"):

print $_ foreach (1, 2, 3);

JavaScript 1.7 added array comprehensions for array initialization:

var squares = [item * item for each (item in [1, 2, 3])];

I just realized I can (ab)use comprehensions to iterate arrays with Perl-like syntax by throwing away the result:

[alert(item) for each (item in [1, 2, 3])];

I can iterate object properties the same way:

var obj = { foo: 1, bar: 2, baz: 3 };
[alert(name + "=" + obj[name]) for (name in obj)];

Sweet!

Snowl Integration with Places

2009-04-09T15:00:00.000-07:00

alta88 has recently done a bunch of work to integrate Snowl with Places. His initial efforts have focused on getting sources and people into Places, as he describes in this post to the discussion forum.

This will make the list of collections in the next version of Snowl work like Firefox's list of bookmarks in the Bookmarks Sidebar, and sources/people will also show up in the AwesomeBar:

Snowl doesn't yet register the snowl: protocol, however, so selecting those items doesn't do anything (yet). What do you think it should do?

And what additional Places-backed features would be useful?

(Picking up on something dietrich said on Twitter last week, I'd love to see Places get populated with tweets that reference a URL, setting the title of the Places entry to the content of the tweet, so you can use the AwesomeBar to find that site you saw last week in a tweet, as other folks have done with Delicious.)

Snowl development builds

2009-04-07T16:09:00.002-07:00

I've created a distribution channel to get Snowl changes out to its testing community faster. I'll push development snapshots to the channel on a regular basis as significant fixes are committed (or perhaps nightly, if I can automate the process).

Once you install a development build, Firefox will automatically check for newer ones and prompt you to update when it finds one, just as it does for the release builds on AMO.

This is truly the slicing edge. If you use these builds, be prepared for the possibility of regressions and bustage. On the other hand, you'll also get to test new features earlier, like alta88's Places integration work.

Install the latest development build, then discuss it and report bugs you find. (If you don't want to be on the slicing edge, try the latest stabler release instead.)

ISO simple open web image editor

2009-04-07T00:24:00.000-07:00

Is there a simple open web image editor? I know about Pixlr, Picnik, and Aviary, but those all use Flash. It seems like it should be possible to make one using Canvas and other modern open web technologies.

In addition to using it for simple photo manipulations, I'd like one that can be embedded into other applications, like Personas (to make it easy for users to make and refine their own personas) and image hosting sites (so users don't have to go to a separate website and import their images to edit them).

In other words, I want the web-based image editing equivalent of Gecko+Firefox: a great open-source image rendering and editing engine with a flagship product that demonstrates its capabilities and an API for embedding it into other applications.

Is there such a thing? Does anyone want to create one?

The Extension Bootcamp Tutorial

2009-04-06T11:23:00.001-07:00

A few weeks ago, as part of Mozilla Labs' Design Challenge, I taught a tutorial on extension development. My goal was to impart the minimum necessary information to build a simple extension using modern best practices. I called it Extension Bootcamp: Zero to "Hello world!" in 45 Minutes.

The tutorial, which borrowed from other ones (including my ancient Tinderstatus tutorial, the venerable Building an Extension, and the new Firefox addons developer guide), covered:

what extensions are, what they can do, and what they're made of;
how to configure your development environment on Windows, Mac OS X, or Linux;
how to construct an extension, including required files and directory structure;
how to package and test your extension;
a simple extension that inserts XUL into the main browser window via an overlay, styles it with CSS, listens for and responds to a DOM event on it using JavaScript, and localizes its strings with both a DTD and a properties file.

It explicitly excluded:

archiving chrome files into a JAR archive (valuable for large, popular, and stable extensions, but not worth the added complexity for the rest);
developing XPCOM components (most extensions should use JavaScript modules instead);
putting CSS/image files into a separate skin package (Firefox doesn't ship with two themes, which was the original impetus for segregating these files thusly, and most extensions don't support third-party themes, so this is unnecessary).

To conserve time, I didn't cover:

developing JavaScript modules;
accessing XPCOM components;
using tools and documentation to introspect and learn Mozilla code and APIs.

Developing a Firefox extension is pretty complicated, since there are so many different pieces to it, and the tutorial ended up taking two hours, including time for the students to practice each lesson on their own computers (for which my colleague Jono's assistance in fielding questions was invaluable).

The video of the tutorial, edited down to 1.5 hours, as well as the slides and accompanying files are available. If I had time to work more on this, I'd expand it to include the essential topics I didn't cover. I'd also update the Building an Extension tutorial to adopt the pedagogy of this one. If you'd like to do either of those things, please do!

Personas in Thunderbird

2009-03-31T10:18:00.002-07:00

Huang Yaoquan recently contributed a patch to make Personas work in Thunderbird! Here's Thunderbird with the Groovy Blue persona selected:

The patch requires a Thunderbird 3 beta, and it landed after the cutoff for the Personas 1.0 release, so it's not in the version on AMO, but it is in the latest development build, so give that one a spin if you're developmentally inclined, and provide feedback in the discussion forums or by filing bugs.

JSON JS module for Firefox 3.0/3.5 API compatibility

2009-03-25T18:23:00.000-07:00

I've written a JSON module to wrap the incompatible Firefox 3.0 and 3.5 JSON APIs, presenting the 3.5 API on both versions of Firefox, which is useful for extensions that support them both. Import this module to parse and stringify JSON in both 3.0 and 3.5 without having to check the host application's version each time.

Note: don't import this into the global namespace! If you do, you'll hork native Firefox 3.0 code that expects the 3.0 API. Instead, import it into your own object like this:

let MyExtension = {
JSON: null,
...
};
Components.utils.import("chrome://myextension/modules/JSON.js", MyExtension);
// Now MyExtension.JSON is an object implementing the Firefox 3.5 JSON API.

The module also works in Thunderbird 3.0, which uses Firefox 3.5's API (although there's no need to use it for an extension that only supports Thunderbird 3.0). I use it in Personas, which supports both Firefox 3.0 and 3.5 as well as Thunderbird 3.0.

The Firefox 3.5 JSON API is documented in the Mozilla Developer Center article Using JSON in Firefox. This JSON module is documented on Mozilla Labs' JS Modules wiki page.

Merging with Mercurial 1.0 and Meld

2009-03-18T18:32:00.000-07:00

In an earlier post, I described my experience merging changes with Mercurial 0.9.5 and Meld 1.1.5.1 on Ubuntu 8.04.

Things have changed in Ubuntu 8.10, which ships Mercurial 1.0.1. Now Meld displays your version of the file on the left, the other version on the right, and the base version (from which yours and the other are descended) in the middle.

Here it is handling the Merging conflicting changes example from the hg book:

Your job is to reconcile the conflicts between your version on the left and the other version on the right, turning your version on the left into the result of the reconciliation, which you then save to your working directory.

I understand why it can be useful to show the base version, and I think it's ok to use the same pane to show both your version and the final result (although kdiff3 provides a fourth pane beneath these three for the latter, which seems better).

But I don't understand why the base version is in the middle, given that one's primary task in this situation is to compare the differences between the two versions on either side of the window, copying changes from one to the other as needed. Putting another file in the middle just gets in the way of that process (which is presumably why kdiff3 puts the base version on the left and your version in the middle according to the screenshot in the aforementioned example in the hg book).

In fact, I frequently find myself starting a merge by copying the other version of the file into the base version in its entirety. That destroys my ability to refer to the base version, but it places the other version and my version side by side, at which point it's much easier to compare the two and edit my version accordingly:

But maybe I'm missing the point or misusing the tool, and there's some valid reason for the way Mercurial uses Meld. Can someone tell me what it is?

A XULRunner-Based Unit Test Harness

2009-02-04T17:48:00.001-08:00

Prologue

After finding a couple bugs in the Observers JS module recently, I looked around for a simple test harness to exercise it and other JS modules. Weave's (based on Mozilla's xpcshell harness) seemed promising, although it has to hack around various limitations of xpcshell.

Atul Varma had suggested some time ago that a XULRunner-based harness might make sense, so I decided to see if I could modify Weave's harness to run JS module tests using XULRunner instead of xpcshell.

Act 1

That proved doable, although some details were tricky. Sometimes I wondered whether I wouldn't have been better off reusing MochiTest, although it's heavier weight. Other times I wondered why I didn't find this bug about XULRunner-based unit tests sooner, although it's not clear that reusing the attachment in that bug would have been faster.

In any case, I got the harness working with XULRunner and wrote an initial set of tests of the Observers module. To run them, pull the jsmodules repository, change to its test/unit/ subdirectory, and run make with the location of your XULRunner executable (or set the XULRUNNER_BIN environment variable to its location):

hg clone https://hg.mozdev.org/jsmodules
cd jsmodules/test/unit/
make xulrunner_bin=/usr/bin/xulrunner

Since Firefox 3.0 comes with XULRunner, you can use any Firefox 3.0 (or newer) executable instead (except on Linux distros that ship Firefox as a XULRunner app--you have to use the XULRunner executable on those), like /Applications/Firefox.app/Contents/MacOS/firefox-bin on Mac and /c/Program\ Files/Mozilla\ Firefox/firefox.exe on Windows.

Epilogue

Once I got the harness working for jsmodules, I backported it to Weave, where it mostly works, except that IWeaveCrypto.generateKeypair (which calls PK11_GenerateKeyPair) fails, apparently because of SEC_ERROR_TOKEN_NOT_LOGGED_IN.

I have no idea why, but if you do, comment here or in bug 476539, since that's the last blocker to Weave using this harness.

Observers JS Module Improvements

2009-02-04T17:44:00.000-08:00

I've updated the Observers JS module with a number of changes. In particular, it's now possible to specify a this object when adding an observer that is a method, so instead of doing this:

let MyObserver = {
  onFoo: function(subject) {...},
  onBar: function(subject) {...},
  onBaz: function(subject) {...},
  QueryInterface: XPCOMUtils.generateQI([Ci.nsIObserver]),
  observe: function(subject, topic, data) {
    switch(topic) {
      case "foo":
        this.onFoo(subject);
        break;
      case "bar":
        this.onBar(subject);
        break;
      case "baz":
        this.onBaz(subject);
        break;
    }
  }
}

Observers.add("foo", MyObserver);
Observers.add("bar", MyObserver);
Observers.add("baz", MyObserver);

You can do this:

let MyObserver = {
  onFoo: function(subject) {...},
  onBar: function(subject) {...},
  onBaz: function(subject) {...}
}

Observers.add("foo", MyObserver.onFoo, MyObserver);
Observers.add("bar", MyObserver.onBar, MyObserver);
Observers.add("baz", MyObserver.onBaz, MyObserver);

Other changes include a couple of bug fixes and some backwards-incompatible API improvements. See the documentation for all the details (including steps to update from the previous version of the module).

Snowl Roadmap

2009-01-09T17:55:00.000-08:00

Based on Snowl's potential futures, and in line with its principles, I've put together a roadmap for its next three releases (beyond the imminent 0.2 release).

Generally, I'm aiming for six week development cycles that are that ambiguous mixture of date-, feature set-, and quality-driven, so the goal is to hit the target date with high quality implementations of the specified set of features. But work may be pushed to later releases or the schedule extended as appropriate.

Each cycle has a theme on which I'll focus my efforts, and I'll publish regular preview releases, which will be numbered with consecutive "pre[n]" suffixes (f.e. 0.3pre1, 0.3pre2).

Most of the work on the roadmap is stuff I'm planning to do myself, although it also lists tasks that other developers have expressed an interest in tackling. And it's ambitious, so I probably won't get to all of these tasks (although I'll probably do other things that come up along the way).

If you want to do something with Snowl, whether or not it's on the roadmap, check out the source code and go for it! (And let us know what you're doing so others don't duplicate your work.)

Caveat: nothing is written in stone, it's the best laid plans of mice and men, take it with a grain of salt, etc. Among other things, this roadmap doesn't account for time I may spend working on other projects, nor am I any better than other hackers at estimating the time it takes to do work I haven't already done. And the farther out the roadmap goes, the fuzzier its plans.

Also, your suggestions are welcome. Are you missing something from the roadmap? Is there something on it that shouldn't be there? Let us know!

The latest version lives on the Mozilla wiki at https://wiki.mozilla.org/Labs/Snowl/Roadmap. Currently, it looks like this:

Priorities

As with Firefox development (f.e. with Firefox 3.1 features), Snowl tasks are prioritized. For Snowl, the fuzzy prioritization scheme is:

P1: really want
P2: want
P3: nice to have

Releases

These are the target release dates for the imminent 0.2 release and the three releases after that:

0.2: Monday, January 12
0.3: Monday, March 9
0.4: Monday, April 20
0.5: Tuesday, May 26

(Note: The 0.3 cycle is eight weeks long to accommodate twoish weeks of vacation I have planned in January and February.)

0.3: simpler and smarter

Feeds

P1 expose feeds from frecent (frequent + recent) websites [atul]
P1 pick a feed automatically from among feeds with equivalent content
P1 let users subscribe to feeds by entering the location of a website
P2 automatically subscribe to feeds in live bookmarks [atul]
P2 automatically discard old feed items and tweets
P2 automatically determine how often to refresh feeds
P2 let users specify how long to retain feed items and tweets

Twitter

P1 support replying to messages
P2 support direct messages (private messages to specific users)
P2 automatically subscribe to Twitter account in browser history
P3 shorten long URLs when composing messages

Architecture/Extensibility

P1 simplify composition of messaging service implementations from Source and Target prototypes

Misc

P1 split collections view into resizable accounts/people sections [alta88]
P1 support email protocols [alta88]
P2 integrate Snowl into Thunderbird
P2 organize river view into editions
P3 use JS templates to build stream, river views

0.4: people and extensibility

People

P1 add a person-centric view
P1 automatically associate identities that represent the same person
P1 let users associate identities that represent the same person

Extensibility

P1 optimize database schema for sparse/custom attributes (f.e. username for Twitter, website URI for feeds)
P2 move OPML import/export processing into messaging service connectors so they can provide custom attributes (f.e. username for Twitter)
P2 compose subscription dialog from messaging service connector-specific overlays
P3 factor out common functionality from Feed and Twitter implementations into Source and Target prototypes

Conversations

P2 derive conversations from message metadata
P2 display messages in the contexts of the conversations to which they belong

Misc

P2 add a wordle-like view of messages

0.5: broader horizons

Feed Conversations

P1 retrieve comments on feed items automatically
P2 enable users to post comments on feed items

People

P1 consume contact info from Portable Contacts service providers
P2 provide contact info to Portable Contacts consumers

Misc

P1 support several additional messaging services/social networks

P2 support categorization of accounts and messages via tagging
P2 synchronize subscriptions to a cloud datastore
P3 synchronize all message data to a cloud datastore

Snowl Principles

2009-01-08T21:34:00.000-08:00

Over the course of working on Snowl, I've established a set of principles to guide its development. They're aspirational, but they have many practical ramifications. And while the current implementation doesn't embody them yet by a long shot, I expect it conform more and more closely to them over time.

The principles are:

feeds are a transport format, not a feature;
people and conversations are first-class objects;
one app can satisfy a broad range of users;
searching is better than pre-categorization;
browser features are useful for messaging;
a messaging app is a platform.

feeds are a transport format, not a feature

Feeds are currently presented in many application interfaces (including Firefox's) as a user-facing feature of the web. But feeds are data formats defining generic mechanisms for syndicating content. They're designed to be machine-readable, not human-consumable, and interfaces that expose them directly are often too technical, like the way websites publish both RSS and Atom feeds and Firefox asks users to pick between them (as aptly described by Atul Varma).

Treating feeds as a transport format means that rather than exposing them directly, Snowl should use them internally to give users a way to do the things they want to do, like find out when a website changes or keep up with what their friends are doing on the web.

So when a website provides RSS and Atom feeds of the same content, Snowl should pick a format instead of asking the user to do so. And when a website provides comment feeds, Snowl shouldn't expose those feeds to users, it should just show the conversations embodied in them to the users following that site.

The same is true for feeds provided by social network APIs. For example, the Last.fm API exposes the listening habits of its users' friends as individual feeds, but Snowl shouldn't make users manage individual feed subscriptions when they're really just interested in what their friends are listening to. Instead, it should present those individual feeds as Last.fm friend updates, separating them by friend--not feed--where appropriate.

people and conversations are first-class objects

There's no unique identifier for people on the internet. We each have an ever growing number of email addresses, usernames, and nicks on various websites, social networks, and messaging services. But we think of each person's various identifiers as representing the same being, and so should Snowl, so its users never have to figure out how they received a message (email, blog post, tweet, etc.) when looking for one from a particular person.

Also, context is crucial in conversations, and it's rarely obvious which message contains a particular portion of an extended conversation, so Snowl should let users interact with conversations rather than just their constituent messages, searching and browsing them as easily as they search and browse individual messages, and displaying messages in context.

one app can satisfy a broad range of users

Most users don't want to configure their experience, they just want things to work. And Snowl shouldn't force them to make unnecessary decisions. In fact, Snowl should need no preferences, because it always knows exactly what its users want.

Realistically, that's not possible, but the closer we can get, the better (while not trying to be too smart in ways that frustrate users). For example, a number of feed readers let users specify how often to update each feed. But this is information that it's possible to derive based on how often a feed is updated. There's no reason for Snowl to expose this question to users who shouldn't need to answer it.

Some users, however, do want to configure their experience. Others have unusual needs. And some have great ideas about how to create better experiences.

Firefox has shown that it's possible to build an application that satisfies all these users, and Snowl should follow in its pawsteps, using hidden preferences, subtle affordances, extensibility APIs, and other techniques to make it possible for those users to customize and modify Snowl to their hearts' content, while keeping the interface simple for everyone else.

searching is better than pre-categorization

Traditional messaging interfaces use pre-categorization to help you find messages. But the evolution of web search engines has shown that it's faster and simpler in many cases to simply search for what you want. Snowl should make it trivial to find any message quickly by searching for its content or metadata.

browser features are useful for messaging

Web browsers have developed a variety of features over the last dozenish years to make it easier for users to browse, find, save, and recall the web pages they visit. Snowl should take advantage of all these innovations to make accessing messages better, utilizing features like tabs for opening multiple messages, bookmarks to flag them for later reference, tagging for categorization, and the awesomebar for searching them.

a messaging app is a platform

Snowl was never intended to be just a feed and tweet reader. Feeds and Twitter are just two popular services with open APIs whose disparate kinds of messaging were good testbeds for early architectural and user experience decisions. Communication over the internet is evolving rapidly, and messaging apps have to evolve alongside it.

The best way to do that is to make Snowl a platform that others can extend to new messaging services and user experiences.

So Snowl comprises a datastore of subscriptions, messages, and people; a set of APIs for retrieving, storing, and sending messages; and a variety of interfaces for accessing them. And it should employ a flexible schema (like CouchDB's) to make it trivial to extend its data model; expose an API that makes it easy to add support for new messaging services; and provide useful primitives for building innovative new ways of looking at messages.

Snowl Futures

2009-01-07T23:59:00.000-08:00

With Snowl 0.2 almost out the door, I've been thinking about potential directions for its future development, and I've identified three possibilities.

Message Reader

The first is the idea that started the project: a message reader integrated into the browser, which uses Firefox features like tabs, bookmarks, history, the awesomebar, the sidebar, etc. to present messages.

Internet users live in their browsers, and much of their messaging is web-centric (feed items, tweets, messages sent via social networks, etc.). A message reader built into Firefox can aggregate those messages, store them locally (where their recipients have control over them), and provide a unified interface to them no matter where they come from and what protocols delivered them.

And it can help users uncover unknown sources of useful messages, like feeds from websites they frequent and updates from social networks to which they belong.

Snowl's biggest weakness relative to web-based alternatives (webmail, web feed readers, messaging apps built into social networks) is that it doesn't persist messages to a remote datastore, so users can't access them from anywhere. But it certainly could do so using Weave, CouchDB, or some other technology for synchronization to cloud storage.

Social Activity Tracker

The second potential direction for Snowl is for it to utilize feeds and social network APIs to keep users informed about what their friends are doing on the web. In other words, it could take up the mantle of The Coop, the idea for a Firefox-based social activity tracker that labs prototyped in 2007.

Just as with a message reader, a social activity tracker built into the browser would be in a unique position to expose useful updates to users across all of the sites they visit.

Snowl would need some modifications to behave in this way. In particular, it would need to get better at retrieving and aggregating lists of friends across multiple social networks. But its architecture is already well-suited to the task in a number of respects, since social activity is frequently exposed through messaging protocols, which Snowl has been designed to support.

Better Feed Reader for Thunderbird

The third potential is to integrate Snowl into Thunderbird as a better feed reader for that application.

Thunderbird's current feed reading feature is based on the Forumzilla feed reading extension I originally started working on in 1999/2000, and it suffers from a number of limitations and archaicisms. For example, it confines its messages to a separate "server" within the Thunderbird folder pane, and it provides only the traditional three-pane interface for reading them.

Snowl integrated into Thunderbird could break feed messages out of their "separate folder" ghetto, integrating them into the stream of messages from other sources, and provide a variety of useful views on them. And it would bring Twitter messaging to Thunderbird, too. Plus, Snowl developers would gain exposure to some of the interesting and innovative work happening in Thunderbird these days, which could benefit Snowl on Firefox.

The biggest challenge would be integrating Snowl into the very different capabilities provided by Thunderbird. Recent improvements to Thunderbird's extensibility and support for browser-like features such as tabs will make this easier, however.

I think all these directions have potential and are worth exploring. And while I'm going to remain focused primarily on Snowl as a message reader for now, I'll be conducting experiments along the way to learn more about the other possibilities. I'd also love to see other folks tackling them, so if you're interested in doing so, let's talk!

Snowl 0.2rc2

2009-01-07T23:47:00.001-08:00

The second release candidate of Snowl 0.2 is now available. This version fixes a few minor issues discovered in the first release candidate. Barring the discovery of additional issues that would block the release, this is the build that will be released as Snowl 0.2.

Give it a try, and let us know what you think.

Install Snowl

Snowl Central | Discussion Forum | IRC Channel | Bug Reports | Report a Bug | Source Code

Snowl 0.2rc1

2008-12-31T17:37:00.001-08:00

The first release candidate of Snowl 0.2 is now available. This version sports no new features relative to the last preview release, just a bunch of bug fixes. Barring the discovery of unexpected issues, this is the build that will be released as Snowl 0.2.

Notable bug fixes include:

an updated visual design with platform-specific icons for feeds and people;
Firefox is more responsive when Snowl is refreshing sources and building views;
multiple Twitter accounts no longer occasionally get confused for each other;
views update more consistently when messages and sources are added or removed;
the river and stream views always show messages in the order they were received.

As of this release, I've also switched to hosting Snowl development on the same infrastructure (source code repository, bug tracker, discussion forums, etc.) as other labs projects to make it easier for participants in those other projects to contribute to Snowl.

Give the release candidate a try, and let me know what you think.

Install Snowl

Discussion Forum | Bug Reports | Report a Bug | Source Code

simpler, easier string bundle API

2008-11-30T22:35:00.000-08:00

In the process of improving localizability in Snowl today, I put together a StringBundle JS module to make it easier to access string bundles from other JS modules (and JS XPCOM components and chrome JS, for that matter).

The module has a simple, easy-to-use API with a single get method that retrieves both plain and formatted strings:

let strings = new StringBundle("chrome://example/locale/strings.properties");
let foo = strings.get("foo");
let barFormatted = strings.get("bar", [arg1, arg2]);
for each (let string in strings.getAll())
  dump(string.key + " = " + string.value + "\n");

However, it also supports the API for the stringbundle XBL binding to make it easier to move from the binding to the module:

let strings = document.getElementById("myStringBundleElement");
              new StringBundle("chrome://example/locale/strings.properties");
let foo = strings.getString("foo");
let barFormatted = strings.getFormattedString("bar", [arg1, arg2]);
let enumerator = strings.strings;
while (enumerator.hasMoreElements()) {
  let string = enumerator.getNext().QueryInterface(Ci.nsIPropertyElement);
  dump(string.key + " = " + string.value + "\n");
}

The module is available with the other useful JS modules in the jsmodules project on mozdev.org. Documentation is embedded inside the module as well as on the project wiki.

Third (and Final) Preview of Snowl 0.2

2008-11-26T23:14:00.000-08:00

I've pushed a new preview release of the next version of Snowl. It includes an updated design for the "river of news" view with many improvements suggested by Alex Faaborg and others.

It also supports sending tweets in addition to receiving them via an interface built into the river and stream views. Just click on the "write a message" icon to expose it:

And it supports multiple Twitter accounts (thanks to Boris Zbarsky, who helped resolve a final vexing issue with HTTP auth), so if you happen to have more than one, you can keep track of all of them:

alta88 also contributed a bunch of improvements and bug fixes, including a context menu for sources with some common commands (like Refresh and Unsubscribe). And Atul Varma contributed a patch that will make it easier to plug in support for other sources of messages.

Finally, I fixed a big performance bug that was slowing down database queries for message content linearly with the database's growth in size. It looked something like this (courtesy beneneuman):

This preview is feature complete, which means no additional features will land in 0.2. From here on out, it's only important bug fixes, minor look and feel enhancements, localization support, and other low risk/polish/completeness issues until the release of 0.2.

So try it out, and let me know if you run into any of those kinds of problems (or any others, for that matter, as 0.3 development will be gearing up soon).

Install | Discuss | Chat | Bug Reports | Report a Bug | Source Code

Warning: this version is a primitive implementation with many bugs, and subsequent versions will include changes that break functionality and delete all your messages, making you start over from scratch.

making OS X stop prompting me for my wireless password

2008-11-19T12:02:00.000-08:00

I've long had problems automatically reconnecting to a certain WPA Enterprise wireless network from my MacBook Pro running Mac OS X 10.5.

Despite deleting the network from System Preferences > Network > AirPort > Advanced... > Preferred Networks, deleting its password from the login and System keychains using Applications > Utilities > Keychain Access, and then saving the network and credentials anew, the OS would prompt me to reenter my password for it every time I reconnected to it.

Some digging around on the web suggested that folks having a variety of Mac OS X wireless networking problems can sometimes solve them by deleting various configuration files, including /Library/Preferences/SystemConfiguration/com.apple.airport.preferences.plist.

I checked that file, and strangely, it listed a very old password for that wireless network. Turning off AirPort, deleting that file (I made a backup first just in case), and turning AirPort back on has resolved the problem. I can now reconnect to the network repeatedly (including after a reboot) without being prompted for my password.

news: links to Google Groups

2008-11-17T22:54:00.000-08:00

(Note: updated with a more reliable representation of the link.)
(Note: updated again to support news:<newsgroup-name> links.)

Here's an obscure hack.

Firefox 3 supports web protocol handlers (f.e. sending mailto: links to Gmail), while Google Groups archives Usenet. But as far as I can tell, Google Groups doesn't support news: links, even though Firefox can send them to it. It does, however, support retrieving messages by their IDs or newsgroup names, which news: links contain.

So how could we get Google Groups to load news: links passed to it by Firefox?

The solution is to pass them to an intermediary that extracts the IDs/names from the links and passes them to Google Groups. Here's a data: URL that does that:

data:text/html,\
  <html><body><script>\
    var url = '%s';\
    var stripPrefix = /^news:(\\/\\/[^\\/]+\\/)?/;\
    if (/@/.test(url))\
      window.location = 'http://groups.google.com/groups?selm='+url.replace(stripPrefix, '');\
    else\
      window.location = 'http://groups.google.com/group/'+url.replace(stripPrefix, '');\
  </script></body></html>

Unfortunately, it isn't easy to configure Firefox to use this URL as the news: protocol handler. Manually hacking mimeTypes.rdf is most unpleasant, while window.navigator.registerProtocolHandler only registers http(s): URLs.

But it's possible to construct a javascript: URL that does the same thing registerProtocolHandler does, namely call the handler service and register the handler with it directly.

The only trick is that you have to enter and run it in the Error Console, which has chrome privileges. Otherwise the URL won't have the privileges it needs to register the handler.

Here's a javascript: URL that registers the handler:

javascript: var Cc = Components.classes; var Ci = Components.interfaces; var handler = Cc['@mozilla.org/uriloader/web-handler-app;1'].createInstance(Ci.nsIWebHandlerApp); handler.name = 'Google Groups'; handler.uriTemplate = "data:text/html, <html><body><script> var url = '%s'; var stripPrefix = /^news:(\\/\\/[^\\/]+\\/)?/; if (/@/.test(url)) window.location = 'http://groups.google.com/groups?selm='+url.replace(stripPrefix, ''); else window.location = 'http://groups.google.com/group/'+url.replace(stripPrefix, ''); </script></body></html>"; var eps = Cc['@mozilla.org/uriloader/external-protocol-service;1'].getService(Ci.nsIExternalProtocolService); var handlerInfo = eps.getProtocolHandlerInfo('news'); handlerInfo.possibleApplicationHandlers.appendElement(handler, false); handlerInfo.alwaysAskBeforeHandling = true; var hs = Cc['@mozilla.org/uriloader/handler-service;1'].getService(Ci.nsIHandlerService); hs.store(handlerInfo); window.location = "data:text/html, <body> <p>The news: link handler for Google Groups has been installed. Try it with these links:</p> <ul> <li><a href='news:mozilla.support.firefox'>news:mozilla.support.firefox</a></li> <li><a href='news://news.mozilla.org/mozilla.support.firefox'>news://news.mozilla.org/mozilla.support.firefox</a></li> <li><a href='news:37604264.65F502CD@netscape.com'>news:37604264.65F502CD@netscape.com</a></li> <li><a href='news://news.mozilla.org/37604264.65F502CD@netscape.com'>news://news.mozilla.org/37604264.65F502CD@netscape.com</a></li> </ul> </body>";

To use it, copy the URL, open the Error Console (Tools > Error Console or Ctrl/Command+Shift+J), paste the URL into the evaluation field, and press the Evaluate button.

Once you've done that, try it out with these links:

When you click one, Firefox will ask you what you want to do with the link, and Google Groups should be an option on the list.

For the curious, here's a nicely formatted version of the code inside that javascript: URL, which is based on WCCR_addProtocolHandlerButtonCallback:

javascript:
var Cc = Components.classes;
var Ci = Components.interfaces;

var handler = Cc['@mozilla.org/uriloader/web-handler-app;1'].createInstance(Ci.nsIWebHandlerApp);
handler.name = 'Google Groups';
handler.uriTemplate = "data:text/html,\
  <html><body><script>\
    var url = '%s';\
    var stripPrefix = /^news:(\\/\\/[^\\/]+\\/)?/;\
    if (/@/.test(url))\
      window.location = 'http://groups.google.com/groups?selm='+url.replace(stripPrefix, '');\
    else\
      window.location = 'http://groups.google.com/group/'+url.replace(stripPrefix, '');\
  </script></body></html>";

var eps = Cc['@mozilla.org/uriloader/external-protocol-service;1'].getService(Ci.nsIExternalProtocolService);
var handlerInfo = eps.getProtocolHandlerInfo('news');
handlerInfo.possibleApplicationHandlers.appendElement(handler, false);
handlerInfo.alwaysAskBeforeHandling = true;

var hs = Cc['@mozilla.org/uriloader/handler-service;1'].getService(Ci.nsIHandlerService);
hs.store(handlerInfo);

window.location = "data:text/html,\
  <body>\
    <p>The news: link handler for Google Groups has been installed.\
       Try it with these links:</p>\
    <ul>\
      <li><a href='news:mozilla.support.firefox'>news:mozilla.support.firefox</a></li>\
      <li><a href='news://news.mozilla.org/mozilla.support.firefox'>news://news.mozilla.org/mozilla.support.firefox</a></li>\
      <li><a href='news:37604264.65F502CD@netscape.com'>news:37604264.65F502CD@netscape.com</a></li>\
      <li><a href='news://news.mozilla.org/37604264.65F502CD@netscape.com'>news://news.mozilla.org/37604264.65F502CD@netscape.com</a></li>\
    </ul>\
  </body>";

Of course Google Groups could make this hack moot by simply supporting news: links and protocol handler registration natively. For info on how to do that, Google Groups, see the news: URL standard in RFC 1738 and these subsequent IETF drafts, then read the Devmo docs on window.navigator.registerProtocolHandler.