Developing Tools for Watson

IMPORTANT NOTICE: Watson has been licensed by Sun Microsystems, and the Mac version is no longer available from nor supported by Karelia Software. Developing tools for the Mac-only version of Watson is still possible, but not encouraged. This section of our web site is here for historical and reference purposes only.

How does Watson's plug-in architecture compare to Sherlock 3?
We've taken a look at Sherlock 3's released plug-in architecture, and it's quite a bit different from Watson's. Although some developers are sure to prefer Sherlock's approach, our reaction is lukewarm at best. You can't exactly expect us to be impartial judges, but here are some differences as we can determine:

Sherlock plug-ins, and the list of which ones are available, are downloaded dynamically over the Internet. (With Watson, plug-ins are bundled with the application itself, or installed later and placed outside of the application bundle.) The advantage of Sherlock's approach is that tools can be updated on the server, and the client can automatically replace its cached version if it has changed, transparent to the user (except for any additional download time, making Sherlock get sluggish). With Watson, tool updates are more explicitly downloaded, and due to constraints with NSBundle, the application has to be re-launched to make use of the new updates. Winner: Sherlock. (If NSBundles could be "unloaded", it would probably be a tie here.)
Sherlock plug-ins are implemented in a scripting language ... OK, two scripting languages -- JavaScript (with extensions) and XQuery (with extensions). Watson plug-ins are little Cocoa programs. Having to use these two scripting languages makes it tedious for the Sherlock programmer. On the other hand, every Cocoa developer we've met is using their favorite development platform! Winner: Watson.
Written in scripting languages rather than Objective C, Sherlock plug-ins are really, really slow. Winner: Watson.
Sherlock Plug-ins are more-or-less platform-neutral; somebody could theoretically build a Sherlock client applications on another operating system. That is good because it follows the platform-neutral vision of the World Wide Web, but it's bad if Apple really wants the concept of "web services for the rest of us" to only work for Mac users. Winner: Potentially, Windows users
The actual parsing of Web content -- HTML scraping -- is done with lots of lines of scripting code in Sherlock. Watson employs the "parsing dictionary" technique, using a data description of a Web page's contents to convert an HTML (or XML) page into convenient data structures. Though a parsing dictionary can't cover all possible cases, it tends to make HTML scraping and XML processing pretty easy. Scripting code, on the other hand, is subject to programming errors much more than a data description. Winner: Watson.

(If any developers have any other points, feel free to contact us and let us know what we should add to the list....

Watson is fully extendable, with an open architecture so that third-party developers can develop their own modules. Qualified tools can be made available to all Watson users by means of the program's built-in downloading mechanism. Non-programmers can get in on the fun by creating "parsing dictionaries" (descriptions of how to load a Web site) for use by existing tools -- for instance, to add better international support.

What kind of Web Site would make sense as a Watson plug-in?
Only a small handful of Web sites are appropriate for being a Watson plug-in. If there is a site that has a lot of information, with just a search ability, it may not be worth the overhead of programming a Watson plug-in. (Before Sherlock 3, we would have suggested writing a Sherlock 2 plug-in -- but that capability is no longer available.) If the site has online forms to fill out or data to browse that would be easier done with a Mac-like interface than a series of Web pages, then making a Watson plug-in may be a good idea.

What sites should your Watson plug-in link to?
Recognizing that there is often more than one Web site available to perform a specific function, and that some Web sites are more appropriate than others (based on a user's preferences, or their geographic location), we encourage plug-ins to be developed that don't rely on a single source. This isn't always the case, when there is a clear dominant provider (such as eBay), but when there are a number of sources (such as telephone directories), it makes sense to build your plug-in so that it will accept one or more data sources, especially across a range of geographies. (Future enhancements to the SDK will make it easier to build such plug-ins.)

Can I build a plug-in that connects to a site that uses Cookies? Connects over SSL?
SSL and cookies should be possible without much work, though we have managed to avoid these so far. let us know and we'll work with you to make it a reality.

Developer Kit Downloads

Download and decompress these files to develop your own tools. Do not use Stuffit Expander, even the new 7.0 version -- it doesn't handle the file names properly! Use the command line, or OpenUp.

Sample Project	This is the full source code for the "Exchange" tool, to use as an example for writing your own.
Project Template	Place this folder into `~/Developer/ProjectBuilder Extras/Project Templates/` (creating subfolders as necessary) in order to create new Watson tools from Project Builder.
Test Tool	Place this in `~/Library/Application Support/Watson`; it will now be available just like any other Watson Tool. Developers and creators of "parsing dictionaries" use this to test loading of a Web site.
APIs	This is a folder of HTML documentation files, the same as available from the links below.

General Guidelines

If you are interested in developing a tool for Watson, you should follow these guidelines:

A Watson Tool is written in Cocoa as an NSBundle, and must conform to the WTool protocol, and will usually descend from GeneralTool; these headers are found in GeneralTool.framework bundled with the application. If you understand this, you may proceed. :-)
Contact us to tell us what you are planning on working on. (If somebody else is already working on such a tool, it may not be worth duplicating their effort.)
Join the Watson Developers' Discusssion Group (now closed) so you can exchange questions and answers, search the posting archives, and so forth. Hint: Scan the archives for some useful tidbits!
Find "GeneralTool.framework" within the Watson application. (Its path may be something like /Applications/Watson.app/Contents/Frameworks/GeneralTool.framework.) Either copy that to /Library/Frameworks, or make a soft-link from there to where it lives in the application. This will allow your projects to link to Watson.
Create your new project in Project Builder using the project template. Choose a simple name for the project (spaces are OK) since it affects the file names generated.
The project is essentially an empty tool, ready to run. You will need to set the class of the "File's Owner" in the nib to be your tool's class, however.
Write and build your project. See "Creating Parsing Dictionaries" for a discussion of using the Test Tool for building your parsing dictionaries if the tool does any HTML "scraping."
Be sure to set the bundle properties appropriately (use the "Expert" mode for Watson-specific properties.)
To make it easy to develop your tool, you can soft-link the entire ~/Library/Application Support/Watson directory to point to your Project Builder's "build directory", or put a soft-link of your built tool into ~/Library/Application Support/Watson.
Tool projects assume the Watson application is stored in /Applications. To debug your tool with Watson installed elsewhere, change the executable associated with the target.
Feel free to send early versions of your project to us for feedback.

Making Your Tool Available

If you want your tool to be available for downloading directly from the program, keep in mind the following acceptance criteria:

It must be free for downloading and basic use, though you retain full copyright. If it is to be shareware, it will need to manage its own licensing, reminders, and so forth. (For example, notice how the "Recipes" tool allows users to donate to RecipeSource.)
The intent and functionality must be acceptable to us. (We provide the direct downloading capability as a courtesy, and we're not going to encourage the downloading of modules we don't find tasteful.)
The tool must be acceptably bug-free, and have a good user interface.
The tool must not send any personal information about the user to any server unless it is explicitly set by user action. (For instance, the "Movies" tool fills in the user's zip code from the preferences, but it is not actually used until the user hits the "Save" button.) We will examine the source code to verify this.
We reserve the right to stop distribution of any third-party tools at any time.

When your tool is ready, send us the entire project, along with permission to distribute the tool online. If your tool is deemed to be acceptable, we will put it up on our servers for downloading directly from Watson.

Of course, you can distribute the tool yourself for manual installation.

Creating Parsing Dictionaries

Existing tools that access regional Web sites (such as phone numbers) need to connect to a number of sites in order to make a tool useful for users around the world. If you are technically inclined, even if you are not a programmer, you can help make Watson work in your area by constructing a parsing dictionary that helps Watson load your provider's web site and interpret the results! That parsing dictionary can then be incorporated into a subsequent version of a tool for others to make use of.

If you want to contribute a parsing dictionary, here's how:

As above, contact us to avoid duplicated effort, and join the discussion group, so you will have access to help.

Download and install the Test Tool from the list above. This will enable you to build your dictionary and see the results.

Read up on the Parsing Dictionary in the Parse Manager.
In the Finder, open up the tool for which you wish to create a module using the contextual menu (by control clicking on Watson and choosing "Show Package Contents."). Navigate into Contents -> Plug-Ins. Then control-click on the "bundle" file of the tool that you wish to create a parsing dictionary for, then navigate into Contents -> Resources, and notice the files ending with ".plist". Make a copy of these files (option-drag them somewhere). Now you have some sample parsing dictionaries that parse existing Web sites and produce standardized results.
Paste the contents of a parsing dictionary file into the top pane of the Test tool. Using any comments in the plist files (between /* and */) and the entries that end with _input, you can build up a test query, that represents the user input, in the small text field at the top of the tool. For example, city_input = C; means that you need to specify the city with an input called "C", as in { C = "Toronto"; }.
First see if you can get the existing parsing dictionary working with real data, and compare the results in the bottom half of the Tool window to the results you get from using the Web site directly. Unfortunately, Mac OS X is really picky in parsing this format, and it's hard to see what you did wrong! Be sure to end lines with ; and escape your quote marks by preceding them with \. It may be helpful to get TextExtras for easy format validation.
As you get comfortable with what the dictionary produces, you can now try and build up a parsing dictionary for another site. This requires loading pages in your web browser, examining the query strings, HTML source, and so on. It's dirty work, but quite gratifying when you see the data parsed out.
When the parse dictionary works for all kinds of data (and deals with no results gracefully), copy it out of the tool, paste it into a Text file, and e-mail it to us.

Documentation

We have made every effort to ensure that the documentation here is accurate. But there may be something missing. Let us know if anything needs clarification. (These pages were generated by AutoDoc.)

General APIs

WTool
: Protocol that all tools must follow. Describes functions your tool must implement, properties you can define for your bundle, and look & feel guidelines.
GeneralTool
: The easiest way to build a tool is to subclass GeneralTool. This describes the default implementations, the extra functionality you get by subclassing this, and how to subclass.
WToolbox
: WToolbox is the object handed to your tool when it initializes; it contains extra utility functions.

Page Loading and Parsing

URLLoader
: Wrapper around NSURLHandle that manages background loading in a somewhat simpler way.
URLManager
: Manages downloads of multiple URLs, returning NSData.
URLStackManager
: Manages multiple loads of URLs, limiting the number of concurrent loads.
ParseManager
: Manages downloading and parsing pages by making use of "parsing dictionaries" to break down the content of a Web page. The workhorse of all Watson tools. This page provides detailed information on how to build and use parsing dictionaries.
ImageViewManager
: Manages NSImageView objects with their contents loaded from remote URLs.
MultiParser
: Manages multiple, concurrent loads of pages with similar function.
LoadBroker
: Superclass of most of the classes above

Foundation Kit Extensions

NSArray
: Category for creating mutable copies
NSBundle
: Bundle utilities for loading parsing dictionaries
NSData
: Category for initializing from an XML string
NSSet
: Category for displaying contents
NSString
: Category for string searching and other useful functions
NSCalendarDate
: Category for returning a localized, relative ("Today"/"Yesterday"/"Tomorrow") description of a date.
NSColor
: Category for initializing from a Web-style RGB string.

Application Kit Extensions

DeletableTableView
: Subclass of NSTableView that responds to the delete key.
MergedColumnTableView
: Subclass of NSTableView in which some columns span across more than one cell.
HidingButton
: Subclass of NSButton that hides itself when it's disabled.
HidingProgress
: Subclass of NSProgressIndicator that hides itself when it's at zero.
NSBrowser
: Category for accessing the associated scroller.
NSImage
: Category to convert a bitmap NSImage to 72 DPI.
NSTableView
: Category for copying the selected row in a table to the clipboard.
NSView
: Category for maintaining a view with a single subview.
NSWorkspace
: Category for getting user's temporary directory and opening URLs
PlacardScrollView
: Subclass of NSScrollView for installing a "placard" view.

Need more information?

If you have any development questions or ideas on how this documentation can be improved, don't hesitate to contact us, either through the Watson Developers' Discusssion Group (now closed) or via our contact form.