Sheep Systems

In order to more easily implement some of the features that users have been requesting and provide an application which is befitting Mac OS 10.5 and 10.6, early this year I decided to succeed Bookdog with a new application.  Although Bookdog will continue to be updated as needed to maintain system compatibility, new capabilities and features are being designed into the new application only.  I've been working on it for several months, and things are now coalescing to the point where it's time to publish a specification for public review, stating the differences between the new application and Bookdog.  This page and the links on it are that specification.

Crossgrade Path for Bookdog Users

Our server is now set up to offer recent Bookdog licensees free crossgrade licenses within the new application with two clicks, and pro-rated discounts will be offered to longer-term Bookdog users.  The new application automatically imports application preferences, document preferences and item preferences (unordered folders, false alarms, etc.) from Bookdog.

System Requirements

The new application requires Mac OS X 10.5 (Leopard) or later.  Supported web browsers are:

• Camino
• Firefox 3.0 or later
• OmniWeb 5.0 or later
• Opera 9.5 or later
• Safari
• Shiira 2.0 or later

Supported web services are:

• del.icio.us
• Google Bookmarks

The old Firefox 2 and Netscape Navigator browsers are no longer supported.  Users of these old browsers can easily use Bookdog to migrate their bookmarks to a supported browser or service, all of which are free, so there is no reason not to use them.

Name and Icon

There seem to be two kinds of Bookdog users:  Those that love the icon, and those that hate it.  An app should not be judged by its icon, and the fuzzy fur poses an unnecessary technical and artistic challenge.  Therefore, the new application will have a new name, and a simple icon that may be beautiful, but will not evoke strong emotions either way.

Documents

The concepts of bookmarks documents and migration documents used by Bookdog can be confusing to users, and has limited what we can do.  Therefore, the new application has one unified document called a  bookmarks book, presented in a tabbed document window.  

Three background tools and schedules replaced by on-demand Librarian.

Inside Bookdog is a companion Bookwatchdog which "watches" bookmarks for cues that ordering should be done.  Migrations, can be scheduled using either Automator workflows saved as iCal plug-ins, or as launchd jobs scheduled by the Bookdog Migration Scheduler.  With the new bookmarks book, these operations can now be performed by the same tool, called the Librarian, and the new application can use launchd in Mac OS X 10.5 to launch it only when a task needs to be done.  After Librarian is done with this task, it terminates itself.  Having one tool to do both ordering and migrating allows a wider mix of tasks and triggers, frees users from having to see Bookwatchdog in their Login Items, and frees me from having to write and maintain lots of code that is necessary to add it, remove it, launch it, check it, and quit it when necessary.  

RSS Live Bookmark Feeds

Live Bookmark RSS feeds from Firefox or OmniWeb bookmarks are handled in a way that makes them useful when migrated or copied among Firefox, OmniWeb and Safari, and, by extension, the iPhone's RSS reader.  The last-updated list of articles, which are typically yesterday's news, which were shown in Bookdog, are not shown, and similarly these articles are not migrated or copied to other browsers as bookmarks.  If the other browser supports storing the current list of articles in its bookmarks, as do Firefox and OmniWeb, they are migrated or copied as articles.  Instead of being shown as folders, the RSS feed:// url is shown, and also it is copied to the destination during a migration, move or copy/paste operation.

Inspector Window

In Inspector Window, junk has been removed and the layout improved.

In order to accomodate the additional document functions which have been rolled in to the bookmarks book from Preferences and Migrations, the document window in the new application is tabbed.  The tabs are

• Content.  Shows the bookmarks and folders in hierarchical or flat view, with search fields; basically the Bookdog document window as it is now.
• General Preferences.  Same as Bookdog's Preferences > General, but laid out to only the attributes of the (one) subject bookmarks book document, so there is no table.  In addition, four checkboxes will be used to specify whether or not the document has a Bookmarks Bar, Bookmarks Menu, Unfiled Bookmarks, and/or Shared Bookmarks collection.  The first two collections are to mirror the collections featured by most web browsers, Unfiled Bookmarks is a feature of Firefox 3 and Shared Bookmarks is a feature of OmniWeb.  It is hoped that these four collections, or variations of them, will be sufficient to mirror the collections of any browser or web service for the foreseeable future!
• View Preferences.  Same as Bookdog's Preferences > View, but laid out to only the attributes of the (one) subject bookmarks book document, so there is no table.
• Order.  Same as Bookdog's Preferences > Sorting Order, laid out to only the attributes of the (one) subject bookmarks book document, so there is no table.
• Sync In/Out.  See detailed description.
• Automation.  Subsumes Bookdog's Open ("do upon launch") preference pane, the Bookwatchdog preference pane and the scheduling of Migration actions which must be done in Bookdog using Automator or the Bookdog Migration Scheduler.  In this tab, the user configures a list of background jobs which are done while the new application is not running.  Each job must have one or more triggers which may be configured configured to execute a sequence of commands.
  
  The following triggers are available in a popup menu in the Type column:
  • When I launch New Application.
  • When I 'Save' this Document.
  • Bookmarks Changed in Browser.  When this is chosen, it is followed in the Detail column by another popup.  The items in the second popup are a selection of installed browsers.
  • Scheduled Time of Day.  When this is chosen, it is followed in the Detail column by another popup which has a time-and-minute selector, NSCalenderSelector or whatever they call it.
  • When I Log In to my Mac Account.
  Note that the new application's Background Worker may demand conditions and thus will not always perform a job when a trigger is satisfied.  For example, most browsers (except Safari) behave badly or at least ignore changes if their bookmarks are altered, as is done by Sync Out, while they are running.  Therefore, the Background Worker will not perform such a synchronization as described in the second example job above unless both Camino and Firefox are not running.  Recalling the first example job, if a user is editing bookmarks in Safari's Show All Bookmarks window, this job would trigger ordering every time the user makes a change, which would be undesirable.  As with the old Bookwatchdog, a lockout timer of 3 minutes prevents it from going into a orderizing craze.  A note displayed under this setting will explain how it works...
  
  
  Note: "Change in Safari Bookmarks" triggers 3 minutes after the actual change.  Change in Other Browsers bookmarks triggers when the browser quits.
  
  The available commands are:
  • Sync In.  Followed in the Detail column with several choices:
    • Yield to browser.
    • Ask user if OK to quit browser
    • Quit browser.
    • Kill browser.
  • Sort
    
  • Find Duplicates
  • Verify
  • Sync Out.  Followed in the Detail column with three choices:
    • Yield to browser.
    • Ask user if OK to quit browser
      
    • Quit browser.
    • Kill browser.
  The Sync In and Sync Out commands will be performed as configured in the Sync In/Out tab.

  The Yield/Ask/Quit/Kill browser details handle the conflict which sometimes occurs if a browser is running.  The problem is that, of the desktop web browsers, only Safari can handle either a Sync In and Sync Out while it is running. Other browsers can handle a Sync In only, except for Firefox 3 which can handle neither Sync In nor Sync Out.  So, if a Sync In or Sync Out command in a job would be something that a running browser cannot handle, the new application (or Background Worker) needs to know what to do.  The default is to Yield to the browser and abort the job; it doesn't get done.  The second choice is to ask the user in a dialog.  Of course, this option is not available for jobs started by unattended triggers.  The third choice is to send the browser(s) Quit message(s).  But web browsers frequently display stupid dialogs when you ask them to quit, for example:  There is a window open.  Are you sure you want to quit?  If a user has scheduled a job to run overnight, there will be no one to answer this dialog.  Therefore, the final option, useful in the overnight-job situation, is to Kill the browser, with no questions asked.

  For example, to orderize Safari bookmarks whenever they are changed, the trigger would be Change detected in Safari, and the commands would be:
  • Sync In (which is configured in Sync In/Out to sync in from Safari only).
  • Orderize all items.
  • Sync Out (which is configured in Sync In/Out to sync our from Safari only)
  For another example, consider a user that wishes to synchronize Firefox, Camino and Safari, whenever Firefox is quit.  The trigger would be Firefox was quit.  This case requires only two commands:
  • Sync In (which is configured in Sync In/Out to sync in from Safari, Camino and Firefox).
    
  • Sync Out (which is configured in Sync In/Out to sync out to Safari, Camino and Firefox).

  Note that, due to the unification of functions which were separate in Bookdog, many possible previously-impossible automatic jobs, such as the above example of synchronizing when a browser is quit, are now possible.
  

  Before executing any command, New Application determines if it is necessary.  Generally, if the bookmarks involved have not changed in any way that could affect the result since the last time the command was executed, command execution is not necessary and is skipped.  The details of this determination are kind of complicated.

More to come.

The Need for Adjustments

Not all browsers and services allow bookmarks to be stored in the same way.  In Bookdog, each bookmarks document is wysiwyg (what you see is what you get) with respect to the browser or service.  For example, Camino does not allow loose bookmarks at its root level.  It only allows folders.  Therefore, if a user drags, migrates, moves or copies some loose bookmarks into the root level of a Camino document in Bookdog, knowing that these are Camino bookmarks, Bookdog retargets them during the drop to the Bookmarks Menu of the Camino document, and the user is immediately aware of the issue.

This wysiwyg behavior, while nice, is not be available in the bookmarks book, because it is an independent document, not restricted by the requirements of any particular browser or service.  The moves or other adjustments necessary to satisfy the destination browser or service will be made during Sync Out, and will not be shown to the user until the user launches the browser or service and sees the result. 

Sync In and Sync Out Process Details

Sync In is a two-step process internally.  First, the data is read in from the external browser or service to an external managed object context.  Second, it is internalized from there into the bookmark book's document managed object context, making any adjustments that are required or specified.

Sync Out is a three-step process internally.  The first step is externalize items from the bookmark book's document managed object context into the external managed object context, again using the old migrate routine to make whatever adjustments are necessary.  The next step is to actually write out the data.  For browser/services with atomic stores, this means to simply overwrite the old file.  For browser/services with nonatomic stores such as Firefox 3 or any online service, this means to compare/insert/update/delete items on the server or in the database to match the items in the external managed object context.  The third step, required for browsers or services which assigns external identifiers to new which are unpredictable, is to retrieve from the browser or service the external identifiers it assigned to any new items, and backfill those into the bookmarks book. 

Some of the adjustments are optional.  For example, when syncing in from a browser that does not support tags, some may want tags to be added to items based on their folder lineage.  The bookmarks book window will have a tab for configuring these options.  These options are generally those that are in the Translating tab of Bookdog's Preferences window.

Merging

To perform a Sync In or Sync Out operations, the new application needs to identify the path, file format, and protocols required to access the external store, which is the source of data for a Sync In command or the destination of data for a Sync Out command.   Physically, the external store is a file or directory, on a local drive or, for web services, on a remote server.  The information about the external store is stored in a data object called a browser service profile.  The attributes in this object are:

• External Store Format (exformat).  This tells the application or worker what file format or API to use.  Examples: Safari, Firefox, Delicious.
• Profile name.  This is for browser names that have user profiles.  Currently only Firefox uses this attribute.
• Store media.  This identifies where to get the data.  Possible values are: relative to user's home folder, relative to  other user's home folder (possibly on other Mac), relative to remote server, relative to startup volume root.
• Other Mac Account  This is required for other Mac account sources or destinations; bookmarks owned by desktop browsers on other accounts or networked accounts.  
  
• Other Mac Volume  This is optional, for other Mac account sources or destinations. It is used when bookmarks were accessed by mounting the root level of a networked hard drive.  
  
• Standalone path.  This is required for standalone sources or destinations.  These are "loose" bookmarks files.  Although they were created using the file format of a particular browser and thus have a browser or service name, are not known to that browser.
  

Jerry Krinock

Automatic Operations done by Bookdog

There are two operations that users have requested Bookdog be able to do automatically, ordering the bookmarks of one browser and migrating between browsers.  These features are avalailable, sort of, in current versions of Bookdog.

How Bookdog does it, and how it Got There

Automatically Ordering.  When I first wrote SafariSorter, and later the first versions of Bookdog, ordering of Safari bookmarks could be scheduled as a cron job.  This was rather crude, but basically I was in a hurry and this is the only thing I knew.  Later versions of Bookdog learned how to orderize when bookmarks were modified (Safari only), when browsers were quit, or at login.  The capability to orderize on a schedule was dropped, since this would have complicated things too much and the three new options were way better.  The actual ordering is done by Bookwatchdog, a background agent, specifically a Login Item,  which is always running (if the user has scheduled any such ordering jobs).

Automatically migrating.  Following Apple recommendations, I made the migration documents and the migrate command in Bookdog AppleScriptable, created a couple of Automator actions, and published instructions for scheduling Automator workflows containing bookmarks-migrating actions as iCal events.  Well, although I've tried hard to believe in Automator, after working with users on it for a year, I decided that there must be a better way and found that AppleScripts could be scheduled as launchd tasks.  So, I hacked together a quick AppleScript named Bookdog Migration Scheduler and also put a BookdogMigrationPerformer AppleScript in the Bookdog package.  The Bookdog Migration Scheduler opens a migration document in Bookdog, asks the user to configure it, and then stores it into a special folder in the user's Library > Application Support.  It then further schedules BookdogMigrationPerformer to run as a launchd task at a certain time every day.  When BookdogMigrationPerformer runs, it runs the migrations in the special folder and voila, we have a daily migration that does not require Automator and does not appear in iCal,  which is really annoying.  (Well, maybe it's not too annoying because, just between you and me, I can't imagine anyone in a sales-type job that seriously needs to keep track of appointments and deadlines using something as lame as iCal.)  Note now that, however it is scheduled, migrating is done by the Bookdog main application in the foreground.  That's not necessary.  Users who are most happy with the process schedule it for 4:00 AM or so.

A single tool to perform both ordering and synchronizing

First of all, in accordance with more modern usage, we're going to call copy/matching bookmarks items between and among browsers synchronizing and drop the term migrating.  And, both operations will be done in the background by a tool named[AppName] Librarian tool.  Unlike Bookwatchdog, the [AppName] Librarian will not be a Login Item and thus will not be visible to the user.

Triggers:  Does [AppName] Librarian need to be always running?

The launchd tool, which is running all the time in Mac OS X, can schedule tasks to run on day/time triggers.  It can also now watch directories, like the older kqueue used by Bookwatchdog.  I believe this capability is new in Leopard's launchd.  Anyhow, it's new to me!  And it makes it possible to do all automatic operations without a constantly-running background agent, with all the headaches that entails.  Instead, we can have a [AppName] Librarian, launched by launchd when necessary, and it can terminate itself whenever its task is complete.  Here is how we implement the various triggers.  

• Day/Time.  The launchd tool is easily configured for this trigger.
• Login.  The launchd tool is easily configured for this trigger.
• Change in [some browser's] Bookmarks.  We use the new WatchPaths trigger in launchd to watch the bookmarks file or directory.  ALSO, Experiments have shown that all web browsers, except Safari, reliably write a certain file or change the contents of a certain directory whenever they are quit.  For some browsers, this is the bookmarks file.  For others, it is a preferences file or some other file.  Thus, if we tell launchd to launch our [AppName] Librarian whenever this certain path has been altered, and then our [AppName] Librarian waits a few seconds to see if the browser is running, and if it is not, then our [AppName] Librarian knows that the browser has just been quit.  However, fortunately Safari is the only browser which allows us to change its bookmarks file while it is running.  Therefore, to simplify the user interface compared to Bookdog, the same trigger is for Safari as for other browsers.
• [AppName] launched.  [AppName]  stores in its Preferences a list of documents (paths and aliases) which have a job with an appLaunched trigger.  When it launches, unless it was launched by either Librarian or the user opening a document, it iterates through this list, opening the indicated documents and, for each, performing the job which has the appLaunched trigger.