3.4  Duplicates

The app has tools to manage duplicate bookmarks, which we call Duplicate Groups, and also tools to manage Duplicate Folders.

It is important to understand that the app can’t work on the bookmarks in Safari, Firefox or Chrome directly.  In addition, the precise definition of what makes a duplicate requires some discretion.  Here is how you eliminate duplicates from bookmarks in a web browser…

       Topics on this Page

3.4.1  Duplicate Groups (Bookmarks)

A group of bookmarks in a Bookmarkshelf which have the same URL is said to form a duplicate group.  (Note that the app normalizes URLs.)  The app is able to find duplicate groups within a Bookmarkshelf and displays a list of them in the tab Reports > Duplicates.

Note that some browser apps (for example, Google Bookmarks, Delicious, and Firefox) do not allow duplicate groups within your account. If you create within or export a new bookmark into one of these apps which creates a duplicate group with an existing bookmark, one of them (usually the existing one) will be immediately and silently deleted.

3.4.2  Finding Duplicates (Bookmarks)

Because the process of finding duplicates needs to compare each of your bookmarks with every other bookmark, it can take quite a few seconds of intense computation when the number of bookmarks is large.  (For example, if you have 1000 bookmarks, 1000 URLs may need to be computed, and 1000 × 999 / 2 = 499,500 comparisons must be made.)  Therefore, the app does not always keep this report updated, and instead does so only when commanded to update the report.

To find duplicates and thus update the report, you may click in the menu Bookmarkshelf > Find Duplicates,

or else click the hyperlink in the tab Reports > Duplicates.

If these controls are disabled, that means that no change which could have added duplicates has occurred since the last such update, and therefore no update is necessary.

Deleting duplicate groups in response to changes is not as laborious as finding them, so the app observes such changes and deletes defunct duplicate groups “on the fly”.

If you think that the app is not finding all the duplicates that you think are duplicates, see Aggressively Normalize URLs, below.

3.4.3  Allowing Some Duplicates (Bookmarks)

Although duplicate bookmarks are undesirable in general, you may wish to allow duplicates of certain bookmarks for easy access.  the app provides two mechanisms for you to do this.

Ignore Duplicates in different Hard Folders

Ignore Duplicates in different top-level Hard Folders is a setting which is useful if you’d like to keep a few commonly-used bookmarks in one hard folder, for example the Bookmarks Bar, but also have duplicates in another hard folder such as the Bookmarks Menu.  If this box is checked, Duplicate Groups which involve bookmarks in different Hard Folders are not reported.

Bookmarks whose lineage does not include one of these four hard folders, such as bookmarks in root directly, or bookmarks which are in a soft folder which descends from root, are considered to be in a “root” hard folder.  Duplicate Groups comprised of one bookmark in, for example, Bookmarks Bar, and another in the “root” will be ignored if this setting is checked ON.

In Smarky, Synkmark and Markster, this setting is a checkbox in Preferences > General.  In BookMacster, the checkbox is in a document’s Settings > General.

Duplicate Allowed Attribute

Duplicate Allowed is an is an attribute which may be switched on in any bookmark.  A bookmark so marked will never appear in a Duplicate Group.  There are several ways to mark (or unmark) a bookmark with this attribute:

will set the Duplicate Allowed attribute of the selected bookmark(s).

3.4.4  Removing Duplicates (Bookmarks)

Of course, the direct way to remove a duplicate group is to delete all of the bookmarks in it except one.  You can do this in either the Content View or, as shown here, in Reports > Duplicates…

Delete All Duplicates

But if you find yourself with more duplicate groups than you care to deal with individually, you can command the app to Delete All duplicate groups…

If you click Delete All, the app will step through each of the group and delete all except one in each.  (This is the same as clicking from the main menu > Bookmarkshelf Delete All Duplicates….)

Although this is fast, it may in some cases delete bookmarks whose name and/or location you would have preferred.  It is undo-able, so you can experiment.

Tags of deleted bookmarks.  If a deleted bookmarks has any tags, the app adds its tags to the tags of its surviving bookmark in the group, so that no tags are lost in this process.

Which one will survive?  For each group of less than 12 bookmarks, the app decides which one one has higher item quality.

For groups of 12 bookmarks or more, the app simply chooses one at random and deletes the remainder.  This is done to reduce the amount of time that it takes.

3.4.5  Aggressively Normalize URLs

When the app enters a bookmark into a document, either because it was imported or because you typed it in, the app normalizes its URL.  One of the effects of this is that all bookmarks in a document which must refer to the same web page have the same URL.  When the app finds duplicates, therefore, because normalization has already been done, it simply looks for identical URLs.

Users often have some bookmarks which may technically refer to different web pages, but in fact refer to the same web page.  So the app will not recognize them as duplicates as the user would like.  The most notorious example is bookmarks whose path portion of its URL has one or more components, ending in a slash.  Here is an example, showing two URLs for a bookmark whose path portion has one component, “/bar”

Technically, according to IETF standards, these two URLs may refer to different web pages.  Therefore, the app’s normalization does not remove a trailing slash from the end of a URL’s path if it has one or more components, and the app will not recognize these as duplicates.

If you do want the app to recognize these as duplicates, you can use the menu item Bookmarkshelf > Aggressively Normalize URLs, which will strip any trailing slashes from the ends of the URLs of all bookmarks in the document whose path has one or more components.  Note that this does not widen the app’s definition of a duplicate.  Instead, it is a one-time operation which checks all of the URLs in a document, and modifies some of them.

To aggressively delete duplicates, including such “near” duplicates in a document,

Note that if a URL has no path components, the app will add a one-character path, “/”  Example:

The above two URLs are technically the same, and the app will automatically add a slash to the end of the first one immediately upon it being imported or typed in to a document.

3.4.6  Consolidate Folders

The app’s definition of duplicate folders is more restrictive than that of a duplicate bookmarks.  To be considered duplicates, folders must (a) have the same parent folder, that is, be siblings of one another and (b) have the same name, where uppercase and lowercase differences are ignored so that, for example, “News” = “NEWS”.

In its main menu, the app provides Bookmarkshelf > Consolidate Folders menu item.  Clicking this menu item will search for all such such duplicate folders, select the one with the higher item quality, and merge all bookmarks from like folders into this one, emptying the others.  It will also search for and delete all empty soft folders.  Empty folders are deleted whether they were made empty by the merging, or whether they were empty to begin with.

Note that merging some duplicate folders and deleting some empty folders is also performed in the normal course of any Import or Export operation, but in this case the merging and deleting are only performed on folders which became candidates as a result of the Import or Export.

3.4.7  Automatically Detect New Duplicates (BookMacster only)

Because one person’s duplicate may be another person’s favorite bookmark, manually deduplicating, as described above, is recommended.

If you have BookMacster, however, and are using Agents to sync or sort bookmarks, you can set it to check for duplicates whenever bookmarks are changed, as part of each syncing job.  The same rule regarding Hard Folders, and any Duplicate Allowed attributes you have set on special bookmarks are respected.

To configure automatic duplicate detection,