First, an important update: if you are not subscribed to my feedburner URL that I switched to a long time ago, please switch your subscription to http://feeds.feedburner.com/AlexBosworth. I am moving my blog to SWiK: increased growth of SWiK traffic over the past few months has meant that we have needed to switch our setup to a more load-balanced setup across more webservers. Rather than struggle with MovableType, which has been having a series of problems, I’m just going to blog solely on this blog, which I was only cross posting to before. If you are subscribed to the RSS feed, you shouldn’t even notice the difference though.

Boz news

It’s been a month since I released boz – an aes encrypted, (open source) private bookmarking service. Today I rolled out the latest update, adding bookmarks export, rss support, and n-way tagging, as well as a tag search box.

RSS support was a tricky problem from the encryption standpoint. I use Firefox RSS integration with my del.icio.us bookmarks, and I really wanted the same functionality in boz. There are a handful of bookmarks that are sensitive and certainly not ‘social’, that I still want to have in my firefox livebookmarks. But Firefox can’t decrypt an encrypted RSS feed. The solution is to attach labels to bookmarks that should be included in an RSS feed. The labels can be generic if I don’t want to give away the purpose of the link, but in any case the real title and link of the bookmark is protected by the encryption.

• Encrypted tagging
  • tag cloud
  • tag autocompletion
  • tag search
• Starred bookmarks – ala gmail
• RSS feed – add cleartext labels to bookmarks to have them appear in an rss feed. The link address and title remain encrypted.
• export: pull out an encrypted stream of bookmarks

I’m biased, but I have to take back what I said earlier about private and public bookmarks not mixing. I’ve found that they do, because I’ve found myself only bookmarking encrypted bookmarks on boz and not really anything more on del.icio.us. I like to bookmark stupid stuff as well as internal intranet urls, and I am too lazy to figure out if it should be posted to del.icio.us or boz, so sorry if you are one of my network buddies on del.icio.us but this is part of the reason for the rash of updates to boz this month.

boz was originally written to replace a previous project that del.icio.us put the kibosh on called priv.at, and I wrote an importer for people who wanted to migrate bookmarks they saved there that were wiped by del.icio.us, but some have asked for a migration tool to encrypt their existing normal del.icio.us bookmarks in boz. I’m not sure if this is a big demand, but if people want to migrate their bookmarks to boz, let me know, it wouldn’t be too hard to adjust my existing migration script.

VideoBookmarks

I’ve updated VideoBookmarks, the application I built on wikiality to display the most recently bookmarked videos. The most recent update deals with a problem that I’ve had to deal with with delimages—people bookmark the same thing over and over.

VideoBookmarks now tries to only show the same video once.

It seems to be a trend: if you’re using web applications, forget about privacy. Forget about owning your own data.

A while ago, I had an issue with this aspect of del.icio.us. Once upon a time, there was no way to save private bookmarks: it’s still pretty obtuse. Well, I hacked up something to get around the limitation and create bookmarks anonymously or semi-privately if not privately.

It was just meant to be something to bookmark for myself and maybe something to blog about, but people took to it, and over 11k private bookmarks were saved using it in the space of 8 months. However a short time ago, Yahoo/del.icio.us blocked the service: not only could I not post new private bookmarks for myself, all the ones I and everyone else had saved were wiped out.

Well ok, lesson learned. But I still want to bookmark privately, and I don’t like the way del.icio.us does it: public and private bookmarks are not chocolate and peanut butter, they should be separate. And private bookmarks should be really private, I don’t even want to trust the servers with them.

So I’ve coded up an open source solution: a web service that lets you post bookmarks that even the server doesn’t know about.

How it works is that just before you post a bookmark, your browser encrypts the data and sends the bookmark information encrypted with your private key to the server. To browse your bookmarks, the server sends them back encrypted and your browser then decrypts them.

A special bookmarklet can be used on remote web pages to post, or you can post directly via the interface.

Because of the heavy use of browser encryption and decryption, the entire application is written in JavaScript/Ajax.

Also, if you have used my previous private bookmarks solution, please email me at alex.bosworth+projects at gmail – I’ve set aside your username and I’ll import your bookmarks that were blanked into the new service.

For those interested, I’ll go into some details on how the application was developed:

Developing an encrypted bookmarks web application

I’ve been working on this project in my spare time since a few months ago when Yahoo/del.icio.us gave me the final word that priv.at was blocked for good. There are several challenges to an encrypted bookmarks service that needed to be overcome:

1. Paucity of quality open source web browser encryption and decryption libraries
2. Developing a bookmarklet that doesn’t betray the url to the server
3. Storing an encryption key on the browser beyond a single page load, without persisting it to the server
4. Ensuring that encrypted data doesn’t become corrupted
5. Keeping a javascript application fast
6. Dealing with pages with content that all has to be decrypted
7. Search and tagging without the database knowing what it’s searching for

The first problem of finding a suitable library was just a matter of going through various libraries, looking at the code and running them through unit tests. Most javascript encryption libraries however were designed as proof of concepts, or coded in very ugly ways, or not for any kind of performance and are therefore very difficult to adapt. This just took methodical testing to find one I liked. I then extended the string object with a .encrypt and a .decrypt method, this allows for encryption to be a simple component of the application.

The problem of developing a bookmarklet is that traditionally you encode the url in a get parameter, and then the server echoes what you asked it to ‘get’ when you hit the post screen. But that implies that the server knows what urls you are interested in. I wanted to avoid that, so my bookmarklet uses the only part of a url that is not passed to the server: the hash.

Storing an encryption key on the browser was another issue that I didn’t really anticipate. It is quite annoying to have to type in your encryption key every time you want to see your bookmarks or post a new one, you get used to being just logged in and having the server remember that you authenticated and it can send you privileged information. But that doesn’t work in this case, you must never tell the server what your private key is, but somehow have JavaScript remember it from page load to page load, which is not something that JavaScript seems to have been designed for. Luckily, the dojo toolkit provides a JavaScript to Flash bridge that allows for permanent storage on the browser, something normally of limited use, but perfect for my purposes.

This was my first practical use of the dojo javascript toolkit, and I have had a mixed experience. I have found on the one hand it’s fairly elegant as an API, provides the functionality I need, and is generally very powerful. On the other hand, it doesn’t always work quite like it should and it creates problems for Safari and Opera: I haven’t even tried IE yet. I decided the tradeoff of having to type your key in over and over was worth losing the minority browsers temporarily, and I’ll look at fixing that at a later stage.

Another problem that I ran into during development of the project was the fact that strange corruptions were taking place in some posts of bookmarks. I would post a bookmark, and it would sometimes return from the server garbled. I could post something 5 times in a row, and 4 could return fine and the fifth would be corrupt. This made it one of the more frustrating issues to pin down. One issue that was obvious is that I had forgotten that encrypting the strings would make them too large for MySQL’s maximum varchar space of 255 characters, which is usually ok for a title and a url.

Another issue is that the encryption library I use doesn’t encode to hex, so it makes data transmission and application design a little more complicated. My normal style of writing JavaScript is to keep everything in the document. Building web applications, you might have various stages of data representation: a database schema, an object schema, a javascript object schema, and finally a document schema. A bookmark is one thing in the database, another as a server object, another as a javascript object, and another as an html node. Because of this, my practice is to generally avoid JavaScript variables and store everything right in the html. I also try to avoid generating html in JavaScript, I prefer to keep things simple and leave all the html generation to PHP.

Except that I discovered that storing the encrypted bookmarks in the document would corrupt them. This meant that PHP had to become a generator of JSON instead of HTML, and JavaScript would then take over the job of generating the pages. The data transmission issue was solved by tracking down the appropriate escape functions in JavaScript and storing the bookmarks in the database escaped for JavaScript.

Finally, all this JavaScript made the application slow, encryption is a processor intensive business and Firefox’s JavaScript engine is sluggish at best, so I’ve limited the number of bookmarks on a page to 15 and tuned the JavaScript to avoid excessive DOM manipulation, which is the biggest CPU killer out there.

Oh yes, lest I forget: tagging. I decided to leave that out for the time being. The server can’t search for a tag, because the encrypted text is different even for the same word encrypted with the same key. Even the same word encrypted twice in a row is not the same. This of course means that I can’t prevent users from posting the same bookmark twice.

If I wanted to implement tagging/search, I would need to either use a different type of encryption that gave back the same result for the same input, or I would need to burn CPU on maintaining a dictionary on the browser side. The original priv.at del.icio.us bookmarks didn’t have tagging, I can still look through dates and page quickly through my bookmarks, so I have left that feature out as being too CPU intensive for a first pass at creating a quick bookmarking application.

Of course I have also published the project source as GPL v2 for those interested.

Still no news on the Yahoo/del.icio.us conundrum.

If you’ve seen the source code, you might see that I don’t actually store any of the private bookmarks you have saved, otherwise it might be easier to recover the missing bookmarks.

In implementing priv.at this was a deliberate decision to help protect the privacy of bookmarkers, however now it seems to be somewhat of a mistake relying on a third party webservice to store the data.

If all comes to naught with del.icio.us, I will begin work on using the del.icio.us API to reconstruct the private bookmarks and migrate to a new account, I will keep you updated through this wiki page.

-alex

As you might have noticed, priv.at is working ok and has been for a little while now, Joshua lifted the ban on our IP and we’ve just bookmarked our 7,457th private bookmark.

Stay classy

OK del.icio.us is back up and everything is working again – LiveMarks, delimages, priv.at, even BozPages tag search.

I’ve added in redundant social bookmarking services so LiveMarks isn’t so much at the mercy of del.icio.us. Now if you bookmark something on del.irio.us or any of the other social-bookmarking sites that are wise enough to offer a feed of their bookmarks, it will scroll by just like the del.icio.us bookmarks.

I’ve been having some problems recently accessing LiveMarks, it seems sometimes it will just time out. Sometimes it works again if I quit and restart FireFox completely, but that might just because I switch between internal and external networks.

I think it must be something with DNS or with Apache or FireFox, or maybe a combination of different things.

If you are seeing any similar problems, please post a comment to let me know it’s not just me.

priv.at works in two stages. (source code at the bottom of this page)

Generating a bookmarklet

priv.at requires that you enter your username, and then a piece of javascript creates a bookmark for you dynamically.

This bookmarklet is essentially your username and the url + title (which is taken from the html title) of the page you are looking at.

Posting bookmarks to del.icio.us

It’s possible to tag something for:userName and have that bookmark sent to the userName. So priv.at simply uses the del.icio.us API to post a bookmark.

The tricky part is what if two people bookmark the same url? Then the bookmark can’t be added, it has to be modified to include the new person.

In order to do this, priv.at keeps a record of every bookmark and every bookmarker, and joins them in a many to many relationship.

When bookmarks are privately posted, priv.at looks to see if the bookmark has been saved privately already, and if it has, then it appends the username/bookmark pair to the list and reconstructs the bookmark and posts it to del.icio.us.

This is why it’s important that the local database and del.icio.us never get out of sync.

Source Code

• source php file

Database schema:

-- Database: `bookmarks`
-- 

-- --------------------------------------------------------

-- 
-- Table structure for table `pmarks`
-- 

CREATE TABLE `pmarks` (
  `id` int(11) NOT NULL auto_increment,
  `hash` varchar(255) NOT NULL default '',
  PRIMARY KEY  (`id`),
  UNIQUE KEY `hash` (`hash`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=4604 ;

-- --------------------------------------------------------

-- 
-- Table structure for table `pusers`
-- 

CREATE TABLE `pusers` (
  `pmarks_id` int(11) NOT NULL default '0',
  `username` varchar(255) NOT NULL default '',
  KEY `pmarks_id` (`pmarks_id`,`username`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

• Bookmarks posted by priv.at are visible, only who posts them is kept from public view.

• Bookmarks cannot be removed from your ‘for’ list.
• Who bookmarked an url is stored in the clear on both del.icio.us servers, sourcelabs servers, and also the account used to bookmark things privately is accessible by sourcelabs.
• Bookmarks can’t be sorted or tagged, etc.

• Users could ‘authenticate’ themselves potentially by bookmarking a special token – the server would then check to see if the account had bookmarked the token, confirming that the user did indeed have control of the del.icio.us account.
• Users could also potentially authenticate themselves through the del.icio.us API, by handing over their credentials.