First, an important update: if you are not subscribed to my feedburner URL that I switched to a long time ago, please switch your subscription to http://feeds.feedburner.com/AlexBosworth. I am moving my blog to SWiK: increased growth of SWiK traffic over the past few months has meant that we have needed to switch our setup to a more load-balanced setup across more webservers. Rather than struggle with MovableType, which has been having a series of problems, I’m just going to blog solely on this blog, which I was only cross posting to before. If you are subscribed to the RSS feed, you shouldn’t even notice the difference though.

Boz news

It’s been a month since I released boz – an aes encrypted, (open source) private bookmarking service. Today I rolled out the latest update, adding bookmarks export, rss support, and n-way tagging, as well as a tag search box.

RSS support was a tricky problem from the encryption standpoint. I use Firefox RSS integration with my del.icio.us bookmarks, and I really wanted the same functionality in boz. There are a handful of bookmarks that are sensitive and certainly not ‘social’, that I still want to have in my firefox livebookmarks. But Firefox can’t decrypt an encrypted RSS feed. The solution is to attach labels to bookmarks that should be included in an RSS feed. The labels can be generic if I don’t want to give away the purpose of the link, but in any case the real title and link of the bookmark is protected by the encryption.

• Encrypted tagging
  • tag cloud
  • tag autocompletion
  • tag search
• Starred bookmarks – ala gmail
• RSS feed – add cleartext labels to bookmarks to have them appear in an rss feed. The link address and title remain encrypted.
• export: pull out an encrypted stream of bookmarks

I’m biased, but I have to take back what I said earlier about private and public bookmarks not mixing. I’ve found that they do, because I’ve found myself only bookmarking encrypted bookmarks on boz and not really anything more on del.icio.us. I like to bookmark stupid stuff as well as internal intranet urls, and I am too lazy to figure out if it should be posted to del.icio.us or boz, so sorry if you are one of my network buddies on del.icio.us but this is part of the reason for the rash of updates to boz this month.

boz was originally written to replace a previous project that del.icio.us put the kibosh on called priv.at, and I wrote an importer for people who wanted to migrate bookmarks they saved there that were wiped by del.icio.us, but some have asked for a migration tool to encrypt their existing normal del.icio.us bookmarks in boz. I’m not sure if this is a big demand, but if people want to migrate their bookmarks to boz, let me know, it wouldn’t be too hard to adjust my existing migration script.

VideoBookmarks

I’ve updated VideoBookmarks, the application I built on wikiality to display the most recently bookmarked videos. The most recent update deals with a problem that I’ve had to deal with with delimages—people bookmark the same thing over and over.

VideoBookmarks now tries to only show the same video once.

Just rolled out a couple of new featurinos to boz.

• RSS feeds. You can now attach a cleartext label to a bookmark to add it to your RSS feed. Only the label will be sent in the clear, the link is still encrypted and you will need to go through an interim redirect page to decrypt it.

Warning: cleartext means that it is not encrypted. Anyone can read it, so use the labels judiciously. Also, if you lose your password, it’s not a password, it’s an encryption key and I do not keep a copy or ever even see your key, so make sure to remember it!

• Tag search. You can now search your bookmarks through tags, through a box above your tags, type the tag(s) you want and it will show you matching bookmarks.

Woot!

New update to boz:

• tag suggestions powered by bozComplete, you can now see and autocomplete common tags you have used
• tag cloud: your tags now sort themselves by frequency of use and appear bigger if you use them more.
• general caching improvements leveraging dojo.storage to speed load and render time.

Yay upgrade!

They said it couldn’t be done! OK – I said it would be a pain in my last post on releasing boz: encrypted web bookmarks.

To me, encrypted tagging seemed like a complicated problem, how can you keep track of tags if the database can’t be trusted to know what the tags are? If you encrypt the same tag word twice it results in different words, so you can’t search the encrypted text for a tag.

Well that actually seemed like fun to work around, so this weekend I rolled out an implementation of tagging into boz that allows for encrypted tags.

This is actually the third release for boz, the second being the ‘stars’ feature. I like the Gmail concept of starring something to make sure you don’t forget it, and I have missed this feature from del.icio.us. There are a variety of important internal addresses to SourceLabs and to SWiK that I’d rather not put on my del.icio.us bookmarks.

The stars feature lets me keep a short list of bookmarks that I hit more frequently, or ones that I need to do something with in the near future.

Tags is just a feature I’ve copied from del.icio.us, and really it’s essential for a bookmarks manager, and a much better model than folders.

The way that it’s implemented to keep things secure is that the client receives a tag dictionary from the server of all the tags that have been used in the past. When new bookmarks are posted, the client consults the dictionary to see if they should flag it with an existing tag from the dictionary, if not, they post an encrypted request to create a new tag.

Now that I’ve started playing around with encryption, I am starting to think it’s a good idea for a lot more applications. Maybe ‘private’ isn’t the new ‘social’, but in migrating to pure web apps off of desktop apps, encrypting content does away with one of the major reservations in leaving the desktop – giving your security over to some third party server.

It seems to be a trend: if you’re using web applications, forget about privacy. Forget about owning your own data.

A while ago, I had an issue with this aspect of del.icio.us. Once upon a time, there was no way to save private bookmarks: it’s still pretty obtuse. Well, I hacked up something to get around the limitation and create bookmarks anonymously or semi-privately if not privately.

It was just meant to be something to bookmark for myself and maybe something to blog about, but people took to it, and over 11k private bookmarks were saved using it in the space of 8 months. However a short time ago, Yahoo/del.icio.us blocked the service: not only could I not post new private bookmarks for myself, all the ones I and everyone else had saved were wiped out.

Well ok, lesson learned. But I still want to bookmark privately, and I don’t like the way del.icio.us does it: public and private bookmarks are not chocolate and peanut butter, they should be separate. And private bookmarks should be really private, I don’t even want to trust the servers with them.

So I’ve coded up an open source solution: a web service that lets you post bookmarks that even the server doesn’t know about.

How it works is that just before you post a bookmark, your browser encrypts the data and sends the bookmark information encrypted with your private key to the server. To browse your bookmarks, the server sends them back encrypted and your browser then decrypts them.

A special bookmarklet can be used on remote web pages to post, or you can post directly via the interface.

Because of the heavy use of browser encryption and decryption, the entire application is written in JavaScript/Ajax.

Also, if you have used my previous private bookmarks solution, please email me at alex.bosworth+projects at gmail – I’ve set aside your username and I’ll import your bookmarks that were blanked into the new service.

For those interested, I’ll go into some details on how the application was developed:

Developing an encrypted bookmarks web application

I’ve been working on this project in my spare time since a few months ago when Yahoo/del.icio.us gave me the final word that priv.at was blocked for good. There are several challenges to an encrypted bookmarks service that needed to be overcome:

1. Paucity of quality open source web browser encryption and decryption libraries
2. Developing a bookmarklet that doesn’t betray the url to the server
3. Storing an encryption key on the browser beyond a single page load, without persisting it to the server
4. Ensuring that encrypted data doesn’t become corrupted
5. Keeping a javascript application fast
6. Dealing with pages with content that all has to be decrypted
7. Search and tagging without the database knowing what it’s searching for

The first problem of finding a suitable library was just a matter of going through various libraries, looking at the code and running them through unit tests. Most javascript encryption libraries however were designed as proof of concepts, or coded in very ugly ways, or not for any kind of performance and are therefore very difficult to adapt. This just took methodical testing to find one I liked. I then extended the string object with a .encrypt and a .decrypt method, this allows for encryption to be a simple component of the application.

The problem of developing a bookmarklet is that traditionally you encode the url in a get parameter, and then the server echoes what you asked it to ‘get’ when you hit the post screen. But that implies that the server knows what urls you are interested in. I wanted to avoid that, so my bookmarklet uses the only part of a url that is not passed to the server: the hash.

Storing an encryption key on the browser was another issue that I didn’t really anticipate. It is quite annoying to have to type in your encryption key every time you want to see your bookmarks or post a new one, you get used to being just logged in and having the server remember that you authenticated and it can send you privileged information. But that doesn’t work in this case, you must never tell the server what your private key is, but somehow have JavaScript remember it from page load to page load, which is not something that JavaScript seems to have been designed for. Luckily, the dojo toolkit provides a JavaScript to Flash bridge that allows for permanent storage on the browser, something normally of limited use, but perfect for my purposes.

This was my first practical use of the dojo javascript toolkit, and I have had a mixed experience. I have found on the one hand it’s fairly elegant as an API, provides the functionality I need, and is generally very powerful. On the other hand, it doesn’t always work quite like it should and it creates problems for Safari and Opera: I haven’t even tried IE yet. I decided the tradeoff of having to type your key in over and over was worth losing the minority browsers temporarily, and I’ll look at fixing that at a later stage.

Another problem that I ran into during development of the project was the fact that strange corruptions were taking place in some posts of bookmarks. I would post a bookmark, and it would sometimes return from the server garbled. I could post something 5 times in a row, and 4 could return fine and the fifth would be corrupt. This made it one of the more frustrating issues to pin down. One issue that was obvious is that I had forgotten that encrypting the strings would make them too large for MySQL’s maximum varchar space of 255 characters, which is usually ok for a title and a url.

Another issue is that the encryption library I use doesn’t encode to hex, so it makes data transmission and application design a little more complicated. My normal style of writing JavaScript is to keep everything in the document. Building web applications, you might have various stages of data representation: a database schema, an object schema, a javascript object schema, and finally a document schema. A bookmark is one thing in the database, another as a server object, another as a javascript object, and another as an html node. Because of this, my practice is to generally avoid JavaScript variables and store everything right in the html. I also try to avoid generating html in JavaScript, I prefer to keep things simple and leave all the html generation to PHP.

Except that I discovered that storing the encrypted bookmarks in the document would corrupt them. This meant that PHP had to become a generator of JSON instead of HTML, and JavaScript would then take over the job of generating the pages. The data transmission issue was solved by tracking down the appropriate escape functions in JavaScript and storing the bookmarks in the database escaped for JavaScript.

Finally, all this JavaScript made the application slow, encryption is a processor intensive business and Firefox’s JavaScript engine is sluggish at best, so I’ve limited the number of bookmarks on a page to 15 and tuned the JavaScript to avoid excessive DOM manipulation, which is the biggest CPU killer out there.

Oh yes, lest I forget: tagging. I decided to leave that out for the time being. The server can’t search for a tag, because the encrypted text is different even for the same word encrypted with the same key. Even the same word encrypted twice in a row is not the same. This of course means that I can’t prevent users from posting the same bookmark twice.

If I wanted to implement tagging/search, I would need to either use a different type of encryption that gave back the same result for the same input, or I would need to burn CPU on maintaining a dictionary on the browser side. The original priv.at del.icio.us bookmarks didn’t have tagging, I can still look through dates and page quickly through my bookmarks, so I have left that feature out as being too CPU intensive for a first pass at creating a quick bookmarking application.

Of course I have also published the project source as GPL v2 for those interested.

Boz uses:

• dojo toolkit
• prototype toolkit
• md5 script from livejournal
• aes encryption script
• swajax javascript/php framework – from musika. (taken from swik codebase)
• bozComplete autocompletion javascript library

The boz sources are in the /boz2/sources/ directory:

• data.phps
• requires.phps
• util.phps
• view.phps
• controller.phps
• service.phps

Images are from:

• kde nuvola collection (gpl)
• sky wallpaper (cc)
• homepage shot (cc)