sorted by: recent | see : popular

Content Tagged with User:alex + JavaScript

I’ve written another of my little javascript web apps, this time based on Digg’s new expansive API.

The API is very well designed and I’ve only found a couple problems with it: the license is very restrictive and getting back some info requires many multiple calls to digg, but I guess it points to the problems with web apis as just being poor proxies for direct database calls.

The Digg application is a friends browser. – It basically asks the question: “Who’s digging you”.

I wrote this script because i’ve noticed that the front page of Digg is guarded by social networks of people who digg each other’s stories. If you want to get your story noticed, you want to have digg friends.

Enter any top digger name, you will see a high ratio of repeat diggers – the same is true of my submissions as well (Thanks Pawfoots!).

As I wrote it, I was interested really in just browsing around the networks, so I thought that was fun enough for an app. It’s kind of like an earlier script I wrote: delimages.

Anyways my app started with a single function basically that gives you back a digg response object with what you want. If you write a Digg script, you can use my little Digg function:

var Digg = 
{
    apikey : 'http://sandbox.sourcelabs.com/'
};

Digg.request = function (path, count, cbkName, offset)
{
    var url = 'http://services.digg.com/' + path;

    if (!offset) offset = 0; 

    var params = $H(
    {
        'count'    :  count, 
        'type'     : 'javascript', 
        'appkey'   :  Digg.apikey, 
        'callback' :  cbkName,
        'offset'   :  offset
    });

    url += '?' + params.toQueryString();

    var head = document.getElementsByTagName('head')[0];
    var request = document.createElement('script');
        request.type = 'text/javascript';
        request.src = url;

    head.appendChild(request);
}

All of these JSON Apis are really hurting my Javascript OOP sensibilities. XMLHttpRequest is cool with you making an anonymous function on the fly as the callback, but with JSON the best approach probably is having a callback name passed in the object return, which isn’t the cleanest.

I tried to make do in my script as best I could though.

It seems to be a trend: if you’re using web applications, forget about privacy. Forget about owning your own data.

A while ago, I had an issue with this aspect of del.icio.us. Once upon a time, there was no way to save private bookmarks: it’s still pretty obtuse. Well, I hacked up something to get around the limitation and create bookmarks anonymously or semi-privately if not privately.

It was just meant to be something to bookmark for myself and maybe something to blog about, but people took to it, and over 11k private bookmarks were saved using it in the space of 8 months. However a short time ago, Yahoo/del.icio.us blocked the service: not only could I not post new private bookmarks for myself, all the ones I and everyone else had saved were wiped out.

Well ok, lesson learned. But I still want to bookmark privately, and I don’t like the way del.icio.us does it: public and private bookmarks are not chocolate and peanut butter, they should be separate. And private bookmarks should be really private, I don’t even want to trust the servers with them.

So I’ve coded up an open source solution: a web service that lets you post bookmarks that even the server doesn’t know about.

How it works is that just before you post a bookmark, your browser encrypts the data and sends the bookmark information encrypted with your private key to the server. To browse your bookmarks, the server sends them back encrypted and your browser then decrypts them.

A special bookmarklet can be used on remote web pages to post, or you can post directly via the interface.

Because of the heavy use of browser encryption and decryption, the entire application is written in JavaScript/Ajax.

Also, if you have used my previous private bookmarks solution, please email me at alex.bosworth+projects at gmail – I’ve set aside your username and I’ll import your bookmarks that were blanked into the new service.

For those interested, I’ll go into some details on how the application was developed:

Developing an encrypted bookmarks web application

I’ve been working on this project in my spare time since a few months ago when Yahoo/del.icio.us gave me the final word that priv.at was blocked for good. There are several challenges to an encrypted bookmarks service that needed to be overcome:

1. Paucity of quality open source web browser encryption and decryption libraries
2. Developing a bookmarklet that doesn’t betray the url to the server
3. Storing an encryption key on the browser beyond a single page load, without persisting it to the server
4. Ensuring that encrypted data doesn’t become corrupted
5. Keeping a javascript application fast
6. Dealing with pages with content that all has to be decrypted
7. Search and tagging without the database knowing what it’s searching for

The first problem of finding a suitable library was just a matter of going through various libraries, looking at the code and running them through unit tests. Most javascript encryption libraries however were designed as proof of concepts, or coded in very ugly ways, or not for any kind of performance and are therefore very difficult to adapt. This just took methodical testing to find one I liked. I then extended the string object with a .encrypt and a .decrypt method, this allows for encryption to be a simple component of the application.

The problem of developing a bookmarklet is that traditionally you encode the url in a get parameter, and then the server echoes what you asked it to ‘get’ when you hit the post screen. But that implies that the server knows what urls you are interested in. I wanted to avoid that, so my bookmarklet uses the only part of a url that is not passed to the server: the hash.

Storing an encryption key on the browser was another issue that I didn’t really anticipate. It is quite annoying to have to type in your encryption key every time you want to see your bookmarks or post a new one, you get used to being just logged in and having the server remember that you authenticated and it can send you privileged information. But that doesn’t work in this case, you must never tell the server what your private key is, but somehow have JavaScript remember it from page load to page load, which is not something that JavaScript seems to have been designed for. Luckily, the dojo toolkit provides a JavaScript to Flash bridge that allows for permanent storage on the browser, something normally of limited use, but perfect for my purposes.

This was my first practical use of the dojo javascript toolkit, and I have had a mixed experience. I have found on the one hand it’s fairly elegant as an API, provides the functionality I need, and is generally very powerful. On the other hand, it doesn’t always work quite like it should and it creates problems for Safari and Opera: I haven’t even tried IE yet. I decided the tradeoff of having to type your key in over and over was worth losing the minority browsers temporarily, and I’ll look at fixing that at a later stage.

Another problem that I ran into during development of the project was the fact that strange corruptions were taking place in some posts of bookmarks. I would post a bookmark, and it would sometimes return from the server garbled. I could post something 5 times in a row, and 4 could return fine and the fifth would be corrupt. This made it one of the more frustrating issues to pin down. One issue that was obvious is that I had forgotten that encrypting the strings would make them too large for MySQL’s maximum varchar space of 255 characters, which is usually ok for a title and a url.

Another issue is that the encryption library I use doesn’t encode to hex, so it makes data transmission and application design a little more complicated. My normal style of writing JavaScript is to keep everything in the document. Building web applications, you might have various stages of data representation: a database schema, an object schema, a javascript object schema, and finally a document schema. A bookmark is one thing in the database, another as a server object, another as a javascript object, and another as an html node. Because of this, my practice is to generally avoid JavaScript variables and store everything right in the html. I also try to avoid generating html in JavaScript, I prefer to keep things simple and leave all the html generation to PHP.

Except that I discovered that storing the encrypted bookmarks in the document would corrupt them. This meant that PHP had to become a generator of JSON instead of HTML, and JavaScript would then take over the job of generating the pages. The data transmission issue was solved by tracking down the appropriate escape functions in JavaScript and storing the bookmarks in the database escaped for JavaScript.

Finally, all this JavaScript made the application slow, encryption is a processor intensive business and Firefox’s JavaScript engine is sluggish at best, so I’ve limited the number of bookmarks on a page to 15 and tuned the JavaScript to avoid excessive DOM manipulation, which is the biggest CPU killer out there.

Oh yes, lest I forget: tagging. I decided to leave that out for the time being. The server can’t search for a tag, because the encrypted text is different even for the same word encrypted with the same key. Even the same word encrypted twice in a row is not the same. This of course means that I can’t prevent users from posting the same bookmark twice.

If I wanted to implement tagging/search, I would need to either use a different type of encryption that gave back the same result for the same input, or I would need to burn CPU on maintaining a dictionary on the browser side. The original priv.at del.icio.us bookmarks didn’t have tagging, I can still look through dates and page quickly through my bookmarks, so I have left that feature out as being too CPU intensive for a first pass at creating a quick bookmarking application.

Of course I have also published the project source as GPL v2 for those interested.