javascript shrinker

It seems to be a trend: if you’re using web applications, forget about privacy. Forget about owning your own data.

A while ago, I had an issue with this aspect of del.icio.us. Once upon a time, there was no way to save private bookmarks: it’s still pretty obtuse. Well, I hacked up something to get around the limitation and create bookmarks anonymously or semi-privately if not privately.

It was just meant to be something to bookmark for myself and maybe something to blog about, but people took to it, and over 11k private bookmarks were saved using it in the space of 8 months. However a short time ago, Yahoo/del.icio.us blocked the service: not only could I not post new private bookmarks for myself, all the ones I and everyone else had saved were wiped out.

Well ok, lesson learned. But I still want to bookmark privately, and I don’t like the way del.icio.us does it: public and private bookmarks are not chocolate and peanut butter, they should be separate. And private bookmarks should be really private, I don’t even want to trust the servers with them.

So I’ve coded up an open source solution: a web service that lets you post bookmarks that even the server doesn’t know about.

How it works is that just before you post a bookmark, your browser encrypts the data and sends the bookmark information encrypted with your private key to the server. To browse your bookmarks, the server sends them back encrypted and your browser then decrypts them.

A special bookmarklet can be used on remote web pages to post, or you can post directly via the interface.

Because of the heavy use of browser encryption and decryption, the entire application is written in JavaScript/Ajax.

Also, if you have used my previous private bookmarks solution, please email me at alex.bosworth+projects at gmail – I’ve set aside your username and I’ll import your bookmarks that were blanked into the new service.

---

For those interested, I’ll go into some details on how the application was developed:

Developing an encrypted bookmarks web application

I’ve been working on this project in my spare time since a few months ago when Yahoo/del.icio.us gave me the final word that priv.at was blocked for good. There are several challenges to an encrypted bookmarks service that needed to be overcome:

1. Paucity of quality open source web browser encryption and decryption libraries
2. Developing a bookmarklet that doesn’t betray the url to the server
3. Storing an encryption key on the browser beyond a single page load, without persisting it to the server
4. Ensuring that encrypted data doesn’t become corrupted
5. Keeping a javascript application fast
6. Dealing with pages with content that all has to be decrypted
7. Search and tagging without the database knowing what it’s searching for

The first problem of finding a suitable library was just a matter of going through various libraries, looking at the code and running them through unit tests. Most javascript encryption libraries however were designed as proof of concepts, or coded in very ugly ways, or not for any kind of performance and are therefore very difficult to adapt. This just took methodical testing to find one I liked. I then extended the string object with a .encrypt and a .decrypt method, this allows for encryption to be a simple component of the application.

The problem of developing a bookmarklet is that traditionally you encode the url in a get parameter, and then the server echoes what you asked it to ‘get’ when you hit the post screen. But that implies that the server knows what urls you are interested in. I wanted to avoid that, so my bookmarklet uses the only part of a url that is not passed to the server: the hash.

Storing an encryption key on the browser was another issue that I didn’t really anticipate. It is quite annoying to have to type in your encryption key every time you want to see your bookmarks or post a new one, you get used to being just logged in and having the server remember that you authenticated and it can send you privileged information. But that doesn’t work in this case, you must never tell the server what your private key is, but somehow have JavaScript remember it from page load to page load, which is not something that JavaScript seems to have been designed for. Luckily, the dojo toolkit provides a JavaScript to Flash bridge that allows for permanent storage on the browser, something normally of limited use, but perfect for my purposes.

This was my first practical use of the dojo javascript toolkit, and I have had a mixed experience. I have found on the one hand it’s fairly elegant as an API, provides the functionality I need, and is generally very powerful. On the other hand, it doesn’t always work quite like it should and it creates problems for Safari and Opera: I haven’t even tried IE yet. I decided the tradeoff of having to type your key in over and over was worth losing the minority browsers temporarily, and I’ll look at fixing that at a later stage.

Another problem that I ran into during development of the project was the fact that strange corruptions were taking place in some posts of bookmarks. I would post a bookmark, and it would sometimes return from the server garbled. I could post something 5 times in a row, and 4 could return fine and the fifth would be corrupt. This made it one of the more frustrating issues to pin down. One issue that was obvious is that I had forgotten that encrypting the strings would make them too large for MySQL’s maximum varchar space of 255 characters, which is usually ok for a title and a url.

Another issue is that the encryption library I use doesn’t encode to hex, so it makes data transmission and application design a little more complicated. My normal style of writing JavaScript is to keep everything in the document. Building web applications, you might have various stages of data representation: a database schema, an object schema, a javascript object schema, and finally a document schema. A bookmark is one thing in the database, another as a server object, another as a javascript object, and another as an html node. Because of this, my practice is to generally avoid JavaScript variables and store everything right in the html. I also try to avoid generating html in JavaScript, I prefer to keep things simple and leave all the html generation to PHP.

Except that I discovered that storing the encrypted bookmarks in the document would corrupt them. This meant that PHP had to become a generator of JSON instead of HTML, and JavaScript would then take over the job of generating the pages. The data transmission issue was solved by tracking down the appropriate escape functions in JavaScript and storing the bookmarks in the database escaped for JavaScript.

Finally, all this JavaScript made the application slow, encryption is a processor intensive business and Firefox’s JavaScript engine is sluggish at best, so I’ve limited the number of bookmarks on a page to 15 and tuned the JavaScript to avoid excessive DOM manipulation, which is the biggest CPU killer out there.

Oh yes, lest I forget: tagging. I decided to leave that out for the time being. The server can’t search for a tag, because the encrypted text is different even for the same word encrypted with the same key. Even the same word encrypted twice in a row is not the same. This of course means that I can’t prevent users from posting the same bookmark twice.

If I wanted to implement tagging/search, I would need to either use a different type of encryption that gave back the same result for the same input, or I would need to burn CPU on maintaining a dictionary on the browser side. The original priv.at del.icio.us bookmarks didn’t have tagging, I can still look through dates and page quickly through my bookmarks, so I have left that feature out as being too CPU intensive for a first pass at creating a quick bookmarking application.

Of course I have also published the project source as GPL v2 for those interested.

One year ago, I shipped the first version of a project for an open source community resource we initially had codenamed project stratocaster.

It’s been a fun road, and we’ve learned a ton about what it means to run a public community driven site, as well as slowly refining through practical use a very cool wiki platform.

What I’m most happy about is that SWiK is over time becoming a more useful resource, and each month we see more people visiting and contributing than we did the previous month.

Here is a picture of our monthly visitor growth so far, from the beginning of the rewritten swik v2.0:

The History of SWiK

Project Stratocaster was long in planning, and went through various rejected ideas on how we might make a site that was compelling and useful in the web market that is already flooded with interesting and useful sites. Eventually I worked out 3 basic ideas on what we might build SWiK around: a comprehensive compendium of what matters in open source software: a timely resource you could use for up to the minute information about these topics, and a personal resource that anyone make their own, something that wasn’t ours alone but that we let anybody who was interested control.

Over a few weeks I quickly knocked out a prototype and gave a demo of how we could provide this kind of service using RSS and webservices to help SWiK integrate with existing services, tags to help categorize and cross reference, a simple web interface to create and find information. Based on the demo, we decided to go ahead and build out and ship Project Stratocaster.

I had never developed this kind of thing or any real shipping product before, but we had just shipped and were promoting a custom PHP stack, so building SWiK on this stack seemed a natural choice and PHP is a great language for rapid prototyping and development.

After an initial demo, we decided to turn my prototype for SWiK into a shipping product, using the protype code as a base. Work continued on the prototype, but designed as a proof of concept, the prototype had large holes that needed fixing before we could think of shipping it. Most importantly, skipping input validation steps I had left huge security holes all over the code, being unfamiliar with MySQL optimization my pages were not performant to our guesstimates of initial traffic numbers, and generally the product was unpolished. To help get things ready we brought in Marc Wandshneider, who had recently written an excellent book on application development in PHP.

To be frank, the prototype née swik v1.0 was an unholy mess when Marc came on board, and it’s amazing that we were able to get things up and running enough to ship in as short a time as we did. At this point I had no training or practical experience in writing maintainable code, and compounded by the use of PHP 4 instead of 5, I had created a 3000 line 1 file monster without many newlines and very few comments.

Marc however is very experienced in writing good maintainable code, and the 3000 line monster was split and refactored and version controlled and editted into something shippable, which we did ship in June of 2005.

Designing a community site from scratch is quite a difficult task, because you are operating against a large set of assumptions about how the site will actually function when you ship it. A community site specifically, you have to think each step of the way how it can be gamed and attacked by spammers and vandals, who plague all social sites from the first second.

The defense we used against spammers in version 1.0 was two parts: the changelog and the obfuscation defense.

As you can see in sites such as Wikipedia, it’s possible to have a large body of information that is editable by anyone at all, so long as you channel the body of edits behind the pages into a manageable set of recent edits. Users can then keep an eye on this management set of data and act as a filter on new information coming into the site, making sure that although there are thousands of edits, they have all passed through a filter that ensures that they are not spam or vandalism. This anti-vandalism and anti-spam approach led to the ubiqitous undo buttons and the very early development and planning around what is possibly our most complex feature: the changelog.

Unfortunately, there is a simple way to defeat a wiki recent edits filter. Simply bombard it with more changes than can be reasonably monitored, and the filter will break under the strain. The easiest way to do this is to automate vandalism and spam, which is done to great effect by botnets all over the internet. My blog for example now receives a spam an hour, which makes it hard for me to publish any comments over the general cacophany of unwanted crud.

Automated spam engines work by scanning for simple keywords in forms, such as ‘homepage’ or ‘name’ – they then fill these forms with junk and post them, thousands of times over. Thus to prevent these automated bots from catching SWiK’s forms and filling them with spam, all the form names were obfuscated: homepage became H0M3P4G3. This was actually quite effective one year ago, and will still today reduce spam received, as I test with various honeypots.

With these defenses in place, we shipped the project with our new name SWiK.net.

SWiK Version 1

Initially the plan was to test our assumptions about how the site would operate by not announcing SWiK to anyone. The site would only grow hollistically, without any press or announcement of any kind. Users and use trickled in, but I was able to figure out some important lessons about how people use web applications.

First, abuses of the site were way fewer than I had imagined. Social software abuse is a problem of scale, when small numbers of people can effect large numbers of people. When small numbers of people can only effect small numbers of people, there is no motivation for abuse, and thus it doesn’t happen.

Second, wiki markup is an absolute necessity. I had spurned wiki languages in the first version’s design, due to my disatisfaction with most formats, the complexity of developing a new one, and the inability of any wiki format to match the flexibility of straight html. This was a mistake, because editing large amounts of text with no formatting help is almost impossible for the normal editor, annoying for the experienced editor, time consuming for any editor, difficult to do collaboratively, and hard to audit in a wiki recent change log.

Third, folders and hierarchy are very bad things. In the initial design I copied a file system layout, with pages in folders and sub folders. This allowed for deep collections of pages that flowed naturally with a URL structure, but it proved to be a nightmare on the database, a nightmare in the changelog, a nightmare to find what you were looking for, a nightmare for UI design, and a nightmare for general use over the internet due to latency or general web surfing behavior in site navigation.

The final mistake was not having Ajax powered editing. The initial prototype way back when had used JavaScript driven forms to make editing quick and easy, but the complexity both in the code and UI design had made me decide Ajax was inappropriate for use in anything critical or complicated. I published my notes about Ajax development issues while working on SWiK v1.0 in May of 2005. But without Javascript help, web forms are too slow and cumbersome to be used casually, which doesn’t fit with the experience I wanted for SWiK.

Ajax was also needed for another feature of SWiK, which is automatic project creation. In initial prototypes of SWiK, merely searching for an open source project that was not in SWiK’s database would launch a process that would behind the scenes go out, research the project, create it, and then deliver the searcher to the created project page as if it existed all along. This was toned down to require a further confirmation click, but problems with this were apparent as soon as you moved out of the demo space, as people search for the most random of things, cluttering the database with useless garbage and taxing the webservices we used to find the project information. These webservices were also prone to failure, leading to problems servicing the new project requests. It was apparent from this that we needed Ajax to asynchronously provide the information when and if the webservices returned it, and to never block on or require a 3rd party service to function.

Despite these mistakes, SWiK was an immediate success when we first launched publicly a month later attracting traffic beyond our expectations, in fact the attention brought our small initial server to its knees with the number of new projects being entered and information added.

SWiK version 2.0

This success prompted work on an ambitious redesign and complete rewrite of SWiK, version 2.0. Version 2 would involve a longer development cycle and we brought in Jerry to complete the project.

Version 2.0 design was started before v1 was even launched, based on the problems listed above, there were serious design changes necessary to move forward. Complicating things was an increasing spam and vandalism problem as we grew, necessitating the development of countermeasures beyond simple form name obfuscation. Additionally to grow we would need to redesign the pages to scale to much larger targets. To put things into perspective, the rush of traffic we received in the first few months of SWiK are today equalled by a few days of current swik traffic.

Most complicated of all, as we were moving forward, we couldn’t just discard the edits already contributed by visitors, so any redesign would have to take into account migrating from an existing data set.

Additionally, maintenance on the SWiK 1.0 site still in operation would have to continue, even though any code for SWiK 1.0 would have a very short lifetime.

The design for SWiK 2.0 involved a lot of radical changes, focusing on keeping everything that worked in the first version: RSS integration, free tagging, webservice integration, anonymous editing, etc. The hierarchy would go, and be replaced with tags. Every single form would be replaced with an Ajax powered form. The URL schema would change to be simpler. Textile wiki markup would be introduced. Tags and Projects would collapse into a single concept: tags. 5 basic templates for pages would be introduced. A comment system would be attached to all pages. A login system would allow users to take credit for their edits after they had made them anonymously. The RSS backend would need to support thousands of feeds instead of tens or hundreds. Tags would use a more complicated but more scalable heuristic to determine tag popularity. Elements of the interface would be optionally Ajax driven for powerful navigation within pages. The changelog would be radically altered to introduce dynamic change groupings to allow for a more manageable stream of edits. Users would now get their own wiki pages. Ajax would now power the auto project creation. History would now include more details and history display would change dramatically. Wiki links would indicate creation status of linked pages. Many other changes were also implied.

The bulk and scope of these changes and the messiness of the prototype derived version 1 led us to the conclusion that development should start from scratch, in PHP5 to allow for an object oriented maintainable project that could grow to meet future demands. We would redesign the database as well, and use InnoDB, transactions, and foreign key restraints to avoid data corruption that had caused a few problems in the first version.

Furious development ensued, with key difficulties involving the migration of old data that had a fundamentally different schema and history model, the development of the improved tag heuristic algorithms, and a redesigned changelog page.

A week after our end of September goal we took SWiK 1.0 down, migrated all of the old data, and released SWiK v2.0 to the world. It was sorely needed, by the end of SWiK 2.0 development we had virtually ceased maintenance of the SWiK 1.0 codebase that was about to be discarded, and bugs and problems in the first version were making themselves increasingly apparent to visitors and editors.

We formally announced SWiK 2.0 at the beginning of October, and traffic just took off to the new version. The first month of release received five times as much traffic as the month before it, and after that traffic tripled or doubled every month until steadying to slower percentage growth in January of 2006, this also coincided with a development hiatus on the project, which saw Marc leave for travels abroad and our search for a new engineer to take over.

SWiK version 3.0

SWiK version 2.0 was a great release with solid legs, but over time a number of problems with the design of SWiK 2.0 made themselves apparent. During the development hiatus, I took time to work on a new specification, and by this time I had switched to doing all specification on our internal version of SWiK, which SourceLabs now uses for all specification and internal documents. After much searching we found Daryl, and development began anew on the next version of SWiK. (We’re still looking for another developer btw!)

One basic problem with SWiK version 2.0 is that there were so many changes and so much to reimplement we just couldn’t fit everything into the development cycle and a lot of good ideas had to be pushed out.

As traffic rose and time went on, spam and vandalism rose and became more sophisticated, again prompting a need for a more sophisticated defense.

Other problems included a page design that was cluttered with too many features, a frustrating lack of diff highlighting in the changelog, too-flexible wiki templating that didn’t emphasize the right way to create pages, too much Ajax used for navigation, hard to use tag clouds, hard to use tagging, and a myriad of other small complaints and problems, including basic reliability and performance of various pages and components.

To fix a lot of basic problems, I designed a new user interface for all the pages. I cut features all over the place, whatever didn’t work was scrapped, to some people’s chagrin even things that did work were cut back in order to emphasize what was important.

The new design is focused on empty space, this is designed to make getting the central information of a page obvious and easy, you can find it easily as there’s nothing else on the page. Page creation, which used to involve more buttons and knobs, has been trimmed to bare essentials. A large icon now gives an identity to a page, so that different pages about different subjects are easy to identify. Tags now feature prominently and are autosuggested for ease of use. SWiK v3.0 is a release of a thousand little changes and tweaks to streamline and improve the fundamental ideas behind SWiK v2.0.

There’s one important thing that we learned the hard way in SWiK v1 and v2: Don’t release big. Small incremental releases are so key, both to development sanity and for design intelligence. Traffic metrics will tell you with every release how well it worked and what you need to fix, but you can’t get any metrics if you don’t release.

It’s been 1 year of SWiK and we’ve come a long way, thanks to the community of editors and a lot of hard learnt development and design lessons. But it’s definitely rewarding, and I think we’ve built a great product and a good start to what in future major releases will be much much better.

And it’s certainly gratifying that for SWiK’s first birthday we just passed 1 million unique visitors served, and went over 10,000 wiki pages created. Thanks for your edits and contributions to making a useful free resource for people who are using open source software, and stay tuned for a lot more to come.

There’s a lot that goes in to making software products. A constant battle between shipping pressure and feature completeness that leaves behind a bloody trail of cut features and half-finished products. With every feature, you have to consider that it needs to be added to the test regimen, it may take more development time than projected, it might not actually be used by anyone, it adds clutter, it might impact other code, or that it might be only half thought out. Mostly it’s a good idea to simply cut everything only a small minority will use. Apple especially is ruthless in this regard.

But there’s another class of features, the little detail features that are easy to overlook but critical to making a product people don’t just use, but love. The bud-vase of the VW Bug. The gargoyles on the Chrysler building. The charging battery display on the Prius.

Here are some examples from software of features that whenever I use them, I think, I’m so glad that made the cut.

• Digg’s Digg this. The animation is a big part of why that’s cool, but also the fact that I can see right in front of me how I’m changing the site makes the interactive nature of Digg tangible. (One interesting paradox of Digg is that even though stories are driven to the front page by people ‘digging’ them, the comments are always filled with people saying the stories are complete crap. This may be pent up desire for a negative Digg button, a weakness of the comment moderation system, or maybe it’s just that when everyone feels like an editor, everyone wants their say in what fits and doesn’t fit.)
• Adium’s Flapping duck. When you get a new message, Adium’s duck in the dock starts flapping its wings to get your attention. It’s actually a bit subtle, but it’s a really cool use of the dock.
• Del.icio.us Tag addition. If I want a javascript library, I visit http://del.icio.us/tag/MVC+PHP. It’s actually a feature only 1% of the del.icio.us users use, but it makes the site for me. Even the fact that the URLS are clean and readable is something probably only a fraction of people take notice of, but I love it, and I put it in SWiK: http://swik.net/MVC+PHP.
• GMail’s Keyboard shortcuts. I’m pretty sure that only a small minority of GMail’s millions of users actually know the keyboard shortcuts in Google. Even if lots of people aren’t taking advantage of hitting ! on a message to send it to the spam folder, or ‘o’ to open an email, it’s a feature that makes me love GMail.
• Bloglines’s and Adium’s Friendly error message. Don’t get me wrong, I absolutely hate it when things break, but seeing a fat plumber or a duck going down in flames somehow tempers my ire. I gave a shot at replicating this on SWiK’s error page with soothing language that helps people understand there are people behind the code, but nothing really says we screwed up and we’re sorry like a duck going down in flames.
• Mac OS X’s hibernate feature. When you use Windows on a laptop, you get used to the fact that when you close the lid, your laptop is going to a special hell from which it may never escape. On OSX closing the lid is like turning off a light switch, by the time you’ve opened it, it’s ready to go.
• SWiK’s Register after the fact. I came up with this feature so that even if you’re not logged in, SWiK remembers changes you make to pages, and lets you then create a new account or login to take credit. Every time I forget to log in before making edits, I think, “damn I like that feature.”

Little features that fill in the details make all the difference in the world.

People aren't robots