One year ago, I shipped the first version of a project for an open source community resource we initially had codenamed project stratocaster.

It’s been a fun road, and we’ve learned a ton about what it means to run a public community driven site, as well as slowly refining through practical use a very cool wiki platform.

What I’m most happy about is that SWiK is over time becoming a more useful resource, and each month we see more people visiting and contributing than we did the previous month.

Here is a picture of our monthly visitor growth so far, from the beginning of the rewritten swik v2.0:

The History of SWiK

Project Stratocaster was long in planning, and went through various rejected ideas on how we might make a site that was compelling and useful in the web market that is already flooded with interesting and useful sites. Eventually I worked out 3 basic ideas on what we might build SWiK around: a comprehensive compendium of what matters in open source software: a timely resource you could use for up to the minute information about these topics, and a personal resource that anyone make their own, something that wasn’t ours alone but that we let anybody who was interested control.

Over a few weeks I quickly knocked out a prototype and gave a demo of how we could provide this kind of service using RSS and webservices to help SWiK integrate with existing services, tags to help categorize and cross reference, a simple web interface to create and find information. Based on the demo, we decided to go ahead and build out and ship Project Stratocaster.

I had never developed this kind of thing or any real shipping product before, but we had just shipped and were promoting a custom PHP stack, so building SWiK on this stack seemed a natural choice and PHP is a great language for rapid prototyping and development.

After an initial demo, we decided to turn my prototype for SWiK into a shipping product, using the protype code as a base. Work continued on the prototype, but designed as a proof of concept, the prototype had large holes that needed fixing before we could think of shipping it. Most importantly, skipping input validation steps I had left huge security holes all over the code, being unfamiliar with MySQL optimization my pages were not performant to our guesstimates of initial traffic numbers, and generally the product was unpolished. To help get things ready we brought in Marc Wandshneider, who had recently written an excellent book on application development in PHP.

To be frank, the prototype née swik v1.0 was an unholy mess when Marc came on board, and it’s amazing that we were able to get things up and running enough to ship in as short a time as we did. At this point I had no training or practical experience in writing maintainable code, and compounded by the use of PHP 4 instead of 5, I had created a 3000 line 1 file monster without many newlines and very few comments.

Marc however is very experienced in writing good maintainable code, and the 3000 line monster was split and refactored and version controlled and editted into something shippable, which we did ship in June of 2005.

Designing a community site from scratch is quite a difficult task, because you are operating against a large set of assumptions about how the site will actually function when you ship it. A community site specifically, you have to think each step of the way how it can be gamed and attacked by spammers and vandals, who plague all social sites from the first second.

The defense we used against spammers in version 1.0 was two parts: the changelog and the obfuscation defense.

As you can see in sites such as Wikipedia, it’s possible to have a large body of information that is editable by anyone at all, so long as you channel the body of edits behind the pages into a manageable set of recent edits. Users can then keep an eye on this management set of data and act as a filter on new information coming into the site, making sure that although there are thousands of edits, they have all passed through a filter that ensures that they are not spam or vandalism. This anti-vandalism and anti-spam approach led to the ubiqitous undo buttons and the very early development and planning around what is possibly our most complex feature: the changelog.

Unfortunately, there is a simple way to defeat a wiki recent edits filter. Simply bombard it with more changes than can be reasonably monitored, and the filter will break under the strain. The easiest way to do this is to automate vandalism and spam, which is done to great effect by botnets all over the internet. My blog for example now receives a spam an hour, which makes it hard for me to publish any comments over the general cacophany of unwanted crud.

Automated spam engines work by scanning for simple keywords in forms, such as ‘homepage’ or ‘name’ – they then fill these forms with junk and post them, thousands of times over. Thus to prevent these automated bots from catching SWiK’s forms and filling them with spam, all the form names were obfuscated: homepage became H0M3P4G3. This was actually quite effective one year ago, and will still today reduce spam received, as I test with various honeypots.

With these defenses in place, we shipped the project with our new name SWiK.net.

SWiK Version 1

Initially the plan was to test our assumptions about how the site would operate by not announcing SWiK to anyone. The site would only grow hollistically, without any press or announcement of any kind. Users and use trickled in, but I was able to figure out some important lessons about how people use web applications.

First, abuses of the site were way fewer than I had imagined. Social software abuse is a problem of scale, when small numbers of people can effect large numbers of people. When small numbers of people can only effect small numbers of people, there is no motivation for abuse, and thus it doesn’t happen.

Second, wiki markup is an absolute necessity. I had spurned wiki languages in the first version’s design, due to my disatisfaction with most formats, the complexity of developing a new one, and the inability of any wiki format to match the flexibility of straight html. This was a mistake, because editing large amounts of text with no formatting help is almost impossible for the normal editor, annoying for the experienced editor, time consuming for any editor, difficult to do collaboratively, and hard to audit in a wiki recent change log.

Third, folders and hierarchy are very bad things. In the initial design I copied a file system layout, with pages in folders and sub folders. This allowed for deep collections of pages that flowed naturally with a URL structure, but it proved to be a nightmare on the database, a nightmare in the changelog, a nightmare to find what you were looking for, a nightmare for UI design, and a nightmare for general use over the internet due to latency or general web surfing behavior in site navigation.

The final mistake was not having Ajax powered editing. The initial prototype way back when had used JavaScript driven forms to make editing quick and easy, but the complexity both in the code and UI design had made me decide Ajax was inappropriate for use in anything critical or complicated. I published my notes about Ajax development issues while working on SWiK v1.0 in May of 2005. But without Javascript help, web forms are too slow and cumbersome to be used casually, which doesn’t fit with the experience I wanted for SWiK.

Ajax was also needed for another feature of SWiK, which is automatic project creation. In initial prototypes of SWiK, merely searching for an open source project that was not in SWiK’s database would launch a process that would behind the scenes go out, research the project, create it, and then deliver the searcher to the created project page as if it existed all along. This was toned down to require a further confirmation click, but problems with this were apparent as soon as you moved out of the demo space, as people search for the most random of things, cluttering the database with useless garbage and taxing the webservices we used to find the project information. These webservices were also prone to failure, leading to problems servicing the new project requests. It was apparent from this that we needed Ajax to asynchronously provide the information when and if the webservices returned it, and to never block on or require a 3rd party service to function.

Despite these mistakes, SWiK was an immediate success when we first launched publicly a month later attracting traffic beyond our expectations, in fact the attention brought our small initial server to its knees with the number of new projects being entered and information added.

SWiK version 2.0

This success prompted work on an ambitious redesign and complete rewrite of SWiK, version 2.0. Version 2 would involve a longer development cycle and we brought in Jerry to complete the project.

Version 2.0 design was started before v1 was even launched, based on the problems listed above, there were serious design changes necessary to move forward. Complicating things was an increasing spam and vandalism problem as we grew, necessitating the development of countermeasures beyond simple form name obfuscation. Additionally to grow we would need to redesign the pages to scale to much larger targets. To put things into perspective, the rush of traffic we received in the first few months of SWiK are today equalled by a few days of current swik traffic.

Most complicated of all, as we were moving forward, we couldn’t just discard the edits already contributed by visitors, so any redesign would have to take into account migrating from an existing data set.

Additionally, maintenance on the SWiK 1.0 site still in operation would have to continue, even though any code for SWiK 1.0 would have a very short lifetime.

The design for SWiK 2.0 involved a lot of radical changes, focusing on keeping everything that worked in the first version: RSS integration, free tagging, webservice integration, anonymous editing, etc. The hierarchy would go, and be replaced with tags. Every single form would be replaced with an Ajax powered form. The URL schema would change to be simpler. Textile wiki markup would be introduced. Tags and Projects would collapse into a single concept: tags. 5 basic templates for pages would be introduced. A comment system would be attached to all pages. A login system would allow users to take credit for their edits after they had made them anonymously. The RSS backend would need to support thousands of feeds instead of tens or hundreds. Tags would use a more complicated but more scalable heuristic to determine tag popularity. Elements of the interface would be optionally Ajax driven for powerful navigation within pages. The changelog would be radically altered to introduce dynamic change groupings to allow for a more manageable stream of edits. Users would now get their own wiki pages. Ajax would now power the auto project creation. History would now include more details and history display would change dramatically. Wiki links would indicate creation status of linked pages. Many other changes were also implied.

The bulk and scope of these changes and the messiness of the prototype derived version 1 led us to the conclusion that development should start from scratch, in PHP5 to allow for an object oriented maintainable project that could grow to meet future demands. We would redesign the database as well, and use InnoDB, transactions, and foreign key restraints to avoid data corruption that had caused a few problems in the first version.

Furious development ensued, with key difficulties involving the migration of old data that had a fundamentally different schema and history model, the development of the improved tag heuristic algorithms, and a redesigned changelog page.

A week after our end of September goal we took SWiK 1.0 down, migrated all of the old data, and released SWiK v2.0 to the world. It was sorely needed, by the end of SWiK 2.0 development we had virtually ceased maintenance of the SWiK 1.0 codebase that was about to be discarded, and bugs and problems in the first version were making themselves increasingly apparent to visitors and editors.

We formally announced SWiK 2.0 at the beginning of October, and traffic just took off to the new version. The first month of release received five times as much traffic as the month before it, and after that traffic tripled or doubled every month until steadying to slower percentage growth in January of 2006, this also coincided with a development hiatus on the project, which saw Marc leave for travels abroad and our search for a new engineer to take over.

SWiK version 3.0

SWiK version 2.0 was a great release with solid legs, but over time a number of problems with the design of SWiK 2.0 made themselves apparent. During the development hiatus, I took time to work on a new specification, and by this time I had switched to doing all specification on our internal version of SWiK, which SourceLabs now uses for all specification and internal documents. After much searching we found Daryl, and development began anew on the next version of SWiK. (We’re still looking for another developer btw!)

One basic problem with SWiK version 2.0 is that there were so many changes and so much to reimplement we just couldn’t fit everything into the development cycle and a lot of good ideas had to be pushed out.

As traffic rose and time went on, spam and vandalism rose and became more sophisticated, again prompting a need for a more sophisticated defense.

Other problems included a page design that was cluttered with too many features, a frustrating lack of diff highlighting in the changelog, too-flexible wiki templating that didn’t emphasize the right way to create pages, too much Ajax used for navigation, hard to use tag clouds, hard to use tagging, and a myriad of other small complaints and problems, including basic reliability and performance of various pages and components.

To fix a lot of basic problems, I designed a new user interface for all the pages. I cut features all over the place, whatever didn’t work was scrapped, to some people’s chagrin even things that did work were cut back in order to emphasize what was important.

The new design is focused on empty space, this is designed to make getting the central information of a page obvious and easy, you can find it easily as there’s nothing else on the page. Page creation, which used to involve more buttons and knobs, has been trimmed to bare essentials. A large icon now gives an identity to a page, so that different pages about different subjects are easy to identify. Tags now feature prominently and are autosuggested for ease of use. SWiK v3.0 is a release of a thousand little changes and tweaks to streamline and improve the fundamental ideas behind SWiK v2.0.

There’s one important thing that we learned the hard way in SWiK v1 and v2: Don’t release big. Small incremental releases are so key, both to development sanity and for design intelligence. Traffic metrics will tell you with every release how well it worked and what you need to fix, but you can’t get any metrics if you don’t release.

It’s been 1 year of SWiK and we’ve come a long way, thanks to the community of editors and a lot of hard learnt development and design lessons. But it’s definitely rewarding, and I think we’ve built a great product and a good start to what in future major releases will be much much better.

And it’s certainly gratifying that for SWiK’s first birthday we just passed 1 million unique visitors served, and went over 10,000 wiki pages created. Thanks for your edits and contributions to making a useful free resource for people who are using open source software, and stay tuned for a lot more to come.

recmmendations engine

Swik is an opensource wiki site listing allsorts of fantastic projects, searchable by tags. Very handy indeed - I’ve found quite a few links to ajax and cms sites.

Swik es un buscador donde podremos buscar proyectos de c�digo abierto, el cual podemos comentarlo, seguirlo mediante RSS e incluso editarlo.

SWiK �r en wiki f�r Open Source projekt.

SWiK - "A Wiki for every open source project." Neat wiki implementation.

To blog or wiki...OSS development, born in the sea of distributed collaboration, in October gained a new primordial stew courtesy of SourceLabs. SWiK overlays the full Web2.0 foundry of blogs, wikis, tags, and feeds on the venerable Sourceforge repository

people's feeds I watch

swik and livemarks blogpost

swik blogpost

swik/livemarks blogpost

Welcome!

I’m, Jerry Kuch, one of the developers (along with Marc and Alex) of SWiK. I’ve worked on many things including operating systems, multimedia, security and crypto software, niche embedded stuff, broadcast video, a couple of web applications, mathematical programming environments (computer algebra and statistics mainly), virtual machines, network appliances and likely some other things that aren’t coming to mind just now.

Recently I’ve been working in and around:

• The Java virtual machine
• Functional programming languages, particularly targeting the JVM
• Statistical computing, R and SPLUS
• Haskell, type systems, monads, concurrency
• A smidgeon of Ruby, Rails and scripting on the side.

I’ve recently been working on a couple of projects at once, but am now reaching the point that I can take a bit of a breather and look at something else. But I’m nearly always open to yakking about anything interesting, with somebody interesting.

I’m usually easy to reach at jerrykuch AT gmail.com.