sorted by: recent | see : popular

Content Tagged with User:alex + del.icio.us

Even though I’ve moved off of del.icio.us and onto my encrypted web bookmarks manager boz, I still have hundreds of bookmarks saved on del.icio.us, and I use the site for other things like a handful of projects that use the site’s api to do some interesting stuff.

I’m adding a new one to the list today, and updating an old one to start Delicious Tools.

• A dead link checker – lots of your urls are out of date or broken: check that your bookmarks are still valid.
• Google sync – synchronize del.icio.us with your Google search results.

(thx osde-info for the screenshot and braving Flickr’s policies against them)

• LiveMarks – streams social bookmarks from various services and does some frequency analysis as well to highlight new and interesting stuff.
• Delimages – hotlinks images from del.icio.us image streams.

Notes

Fixing my database

Over my Thanksgiving vacation I stopped checking on my projects for a week or so and that’s when everything decided to go to hell. The MySQL database dedicated to the del.icio.us projects ran out of space, and when I tried to fix it I wiped it :/ I’m pretty lax with these projects and I don’t do sensible things like use a VCS or make comprehensive backups, so it took a bit of work just to get things back to normal. There was also some code rewriting required, and I revisited a lot of queries in various services to bring them back online, such as in LiveMarks which now has a different popularity query and for delimages which now has a simpler tag query. Things should be back to normal though, so if you use any of these services, send me mail if you don’t like the changes or something is broken.

Making the dead link checker

The dead link checker turned out to be a little bit annoying, given that a lot of servers don’t respond properly to HTTP HEAD requests.

The checker works by making a bunch of prototype-driven Ajax calls to a little JSON outputting PHP script that makes a socket connection to the server to do a HEAD request for the URL.

• Server just hangs on Connection: close
• Server decides to put the entire page HTML in the HEAD response (i’m looking at you IIS 6.0)
• Server always returns a wrong response code for the page (Wikipedia and Google Video)

I didn’t do that much about special-casing completely wrong codes, so you’ll have to verify some yourself.

Updates to Kibbutz

I already had a Google sync project called Kibbutz which drives the XML translation of del.icio.us bookmarks to Google Co-op search results.

One thing you might have noticed however if you used it – it didn’t index all of your posts. Del.icio.us has a purposely limited API that requires login credentials to access your full list of public bookmarks, and I didn’t feel like writing an importer. Well I finally wrote one, and now you should be able to get all of your bookmarks indexed.

Now that I’ve written the importer script, it should be straightforward to write some more tools for working with your bookmarks – these will go under delicious-tools, watch that space.

It’s easy to get on del.icio.us popular and drive a bunch of traffic to your stuff: just set up various fake accounts and all bookmark the same thing at once.

Check this recent bookmark out for example: http://del.icio.us/url/b80e64803e8e83449aa519a6ef13080a

If you don’t notice why it’s suspicious, look at the first ten+ people who have bookmarked it, they have just joined in the past 2 days and have all bookmarked similar things.

I’ve started filtering some del.icio.us spam from LiveMarks, but this kind of thing is hard to stop automatically.

del.icio.us appears to be doing fine to me :/

I don’t know what Michael Arrington is smoking when he says quote they’ve tanked completely

By my internal counters, del.icio.us has well over 300k users who actively post bookmarks. This is different from Digg or really any of the social sites out there besides the picture and myspace sites – these are people who are actively contributing to the del.icio.us resource with quality stuff.

Digg is pretty ephemeral by comparison, with Diggs occupying a perhaps similar volume of traffic thanks to the easy button, but the tagging and focus on new content means that Digg traffic lives and dies quickly, which may account for why Digg has eclipsed del.icio.us in traffic.

However look at the contributor numbers, people have recently claimed that Digg only has a thousand or far fewer active bookmarkers, and my internal numbers from my feeds back that up. LiveMarks has scrolled 60k digg bookmarks, and 4 million unique del.icio.us links over the past year or so.

What really seems to have improved at del.icio.us since Yahoo bought them is something that Michael Arrington likely would have a hard time selling as copy: their performance has improved drastically from a near cataclysm of broken pages, timeouts and database failures that were striking del.icio.us shortly before they were sold.

The feature improvements have indeed been scarse and the design is still unfortunate, but it’s still the best public bookmarking service out there by far (of course the new boz is the first and best and only truly private bookmarking service :P)

Sometimes features and visible improvements take a long time to brew, such as before Flickr released their Gamma update. There is a ton of work that needs to be done on del.icio.us, but I wouldn’t count Yahoo as having killed the goose yet.

PS: The comscore high Arrington refers to is likely bumped artificially high by press regarding the Yahoo aquisition, not droves of users suddenly signing up and using the service.

or “Kibbitzing the Google Kibbutz”

Google search recently launched a product called “Google Coop”.

I’m one for the odd mashup, so this project sounded fairly interesting – you get to make little vertical searches on top of Google to which other people can subscribe, kind of a mix between RSS and Adwords.

However the documentation for this project was clearly written by people who hate me and want to make my life miserable. The Phds up in the GooglePlex clearly have no need to explain their genius to common folk like me, instead opting for really helpful headings like:

“For Contributors”, “Topics”, “Subscribed Links”
Definition of the project: “Google Co-op is a platform which enables you to use your expertise to help other users find information. This is a work in progress; over time, you can expect to see evolution in both Co-op’s structure and the means by which you can contribute your expertise to our goal of making information more discoverable for millions of people. ” (ok but what does it actually do?)

I scratched my head for a bit thinking what these things might actually mean, luckily the excellent Google Blogoscoped quickly came out with a primer, that explained what the heck google coop actually did.

This weekend, looking for another Sunday project I revisited Google Co-Op – and built a little integration between personal del.icio.us bookmarks and Google. How it works is that you visit Google Co-Op and subscribe to my ‘subscribed links’ feed.

Here is a page that explains how to set your del.icio.us bookmarks up with Google Co-Op

If you just want to integrate your del.icio.us bookmarks with Google, you can stop reading here, I won’t hassle you with Google’s screwy terminology or how the annoyances I’ve found working with Google Co-Op.

And there are a number of these annoyances, even going beyond the documentation’s sore need of a basic glossary.

Basically, what I used is the ‘subscribed links’ functionality offered by google coop. This allows you to set an XML file with preset matches to search results. If someone searches for “What is the air speed velocity of an African Swallow”, you can specify they should get a link up at top of their search to the gorgeofnothingness.com

The fun part in setting these up is that Google won’t let you have a sandbox to play around creating these subscribed links. In order to test anything, you have to upload your xml file to them and add it to your subscribed links. You then have to wait 5 minutes until they go and get your file. Then you have to study messages like “xml entity not recognized” (we don’t know how to deal with unicode). Then delete the subscription you added, re-add/re-upload and wait 5 minutes to test again. It’s a lot of fun. Did I mention sometimes if you delete a subscription and re-add it, Google retains its cache of the old subscription and won’t reload the new one?

Other than the hassle of getting started, the issues are fairly minor, mainly my opinion of their xml schema. I don’t like data in XML attributes, it’s a lot cleaner when it’s in nodes. And if you set up counting notations, stay consistant, either go 0,1,2,3 or 1,2,3—don’t mix both please.

Anyways, this was a quick project that has kept me up way too late, but if you want to see the simple source to the project you can visit the SWiK – Kibbutz wiki page.

If you want to see improvements to this project, shoot me a message, I will be looking at adding more or betterer information to the integration and I’d like feedback on if this is actually useful or what aspects of del.icio.us I could integrate with Google further.

Also if Joshua Schachter would expose an RSS feed for del.icio.us search as there is on most other things, or if there were more XML access to a user’s bookmarks, I could probably do a better job integrating the two services.

Social bookmarking is currently in a high growth pattern. I notice it all the time running LiveMarks, the rate of social bookmarking is increasing every single day.

The more people flock to something, the bigger target it is for abuse. The more people turn to del.icio.us to find useful bookmarks, or digg.com to read the latest news, the more tempting targets those services become for spammers and vandals.

As Clay Shirky has noted, “Social software is stuff that gets spammed.” Perhaps as an addendum to that aphorism it should be noted that successful social services are those that can resist spam successfully.

The spamming foes of social software are just starting to take shape. Witness a new service called ‘TagExplosion’, TagExplosion describes itself:

“[TagExplosion] helps get your program, blog, advertisement or website on the several “Social Bookmarking” sites so you can get on 100,000’s of computers, worldwide. This will help drive more traffic to your website by utilizing the “word-of-mouth” aspect of Social Bookmarking.

Services like these are bound to multiply and exploit every weakness in social bookmarking’s defenses against spam and general social filters such as Digg’s front-page promotion mechanism or del.icio.us’s popularity or aggregation filters.

At the moment, the defenses of the social bookmarking sites are fairly weak. Currently del.icio.us has certain filters that destroy obvious spammers, prevent excessive tagging from aggregating onto every tag page, and prune the front page’s new bookmarks list. But del.icio.us’ only solid defense against automated attack is their signup captcha. As we’ve seen with BlogSpot, a signup captcha is only worth what it costs a spammer to pay an Indian or Filipino company to fill in.

del.icio.us may soon have to contend with spammers that post automatically or through peer to peer schemes such as TagExplosion or hide their spam like steganographers, continuously studying filters to evade and manipulate them.

Digg has similar problems, and is currently using similar techniques with similar vulnerabilities. As Digg is a more socially oriented site, it also uses more socially oriented solutions to fight off the spam problem. Users are encouraged not to post duplicate stories by having to first wade through other users’ submissions, and once posted other users quickly respond and post complaintative comments against problematic links. Users of Digg also help fend off spam by reporting what is wrong to the administrators and to the other users watching the live queue.

As frequent Digg users are well aware, Digg has had a tough battle in the recent times fighting vandalism. So far however, only a tiny minority vandalize the site, spammers are still thankfully somewhat in the dark as to how they might abuse the system.

This is all very reminiscent of the rise of search engines, Compare a search on Altavista for viagra to the same search on Google. Search engines that failed, failed in large part because spammers eventually reverse engineered the algorithms they used to determine relevance and simply rearranged their pages to suit those algorithms. The top result for viagra: ‘t-e-x-a-s-poker.com’ on altavista is the end result of the eventual failure of the first generation algorithms.

It’s likely that to survive and remain useful, social services will have to follow in this mold and emulate Google by switching their filters to factors that are more expensive and more complicated to engineer, such as third party references, longevity, references from known trusted sources, and patterns of human created content. Or they may be forced to become less social, such as with Wikipedia’s recent policies. Even then, it’s certain to be an ongoing battle.

I started another project this weekend, called “Delimages”.

del.icio.us exposes a special tag called system:filetype:jpg, but it’s pretty boring to subscribe to these feeds, you can’t see any of the images, just the titles. I wanted a way to be able to see the feed as images, so I coded up Delimages which can show del.icio.us tag or user image feeds as images.

Also, I found that image posting on del.icio.us can be fairly repetitive, so I cleaned up the repetitions.

Check it out. (Be forewarned, this is the raw unmoderated image stream with all that implies.)

Kibbutz is a simple project to integrate Google and del.icio.us through Google co-op.

To have your del.icio.us bookmarks appear next to your Google search results, follow the instructions outlined here.

Kibbutz may be expanded in the future, subscribe to the rss feed for this page to be updated on future capabilities.

The code for Kibbutz is at the moment quite simple, it leverages the livemarks and magpierss projects to handle del.icio.us bookmarks.

External Links

• source

priv.at is a project to allow users to save bookmarks anonymously on del.icio.us – the social bookmarking service.

Notice – If you have bookmarks saved with priv.at, please visit Boz – this is the successor to this project and your missing bookmarks can be imported here.

priv.at is now deprecated, although the code should still work if you wish to use your own private.bookmarks account.

Users can create a unique bookmarklet for themselves, which they then trigger to post the bookmark to their account.

This hack takes advantage of del.icio.us’ for: tag feature, which allows users to bookmark urls for their friends invisibly.