I like open source as much as the next guy, probably more.

But being open source, under whatever common definition of the term one wants to use, is neither a necessary nor a sufficient condition for something being good, or even adequate. Evaluating a given piece of OSS for use, especially when you’re in a hurry, can be tricky, and sometimes there are signs and portents that suggest more promise than is really present.

Things that Cross my Mind When I Look at an OSS Project for the First Time:

• The presence of a vibrant and active community of contributors: It’s nice to see active discussion forums, wikis, bug databases, documentation updates and regular releases. If the activity isn’t all gnashing of teeth by plausibly competent people who are making credible and well supported arguments that things don’t work worth a damn, all the better.
• Something that Builds: A source distribution should be as self-contained as possible. If there are external dependencies you must obtain separately, they should be clearly called out in READMEs, the documentation, the source and pretty much anywhere a user might be looking when one turns out to be absent. A credible, working and usable Ant or Maven build, or autoconf/automake setup, or whatever should be a requirement.
• Crap that doesn’t build: No automated build? In this case why even consider it real software? It’s offered you no proof to the contrary. Are the build files or scripts full of hardcoded paths to ’/home/rjnutsack’, or anybody else’s home directory? Anathema. Did you get nothing more than a vague description of all the places in DevStudio, IntelliJ, Eclipse, Borland Power Snogger Mega Delphi Turbo Modula-2 Builder ‘99 or some other IDE that you need to click to have a prayer of seeing it build and run? The suxx0r! Any sign that the stuff hasn’t been built or run on any machine beyond that of the putz who apparently prematurely loosed it on the world falls in the no es bueno category.
• Something with Good Tests: Are you going to use this stuff? Maintain it? Change anything and not to have to pray to Great Father Dagon that you didn’t just break or regress something in some way subtle or gross? No tests mean no confidence. Whoever put this thing out either didn’t test it, or did so in a non-reproducible, half-assed way that can’t be easily replicated by you and probably even them. It’s possible they had an all-singing, all-dancing, crap-of-the-world regression suite so clean you could eat off of it, but if they really had such a thing, why wouldn’t they be sharing it? Want to come over tonight and see the supersonic, lawn-clipping fueled rocket car I’ve been testing in the state of the art wind tunnel in my garage? Oh, sorry, I won’t be home. Maybe we can reschedule for some unspecified time in the future.
• No tests of any kind, or obviously inadequate tests: Even if it builds, why should that alone make you consider it to be software of decent quality? It’s offered you no proof of that. So now you have no evidence that it works worth for poop, and no easy way to determine that it may have stopped working after you made your first trivial seeming whitespace change in a header file.
• Bad Internal or External Documentation: If nobody bothered writing a single word about the stuff outside of the source code, and if the only comments in the source aren’t discussing the actual ideas and concepts that hold the whole thing together and rationalize its structure, but instead look like the following, it may be time to node, smile politely, and back away from the thing.

i++;  // increment 'i' (by one!)

And there’s more…

• Puzzling genealogy: If the project looks like a mutant offshoot of a bastard child of some collection of other half-orphaned projects by quasi-alienated misanthropes whose working relationship fell apart before the project became valuable enough for someone well adjusted to keep it moving forward, what do you think the odds are that it’s in good hands now?
• The Reputation of its Producers: Knowing the code comes from an organization that’s serious in its intent and scrupulous in its development practices goes a long way. On the flip side of this, some organizations tend to jump the shark with their projects living on long past the ability of the organization to do good work. This can happen through fatigue, attrition, too rapid of hiring or induction of new developers, bad luck, ego, vanity, sloth, pride the second system effect and a host of other factors. Things that call themselves “frameworks” tend to be particularly susceptible to this sort of failure mode since many framework builders have a tendency to restructure their entire intellectual universe around their framework, and assume that you’re not only willing but eager to do the same thing. You don’t want to use their stuff to make one of your problems go away. You want to use it so that you can worship in their gilded, rathole cathedral of self-indulgent, technical pretentiousness. Duh, of course. Once that’s started to happen the project is standing at the platform waiting for the next train to Uselessville Junction.

Claims, benchmarks, or (even worse) “research papers” claiming to compare the performance of two things, usually making some definitive, strong and universally quantified statement in favor of one or the other despite:

• Making totally irrelevant apples to razor blades comparisons;
• Showing total statistical illiteracy in the analysis of results (two distributions with the same mean are obviously the same right? There are no other interesting moments to consider);
• Using one of the systems under comparison in such an inappropriate, unnatural, or bizarre manner that only a huckster or complete numbnuts would contrive to do so (quite deliberately in one case, quite mouth-breathingly in the other);
• More than one of the above.

Usually the inanity of these pieces of work is rivaled only by their smugness, something that would be hard to sustain if what one was saying was more obvious in its isomorphism to:

• “I mowed the lawn with my Porsche and it made an awful mess… clearly it’s a terrible machine relative to my Toro Lawninatrix 2000.”

or…

• “I tried brushing my teeth with a Husqvarna chainsaw and the results were clearly inferior to those obtained with my Sonicare.”

or…

• “Wow this supposed to be real-time, 3D, ray traced first person shooter I wrote in Prolog really sucks—-what a stupid language.”

One of these crimes against reason can be remedied by taking to heart some of the advice here. The cures for some of the others are less obvious, but it’s possible that chainsaw would come in handy.