We recently had an error in the Sourceforge Public Areas data dumps for October.

New files available for you to download. They are marked "2".

The October data are available at the following locations:

Google Code Downloads Page

Notes:

• SF Stats are unavailable this month because the server was not reliably reporting stats when we did our collection.
  
• There were a few errors in the file uploads. These files can't be taken off of Google Code, so I've just marked the files "DO NOT DOWNLOAD". You can usually tell these files by their very tiny filesize.
  
• SF Developers and SF Developer_projects did not make the datamart build. I'll release these separately -- UPDATE: these are on our download site now

Hello moles,

I'm excited to give you all a heads up that the entire flossmole database is now available directly via a MySQL server.

We have transferred the database to the NSF TeraGrid Data Central hosting site [1] (based at the San Diego Supercomputing centre). It's a bigger machine and professionally administered, which was much better than we could offer ourselves. See below for access procedure.

The process of transferring the database also enabled us to prepare comprehensive datamarts for each datasource in the database. These are mysqldump files which can be used for local access to parts of the database; there are two for each datasource, one containing the raw html pages and one, substantially smaller, containing just the parsed data points. These will be available shortly and will be an option for those who want to install a local copy of the DB; although we'd be very interested in reasons people find to do that, we'd like to have people sharing useful transformations of the data and the Data Central database should be pretty quick.

So now we have three great options for accessing the FLOSSmole data:

1. The traditional monthly flat files
2. Direct MySQL access to the full database @ DC.
3. Comprehensive datamarts for local access

Database access further info
------------------------------

In order to demonstrate usage to NSF and to monitor run-away queries (hey, I write them myself. Often :) interested users need to contact the FLOSSmole project to request a personal username and password, which should not be shared. Other than that simple request, we're not introducing any new AUPs or conditions.

Initially requesters should join and email their request to the ossmole-discuss list, with a preferred username. We can review using the list that way if the traffic spikes. Turnaround should be no longer than a business day or two (we email the db admin at Data Central with the request).

OTOH when we, and hopefully you, publish workflows using the database, we would like them to work 'out of the box', without a potential user needing to request a user/pass. To enable this, in addition to the full database we are in the process of creating a small database, with very limited data in each table (~20 rows in each table, just demo data). This is to allow querying through a single, public, shared login which we urge people to use when publishing their workflows; once potential users wish to go beyond the sample data they should request their own user/pass and plug it into the workflow. We're still figuring out the best way to do this (finding 20 projects with total data coverage is actually quite hard :)

Hopefully this improves the accessibility of the datasets, and will likely result in finding more bugs; both from the migration [2] and within the dataset. We're asking people to file bugs and request for documentation in the Sourceforge Trackers; although discussing them on this list is always welcome as well.

So, have at it.

--J

[1]: http://datacentral.sdsc.edu/ . The Machine (thor) is on Internet2 and has >80G of RAM.

[2]: If anyone wants to chat about ways to confirm data integrity while migrating 300GB+ databases with some very large tables, ping me :) I think we got it sorted, with the much appreciated help of our Master's student, Vinay Venugopal.

I am at Oxford in the UK (staying at Hertford College Graduate Center) for a few days for the OSS Watch workshop on Profiling Open Source Communities.

OSS Watch is the National Advisory Service on open source for UK Further Education (FE) and Higher Education (HE). As such, it is part of our remit to help FE and HE institutions and projects who want to engage with open source development, and a key factor for that is the development of open source communities.

I'm giving The Standard FLOSSmole Talk today at 11:45GMT.

June Sourceforge data has been posted.

This time, I've released the files onto our Google Code project page. Let's see how we like using Google Code for this (psst, I can tell you as the person who does the file releases: it's a lot easier to use Google Code than Sourceforge for this particular part of the job!)

Enjoy!

UPDATE: Because of file size limits on Google Code, I've had to re-release our code onto Sourceforge.

Here is the link to the Flossmole SF Downloads Page, where you can get ALL the June files, including the huge datamart files.

Hi moles! April 2008 data has been released on our SF project page. Enjoy it! (Debian data included!)

UPDATE 08-JUL-2008: new files released to get past problem with data quality on these April files.

Hi Moles!

The January data files have been released, and I went back and released the December data marts too (sql statements to make your own database).

Have fun with it! Grab the data from our Sourceforge project page.

Files for November 2007 have been released here:

Sourceforge File Release Page