Sun advances GlassFish. Red Hat announces JBoss momentum. Mandriva releases Linux update. (and more)

Sun Microsystems Announces Technology Preview of Open Source Sun GlassFish Enterprise Server and New Sun GlassFish Communications Server, Sun Microsystems (Press Release)

Red Hat Continues Middleware Industry Momentum, Red Hat (Press Release)

Mandriva presents its latest distribution: Mandriva Linux 2008 Spring, Mandriva (Press Release)

Hyperic Announces MySQL Performance Study Results, Hyperic (Press Release)

Sun Microsystems Joins Liferay Open Source Community, Liferay (Press Release)

EnterpriseDB Releases Java Application Generator Plug-in for Postgres Studio, EnterpriseDB (Press Release)

SourceForge Implements OpenID Technology, SourceForge (Press Release)

Reinventing OpenSolaris, eWeek, Jason Brooks (Article)

Free Flash community reacts to Adobe Open Screen Project, Linux.com, Bruce Byfield (Article)

Google?s open source problem is Affero, ZDNet Linux and Open Source, Dana Blankenhorn (Blog)

The GPL Wins Again - Welte vs. Skype Technologies SA (Germany), Groklaw, Pamela Jones (Blog)

Josh Berkus on how to destroy a community, InfoWorld Open Sources, Zack Urlocker (Blog)

The Curse of Open Source License Proliferation, Socialized Software, Mark Hinkle (Blog)

Last year open source analyst Michael Coté of Redmonk coined the term Little Four to describe four up-and-coming open source management vendors and as a foil to the Big Four of systems management.

In the open source space, the 4 names that come up each time ? usually from people I?m talking with even before I say anything ? are: Zenoss, Hyperic, GroundWorks, and openQRM.

This week Qlusters/openQRM announced they would no longer be developing their open source project openQRM and leaving it to the community at large. I guess that leaves the remainder of the band of four to be labeled the “Little 3″. This isn’t all that surprising. The Qlusters team that originally launched openQRM is gone. Ofer Shoshan is no longer CEO, Qlusters CTO whurley went to BMC, Fred Gallagher went to open source database maker Ingres, and former Red Hat sales exec Don Langley has moved on. So I suspect that the mindset and commitment to further the project has departed with them.

The shame is that the openQRM software is good and hopefully openQRM project lead Matt Rechenburg will continue on with the project. openQRM is an excellent tool for someone who wants to provision testing laboratories and with more maturity be able to provide data center automation to the more demanding data centers (a classic rise by disruptive technology as described here). Perchance Qlusters set their sights too high trying to draft the success of a BladeLogic IPO (BladeLogic was since gobbled up by BMC) and they didn’t service a market that VMware has started to abandon as they focus on server consolidation.

With Qlusters turning their attention away perhaps there’s an opportunity for someone to lend their support to Rechenburg’s efforts. Personally, I have been impressed by Enomaly, a Toronto-based virtualization services vendor, that makes Enomalism a management platform for elastic computing. Maybe there is some synergy between the two projects. At one time the openQRM project was very active fronted by my friend and sometime coconspirator whurley who now jets around as BMC’s open source architect (BMC is one of the Big Four). I gave him a call and see if he had any thoughts. Given BMC’s anemic open source offerings I thought maybe he would be stepping up to sponsor the project. Of course now being a corporate guy he just chuckled and gave me the official: “No Comment”. I guess he’s happy to make proprietary software while carrying around an open source title.

A Little History

Shortly after the launch of OpenQRM Qlusters along with prominent open source management projects Nagios, Webmin and software vendors Symbiot, Zenoss and Emu Software started a grass roots effort to to raise awareness of open source systems management as an alternative to expensive proprietary software suits via the Open Management Consortium. The band resulting organization drew together over 40 companies and projects to discuss systems management along with thousands of end-users.

The result was heightened awareness of open source systems management solutions and conversations among the projects and companies that produce them. No working groups, no marketing efforts, just a banner and a place to converse, a very humble set of goals.

You see systems management is a broad category with sweeping subcategories like provisioning, monitoring, configuration, capacity management, storage management, inventory, network management, virtualization management, and the so-on. The goal was to encourage collaboration which I believe is happening organically, though more on that later.

The Sad State of Proprietary Software

This week at the Gartner Emerging Technologies Conference analysts Michael Silver and Neil MacDonald discussed what was broken with the Windows operating system.

Microsoft’s operating system (OS) development times are too long and they deliver limited innovation; their OSs provide an inconsistent experience between platforms, with significant compatibility issues; and other vendors are out-innovating Microsoft. That gives enterprises unpredictable releases with limited value, management costs that are too high, and new releases that break too many apps and take too long to test and adopt.

This is the same issues exist with proprietary systems management. There’s a trainwreck coming in a rapidly growing management software market:

IT operations management software market revenue will reach $18.1 billion in 2012, reflecting a compound annual growth rate of 7.2% from 2007 through 2012.

Gartner
Forecast: IT Operations Management Software, Worldwide, 2007-2012

Systems management users have to three choices. Adopt solutions that are monolithic, expensive, and hard to integrate, use low-end solutions that are lack needed features and breadth, or choose open systems that are flexible, easy to unify and can be used and consumed on the terms of the consumer not the vendor.

Open Source Systems Management Thriving

In a report by the 451 Group last summer released a report on the commercial adoption of open source, Managing in the Open: The Next Wave sums up the current state of systems management.

The ‘Big Four’ systems management vendors (BMC, HP, IBM and CA) are ripe for a shake-up. In the past 12 months, the large-scale, commercially supported application of open source development methods has been applied to systems management software problems. Open source could give potential challengers the cost and scale advantages they need to take on such formidable opponents while offering users a potential avenue for cost reduction. Yet open source vendors and software still face an uphill battle against entrenched players with their existing integrated suites and supportive relationships.

Three of the little four, Groundwork, Hyperic, and Zenoss all have steadily added customers and received venture capital to accelerate the growth of their businesses. Beyond that there a plenty of other thriving open source projects in systems management. While the big boys are hardly shaking in their boots it’s evident that they are going to see pressure from those vendors executing an open model for systems management.

A look at popular open source software site SourceForge shows that systems management boasts plenty of active projects.

• Nagios - Nagios is perhaps the most popular of all open source monitoring projects and also the oldest. Recently releasing version 3.0 Nagios install base likely exceeds six figures and well over 1.5 million downloads. It also has over 100 sub-projects that support the development and extensibility of the main project. Project lead Ethan Galstad, announced in November of last year that he had formed Nagios Enterprises to provide professional services for Nagios and since has been announcing a steady stream of partners.
• OpenNMS - Another popular open source SNMP monitoring project has a solid reputation, and actually beat out at least two of the Big Four, HP OpenView and IBM Tivoli, in the SearchNetworking.com Product Leadership awards for 2007. Voted upon by systems management users it shows a growing appetitie for open source solutions especially in monitoring.
• RRDTool - RRDTool is the Round Robin Database tool and a standard for collecting time series data that is sponsored by Groundwork Open Source and Zenoss. It’s very popular and is the standard tool for collecting data for open source performance management.
• Net-SNMP -This project consists of tools and libraries relating to the Simple Network Management Protocol including: An extensible agent, An SNMP library, tools to request or set information from SNMP agents, tools to generate and handle SNMP traps, etc. It’s also provide the ability for open source and proprietary software alike to monitor things like Linux servers.
• Webmin - (40th most downloaded application of all-time on SF.net with close to 10 million downloads) - A configuration management software for Linux and Unix is one of the most popular and earliest examples of a web services front-end for server and application management.
• phpMyAdmin - (25th most downloaded application all time with close to 16 million downloads) -A web-based interface to administer Sun’s open source database, MySQL.

Beyond the downloads these open source programs are distributed by many other methods including Linux distributions and other software repositories. Beyond that they are used to solve problems by millions of users.

Integration and Re-Usable Code, User Driven Standards

Systems management is such a complex problem especially for large enterprises that large product suites from a single vendor will struggle to keep up with the demand for a constantly evolving systems management landscape. I was struck by this realization as I was attending Usenix Large Installation Systems Administration (LISA) Conference last fall. During a session on configuration management led by open source project lead Luke Kanies of Puppet the attendees indicated that they used a wide breadth of solutions. Many of the users indicated that they used homegrown solutions built on top of bits of open source code. Others indicated their use of open source projects, CFengine and BCfg2.

There was no clear commercial winner or open source one for that matter, and their probably never will be. The availability of tools and bits of code to build highly complex and customized software configuration and deployment platforms was a key part of most everyone’s strategy. Having the freedom to integrate and use existing products and solutions that adhere to open source methodologies and open standards should be a requirement of systems management users–along with ease-of-use and overall value provided.

In the open source world it is common for projects to support and leverage the work of others. Nagios who has been around longer than any of the monitoring solutions mentioned here they have a large base of plugins and tests used to checks status. Hyperic, Groundwork, OpenNMS, and Zenoss all support Nagios plugins as it is the most utilitarian approach to expanding their products rather than create new standards that might prevent users from using previous customizations and gives flexibility to try new solutions. This adherence to standards enforced (or at least motivated) by users rather than vendors is a bit of a novelty.

There’s plenty of other integration going on as well. Early on Hyperic integrated with JBoss to provide management tools. And since leading Linux vendor Red Hat has acquired JBoss they have launched the RHQ project in conjunction with Hyperic to help provide a common infrastructure management platform for Red Hat Linux.

Many monitoring providers including OpenNMS, Groundwork, and Zenoss include RRDTool in their solutions. Vendors like Groundwork provide the glue for a lot of open source projects though they lack the sizeable communities that power many other OSS vendors.

As much as I like to rib the big guys about their solutions it’s going to be necessary to work together to best serve the requirements of end-users. I know Zenoss users are already integrating with their legacy HP Openview installation. At BarCampESM (as in Enterprise Systems Management) representatives from Alterpoint, BMC, IBM (including Tivoli Monitoring product manager Heath Newburn), Netcool, OpenNMS, and Zenoss collaborated with end-users on how they used our products. They made it clear that integration and cooperation was definitely in their best interests.

What’s Next for Open Source Systems Management?

Open source systems management is a nascent approach to an old industry. The Little 3: Groundwork, Hyperic, and Zenoss aren’t so little anymore with fast growing customer bases and many thousand users in their communities. As companies are tasked with measuring even more infrastructure and new technologies large vendors will be hard-pressed to deliver complete enterprise solutions. New technologies, such as cloud computing, are severely in need of tools to manage the new utility-based technologies. The growing success of open source management technologies is the wave of the future both augmenting and replacing expensive, antiquated proprietary solutions as well as quickly adapting to a growing IT landscape.

Other coverage of OpenQRM/Qlusters

• 451 Group: Qlusters waves to openQRM
• Internet News: Qlusters Dumps Open Source openQRM Systems Management Project
• Matt Rechenburg’s blog: OpenQRM is available
• Virtualization .com: Qlusters releases openQRM 3
• Virtualization for Everyone: openQRM 3.5 released: Control your heterogenous virtualization jungle

[Disclosures: I am the VP of Community at Zenoss and the President of the Open Management Consortium]

Technorati Tags: Big 4, Bladelogic, BMC, enomalism, Enomaly, Hyperic, Little 3, Little 4, Nagios, Open Source, OpenQRM, Qlusters, RHQ, Sybmiot, Symbiot, Systems Management, webmin, Zenoss

ShareThis

The built-in HQ database is PostgreSQL. Recently, users have been discovering PostgreSQL has a certain limitation: it will not execute more than 2 billion transactions between vacuums. In rare cases, an HQ built-in database can get into this state.

If this happens, the database will stop accepting connections and HQ, which needs a data store, will obviously cease to operate properly. The immediate symptom will be that users will not be able to log in to HQ and the message displayed on the screen will be The backend datasource is unavailable.

That error is not enough to say for sure that the problem is PostgreSQL avoiding wraparound failure by not accepting connections. A quick look at the hqdb.log can confirm. The telltale log entries look like this:

FATAL: database is not accepting commands to avoid wraparound data loss in database "postgres"
HINT: Stop the postmaster and use a standalone backend to vacuum database "postgres".

Happily, postgres tells you how to solve the problem. It’s not very specific, though so I’ll add some details.

The first thing to do is immediately shut HQ down (including the built-in database). Next, start just the database in single-user mode. Technically, you only really need to run a VACUUM, but since we’re here, we might as well be thorough and run VACUUM FULL ANALYZE.

I’ve created a script to start the built-in HQ database in single-user mode. It’s a slightly modified version of the db-start.sh script that is included in HQ installations. It takes one argument which is the name of the database to start up. The database name to VACUUM is specified in the hqdb.log message (it is “postgres” in the message above). Start the database with the script and you will get dropped into the psql command line. Run the VACUUM and exit. Looks something like this:

$ bin/db-start-single-user-8.1.sh postgres
PostgreSQL stand-alone backend 8.1.2
backend> VACUUM FULL ANALYZE;
VACUUM
backend>

When it’s done, you send an EOF (usually control-D) to exit the shell and shut down the database. At this point, you’re done and you can start HQ normally. Things will work and will usually perform much faster.

For more information on this PostgreSQL, check their documentation on how to avoid running into it to begin with.

In the spirit (pun intended) of Halloween, Hyperic sponsored a Nightmare on Web Street contest, where folks were encouraged to tell their grim tales of IT woe for a chance at trick-or-treating for a Wii! The day has come, and the contest winner has been selected. Without further ado, I would like to congratulate “Mr Anderson” for his nightmarish tale of servers and HTML. Here’s the winning entry folks:

Oh by the way, your websites will no longer be hosted in 4 days

In spring of 2002 I was called in by a Company to help them with their websites (I wasn’t an employee at that time, I had a small shop of about 10 guys doing web development/business systems).

The issue? Their Host went bankrupt and they had 4 days to move 100+ sites with 50+ pages each off their servers before the plug was pulled.

If this didn’t go right, 80,000 plus doctors around the country would be very unhappy - they use the sites to look for Continuing Medical Education seminars, register, make payments, keep track of their certificates, etc. the Company would lose a LOT of money.

So - no big deal, right? Just access the servers and pull the code off and put it on our servers.

SORRY - the Hosting Vendor of the Company actually outsourced the hosting to another vendor whom they hadn’t paid and they refused to give us access at all! With the short amount of time that we had, I pulled my entire team together to go to every page, save as html, fix the image src tags and everything else and rebuild the sites by hand as static HTML (no database driven content). We got their sites back up in 1/2 day before the plug got pulled, and then a week or two later launched a Content Management system for them to be able to keep the content up to date.

This took a team of 6 72+ hours of work. Non-stop. 3 all nighters in a row with interspersing of Colin Powell 20 Minute Power Naps.

Needless to say, the Company was very happy we could rescue them, and now that I’ve moved on from owning my own company they have very happily employed me.

Congratulations, Mr Anderson, your Wii will be in the mail this week!