We’re happy to announce the first version of the Gearman rewrite in C, along with some interesting new MySQL UDFs based on the C library. Check out the Gearman wiki for an overview of what this is, download details, and API documentation.

For the anxious, here is a quick how-to:

Download: http://launchpad.net/gearmand/trunk/0.1/+download/gearmand-0.1.tar.gz
tar xzf gearmand-0.1.tar.gz
cd gearmand-0.1/
./configure
make
make install

You should now have the job server (gearmand) installed in /usr/local/bin, along with headers in libraries in /usr/local/lib & include. You can now see it work by running some simple clients and workers in the examples/ directory:

gearmand &      (this is assuming /usr/local/bin is in your path)
cd examples
./reverse_worker &
./reverse_client "Hello, Gearman!"

If everything went well, your terminal should look something like this:

> gearmand &
[1] 2270
> ./reverse_worker &
[2] 2301
> ./reverse_client “Hello, Gearman!”
Job=H:lap:1 Workload=Hello, Gearman! Result=!namraeG ,olleH
Result=!namraeG ,olleH

What happened is the reverse_client program sent the string “Hello, Gearman!” to the job server (gearmand), telling it to run it with a function named “reverse”. The job server then found a worker is registered (out reverse_worker we started), and forwarded the job on to it. The reverse_worker program then reversed the string, and sent it back to the job server, which then sends it back to the original reverse_client. The “Job=…” line is the output from the reverse_worker, saying it received a job with the given workload, and sent a result back. The “Result=…” line is the final output from the reverse_client. You have now run your first Gearman job!

Before getting into the MySQL UDFs, lets look at how this could scale in a useful way. Imagine your reverse_client is actually an client inside of your PHP or Perl script running on your webserver. Next, your could have multiple job servers to spread the load and get redundancy, and then you could have a farm of machines running the workers. You now have a simple distributed computing framework to do time and/or resource intensive jobs like document or image conversion.

Now lets see how we can make MySQL run Gearman jobs:

Download: http://launchpad.net/gearman-mysql-udf/trunk/0.1/+download/gearman-mysql-udf-0.1.tar.gz
tar xzf gearman-mysql-udf-0.1.tar.gz
cd gearman-mysql-udf-0.1/
./configure --with-mysql=/usr/local/mysql/bin/mysql_config --libdir=/usr/local/mysql/lib/plugin/
make
make install

You may need to change your configure line depending on where you have MySQL installed (this is assuming /usr/local/mysql). Configure needs to know where the mysql_config tool is and where to install the plugin. The above assumes that your plugins should be in lib/plugin, but they may just be in lib. You will need to check your MySQL installation to see what paths you should use here.

Once you get the paths correct and everything installed, you can now load them through the MySQL command line tool:

CREATE FUNCTION gman_do RETURNS STRING
       SONAME "libgearman_mysql_udf.so";
CREATE FUNCTION gman_do_high RETURNS STRING
       SONAME "libgearman_mysql_udf.so";
CREATE FUNCTION gman_do_background RETURNS STRING
       SONAME "libgearman_mysql_udf.so";
CREATE AGGREGATE FUNCTION gman_sum RETURNS INTEGER
       SONAME "libgearman_mysql_udf.so";
CREATE FUNCTION gman_servers_set RETURNS STRING
       SONAME "libgearman_mysql_udf.so";

If you get errors, the most likely case is that MySQL cannot find the module in the library path. Try a different libdir in the configure above and reinstall. Once these are loaded, you can now tell it where to find a job server and then run a Gearman job from a query. This test assumes you still have gearmand and reverse_worker running in the terminal from the previous example.

mysql> SELECT gman_servers_set("127.0.0.1");

+-------------------------------+
| gman_servers_set("127.0.0.1") |
+-------------------------------+
| NULL                          |
+-------------------------------+
1 row in set (0.00 sec)

mysql> SELECT gman_do("reverse", Host) AS test FROM mysql.user;
+-----------+
| test      |
+-----------+
| 1.0.0.721 |
| pal       |
| pal       |
| tsohlacol |
| tsohlacol |
+-----------+
5 rows in set (0.00 sec)

Now you have a MySQL UDF that runs as a normal Gearman client! See the README on more information on how to run other types of jobs. I’ll be writing another blog entry shortly with a more interesting use case using these UDFs.

So what are the next steps for Gearman? We’ll be improving the server and clients a bit more, a new PHP extension based on the C library, more language wrappers using SWIG, persistent queues, queue replication, and also seeing how we’ll be able to plug this into other applications.

I spent this past week down in San Jose, CA at my employer’s office for team meetings and to officially kick-off my next big project. The design and architecture was very well received, and I drummed up some excitement with Gearman and working with the OSS community in general (which we’ve not done too much of in the past). We’ll be developing it entirely on Launchpad under GPLv2, and I’ll be writing a number of blog posts covering each component in detail. Why would anyone else find this interesting? It covers many topics of how to write a high-performance application in the cloud. Specific topics will include Gearman, persistent Gearman queues, eventual consistency data models (and related schemas), lightweight Map/Reduce for real-time applications, and how to combine all this with MySQL and/or Drizzle to build an e-mail storage infrastructure in the cloud. While the use case may be specific to e-mail, its concepts (and much of the code) will be easy to translate to a number of other applications. Also, there will be a number of other projects that will get spawned from this, such as a faster Gearman job server written in C (currently it is in Perl), generic persistent queuing system based on Gearman, and Gearman UDFs for MySQL and Drizzle (those are coming very soon).



This was the final result of a long white-boarding session. Now off to code it…

In case you don’t know, there is a trend on the internet known as Rickrolling:

Rickrolling is an Internet meme typically involving the music video for the 1987 Rick Astley song “Never Gonna Give You Up”. The meme is a bait and switch: a person provides a Web link they claim is relevant to the topic at hand, but the link actually takes the user to the Astley video.

When a person clicks on the link given and is led to the web page he/she is said to have been “Rickrolled” (also spelled Rickroll’d). By extension, it can also mean playing the song loudly in public in order to be disruptive

So there I was, watching the Macy’s parade. The dopey Cartoon Network float is rolling by, and in the midst of whatever song they were doing, the whole things stops and out pops Rick Astley. I have not laughed that hard in at least a year.

What’s the new libdrizzle? It’s a complete rewrite of the client library for the Drizzle project, but it also has full support for the current MySQL protocol (4.1+). Right now Drizzle uses the same protocol as MySQL, but work is being done to design a new, more robust protocol. Even when the Drizzle protocol changes, I plan to keep full support for the MySQL protocol since there is a need for a good low-level non-blocking client library. Also, once libdrizzle turns into a full protocol library (server and client packets) it could make for some interesting Drizzle/MySQL hybrid proxies.

Over the weekend I made a lot of progress on the new library, and last night I just finished up the first pass at the core functionality. It can now do full query and result processing. Currently only non-cached results are being used (no store_result() function), but the cached interfaces will be easy to add in. The nice thing about the non-cached results is that besides a few core structs, no malloc()s are happening. This can be very useful for applications that just need to stream data and don’t want to allocate a huge chunk for each BLOB.

Here are two tests using the new library and example client, each opening up 50 connections (-c 50 option) to the server and running a query. The first uses non-blocking I/O and allows the connections to run concurrently, while the second uses blocking I/O (-b option) and performs each connection/query serially.

lap> time ./client -h host -u sql -p pass -c 50 "SELECT * FROM mysql.user" > /dev/null
0.020u 0.024s 0:00.30 13.3%     0+0k 0+0io 0pf+0w

lap> time ./client -h host -u sql -p pass -c 50 -b "SELECT * FROM mysql.user" > /dev/null
0.020u 0.012s 0:09.70 0.3%      0+0k 0+0io 0pf+0w

Now, this test doesn’t show too much besides the amount of network latency between my laptop and the server, but it does show how the non-blocking interface allows a single thread to run concurrent connections with relatively little overhead. Now, for more practical usage, imagine a PHP script that needs to make multiple, time consuming queries. If they are all happening in parallel, your total page response time is only as long as the most time consuming query (plus a little overhead).

Next steps:

• Develop cached interfaces.
• Rework existing client utilities to use new library.
• Develop PHP module, recruit people to help with other language interfaces.
• Give talk at OpenSQL Camp about the new library, asking for feedback and recommendations.
• During above steps, work with others to design the new Drizzle protocol.
• Implement new Drizzle protocol.
• Other protocol layers? HTTP? Proxy server? Anything else client/protocol related?