sorted by: recent | see : popular

Content Tagged with records + Data

Following on from the announcement at the MySQL conference where Sun and Kickfire jointly announced data warehousing benchmark records, we have just announced new TPC-H benchmark records. Specifically, the Kickfire Database Appliance 2400 is the highest price/performance offering at 300GB, again breaking the $1 barrier for the first time coming in at 89 cents per QphH (Queries per hour on the TPC-H benchmark). The 2400 is also the highest performance (non-clustered) offering at 300GB.

I’m not going to further dwell on the numbers in this post other than to quickly point out another aspect of this achievement that Justin noted in his blog related to the energy savings the Kickfire appliance delivers in addition to the performance and price/performance. What I want to address is why we decided to do these benchmarks in the first place and what we believe their relevance to be. The reason is that as we continue publishing these benchmarks we occasionally get questioned about their importance (or lack thereof). Here?s my take.

First of all, they?re benchmarks and so, by definition, are limited. Let?s get this one out of the way. No benchmark, no matter how thorough, is going to cover every possible real-world scenario. But just because benchmarks have limitations is not a reason to discard them, particularly if they are thoughtfully conceived and rigorously applied as is the case for the TPC-H benchmarks.

Some vendors have been pushing the idea that only POCs count. Not surprisingly these are the vendors who haven?t published their results (and the reason for this should be obvious). Whereas I would agree that POCs are clearly a critical part of an evaluation process, much as test driving is when buying a car, it doesn?t mean you should discard the objective comparison of pertinent metrics. Going back to our car buying analogy, it would be like throwing away the information sheet you find on the car’s passenger window at the dealership. It would seem to me that prospective buyers would want to know about things such as the MPG rating or horsepower and how these compare to other cars they are considering.

To speak more specifically about the TPC-H benchmarks, it is not immediately obvious (unless you have been through the process) how extensive and rigorous they are. First, the 22 queries test a broad spectrum of SQL complexity spanning everything from simple reporting-type queries to deep analytic-type queries with multi-table joins, correlated sub-queries and the like. Second, the system performance is measured on a single query stream (the Power Run) but also on concurrent queries (The Throughput Run). Third, the load performance (important for data warehousing) is measured. Finally, full ACID compliance is tested for. You can check out the full details of the benchmark specifications here.

The benchmark specification also places extensive restrictions on what is allowed to prepare the test system. As an example, anything that would circumvent a true test of the system?s performance such as pre-built aggregates is disallowed.

The audit is also a rigorous process which is carried out by independent, TPC-sanctioned auditors who certify the benchmark and must sign off on the disclosure report in order for the results to be approved and published.

The point I?m making here is that these are not your homegrown benchmarks too often seen in vendors? marketing material. The fact that there is an independent body, the Transaction Processing Performance Council, which has been around for 20 years and whose sole purpose is to define and monitor these benchmarks, should be a clear indicator that these benchmarks mean business.

Finally, and to make the point that these are serious, I have to be diligent here in how I talk about our performance numbers as there is a fair use clause when speaking about results that must be carefully adhered to in order to avoid penalties. To that end and to wrap up this post, here are some additional details I must disclose in order to be fully in compliance:

The Kickfire Database Appliance Series 2400 delivers a performance of 54,895 QphH@300GB (Queries per hour on the TPC-H benchmark) on the 300GB TPC-H benchmark. The Kickfire Database Appliance has a price/performance of $0.89/QphH@300GB USD on the 300GB benchmark. Kickfire delivers this performance with a 3 year total system cost of $48,790 USD. The Kickfire Database Appliance is in Beta and will be available October 14, 2008.

TPC-H, QphH and $/QphH are trademarks of the TPC. For additional information on the TPCH benchmark, please visit the Transaction Processing Performance Council’s Web site at http://www.tpc.org/.

Viewed 1129 times by 368 viewers