Wednesday, 21 March 2007

TPC-C to die

One of the least reliable and easily distorted benchmarks is nearing the end of its life. Having seen this from the vendor's side, the TPC-C benchmarks of so little relevance to real world database performance as to be virtually useless to companies looking to compare the performance of different vendors. What the benchmark does tell you, is who is investing money in their databases, as running a TPC benchmark costs a lot of money. The reason you hardly ever see Sybase or Postgres or MySQL in the top ten is they can't afford / aren't investing to run the benchmarks. Looking at the top ten this morning you see Oracle, IBM and Microsoft filling up the whole list. No surprises there.

Whether the new benchmark (TPC-E) is any better, is the question. It is based on a financial scenario, which may work against Oracle (which has never been great at high volumes of small transactions which are typical of financial systems) a bit, but its unlikely to be much more real-world than the old one. Good companies will have to run their own benchmarks, like they always have.


Paul Robinson said...

Benchmarks and certifications/qualifications issued by private organizations are almost always done so the company being given the certification can crow about it, having paid a nice chunk of change to get it.

Plus the fact the certifying agency makes nice money off the scam^W deal. I mean, as you mention, the Transaction Processing Council charges quite a bit of money for a company to be certified for TPC benchmarks. I'll bet that if a company fails it simply changes something and tries again, the failures are never announced.

For example, you've probably heard of how companies get awards from J.D. Power and Associates for some quality measure of customer surveys? You could probably get one from them for "Best organized blog on Structured Query Language," or something like that. The only problem you would have to do the same thing all the other "winners" of some J.D. Power and Associates quality award have done: pay $50,000.00 for the privelege!

You have to look at the reasoning behind the standards or qualifications.

A benchmark or a qualification put out by an organization whose only interest in collecting (high) fees ensures that is the standard: whoever can pay the fee.

Companies that want Underwriters Laboratories certification on their products (practically a requirement for most electrical equipment as most building codes require it) has to take into account the cost to do so, which basically requires they submit one or more of the items for serious destructive testing plus pay a fee which can be as much as $10,000. The difference being UL goes through a lot of work to determine if the device passes, which includes basically trying anything they can to make the device have electrical shorts or could cause injury. If the device basically has to be destroyed before it will fail, then it passes and the customer gets the right to use the UL label on their product.

What exactly do TPC and JD Power do? Either they accept the software manufacturer's claims (after their check clears, of course), or they rig the surveys so that they are so narrowly tailored that they could pass even if they were a manure company trying for "best tasting excrement." (I'm being polite here or I'd have used the shorter word.)

Paul Robinson's Blog

Paul Robinson said...

With respect to my prior comment, J.D. Power and Associates operates in the United States; I don't know if they operate in Australia. (It wasn't until after I wrote my first comment that I noticed you were in Australia, and it occurred to me you might not know what I'm talking about.)