Reliable Benchmarking:

With a flood of advertising from top database platform providers like Microsoft, Oracle, Sybase, and Informix, development managers may find themselves looking for reliable benchmarking figures for relational database management systems (RDBMS).

Every database vendor mentions benchmark data that proves its database is the fastest, and each vendor chooses the benchmark test that presents its product in the most favorable manner. So the Transaction Processing Performance Council (TPC) was established in an effort to provide standardized benchmark tests.

Different applications - Different processing signatures:

The TPC also recognized that different types of applications have different processing signatures. They developed a number of benchmarks including;

TPC-C benchmark test:

The TPC-C benchmark test is considered to all intents and purposes standard for online transaction processing database benchmarks, primarily because TPC-C attempts to simulate real-world OLTP database transactions.

Why is benchmarking of different products difficult?

TPC was unable to avert database vendors from fudging benchmark results. The biggest concern is the unlike hardware platforms databases run on. For example, IBM's UDB database formerly DB2 only runs on the IBM 3090 mainframe, while the Informix database only runs on UNIX. So it's hard to evaluate benchmarks for these products because of the vastly different machine architectures.

Tricks to improve the processing speed of Benchmarks:

In the efforts to prove that the database product is superior to the competition, database vendors take up many tricks to improve the processing speed of their benchmarks. Nearly all of the tricks in use by database vendors involve caching data and SQL in RAM.

Remember, database performance is all about disk I/O, and vendors use large RAM memory regions to preload critical benchmark components to avoid any disk access.

Using tricks, all database vendors find ways to make their product better-quality to the competition. Therefore, most database benchmark data should be viewed with healthy cynicism. Some of the tricks include:

Pre-loading the data into the RAM buffers:

By preloading the data into the RAM buffers, database can access the information thousands of times quicker then a disk I/O access.

Pre-parsing and Pre-computing the execution plans:

By pre-parsing and pre-computing the execution plans for the SQL, the database vendors find a way around the overhead of parsing and invoking the SQL optimizer to produce the execution plan.

Pre-aggregation mechanisms:

Some database products have special pre-aggregation mechanisms to pre-join tables. For example, Oracle has Materialized Views that can store up the results of an n-way table join, allowing super-fast data access.

High-speed machines and Cluster Architectures:

Database vendors can significantly improve benchmark speeds by using special high-speed machines and cluster architectures.

Database speed:

For the manager responsible for comparing database performance, benchmark studies present a great challenge. While it's not fair to say that database benchmarks are worthless, it's very hard to use benchmarks to establish that any single database product is faster than another. To see how difficult it is to compare database performance, just look at the TPC publication on a variety of TPC-C benchmarks for various hardware and database platforms.

Evaluating and choosing a database:

Performance is only one of many factors in evaluating a database. You must also consider the availability of trained DBAs, the vendor's technical support, and total cost of ownership, among other factors. Most databases can be configured to process hundreds of transactions per second. Based on that, this is what we can say about database performance:

Hardware to improve Database performance:

Database performance is mainly a function of hardware. A poorly performing database can be made to appear fast with cached disk arrays and super-fast processors. Even “independent” benchmarks can be deceptive because of the diverse hardware, disk, and network configurations.

High-speed database performance:

Delivery of high-speed transactions needs information of the application. High-speed database performance is generally achieved through complicated caching tricks and pre-aggregation schemes. This requires thorough information of the application and its I/O signatures.

Final Thoughts:

In sum, it's hard to appraise database performance objectively. The savvy manager must cautiously assess all of the database vendors with a cynical eye to separate the hype from the reality.

 More Database Articles
   Database Security: Step by step guideline
   Performing Oracle Automated Tablespace Point-in-Time Recovery!!
   Web Applications: Oracle-SQL Database Security
   Efficiently handle Tricky Data Guard Failures!!
   A Guideline for Oracle Instantiation with RMAN!