Processor performance is a complex subject and cannot be characterized in a meaningful manner with a single number or benchmark score. Some insight into the factors that influence performance can be gained by examining performance across a range of processor characteristics, including frequency, cache size, bus bandwidth, and processor architecture.
The best known database benchmarks are TPC-C for OLTP applications and TPC-H for DSS applications. Unfortunately, TPC benchmarks are very expensive to run and results are available only for a limited number of processors. Further complicating matters is that some results target price-performance rather than absolute performance, making comparisons between results of limited use.
The one benchmark for which an extensive range of results are available is SPEC CPU2000. SPEC CPU2000 is comprised of two benchmark suites. One suite focuses on integer performance and the other on floating point performance. Each suite has a base (SPECint_base2000 and SPECfp_base2000) and peak (SPECint2000 and SPECfp2000) metric. The peak metric represents the true capability of a processor. The base score better represents the performance that can be achieved with compiled code without excessive tweaking. (The peak scores allow complier flags that essentially substitutes hand tuned assembly code in place of recognized code structures from a particular benchmark).
Most server applications are integer intensive, for which performance is better reflected by the SPEC CPU2000 integer suite. Shrink wrap commercial software is generally compiled without special optimizations, so performance will be better reflected by the base metric SPECint_base2000. All SPEC CPU2000 results discussed here pertain only to the integer suite and the base metric.
The SPECint2000 suite is comprised of 12 individual benchmark programs, including GZIP (a compression program) and GCC (GNU C compiler). The results in individual programs shows the range of performance benefit that can be expected with variations in processor frequency, cache size, memory bandwidth and processor architecture.
Compiler and Other Software-Related Changes
The first issue to be aware of is that the compiler can significantly influence performance and that SPEC CPU results typically employs the latest in compiler technology. Except for some open source database applications, most developers will not have the opportunity to recompile the database engine source code. However, compiler-related performance improvements can be expected every few years with major releases.
Figure 1 shows three posted SPECint_base2000 scores for the Intel Pentium III processor, 800MHz, 256K L2 cache and Intel VC820 motherboard. The first set, base, is the composite score. The remaining sets show the individual component scores.
Figure 1. Pentium III 800MHz performance with compiler improvements.
All Intel processor results presented here use the Intel C/C++ compiler. The March 2000 test score used version 4.5. Both the June and November 2000 tests used version 5.0. There is a small performance increase from the June 2000 and November 2000 scores, even though both used the same compiler version. The performance gain in this case stems from other software configuration changes (possibly Intel Application Accelerator).
Figure 2 shows SPECint_base2000 scores for the Pentium 4 processor, 1.5GHz, 256K L2 cache, 400MHz FSB. The November 2000 and August 2001 results are on the Intel D850GB motherboard. The November 2001 result is on the D850MD motherboard and the May 2002 result is on the D850EMV2. The first measurement used the version 5.0 compiler, the second and third used version 5.0.1 and the last used version 6.0.
Figure 2. Pentium 4 1.50GHz performance with compiler improvements.
Figure 3 shows the Pentium III 800MHz percentage change from the version 4.5 compiler to the more recent 5.0. The increase in the composite SPECint_base2000 score is 8%. The EON program shows a very large performance gain, three programs show approximately 10% performance gain and other programs show lesser gain. PARSER actually shows a small loss. Figure 4 shows the Pentium 4 1.5GHz percentage change from the version 5.0 to 6.0 compiler, along with other chipset and motherboard refinements. The composite score gain is 6.9%. Five programs show 10% or larger performance gain, but three programs, EON, PERLBMK and TWOLF, show a performance drop.
Figure 3. Pentium III performance gain due to compiler improvements.
Figure 4. Pentium 4 performance gain from complier improvements.
In a period of just over two years, there is an overall performance gain of close to 15%, most of which, but not all, can be attributed to compiler improvements. Not included in this is the 20-25% in compiler related performance improvements made in 1999 as demonstrated in SPEC CPU95 results. Having established the influence of compiler technology, comparisons of other factors will use results from the same compiler version where possible or practical.