The Whetstone benchmark is a synthetic benchmark for evaluating the performance of computers. It was first written in ALGOL 60 in 1972 at the Technical Support Unit (TSU) of the Department of Trade and Industry (later part of the Central Computer and Telecommunications Agency (CCTA)) in the United Kingdom. It was derived from statistics on program behaviour gathered on the KDF9 computer at the National Physical Laboratory (NPL), using a modified version of its Whetstone ALGOL 60 compiler. The workload on the machine was represented as a set of frequencies of execution of the 124 instructions of the Whetstone Code. The Whetstone Compiler was built at the Atomic Power Division of the English Electric Company in Whetstone, Leicestershire, England, hence its name. Dr. B.A. Wichman at NPL produced a set of 42 simple ALGOL 60 statements, which in a suitable combination matched the execution statistics. To make a more practical benchmark Harold Curnow of TSU wrote a program incorporating the 42 statements. This program worked in its ALGOL 60 version, but when translated into FORTRAN and compiled by IBM's optimizing FORTRAN compiler, calculations whose results were not output were optimized out by the compiler. He then produced a set of program fragments which were more like real code and which collectively matched the original 124 Whetstone instructions. Timing this program gave a measure of the machine's speed in thousands of Whetstone instructions per second (kWIPS). The Fortran version became the first general purpose benchmark that set industry standards of computer system performance. Further development was carried out by Roy Longbottom, also of TSU/CCTA, who became the official design authority. In July 2010, the original Algol 60 program ran once again under the Whetstone compiler, 30 years since the shutdown of the last KDF9 machine. The program was executed by a KDF9 emulator.
Benchmark content and outputs The benchmark employs 8 test procedures:
The original version only reported comprised parameters used for each test, numeric results produced, and the overall KWIPS performance rating. In 1978, the program was updated to log running time of each of the tests. As a result, each individual test could report a score. The three floating-point tests each reported one version of the MFLOPS (Millions of Floating Point Operations Per Second) statistic. The rest each reported a MOPS statistic. In 1980, the MOPS of tests 3, 4 and 7 were combined to form the VAX MIPS, a Millions of Integer Instructions Per Second measure calibrated to read the Digital VAX 11/780 as 1 MIPS. Note that there are other versions of the Whetstone Benchmark available online, some claiming copyright, without reference to CCTA or the design authority.
History before 1997
Initial CCTA results (1972) In conjunction with the undertaking controlled by the Contracts Division, CCTA engineers had responsibility to design and supervise acceptance trials of all UK Government computers and those for centrally funded for Universities and Research Councils, with systems varying from minicomputers to supercomputers. This provided the opportunity to gather verified Whetstone Benchmark results. Other results were obtained via new computer system appraisal activities. CCTA records are now available in The UK National Archives, including technical reports. Original Whetstone Benchmark results are in the 1985 CCTA Technical Memorandum 1182, where overall speeds is only shown as MWIPS. This contains more than 1000 results for 244 computers from 32 manufacturers, including the first for PCs and supercomputers.
The first 1 MIPS minicomputer: VAX-11/780, and VAX MIPS As mentioned above, per item scoring was added in 1978. The VAX-11/780 was the first commercially available 32-bit computer to demonstrate 1 MIPS, as measured by performance relative to the IBM System/370 architecture (it was specifically as fast as the model 158–3). In 1980, Whetstone added a feature where the MOPS of tests 3, 4 and 7 were combined to form the VAX MIPS, millions of instructions using VAX-11/780 as the definition of 1 MIPS.
Vector processing Roy Longbottom converted the original Whetstone Benchmark to fully exploit capabilities of the new vector processors since 1972. Results were included in the paper “Performance of Multi-User Supercomputing Facilities” presented in the 1989 Fourth International Conference on Supercomputing, Santa Clara. This was also repeated in the Harold Curnow paper “Whither Whetstone? The synthetic benchmark after 15 years” presented at the “Evaluating supercomputers: strategies for exploiting, evaluating and benchmarking computers with advanced architecture” conference in 1990, in book .
Further changes In 1987, code changes were also carried out, including by Bangor University, necessary to identify unexpected behaviour, without changing the implementation of the original 124 Whetstone instructions. One necessary change was to maintain measurement accuracy at increasing CPU speeds, with self calibration to run for a noticeable time, typically set for 10 seconds or 100 for early PCs with low timer resolution.
… excerpt ends here. Continue reading the full article.
