Benchmarks

Here is the performance comparison of AlphaVM to some real Alpha systems.

BenchmarkDS10 6/466, 256MB RAMES47 1.15, 2 CPUAlphaVM
OpenSSL speed MD5
8192 bytes.
42636104788234367
OpenSSL speed RSA,
4096 bytes, sign/s.
3.58.714.3
OpenSSL speed RSA,
4096 bytes, verify/s.
232.0575.3961.1
NBench Memory Index3.4178.63512.751
NBench Int Index2.5116.1518.771
NBench FP Index8.34621.0799.191
PI Computations, 40000 digits, sec.499
VUPS245.8977.8
Diskblock, 100 blocks/IO, KB/s 37915849
Diskblock, 200 blocks/IO, KB/s383811088
Diskblock, 300 blocks/IO, KB/s382313902

AlphaVM measurements have run on AlphaVM-Pro 1.4.11 running on a server with the CPU Intel Xeon E3-1270v3 @ 3.5GHz, hyper-threading on and Windows Server 2012R2. AlphaVM is configured as follows: DS10 6/667 (1CPU), 256MB RAM, JIT2, 2048 JIT pages, idle disabled, experimental features disabled, disk caching disabled. This configuration corresponds to one of our reference systems DS10 6/667,256MB RAM.

OpenSSL benchmarking has been performed on freshly installed Tru64 5.1b using the commands:

openssl speed md5
openssl speed rsa.

NBecnch benchmarking has been performed on a freshly installed Tru64 5.1b.

NBench measures performance of several widely used CPU-intensive algorithms. In the end it computes 3 numbers: Memory index, Integer index and Floating Point index.

We have run the tests on Tru64 5.1b. It should be noted that CPU benchmarks behavior does not depend much on the operating system. Thus, the benchmark is also representative for OpenVMS usage.

The benchmark can be found here. It has been compiled on a freshly installed Tru64 5.1b-4. To compile it we edited the makefile to enable DEC C compiler:

# for DEC Unix using cc you can try
CC = cc
CFLAGS = -O3
LINKFLAGS = -s -non_shared

The compiler reports itself as

# cc -V
Compaq C V6.5-011 on Compaq Tru64 UNIX V5.1B (Rev. 2650)
Compiler Driver V6.5-003 (sys) cc Driver.

PI computation benchmarking has been performed on a freshly installed OpenVMS 8.4.

This benchmark measures time needed to compute PI with the precision of 40000 digits. It was introduced by Migration Specialties to benchmark their emulator and real Alpha systems.

The program to compute PI is available in the macro assembler examples on OpenVMS. It is located in sys$examples:macro64$pi.m64. Compile and run it as follows:

$ macro/alpha_axp/object=pi sys$examples:macro64$pi
$ link pi
$ run pi

The executable compiled for 7.3-2 can be found here: pi.exe. This executable also run on other version of OpenVMS, like 8.4. The DCL script to measure time is as follows:

$ t = f$cvtime(f$time(),,"secondofyear")
$ run pi
40000
$ t = f$cvtime(f$time(),,"secondofyear") - t
$ write sys$output "Computed in ''t' sec"

VUPs benchmarking has been performed on a freshly installed OpenVMS 8.4.

VUP stands for VAX Unit of Performance. It is an OpenVMS DCL script originally designed to measure performance of VAX systems.

The benchmark involves a lot of OpenVMS system activity like mode and context switches. In this sense it is different from the usual CPU benchmarks. The result includes not only the plain computational speed, but also the performance of the system activity.

Unfortunately the test has a number of drawbacks. It cannot be considered as an exact measurement.

  • Different boots of the OS produce substantially different results. The difference can be 5% or perhaps more (on real Alpha systems).
  • The test is even less accurate in SMP environment.
  • The test produces very different results on different versions of OpenVMS.

Nevertheless, this benchmark is very popular in the OpenVMS world.

The original DCL procedure was sometimes hanging in a infinite loop on fast CPUs. It has been improved by Volker Halle. We use this improved procedure vups_com.txt.

Diskblock benchmarking has been performed on a freshly installed OpenVMS 8.4 using the version 6.2 of the benchmark.

Diskblock is a free OpenVMS tool for low-level disk IO operation. It has a command to benchmark disk IO performance. We have used the version 6.2 under OpenVMS 8.4. It can be downloaded from here.

Install the tool as follows. Download the files (diskblock062.a, diskblock062.b, …) to your disk, say to dka0:[here_is_the_kit]. Then run the command:

$ @sys$update:vmsinstal diskblock dka0:[here_is_the_kit]

The benchmark has been run as follows

$ mc diskblock
select dka0: /over
test /iosize=1 /queue=20 /readperc

The benchmark has been run on the system disk. Thus, so far only the read of system disk has been benchmarked. This scenario can be reproduced on any system with just the boot disk. The throughput was measured for several IO sizes.

AlphaVM has been configured with file caching disabled. File caching improves performance. When file caching is enabled, the throughput sometimes reaches the gigabytes per second range. However, the result is highly dependent on the history of the previous access to the disk image. That makes it harder to get stable results. The benchmark with the caching enabled will be added later.

The idle mode has been enabled for this test.