Benchmarking caveats

• The blatant cheat
• The targeted optimization
• Flag fiascos
• The special case
• Bait and switch
• Bending the rules
• Rigging the game
• Bogus baselines
• Competing optimizations

The blatant cheat

To guard against this, it is always a good idea to perturb benchmarks a little bit to see if their performance changes. This won't help with compilers that pattern-match the flowgraph; for that, it may actually be necessary to change what the benchmark computes, so that the compiler will not recognize it.

Examples of benchmarks that have been pattern-matched in this fashion in the past include Linpack and eqntott. A somewhat milder form of "blatant cheat" is to silently enable the best set of optimizer options when a benchmark is recognized (see "flag fiascos").

The targeted optimization

One targeted optimization to be especially careful of is the kind that undoes obfuscations added to microbenchmarks to prevent other optimizations. For instance, in the NaturalBridge microbenchmarks, the constant 1 is sometimes written like this:

  static int oddCounter;
  static int one() {
    if ((oddCounter & 1) == 0) return 1;
    else return (oddCounter * 3) & 1;
  }

Flag fiascos

Note that if you are actually motivated to go searching for just the right combinations of flags, then this sort of thing is ok, because it matches your development style.

The special case

To guard against this, try adding a second idle thread to any benchmark, and try running it on a multiprocessor, and compare the modified performance with the original performance. We use this class for an idle thread in our microbenchmarks

class SleepyThread extends Thread {
  boolean die = false;
  Object lock = new Object();
  public void kill() {
    synchronized (lock) {
      die = true;
      lock.notifyAll();
    }
  }
  public void run() {
    try {
      synchronized (lock) {
	while (! die) {
	  lock.wait();
	}
      }
    } catch (InterruptedException e) {
    }
  }
}

  SleepyThread st = new SleepyThread();
  st.start();
  // insert benchmark here
  st.kill();

Another special case sometimes used is noting when a program's output is being discarded, and skipping some or all of the I/O processing that takes place before the actual output. At least one workstation vendor applied this trick to the SPEC sc benchmark.

Bait and switch

Bending the rules

Rigging the game

Bogus baselines

Simple error can result from well-intentioned optimization gone awry. For example, an optimizer might more aggressively unroll and optimize a small loop; this would reduce the (mis)calculated net performance of the large loop's body. If, however, the optimizer does a poor job optimizing the smaller calibration loop, the net performance will be overstated instead. In the benchmarking world there are limits to how quickly the big loop can be executed, but the calibration loop can run arbitrarily slowly.

Competing optimizations

• dead code elimination
• constant propagation
• inlining

// Measure allocation/GC performance
for (int i = 0; i < 1000000; i++)
   int[] x = new x[1000];

Constant propagation and inlining often work hand-in-hand with dead code elimination to undermine measurement. For example, supposing that loop were wrapped in a subroutine to "use" the result

static int[] measureGC() {
  // Measure allocation/GC performance
  int[] x = null;
  for (int i = 0; i < 1000000; i++)
     x = new x[1000];
  return x;
  }

So, trying one more time to make that code not be dead, you decide to use a virtual call to invoke it, everybody knows that those can't be inlined, right?

public int[] measureGCVirtual() {
  // Measure allocation/GC performance
  int[] x = null;
  for (int i = 0; i < 1000000; i++)
     x = new x[1000];
  return x;
  }

static void measureGC() {
  Foo f = new Foo();
  f.measureGCVirtual();
  }

Here's another example, take directly from the JavaGrande Fork-Join benchmark:

// do something trivial but which won't be optimised away! 
double theta=37.2, sint, res; 
sint = Math.sin(theta);
res = sint*sint; 
//defeat dead code elimination 
if(res <= 0) System.out.println(
     "Benchmark exited with unrealistic res value " + res);

Lessons learned

The second, and more important lesson, is that when running benchmarks written or selected by someone else, that you should watch your wallet. The best benchmark is always your own application.