Java Language Benchmarks Simple JMH example
Example
One of the tools for writing proper benchmark tests is JMH. Let's say we want to compare performance of searching an element in HashSet vs TreeSet.
The easiest way to get JHM into your project - is to use maven and shade plugin. Also you can see pom.xml from JHM examples.
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>/benchmarks</finalName>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.openjdk.jmh.Main</mainClass>
</transformer>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>1.18</version>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>1.18</version>
</dependency>
</dependencies>
After this you need to write benchmark class itself:
package benchmark;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;
import java.util.HashSet;
import java.util.Random;
import java.util.Set;
import java.util.TreeSet;
import java.util.concurrent.TimeUnit;
@State(Scope.Thread)
public class CollectionFinderBenchmarkTest {
private static final int SET_SIZE = 10000;
private Set<String> hashSet;
private Set<String> treeSet;
private String stringToFind = "8888";
@Setup
public void setupCollections() {
hashSet = new HashSet<>(SET_SIZE);
treeSet = new TreeSet<>();
for (int i = 0; i < SET_SIZE; i++) {
final String value = String.valueOf(i);
hashSet.add(value);
treeSet.add(value);
}
stringToFind = String.valueOf(new Random().nextInt(SET_SIZE));
}
@Benchmark
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public void testHashSet(Blackhole blackhole) {
blackhole.consume(hashSet.contains(stringToFind));
}
@Benchmark
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public void testTreeSet(Blackhole blackhole) {
blackhole.consume(treeSet.contains(stringToFind));
}
}
Please keep in mind this blackhole.consume(), we'll get back to it later. Also we need main class for running benchmark:
package benchmark;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
public class BenchmarkMain {
public static void main(String[] args) throws RunnerException {
final Options options = new OptionsBuilder()
.include(CollectionFinderBenchmarkTest.class.getSimpleName())
.forks(1)
.build();
new Runner(options).run();
}
}
And we're all set. We just need to run mvn package (it will create benchmarks.jar in your /target folder) and run our benchmark test:
java -cp target/benchmarks.jar benchmark.BenchmarkMain
And after some warmup and calculation iterations, we will have our results:
# Run complete. Total time: 00:01:21
Benchmark Mode Cnt Score Error Units
CollectionFinderBenchmarkTest.testHashSet avgt 20 9.940 ± 0.270 ns/op
CollectionFinderBenchmarkTest.testTreeSet avgt 20 98.858 ± 13.743 ns/op
About that blackhole.consume(). If your calculations do not change the state of your application, java will most likely just ignore it. So, in order to avoid it, you can either make your benchmark methods return some value, or use Blackhole object to consume it.
You can find more information about writing proper benchmarks in Aleksey Shipilëv's blog, in Jacob Jenkov's blog and in java-performance blog: 1, 2.
