Correct Java Micro-Benchmarks with JMH

A reliable Java micro-benchmark measures a very small piece of code while accounting for JVM behavior such as warm-up, JIT compilation, garbage collection, and compiler optimizations. By the end, you will know why JMH is the standard tool for this job, how to choose benchmark modes, and why hand-written stopwatch loops usually produce misleading results.

A micro-benchmark measures the cost of a small operation: for example, parsing one value, looking up one map entry, or calculating a hash.

Java makes this harder than it first appears because code does not always run the same way throughout a program's lifetime:

• Java source is compiled to bytecode first.
• The JVM initially interprets or lightly compiles code.
• The Just-In-Time (JIT) compiler detects frequently executed code and optimizes it while the program runs.
• The optimizer may inline methods, remove unused calculations, eliminate allocations, or specialize code based on observed values.
• Garbage collection, CPU scheduling, cache state, and other running processes can add noise.

A loop timed with System.nanoTime() often measures a mixture of application work and these effects. It can also accidentally let the JIT remove the work being measured.

JMH (Java Microbenchmark Harness) is the standard Java tool for micro-benchmarks. It is designed by OpenJDK contributors and handles important details such as:

• Warm-up iterations before measurement
• Multiple measurement iterations
• Separate JVM forks
• Result consumption to reduce dead-code elimination
• Statistical reporting
• Benchmark modes such as throughput and average time

A benchmark is useful only when it models a meaningful operation and prevents the JVM from optimizing away that operation.

Think of the JVM as a kitchen that learns its busiest orders.

The first few times you order a dish, the cook follows the recipe slowly. After seeing the same dish many times, the cook prepares ingredients in advance and removes unnecessary steps. That is similar to JIT compilation and optimization.

If you start a stopwatch on the first order, you measure training and setup rather than normal service. If you ask the cook to prepare food but throw it away without checking it, the cook may decide not to make it at all. That is similar to dead-code elimination.

JMH lets the kitchen practice first, checks that the dish is actually produced, and then records several normal-service measurements.

JMH benchmarks are methods annotated with @Benchmark.

import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;

@State(Scope.Thread)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class StringLengthBenchmark {

    private String text;

    @org.openjdk.jmh.annotations.Setup
    public void setUp() {
        text = "microbenchmark";
    }

    @Benchmark
    public int stringLength() {
        return text.length();
    }
}

What each important part does:

• @Benchmark marks the method JMH should measure.
• @State(Scope.Thread) gives each benchmark thread its own instance and fields.
• @Setup prepares data outside the measured method.

Consider this benchmark:

import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.Setup;

@State(Scope.Thread)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class ArraySumBenchmark {
    private int[] numbers;

    @Setup
    public void setUp() {
        numbers = new int[] {3, 7, 11, 13};
    }

    @Benchmark
    public int sumNumbers() {
        int sum = 0;
        for (int number : numbers) {
            sum += number;
        }
        return sum;
    }
}

Micro-benchmarks are useful when you have a focused performance question and a realistic workload.

• Data processing: Compare two ways to parse timestamps or convert records in a high-volume import job.
• API services: Measure JSON serialization, validation, token parsing, or a frequently called cache-key function.
• Collections: Compare HashMap, EnumMap, or an array lookup when the key type and access pattern are known.
• Database-adjacent code: Measure object mapping or SQL-string construction separately from actual network and database latency.
• Libraries: Verify whether a change improves a hot method without relying on a full application's noisy performance test.
• Regression prevention: Keep representative JMH benchmarks to detect substantial slowdowns after dependency or implementation changes.

A micro-benchmark does not replace load testing. A web service's real latency also includes networking, database calls, lock contention, request queues, and deployment configuration.

In production codebases, developers usually benchmark a candidate decision, not random snippets.

Benchmark a realistic input

Use @Param to test input sizes or strategies without editing source code each time:

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.Param;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;

@State(Scope.Thread)
public class SearchBenchmark {
    @Param({"10", "1000", "100000"})
    public int size;

    private int[] values;

    @org.openjdk.jmh.annotations.Setup
    public void setUp() {
        values = new int[size];
        for (int i = 0; i < size; i++) {
            values[i] = i;
        }
    }

    @Benchmark
    public int findLastValue() {
        for (int i = ; i < values.length; i++) {
             (values[i] == values.length - ) {
                 i;
            }
        }
         -;
    }
}

Timing a single call

long start = System.nanoTime();
int result = "hello".length();
long elapsed = System.nanoTime() - start;

This is not a useful micro-benchmark. The operation is far smaller than normal timing noise, and the first call may not represent optimized execution.

Avoid it: use JMH, which repeats operations and records multiple measurement samples.

Forgetting warm-up

for (int i = 0; i < 1_000_000; i++) {
    work();
}

Timing this entire loop combines early, unoptimized execution with later optimized execution.

Avoid it: let JMH warm up first. Do not guess a fixed warm-up count in a home-made harness.

Letting work disappear

@Benchmark
public void brokenBenchmark() {
    "hello".toUpperCase();
}

Because the result is unused, the JVM may be able to eliminate or drastically simplify the operation.

• Use JMH for Java micro-benchmarks; avoid hand-written stopwatch loops for small operations.
• Put the measured operation in a method annotated with @Benchmark.
• Use @State(Scope.Thread) for mutable per-thread benchmark data.
• Use @Setup to prepare inputs outside the measured operation.
• Return a result or use Blackhole.consume(value) so work is not optimized away.
• Use @Param to test realistic input sizes and options.
• Use Mode.AverageTime for time per operation.
• Use Mode.Throughput for operations per unit of time.
• Use Mode.SampleTime when latency distribution matters.
• Keep JVM forks enabled unless you have a specific, documented reason not to.
• Benchmark the exact question: include required production work, exclude unrelated work.
• Compare repeatable differences, not tiny changes within noise.

@Benchmark
public int operation() {
    return input.hashCode();
}

@Benchmark
public void operationWithNoReturn {
    blackhole.consume(createValue());
}

What is JMH in Java?

JMH is the Java Microbenchmark Harness. It is a framework for running reliable micro-benchmarks while accounting for JVM warm-up, optimization, and measurement issues.

Why is System.nanoTime() not enough for a Java micro-benchmark?

System.nanoTime() is appropriate for elapsed-time measurement, but a manual benchmark usually fails to handle JIT warm-up, dead-code elimination, multiple samples, forks, and other JVM effects. It is more suitable for coarse timing of larger operations.

Should I measure nanoseconds per operation or operations per second?

Use nanoseconds per operation (AverageTime) when you care about latency of one call. Use operations per second (Throughput) when you care about processing capacity. They are often approximately reciprocal for stable work.

Why does JMH need warm-up iterations?

The JVM optimizes frequently executed code during runtime. Warm-up lets compilation and optimization settle before JMH records the main measurements.

Why should a JMH benchmark return a value?

If a calculation has no observable result, the JVM may remove it as unnecessary. Returning the value, or consuming it through Blackhole, helps ensure the intended work remains observable.

Can I benchmark code that allocates objects?

Yes. Use realistic inputs and consume or return the result. Be aware that escape analysis may remove allocations that cannot be observed; this can be correct if it matches how the code is used in the real application.

Can I run JMH benchmarks from my IDE?

You can, but command-line runs in a stable environment are usually more reproducible. Ensure the benchmark is launched through JMH rather than by directly calling a benchmark method.

• JIT compilation — Explains why Java code changes performance characteristics after it has run repeatedly.
• Dead-code elimination — Explains why unused benchmark results can invalidate a measurement.
• Garbage collection — Allocation and collection behavior can affect benchmark results and application latency.
• Big O notation — Helps predict how algorithms scale, while micro-benchmarks measure actual constants and runtime behavior.
• Java profiling — Profilers identify where an application spends time; micro-benchmarks compare focused implementation choices.
• Load testing — Measures end-to-end system behavior under realistic concurrency and traffic.
• Escape analysis — A JVM optimization that can remove or transform allocations, especially relevant in allocation benchmarks.
• Concurrency in Java — Required when benchmark results depend on contention, shared state, or multi-threaded throughput.