Why GCC -Os Can Beat -O2: Code Alignment and CPU Front-End Effects

-Os is not inherently faster than -O2. In some small, highly repetitive programs, however, its smaller code and reduced alignment padding can happen to produce a more favorable instruction layout for a specific CPU.

This page explains instruction alignment, CPU front-end bottlenecks, why tiny code-layout changes can affect tight loops, and how to measure low-level performance more reliably.

Compiler optimization levels express goals, not guarantees about execution time.

• -O2 enables a broad set of optimizations intended to improve typical performance.
• -O3 adds more aggressive transformations, often increasing code size.
• -Os largely starts from -O2 but avoids optimizations that commonly increase code size, including several code-alignment and block-reordering choices in GCC.

On modern CPUs, execution is not only limited by arithmetic instructions. The processor must first fetch instruction bytes, predict branches, decode instructions, and deliver them to the execution units. This part of the processor is often called the instruction front end.

GCC may insert NOP bytes before a function, loop, jump target, or label so that the following instructions begin at an address boundary such as 16 or 32 bytes. This is instruction alignment. Alignment can help when it lets a frequently executed block fit better into fetch or decode boundaries. But it can also hurt:

• Padding makes the executable larger.
• Moving a loop by a few bytes changes which instructions cross fetch or decode boundaries.
• Moving a call or its target can change branch-prediction and front-end behavior.
• A tiny loop executed hundreds of millions of times magnifies even a very small per-iteration cost.

Therefore, the observed result is plausible: -Os changed the machine-code layout, and that particular layout happened to work better on that CPU. It does not mean that removing alignment is generally faster, or that -Os should normally be preferred for speed.

The example is especially sensitive because it is an artificial, tiny hot loop containing a forced function call. Real programs usually have more work, more varied control flow, and less sensitivity to one exact instruction address.

There is also a C++ correctness issue in the original example: signed int arithmetic eventually overflows, and signed overflow is undefined behavior. A benchmark should use unsigned arithmetic or otherwise ensure values stay in range.

Think of a CPU front end as a delivery entrance that brings instruction packages to workers.

Instruction alignment is like choosing where a conveyor belt begins. Starting a package at a convenient boundary can make delivery smooth. But adding empty space to force that boundary can push the next package into an awkward position.

For a loop that repeats 200 million times, being delayed by even a tiny amount on each trip through the entrance becomes noticeable. Moving a function or adding a few NOPs is like shifting the entire conveyor belt by a few centimeters: sometimes it helps, sometimes it makes the package repeatedly straddle a doorway.

The key idea is that the same instructions at different addresses can have different performance on a real processor.

GCC optimization and alignment options are command-line options:

g++ -O2 benchmark.cpp -o benchmark-o2
g++ -Os benchmark.cpp -o benchmark-os

To investigate whether alignment affects a result, compile an additional variant:

g++ -O2 \
  -fno-align-functions \
  -fno-align-loops \
  benchmark.cpp -o benchmark-o2-no-align

On GCC, -Os disables several optimizations that can increase code size, including alignment-related options such as:

-falign-functions
-falign-jumps
-falign-loops
-falign-labels

A safer small benchmark uses defined unsigned wraparound rather than signed overflow:

#include <cstdint>
#include <cstdlib>

constexpr std::uint32_t loop_bound = 200000000;

[[gnu::noinline]]
std::uint32_t add(std::uint32_t x, std::uint32_t y) {
    return x + y;
}

[[gnu::noinline]]
std::uint32_t work(std::uint32_t x_value, std::uint32_t y_value) {
    std::uint32_t sum = 0;

    for (std::uint32_t i = 0; i < loop_bound; ++i) {
        const std::uint32_t x = x_value + sum;
        const std::uint32_t y = y_value + sum;
        sum += add(x, y);
    }

    return sum;
}

int main(int argc, char** argv) {
    if (argc < 3) {
        return 1;
    }

    const auto x = static_cast<std::uint32_t>(std::strtoul(argv[1], nullptr, 10));
    const auto y = static_cast<std::uint32_t>(std::strtoul(argv[2], nullptr, 10));
    return static_cast<int>(work(x, y));
}

std::uint32_t arithmetic is defined to wrap modulo 2³². The noinline attributes intentionally preserve the call so that the benchmark can study call and layout behavior. In production code, forcing a trivial add function not to inline would usually be undesirable.

Consider this smaller loop:

[[gnu::noinline]]
unsigned add(unsigned x, unsigned y) {
    return x + y;
}

unsigned work(unsigned x_value, unsigned y_value) {
    unsigned sum = 0;

    for (unsigned i = 0; i < 3; ++i) {
        unsigned x = x_value + sum;
        unsigned y = y_value + sum;
        sum += add(x, y);
    }

    return sum;
}

For work(2, 3):

1. sum starts as 0.
2. First iteration: x = 2, y = 3, add(x, y) returns 5, and sum becomes 5.
3. Second iteration: x = 7, y = 8, add(x, y) returns 15, and sum becomes 20.
4. Third iteration: x = 22, y = 23, add(x, y) returns 45, and sum becomes 65.
5. The function returns 65.

At machine-code level, a typical iteration contains instructions to calculate arguments, a call instruction, the few instructions in add, a ret, an addition into sum, a loop counter update, and a conditional branch.

If GCC adds NOP padding before the loop or changes the location of add, every one of those instruction addresses can shift. The source-level execution is identical, but the CPU's fetch, decode, and branch machinery may process the shifted instruction stream differently.

Code layout matters most when a small piece of machine code runs extremely often.

• Serialization and parsing: A JSON, binary-protocol, or text parser may execute a short loop for every byte or field.
• Cryptography and compression: Inner loops in hashing, encryption, decompression, and checksums may be front-end sensitive.
• Game and graphics engines: A math or entity-processing loop can run once per object per frame.
• Databases and analytics: Filtering or aggregating millions of rows can make a small loop hot.
• Runtime systems: Virtual machines, allocators, and garbage collectors contain highly tuned hot paths.

In such code, developers sometimes measure alignment and layout effects. They do so only after profiling has identified a real bottleneck, and they validate results across representative machines and workloads.

In normal codebases, developers should optimize for clear source code and use compiler defaults before applying layout-specific flags.

Useful real-project practices include:

• Profile before changing flags. Use representative inputs and identify hot functions with a profiler.
• Build release code realistically. Include the optimization level, CPU target, link-time options, and libraries used in deployment.
• Use profile-guided optimization when appropriate. PGO provides the compiler with real execution-frequency information, which is more useful than guessing which blocks deserve alignment.
• Keep benchmark work observable. Return a result to a caller that uses it, print a checksum, or use a benchmark framework barrier. Do not let the compiler remove the work.
• Inspect generated code only after measuring. Tools such as objdump -drwC program, Compiler Explorer, and perf annotate can connect source, assembly, and samples.
• Treat alignment flags as experiments. If -fno-align-loops helps one hot loop, test the complete application. It can make other code slower.

For a benchmark suite, record the compiler version, complete flags, CPU model, operating system, input data, and whether frequency scaling was controlled. Low-level measurements are only meaningful when they are reproducible.

Assuming -O3 must always beat -O2

Optimization levels are collections of heuristics and transformations. More aggressive optimization can increase code size, register pressure, compile time, or instruction-cache pressure.

Calling -Os a speed optimization

-Os targets code size. It can win accidentally or on code-size-limited workloads, but it is not a universal speed setting.

Benchmarking undefined behavior

This is unsafe:

int sum = 0;
sum += 2000000000; // Signed overflow is undefined if it exceeds int range.

Use an unsigned type when modular wraparound is intended:

std::uint32_t sum = 0;
sum += 2000000000u; // Defined modulo 2^32.

Allowing the compiler to remove the benchmark

This can disappear entirely under optimization:

for (int i = 0; i < 1000000; ++i) {
    expensive_calculation(i);
}

If the result has no observable effect, the compiler may eliminate it. Make the result observable in a controlled way, such as returning it to a benchmark harness or consuming it through an appropriate framework utility.

Timing one run

One timing can be distorted by CPU frequency changes, interrupts, process migration, cache state, and other activity. Run many iterations, report distributions, and pin down the environment where possible.

Comparing different executables without controlling layout

Adding logging, changing link order, enabling a frame pointer, or merely moving a function can shift code addresses. A result may be a layout effect rather than evidence that a source change is fundamentally better.

Overinterpreting one hardware counter

A counter such as front-end stalled cycles can support a hypothesis, but it is not a complete explanation by itself. Counter names and meanings vary by CPU, events may be approximate, and multiple causes can produce similar measurements.

-Os versus -O2 is not simply “small versus fast.” Smaller code can reduce instruction-cache pressure, while alignment can improve fetch/decode behavior. Which effect wins depends on the CPU, exact addresses, surrounding code, and workload.

A source-level improvement is generally more trustworthy than an accidental layout win. For example, removing unnecessary work from a hot loop is likely to help broadly; changing code only to shift an instruction by 16 bytes may not survive a compiler update or a different build.

• -O2: GCC's common balanced optimization level.
• -O3: adds more aggressive optimizations; can increase code size.
• -Os: favors smaller code and disables some size-growing -O2 optimizations.
• .p2align n: assembler directive that pads until an address is aligned to a power-of-two boundary.
• Alignment padding is usually NOP instructions in executable code.
• A CPU can be front-end limited when instruction fetch, branch prediction, or decode cannot supply instructions quickly enough.
• The same instruction sequence can perform differently at different addresses.
• Test an alignment hypothesis with GCC flags such as:

g++ -O2 -fno-align-functions -fno-align-loops source.cpp

• Do not assume a winning alignment on one binary, compiler version, or CPU will generalize.
• Avoid undefined behavior in benchmarks, especially signed overflow.
• Use repeated runs, representative inputs, and profiling tools such as perf before changing production flags.

Why can -Os be faster than -O2 in GCC?

-Os may emit smaller code and omit alignment padding. On a particular CPU, the resulting instruction addresses can improve instruction fetch or decode behavior in a hot loop.

Does -Os always produce faster programs because they are smaller?

No. -Os prioritizes size and may skip transformations that improve runtime. Its performance depends on the workload and hardware.

What does .p2align mean in GCC assembly output?

It asks the assembler to insert padding until the next code location reaches a power-of-two alignment boundary. For example, an alignment exponent of 4 commonly means alignment to 16 bytes.

Are NOP instructions always bad for performance?

No. NOP padding can help by aligning hot code favorably. It can also hurt by increasing code size or shifting instructions into an unfavorable layout.

Why does moving a function change a benchmark result?

Moving a function changes its address and may change the alignment of the caller, call instruction, branch targets, and surrounding code. CPUs can be sensitive to these positions in tiny hot loops.

How should I benchmark a C++ micro-optimization?

Use defined behavior, prevent dead-code elimination, perform many repeated measurements, control the machine environment, inspect generated assembly when needed, and verify the result on representative hardware.

Should I disable GCC loop alignment globally?

Usually not. It is a target-dependent compiler heuristic that may benefit other code. Disable it only as a measured experiment, then evaluate the whole workload.

Is perf enough to prove an alignment explanation?

No. perf provides useful evidence, including samples and hardware-counter data, but results should be interpreted carefully and combined with controlled binary-layout experiments.

• Compiler optimization levels — Understand the different goals and trade-offs of -O0, -O2, -O3, and -Os.
• Instruction cache — Smaller code can reduce pressure on the CPU cache that stores instructions.
• CPU instruction pipeline and front end — Explains fetching, decoding, and delivering instructions for execution.
• Branch prediction — Calls, returns, and loop branches rely on prediction hardware that can affect throughput.
• Function inlining — Inlining removes call overhead but can also increase code size and change layout.
• Profile-guided optimization — Lets the compiler use real workload profiles to guide layout and optimization decisions.
• C++ undefined behavior — Signed integer overflow can invalidate assumptions made by a benchmark.
• Microbenchmarking — Covers reliable measurement techniques for very small code changes.
• Linux perf — Helps profile hot code and inspect performance-counter evidence.