Why Separate Loops Can Be Faster Than a Combined Loop in C++: Cache, Memory Bandwidth, and Loop Optimization

By the end of this page, you will understand why two loops that do the same arithmetic work can run at very different speeds in C++. The key ideas are CPU cache, memory bandwidth, instruction-level parallelism, and how compilers optimize loops. You will also learn why performance changes as array sizes move from fitting in L1 cache to L2 cache to main memory.

Modern CPUs are much faster at arithmetic than at fetching data from memory. Because of that, simple array code is often limited not by +, but by how quickly data can be delivered to the CPU.

In the example, each iteration reads and writes multiple arrays:

a1[j] += b1[j];
c1[j] += d1[j];

That means the CPU is touching four streams of memory at once:

• read a1[j]
• read b1[j]
• write a1[j]
• read c1[j]
• read d1[j]
• write c1[j]

Even though the code looks compact, it creates pressure on several hardware resources:

• L1/L2/L3 cache capacity
• cache-line fills and evictions
• load/store bandwidth
• hardware prefetchers
• register pressure and instruction scheduling

When you split the loop:

for (int j = 0; j < n; j++) {
    a1[j] += b1[j];
}
for (int j = 0; j < n; j++) {
    c1[j] += d1[j];
}

you reduce the number of simultaneously active memory streams in each loop. Each loop now focuses on only two arrays for input and one for output. That simpler access pattern can be easier for the CPU cache and prefetch logic to handle.

Why this matters

In real programs, many loops are memory-bound rather than compute-bound. That means:

• performance depends more on data movement than arithmetic
• a small source-code change can produce a large speedup
• understanding cache behavior helps explain "surprising" benchmark results

Main performance factors here

1. Cache working set size

A CPU does not fetch one double at a time from RAM. It fetches a whole cache line (commonly 64 bytes). Since a double is 8 bytes, each line holds 8 doubles.

If your loop touches many arrays at once, the active data may no longer fit comfortably in the fast cache. Once the working set exceeds cache capacity, cache misses increase and performance drops.

2. Number of memory streams

The combined loop walks through four arrays simultaneously. The split version walks through only two source arrays plus one destination per loop. Fewer active streams can improve:

• hardware prefetch efficiency
• cache residency
• TLB behavior
• load/store scheduling

3. Memory bandwidth

If the arrays are large, most execution time may be spent moving cache lines between RAM and cache. In that case, the exact number of bytes transferred and the timing of reuse become critical.

4. Compiler optimization

Compilers can optimize simpler loops more aggressively. Separate loops may:

• unroll more effectively
• schedule loads/stores better
• reduce dependency chains
• vectorize more easily in some cases

5. Cache reuse across outer-loop iterations

If arrays fit in cache, data touched in one pass may still be present for the next pass. Whether that happens depends on:

• array size
• cache level sizes
• associativity
• whether the loop interleaves too many streams

That is why benchmark graphs often show several performance regions rather than one smooth line.

Imagine a chef working at a small kitchen counter.

• The CPU is the chef.
• Registers and L1 cache are the counter space right in front of the chef.
• RAM is the pantry in another room.
• Arrays are ingredient trays.

In the combined loop, the chef tries to work with four trays at once:

• tray a1
• tray b1
• tray c1
• tray d1

The counter gets crowded. The chef keeps moving trays on and off the counter.

In the split-loop version, the chef first works with only:

• a1
• b1

Then with:

• c1
• d1

Less crowding means fewer trips to the pantry and less rearranging.

That is often why separate loops can be faster: not because the arithmetic changed, but because the data flow became easier for the hardware to manage.

Combined loop

for (int j = 0; j < n; j++) {
    a[j] += b[j];
    c[j] += d[j];
}

This performs two independent element-wise additions in one pass.

Separate loops

for (int j = 0; j < n; j++) {
    a[j] += b[j];
}

for (int j = 0; j < n; j++) {
    c[j] += d[j];
}

This performs the same total arithmetic, but with a simpler memory access pattern in each loop.

Small example

#include <iostream>
using namespace std;

int main() {
    const int n = 4;
    double a[n] = {1, 2, 3, 4};
    double b[n] = {10, 20, 30, 40};
    double c[n] = {5, 6, 7, 8};
    double d[n] = {100, 200, 300, 400};

    for (int j = 0; j < n; j++) {
        a[j] += b[j];
    }

    for (int j = 0; j < n; j++) {
        c[j] += d[j];
    }

    for (int j = 0; j < n; j++) {
        cout << a[j] << " ";
    }
    cout << "\n";

    for (int j = 0; j < n; j++) {
        cout << c[j] << " ";
    }
    cout << "\n";
}

Output:

11 22 33 44
105 206 307 408

Important point

Both versions are algorithmically equivalent. The speed difference comes from hardware effects, not fewer arithmetic operations.

Consider this traceable example:

const int n = 3;
double a[3] = {1, 1, 1};
double b[3] = {2, 2, 2};
double c[3] = {3, 3, 3};
double d[3] = {4, 4, 4};

for (int j = 0; j < n; j++) {
    a[j] += b[j];
    c[j] += d[j];
}

Iteration j = 0

• read a[0] = 1
• read b[0] = 2
• compute a[0] = 3
• write a[0] = 3
• read c[0] = 3
• read d[0] = 4
• compute c[0] = 7
• write c[0] = 7

Iteration j = 1

• read a[1] = 1
• read b[1] = 2
• write a[1] = 3
• read c[1] = 3
• read d[1] = 4
• write c[1] = 7

Iteration j = 2

• same pattern

Final values:

a = {3, 3, 3}
c = {7, 7, 7}

Now compare with separate loops:

for (int j = 0; j < n; j++) {
    a[j] += b[j];
}
for (int j = 0; j < n; j++) {
    c[j] += d[j];
}

First loop

• only a and b are actively streamed
• cache lines for these arrays are fetched and used in sequence

Second loop

• only c and d are streamed
• fewer simultaneously active streams in each pass

What changes at the hardware level?

The total math is the same, but the CPU sees a different load/store pattern:

• combined loop: more interleaving
• separate loops: simpler streaming pattern

That can reduce cache conflict and improve prefetch behavior, especially for certain array sizes.

This concept appears in many performance-sensitive programs.

Scientific and numerical computing

Code that updates vectors, matrices, and simulation state often spends most of its time in loops like this.

Example:

position[i] += velocity[i];
energy[i] += delta[i];

Whether these are fused into one loop or split into two can affect throughput.

Image and signal processing

Pixel and sample arrays are processed sequentially. Touching too many arrays at once can hurt cache locality.

Data analytics

Column-oriented processing often performs element-wise transforms across large arrays. Engineers must consider memory traffic, not just arithmetic count.

Game engines

Entity updates often loop over large arrays of component data. Structure-of-arrays layouts and cache-friendly loops matter a lot.

High-performance APIs

Libraries such as BLAS, SIMD kernels, and custom numerics code often tune loops around cache size and bandwidth limits.

In real projects, developers rarely assume that "one loop is always faster" or "two loops are always faster." Instead, they use patterns based on data size and measured performance.

Common patterns

1. Loop fusion

Combine loops when:

• the same data is reused immediately
• the working set still fits in cache
• it reduces overhead and improves locality

Example:

for (int i = 0; i < n; i++) {
    x[i] = a[i] + b[i];
    y[i] = x[i] * scale;
}

This can be good because x[i] is produced and used immediately.

2. Loop fission (loop splitting)

Split loops when:

• too many arrays are touched at once
• vectorization is blocked
• memory pressure is high
• each loop becomes simpler for the compiler

Example:

for (int i = 0; i < n; i++) {
    a[i] += b[i];
}
for (int i = 0; i < n; i++) {
    c[i] += d[i];
}

3. Benchmark-guided optimization

Teams often test multiple versions for:

• small input sizes
• medium sizes that fit in L2/L3
• large sizes that go to DRAM

4. Alignment and layout tuning

Developers may improve performance by:

• aligning arrays
• avoiding aliasing
• using contiguous allocation
• choosing structure-of-arrays over array-of-structures when streaming data

5. Compiler-friendly code

Real codebases often use straightforward loops because simple loops are easier for compilers to optimize reliably.

1. Assuming fewer loops always means faster code

A common beginner assumption is:

• one loop = faster
• two loops = slower

That is not always true on modern CPUs. Memory behavior often matters more than loop count.

2. Ignoring cache effects

This benchmark changes dramatically with array size. A version that is faster for small n may be slower for large n.

3. Benchmarking without enough care

Bad benchmarking can mislead you.

Problems include

• not warming up caches
• timing allocations instead of just the loop
• using too small a workload
• letting the compiler optimize away work
• comparing debug builds instead of optimized builds

4. Forgetting that write operations may trigger extra traffic

When writing to a[j] or c[j], the CPU usually needs the cache line in cache first. This can cause read-for-ownership traffic before the write completes.

5. Expecting the same result on every CPU

Different CPUs have different:

• cache sizes
• cache associativity
• prefetchers
• vector units
• memory subsystem behavior

So a benchmark result on one machine may not generalize.

6. Assuming source-code similarity means assembly similarity

Two C++ loops that look similar may produce very different machine code after optimization.

Broken reasoning example

// This is not guaranteed to be faster just because it uses one loop.
for (int i = 0; i < n; i++) {
    a[i] += b[i];
    c[i] += d[i];
    e[i] += f[i];
    g[i] += h[i];
}

This may overload cache and memory bandwidth.

Better mindset

• write clear code first
• measure with realistic data sizes
• inspect compiler output when performance matters
• think in terms of memory traffic, not just operations

Related optimization ideas

Combined loop vs separate loops

Compute-bound vs memory-bound

This example is usually memory-bound once arrays become large enough.

Quick reference

Core idea

Two loops can be faster than one if splitting them reduces cache pressure and improves memory streaming.

Terms

• Cache line: smallest block moved between memory and cache, often 64 bytes
• Working set: data actively needed during a phase of execution
• Memory-bound: limited by data movement, not arithmetic speed
• Loop fusion: combine loops
• Loop fission: split loops
• Prefetcher: hardware that predicts future memory accesses

Example forms

// fused
for (int i = 0; i < n; i++) {
    a[i] += b[i];
    c[i] += d[i];
}

// split
for (int i = 0; i < n; i++) {
    a[i] += b[i];
}
for (int i = 0; i < n; i++) {
    c[i] += d[i];
}

When split loops may help

• arrays are large
• multiple streams exceed cache comfort zone
• compiler optimizes simpler loops better
• hardware prefetch works better with fewer streams

When fused loops may help

• data reused immediately
• working set fits in cache
• fewer loop-control instructions matter

Rules of thumb

• Measure on the target CPU
• Test multiple array sizes
• Use optimized builds
• Keep benchmark setup separate from timed code
• Inspect generated assembly if needed

Important edge case

A result for one n is not a universal truth. Performance often changes in regions corresponding to L1, L2, L3, and main memory behavior.

Why can two loops be faster than one in C++?

Because the split version may reduce cache pressure, simplify memory access patterns, and allow better compiler scheduling or vectorization.

Is the split-loop version always faster?

No. If the data fits well in cache, a fused loop may be just as fast or faster.

Why does array size matter so much?

Because different sizes fit in different cache levels. Once data spills beyond a cache level, latency and bandwidth costs change.

What is the main bottleneck in this example?

Usually memory traffic, not floating-point addition.

Does contiguous allocation of arrays matter?

Yes. Memory layout can affect spatial locality, cache conflicts, and prefetch behavior.

Why do different CPUs show different graphs?

They have different cache sizes, associativity, prefetch strategies, execution engines, and memory bandwidth.

Should I manually split loops in production code?

Only if profiling shows it helps. Prefer clear code first, then optimize based on measurement.

Does SIMD change the answer?

It can. SIMD often improves throughput, but memory access and cache behavior still remain critical.

• CPU cache — explains why nearby and recently used data is faster to access.
• Memory bandwidth — important when array loops are limited by data transfer speed.
• Cache lines — helps explain why CPUs move blocks of data, not single elements.
• Loop fusion — the optimization of combining loops, which is the direct alternative here.
• Loop fission — the optimization of splitting loops, which is the main concept in this example.
• SIMD vectorization — relevant because compilers may optimize simple array loops with vector instructions.
• Data locality — covers spatial and temporal locality, both central to performance here.
• Compiler optimization — helps explain why simple loops may produce better machine code.
• Benchmarking in C++ — useful because performance conclusions depend on good measurement.
• False sharing and cache coherence — related broader cache topics in multithreaded programs.