24. Prefetching

Memory latency is high → if we prefetch accurately early enough we can reduce/eliminate that latency.

24.1 Prefetching

Prefetching is usually done at cache block granularity.
→ It can reduce both:

• miss rate (if data is in cache when needed)
• miss latency (if data is already on its way when requested).

Correctness

It does not affect correctness, we just wasted effort (unlike speculative execution).

Designing a prefetcher involves answering four fundamental questions:

1. What to prefetch? (Address prediction: Which addresses will be needed?)
2. When to prefetch? (Timing: How early or late should prefetches be initiated for optimal timeliness?)
3. Where to prefetch?
   • Where to place the prefetched data (e.g., L1, L2, L3 cache, or a dedicated prefetch buffer)?
   • Where to locate the prefetcher logic itself in the memory hierarchy?
4. How to prefetch? (Mechanism: Is it software-driven, hardware-driven, execution-based, cooperative, hybrid?)

You can only have a good metric in two of the three easily → tradeoffs.

24.1.1 What (Address Prediction)

Prefetching useless data wastes resources: memory bandwidth, cache/buffer space, and energy.

Accuracy

Accuracy is crucial: Defined as (Used Prefetches) / (Sent Prefetches).

Address prediction can be based on:

• Past access patterns: Hardware prefetchers often learn from recent history.
• Compiler/programmer knowledge: Software prefetching leverages static program structure.

The prefetching algorithm determines what to prefetch. Some predictable access patterns include:

• Sequential/Streaming: A, A+1, A+2, … (Stride = 1)
• Strided: A, A+N, A+2N, … (Constant stride N)
• Complex Regular (e.g., Multi-Stride): Repeating sequence of different strides (e.g., A, A+2, A+5, A+9, A+11, A+14, A+18, … where strides are +2, +3, +4, repeating).
• Correlated/Irregular: Patterns based on data structure traversals (e.g., linked lists, trees) or indirect accesses.
  • Pointer chasing linked list for example

24.1.2 When (Timeliness)

Prefetching too early:

• The prefetched data might be evicted from the cache/buffer before it’s used.
• It occupies resources for a longer, potentially unnecessary, period.

Prefetching too late:

• Might not hide the full memory latency; the CPU still waits.

Timeliness is key. Prefetches should arrive “just in time.”

24.3 Where (Placement)

1. Which level of cache to prefetch into?
   • Prefetching directly into L1 is aggressive and can pollute the small L1 with speculative data.
   • Prefetching into L2 or L3 is more common for hardware prefetchers; these levels are larger and act as a buffer.
   • Prefetchers can also exist between cache levels (e.g., an L2 prefetcher fetching data from L3 into L2).
2. Where to place prefetched data within a cache set?
   • Should prefetched blocks be treated the same as demand-fetched blocks (blocks fetched due to an actual CPU request)?
   • Demand-fetched blocks are known to be needed. Prefetched blocks are speculative.
   • A common LRU policy places new blocks in the MRU position. Placing prefetched blocks there might prematurely evict useful demand-fetched data.
   • Alternative: Place prefetched blocks in the LRU position or a less privileged position to reduce cache pollution.
3. Where to place the hardware prefetcher logic?
   • Placing it closer to the CPU (e.g., observing L1 accesses) allows it to see a more complete access pattern (hits + misses), potentially improving accuracy and coverage. However, it processes more requests, increasing its own complexity and bandwidth demand.
   • Placing it further (e.g., observing L2 misses only) means it sees a filtered stream, which might simplify pattern detection for some patterns but miss others.

24.1.4 How (Mechanism)

24.1.4.1 Software Prefetching

• The ISA provides explicit prefetch instructions (e.g., x86 PREFETCHh).
• The programmer or compiler analyzes the code and inserts these instructions to hint to the hardware about future memory needs.
• Works well for regular, predictable access patterns (e.g., array traversals). Less effective for complex, data-dependent patterns.

24.1.4.2 Hardware Prefetching

• Specialized hardware automatically monitors memory accesses (addresses, PCs).
• It learns patterns (strides, correlations) and generates prefetch addresses.
• Examples: Stream prefetchers, stride prefetchers.

At each level we prefetch different sizes of data, due to differences in available size and latency hits.

Modern systems use stride-based prefetchers that search for a predictable stride in data accesses and then automatically load the next addresses:

• they can be instruction-based, i.e. store deltas per instruction
• memory region based, and detect access patterns distribruted accross instructions, when they access the same memory region

24.1.4.3 Execution-Based Prefetching (e.g., Runahead Execution)

• A separate “helper thread” or the main thread in a speculative mode executes ahead of the normal program execution.
• The primary purpose of this speculative execution is to trigger memory accesses that act as prefetches for the main, non-speculative thread.
• Can be very accurate as it follows the program’s control flow.

24.2 Real Systems

Modern processors often employ multiple, different prefetching mechanisms simultaneously → each covers a different access pattern.

This allows specialisation and better timeliness. But this might saturate memory bandwidth → interfere with each other (pollution, contention).

24.2.2 Multi-Core systems

Coherence: prefetching helps figure out if we need to enable coherence policies early → reduces latency even more.

Efficiency is more important (6 prefetchers per core, 10 cores = 60 prefetchers) → we need to be sure to play nice.
→ The prefetchers interfere with each other = we need to throttle them

24.3 Execution-Based Prefetching

Pre-execute a piece of program, solely to predict what data we need to prefetch = speculative thread approach.
This speculative thread (basically it touches the memory that will be needed, to pull into cache) could be executed:

• on a seperate core
• separate hardware thread context (FMT)
• on the same context as the main program (= runahead execution)

we have branch predictions and cache predictions in the above picture.

24.3.1 Runahead Execution

The instruction at the top needs to be done before we can commit the rest (OoO commit).

If we increase window size, we can prevent the cost of such misses. Thus OoO needs large instruction windows to tolerate today’s main memory latencies.

They are however much more costly to build.

Runahead execution is a technique to obtain the MLP benefits of a large instruction window.

• Entry: When the oldest instruction in the window is a long-latency cache miss (e.g., L2 miss):
  • The processor checkpoints the architectural state (registers).
  • It enters “runahead mode” (a speculative pre-execution mode).
• In Runahead Mode:
  • The processor speculatively pre-executes instructions down the predicted program path, without stalling for most long-latency operations.
  • The primary goal is to generate prefetches for both data and instructions.
  • Instructions dependent on the original long-latency miss (whose values are unknown) are marked as “invalid” (INV) and quickly removed from the window to make space for new instructions. Their results are not trusted and do not update architectural state.
• Exit: When the original long-latency miss (that triggered runahead mode) returns from memory:
  • The processor restores the checkpointed architectural state.
  • The pipeline is flushed.
  • Normal (non-speculative) execution resumes from the instruction that caused the original miss.

Example:

This generates very accurate prefetches (both for regular and irregular patterns) because it follows the programs actual predicted control flow.

• No separate hardware context required.
• can “pretrain” the predictors

However:

• consumes energy
• this is limited by branch prediction accuracy!
  • cannot recover from misprediction
• depends on cache misses
  • cannot prefetch if the data depends on the previously unresolved long-latency miss
• inefficient for pointer-based:
  • cannot parallelise dependent L2 cache misses (always need to wait until we get the next pointer from main…)

Pointer chasing We can optimise the pointer chasing issues:
When we see we cannot compute address → mark as invalid.
To fix this, we predict the actual pointer value!

Actual performance: