Cache prefetching is a technique used by central processing units (CPUs) to boost execution performance by fetching instructions or data from their primary or main storage in slower memory to a faster local memory before it is actually needed. Most modern CPUs have fast and local cache memory in which prefetched data is held until it is required. The source for the prefetch operation is usually main memory. Because of their design, accessing cache memories is typically much faster than accessing main memory. Prefetching can be done with non-blocking cache control instructions. Prefetching is based on the principle of data locality.
Data vs. instruction cache prefetching Cache prefetching can either fetch data or instructions into cache.
Data prefetching fetches data before it is needed. Because data access patterns show less regularity than instruction patterns, accurate data prefetching is generally more challenging than instruction prefetching. Instruction prefetching fetches instructions before they need to be executed. The first mainstream microprocessors to use some form of instruction prefetch were the Intel 8086 (six bytes) and the Motorola 68000 (four bytes). In recent years, all high-performance processors use prefetching techniques.
Hardware vs. software cache prefetching Cache prefetching can be accomplished either by hardware or by software.
Hardware-based prefetching is typically accomplished by having a dedicated hardware mechanism in the processor that watches the stream of instructions or data being requested by the executing program, recognizes the next few elements that the program might need based on this stream, and prefetches into the processor's cache. Software-based prefetching is typically accomplished by having the compiler analyze the code and insert additional "prefetch" instructions in the program during compilation itself.
Methods of hardware prefetching
Stream buffers Stream buffers were developed based on the concept of "one block lookahead (OBL) scheme" proposed by Alan Jay Smith. Stream buffers are one of the most common hardware based prefetching techniques in use. This technique was originally proposed by Norman Jouppi in 1990, and many variations of this method have been developed since. The basic idea is that the cache miss address (and k subsequent addresses) are fetched into a separate buffer of depth k. This buffer is called a stream buffer and is separate from the cache. The processor then consumes data/instructions from the stream buffer if the address associated with the prefetched blocks matches the requested address generated by the program executing on the processor. The figure below illustrates this setup: Whenever the prefetch mechanism detects a miss on a memory block, say A, it allocates a stream to begin prefetching successive blocks from the missed block onward. If the stream buffer can hold 4 blocks, then the processor would prefetch A+1, A+2, A+3, A+4 and hold those in the allocated stream buffer. If the processor consumes A+1 next, then it shall be moved "up" from the stream buffer to the processor's cache. The first entry of the stream buffer would now be A+2 and so on. This pattern of prefetching successive blocks is called Sequential Prefetching. It is mainly used when contiguous locations are to be prefetched. For example, it is used when prefetching instructions. This mechanism can be scaled up by adding multiple such stream buffers, each of which would maintain a separate prefetch stream. For each new miss, there would be a new stream buffer allocated, and it would operate in a similar way as described above. The ideal depth of the stream buffer is subject to experimentation against various benchmarks and depends on the rest of the microarchitecture involved.
Strided prefetching This type of prefetching monitors the delta between the addresses of the memory accesses and looks for patterns within it.
Regular strides In this pattern, consecutive memory accesses are made to blocks that are s addresses apart. In this case, the prefetcher calculates the s and uses it to compute the memory address for prefetching. For example, if s = 4, the address to be prefetched would A+4.
Irregular spatial strides In this case, the delta between the addresses of consecutive memory accesses is variable but still follows a pattern. Some prefetcher designs exploit this property to predict and prefetch for future accesses.
Irregular temporal prefetching This class of prefetchers looks for memory access streams that repeat over time. For example, in the stream of memory accesses N, A, B, C, E, G, H, A, B, C, I, J, K, A, B, C, L, M, N, O, A, B, C, ...; the stream A, B, C is repeating over time. Other design variations have tried to provide more efficient implementations.
Collaborative prefetching Computer applications generate a variety of access patterns. The processor and memory subsystem architectures used to execute these applications further disambiguate the memory access patterns they generate. Hence, the effectiveness and efficiency of prefetching schemes often depends on the application and the architectures used to execute them. Recent research has focused on building collaborative mechanisms to synergistically use multiple prefetching schemes for better prefetching coverage and accuracy.
Methods of software prefetching
Compiler-directed prefetching Compiler-directed prefetching is widely used within loops with a large number of iterations. In this technique, the compiler predicts future cache misses and inserts a prefetch instruction based on the miss penalty and execution time of the instructions. These prefetches are non-blocking memory operations; that is, these memory accesses do not interfere with actual memory accesses. They do not change the state of the processor or cause page faults. One main advantage of software prefetching is that it reduces the number of compulsory cache misses. The following example shows the addition of a prefetch instruction into code to improve cache performance. In the following iteration,
the ith element of the array array1 is accessed. The system can prefetch the elements that are presumably accessed in future iterations by inserting a prefetch instruction as shown below:
… excerpt ends here. Continue reading the full article.
