The Speed Cliffs Between Layers
Modern CPUs have multiple cache levels between the registers and main memory. L1 (on-core, ~32KB) hits in 1 nanosecond. L2 (on-core, ~256KB) hits in 4 nanoseconds. L3 (shared, ~8MB) hits in 12 nanoseconds. Main RAM (8-16GB) hits in 100 nanoseconds. Each jump is an order of magnitude slower. An L1 miss that lands in L2 costs 3 extra nanoseconds; an L1 miss that goes to RAM costs 99 extra nanoseconds. That is a 99x slowdown from poor cache placement.
The hierarchy reflects physical reality. L1 is closest to the CPU core, using expensive silicon real estate. L3 is shared across cores to reduce cost. RAM is off-chip, facing the speed of light and PCB traces. The CPU is designed to predict and prefetch likely accesses into L3, but L1 and L2 are fast enough that the CPU can only react to misses, not proactively fill them.
Predictable Access Patterns Win
Sequential memory access (reading array elements in order) is predictable: the CPU prefetches the next cache line before you need it. Random access is unpredictable: you pay the miss cost every time. Tight loops that fit in L1 run at CPU speed; loops that touch random memory run at RAM speed.
Modern optimization is largely cache optimization. Reduce your working set size to fit in L1 or L2. Access memory sequentially. Keep hot data together. Tile algorithms to process small blocks that fit in cache before moving to the next block. Profile tools (like perf on Linux) show you cache misses, revealing where your access patterns fail. Fixing cache misses often yields bigger performance gains than algorithmic improvements.