Large language models (LLMs) are central to modern natural language<br>processing, delivering exceptional performance in various tasks. However, their<br>intensive computational and memory requirements present challenges, especially<br>for devices with limited DRAM capacity. This paper tackles the challenge of<br>efficiently running LLMs that exceed the available DRAM capacity by storing the<br>model parameters on flash memory but bringing them on demand to DRAM. Our<br>method involves constructing an inference cost model that harmonizes with the<br>flash memory behavior, guiding us to optimize in two critical areas: reducing<br>the volume of data transferred from flash and reading data in larger, more<br>contiguous chunks. Within this flash memory-informed framework, we introduce<br>two principal techniques. First, "windowing'" strategically reduces data<br>transfer by reusing previously activated neurons, and second, "row-column<br>bundling", tailored to the sequential data access strengths of flash memory,<br>increases the size of data chunks read from flash memory. These methods<br>collectively enable running models up to twice the size of the available DRAM,<br>with a 4-5x and 20-25x increase in inference speed compared to naive loading<br>approaches in CPU and GPU, respectively. Our integration of sparsity awareness,<br>context-adaptive loading, and a hardware-oriented design paves the way for<br>effective inference of LLMs on devices with limited memory.