{"product_id":"cuda-for-llms-building-fast-transformer-kernels","title":"CUDA for LLMs: Building fast transformer kernels","description":"\u003cp\u003e\u003cb\u003e\u003cspan class=\"a-text-bold\"\u003eGet the eBook free when you register your print book at Manning.\u003c\/span\u003e\u003cspan\u003e\u003cbr\u003e\u003c\/span\u003e\u003c\/b\u003e\u003c\/p\u003e\n\u003cp\u003e\u003cb\u003e“A fundamental guide for CUDA and modern GPU architecture and their application in the field of AI.”\u003cbr\u003e—Bryce Adelstein Lelbach, NVIDIA\u003c\/b\u003e\u003cbr\u003e\u003cbr\u003e\u003cspan\u003eThe difference between a model that infers in milliseconds instead of seconds, or one that trains in days instead of weeks, is in how effectively you use your hardware. To squeeze every last teraflop out of an NVIDIA GPU, you need to write your own CUDA kernels, and \u003c\/span\u003e\u003ci\u003eCUDA for LLMs \u003c\/i\u003e\u003cspan\u003ewill get you there! This book is a friendly hands-on tour of the modern GPU programming stack, written for the latest NVIDIA hardware, and grounded in fundamentals that will outlast any single generation of chips. In 408 fully-illustrated, code-rich pages, author \u003c\/span\u003e\u003cb\u003eElliot Arledge\u003c\/b\u003e\u003cspan\u003e guides you from your first kernel all the way to Flash Attention, tensor cores, and multi-GPU distributed training.\u003c\/span\u003e\u003cbr\u003e\u003cbr\u003e\u003cspan\u003eGPU programming is complicated, so \u003c\/span\u003e\u003ci\u003eCUDA for LLMs\u003c\/i\u003e\u003cspan\u003e builds your skills one layer at a time using a six-level kernel optimization stack. You’ll start by writing a vector addition kernel, learning how grids, blocks, and threads map onto real hardware. From there, you’ll implement the naive versions of the kernels every deep learning model depends on: GEMM, softmax, 1D and 2D convolution, max pooling, and matrix transpose. Then you’ll build and train a full neural network in six progressively faster implementations and dig into cuBLAS, with backpropagation derived and coded from scratch.\u003c\/span\u003e\u003cbr\u003e\u003cbr\u003e\u003cspan\u003eThroughout this book, your focus will stay on real optimization work. You’ll port a transformer to CUDA inference and attack each kernel in turn. Your GEMV decoder gets coalesced access, block-level reduction, and vectorization. Softmax becomes online, warp-shuffled, and fused. GEMM progresses through thread remapping, shared memory tiling, 1D and 2D blocktiling, and vectorized loads. Every optimization is profiled, and every profile is explained.\u003c\/span\u003e\u003cbr\u003e\u003cbr\u003e\u003cspan\u003eThe final chapters take you to the frontier. You’ll program tensor cores using the WMMA and WGMMA instruction sets for Hopper hardware and see how manual tiling gets replaced by hardware-managed data movement. You’ll implement Flash Attention with tiling and online softmax, quantize models, and explore scaling up to 16 GPUs with tensor and pipeline parallelism over NCCL. By the end, you’ll have both a portfolio of more than 50 working kernels and the mental models to write new ones for hardware that doesn’t exist yet.\u003c\/span\u003e\u003c\/p\u003e\n\u003cdiv class=\"product-page-section\"\u003e\n\u003ch2\u003ewhat's inside\u003c\/h2\u003e\n\u003ca name=\"whats-inside\" class=\"anchor\"\u003e\u003c\/a\u003e\n\u003cul\u003e\n\u003cli\u003eCUDA and GPU programming from the ground up\u003c\/li\u003e\n\u003cli\u003eTensor and pipeline parallelism across multiple GPUs\u003c\/li\u003e\n\u003cli\u003ePyTorch C++ extensions for shipping custom kernels\u003c\/li\u003e\n\u003c\/ul\u003e\n\u003c\/div\u003e\n\u003cdiv class=\"product-page-section\"\u003e\n\u003ch2\u003eAbout the reader\u003c\/h2\u003e\n\u003ca name=\"about-the-reader\" class=\"anchor\"\u003e\u003c\/a\u003eFor software and AI engineers comfortable with C\/C++. No prior CUDA experience required.\u003c\/div\u003e\n\u003cdiv class=\"product-page-section\"\u003e\u003c\/div\u003e\n\u003cdiv class=\"product-page-section\"\u003e\n\u003ch2\u003eAbout the author\u003c\/h2\u003e\n\u003ca name=\"about-the-author\" class=\"anchor\"\u003e\u003c\/a\u003e\u003cb\u003eElliot Arledge\u003c\/b\u003e is a GPU performance engineer and educator. Every kernel in this book comes from his workflow of writing, profiling, and verifying CUDA on both consumer and datacenter hardware. He created freeCodeCamp’s 12-hour CUDA course and consults on deep learning performance.\u003c\/div\u003e","brand":"Manning Publications","offers":[{"title":"Print Book","offer_id":51021641449542,"sku":"9781633434899","price":6205.0,"currency_code":"INR","in_stock":false},{"title":"E-book","offer_id":51021641482310,"sku":"9781633434899","price":4412.0,"currency_code":"INR","in_stock":false}],"thumbnail_url":"\/\/cdn.shopify.com\/s\/files\/1\/0754\/2797\/1142\/files\/81pMZk05eiL._SY385.jpg?v=1794550596","url":"https:\/\/cb-india.com\/products\/cuda-for-llms-building-fast-transformer-kernels","provider":"Computer Bookshop (India) Pvt Ltd","version":"1.0","type":"link"}