🔍 Finding a Hidden GPU Synchronization Bottleneck While profiling DeepSpeed Sequence Parallel training with Nsight Systems, I found an unexpected performance gap inside the compute_loss region....

🔍 Understanding GPU-Initiated Networking (GIN) While GPUDirect RDMA optimized the Data-Plane, the Control-Plane remained tethered to the CPU. GIN overcomes this by shifting the initiation of ne...

🔍 Understanding GPU-Centric Data Movement “The fastest GPU is useless if data cannot reach it efficiently.” This post traces the evolution of GPU data movement layers and explains how modern ar...

🔍 Understanding Warp Scheduling and Latency Hiding “Many warps, four partitions — and only one warp per partition can run each cycle.” This post breaks down how SM partitions actually execute w...

🔍 Understanding Modern Network Encapsulation “They look similar, but their purpose, header structure, and encapsulation scope are completely different.” In this post, we break down how GRE, VXL...

🔍Understanding Modern GPU Virtualization “One GPU, many jobs — but not all sharing is the same.” In this post, we’ll explore how cVIDIA MPS, NVIDIA MIG, and AMD MxGPU differ in design, purpose...

🔍Understanding Allocated Active Blocks per SM (AABS) ① What Is AABS? AABS (Allocated Active Blocks per SM) refers to the number of thread blocks that are simultaneously allocated and actively run...

🔍Understanding Occupancy in CUDA ① What Is Occupancy? Occupancy refers to the ratio between the number of active warps running on a Streaming Multiprocessor (SM) and the maximum number of warps t...

🔍GPU Memory Data Paths — How Bytes Actually Move In the previous post, we explored the structure of the GPU’s memory system — how SMs, caches, controllers, and HBM stacks connect. Figure 1. PCIe...

🔍Understanding the GPU Memory Network In the previous post, we traced how commands travel — from the CUDA driver to SMs. Now, to understand how operands (data) move, we must first understand the G...