Non-Blocking Isn’t Parallel: A PyTorch Detail That Matters

Adding non_blocking=True can help a training pipeline—but it doesn’t automatically make data transfers overlap with GPU computation.

Consider this:

batch_gpu = batch_cpu.to("cuda", non_blocking=True)
output = model(batch_gpu)

The subtle question is: non-blocking for whom?

The CPU moves on. The GPU still has an order.

The flag removes PyTorch’s usual CPU-side wait after submitting the copy. It does not create a separate execution stream. PyTorch’s transfer guide.

A CUDA stream is an ordered queue. If the transfer and model execution use the same stream, the copy completes before computation begins. The CPU can submit work ahead of time while the GPU still executes it sequentially.

That dependency is necessary: the model cannot use data that hasn’t arrived.

Overlap happens between different batches

The opportunity is to transfer batch n + 1 while computing on batch n.

For a conventional CPU-to-GPU pipeline, this requires pinned host memory, a separate copy stream, and hardware capable of concurrent copying and computation. Pinned memory keeps the source pages resident in RAM, enabling the asynchronous transfer path. NVIDIA’s explanation.

Before using the next batch, the compute stream must wait for its transfer to finish. Buffers must remain valid until their outstanding operations complete. PyTorch’s stream semantics.

The gain comes from hiding time

Suppose a transfer takes 8 ms and computation takes 12 ms.

Sequential execution takes 20 ms per batch. With ideal overlap, the steady-state interval between completed batches approaches 12 ms—about 1.67× throughput.

These are illustrative numbers, excluding startup and scheduling overhead. Neither operation became faster; one happened while the other was already doing useful work.

Non-blocking removes a CPU-side wait. Correctly scheduled, independent work enables GPU-side overlap.