Lecture 4: CUDA Programming Model - NYU

CSCI-GA.3033-012 Graphics Processing Units (GPUs): Architecture and Programming

Lecture 4: CUDA Programming Model Mohamed Zahran (aka Z) [email protected] http://www.mzahran.com

Behind CUDA CPU (host) GPU w/ local DRAM (device)

Parallel Computing on a GPU •

8-series GPUs deliver 25 to 200+ GFLOPS on compiled parallel C applications –

GeForce 8800

Available in laptops, desktops, and clusters

• •

GPU parallelism is doubling every year Programming model scales transparently

• •

Programmable in C with CUDA tools Multithreaded SPMD model uses application data parallelism and thread parallelism

Tesla D870

Tesla S870 3

CUDA • Compute Unified Device Architecture • Integrated host+device app C program

– Serial or modestly parallel parts in host C code – Highly parallel parts in device SPMD kernel C code Serial Code (host)‫‏‬

Parallel Kernel (device)‫‏‬ KernelA>(args);

...

Serial Code (host)‫‏‬ Parallel Kernel (device)‫‏‬ KernelB>(args);

... 4

Parallel Threads • A CUDA kernel is executed by an array of threads

– All threads run the same code (SPMD)‫‏‬ – Each thread has an ID that it uses to compute memory addresses and make control decisions threadID

0 1 2 3 4 5 6 7

… float x = input[threadID]; float y = func(x); output[threadID] = y; …

Thread Blocks

• Divide monolithic thread array into multiple blocks – Threads within a block cooperate via shared memory, atomic operations and barrier synchronization – Threads in different blocks cannot cooperate Thread Block 1

Thread Block 0 threadID

0

1

2

3

4

5

6

… float x = input[threadID]; float y = func(x); output[threadID] = y; …

7

0

1

2

3

4

5

6

Thread Block N - 1 7

… float x = input[threadID]; float y = func(x); output[threadID] = y; …

0

…

1

2

3

4

5

6

7

… float x = input[threadID]; float y = func(x); output[threadID] = y; …

IDs •

Each thread uses IDs to decide what data to work on – –

•

Block ID: 1D or 2D Thread ID: 1D, 2D, or 3D

Simplifies memory addressing when processing multidimensional data – – –

Image processing Solving PDEs on volumes …

Host

Device Grid 1

Kernel 1

Block (0, 0)

Block (1, 0)

Block (0, 1)

Block (1, 1)

Grid 2 Kernel 2 Block (1, 1) (0,0,1) (1,0,1) (2,0,1) (3,0,1)

Thread Thread Thread Thread (0,0,0) (1,0,0) (2,0,0) (3,0,0) Thread Thread Thread Thread (0,1,0) (1,1,0) (2,1,0) (3,1,0)

Courtesy: NDVIA

Figure 3.2. An Example of CUDA Thread Org

CUDA Memory Model • Global memory

– Main means of communicating R/W Data between host and device – Contents visible to all threads – Long latency access

Grid Block (0, 0)‫‏‬

Shared Memory

• We will focus on global memory for now

– Constant and texture memory will come later

Registers

Registers

Thread (0, 0)‫ ‏‬Thread (1, 0)‫‏‬

Host

Global Memory

Block (1, 0)‫‏‬

Shared Memory Registers

Registers

Thread (0, 0)‫ ‏‬Thread (1, 0)‫‏‬

CUDA Device Memory Allocation • cudaMalloc()

– Allocates object in the Grid device Global Memory Block (0, 0)‫‏‬ – Requires two parameters • Address of a pointer to the allocated object • Size of of allocated object

• cudaFree()

Host

Shared Memory

Registers

Registers

Thread (0, 0)‫ ‏‬Thread (1, 0)‫‏‬

Block (1, 0)‫‏‬

Shared Memory Registers

Registers

Thread (0, 0)‫ ‏‬Thread (1, 0)‫‏‬

Global Memory

– Frees object from device Global Memory

• Pointer to freed object 9

CUDA Device Memory Allocation Example:

Grid

WIDTH = 64; float* Md int size = WIDTH * WIDTH * sizeof(float);

Block (0, 0)‫‏‬

Shared Memory Registers

cudaMalloc((void**)&Md, size); cudaFree(Md); Host

Registers

Thread (0, 0)‫ ‏‬Thread (1, 0)‫‏‬

Block (1, 0)‫‏‬

Shared Memory Registers

Registers

Thread (0, 0)‫ ‏‬Thread (1, 0)‫‏‬

Global Memory

10

CUDA Device Memory Allocation • cudaMemcpy()‫‏‬

– memory data transfer – Requires four parameters • • • •

Grid Block (0, 0)‫‏‬

Pointer to destination Pointer to source Number of bytes copied Type of transfer – – – –

Host to Host Host to Device Device to Host Device to Device

Shared Memory Registers

Registers

Thread (0, 0)‫ ‏‬Thread (1, 0)‫‏‬

Host

Block (1, 0)‫‏‬

Shared Memory Registers

Registers

Thread (0, 0)‫ ‏‬Thread (1, 0)‫‏‬

Global Memory

• Asynchronous transfer 11

CUDA Device Memory Allocation Example: Destination pointer

Source pointer

Grid

Size in bytes

Block (0, 0)‫‏‬

Direction

Shared Memory Registers

Registers

Thread (0, 0)‫ ‏‬Thread (1, 0)‫‏‬

Host

Block (1, 0)‫‏‬

Shared Memory Registers

Registers

Thread (0, 0)‫ ‏‬Thread (1, 0)‫‏‬

Global Memory

cudaMemcpy(Md, M, size, cudaMemcpyHostToDevice); cudaMemcpy(M, Md, size, cudaMemcpyDeviceToHost); 12

The Hello World of Parallel Programming: Matrix Multiplication N

WIDTH

Data Parallelism: We can safely perform many arithmetic operations on the data structures in a simultaneous manner. P

WIDTH

M

WIDTH

WIDTH

13

The Hello World of Parallel Programming: Matrix Multiplication M0,0 M1,0 M2,0 M3,0 M0,1 M1,1 M2,1 M3,1 M0,2 M1,2 M2,2 M3,2 M0,3 M1,3 M2,3 M3,3

M

M0,0 M1,0 M2,0 M3,0 M0,1 M1,1 M2,1 M3,1 M0,2 M1,2 M2,2 M3,2 M0,3 M1,3 M2,3 M3,3

C adopts raw-major placement approach when storing 2D matrix in linear memory address. 14

The Hello World of Parallel Programming: Matrix Multiplication

A Simple main function: executed at the host

The Hello World of Parallel Programming: Matrix Multiplication // Matrix multiplication on the (CPU) host

void MatrixMulOnHost(float* M, float* N, float* P, int Width)‫ ‏‬N { for (int i = 0; i < Width; ++i)‫‏‬ for (int j = 0; j < Width; ++j) { j double sum = 0; for (int k = 0; k < Width; ++k) { double a = M[i * width + k]; double b = N[k * width + j]; sum += a * b; M P } i P[i * Width + j] = sum; } }

WIDTH

WIDTH

k

k WIDTH

WIDTH

16

The Hello World of Parallel Programming: Matrix Multiplication

The Hello World of Parallel Programming: Matrix Multiplication

The Hello World of Parallel Programming: Matrix Multiplication Nd

WIDTH

k

tx

Md

Pd

ty WIDTH

ty

tx

k

The Kernel Function WIDTH

WIDTH

19

The Hello World of Parallel Programming: Matrix Multiplication __device__ float DeviceFunc() __global__ void KernelFunc() __host__ •

•

Only callable from the: device host

host

host

float HostFunc()

__global__ defines a kernel function

–

•

Executed on the: device device

Must return void

__device__ and __host__ can be used together

For functions executed on the device: • No recursion • No static variable declarations inside the function • No indirect function calls through pointers

Specifying Dimensions // Setup the execution configuration dim3 dimGrid(1, 1); dim3 dimBlock(Width, Width);

// Launch the device computation threads! MatrixMulKernel(Md, Nd, Pd, Width);

Important: • dimGrid and dimBlock are user defined • gridDim and blockDim are built-in predefined variable accessible in kernel functions

Tools C/C++ CUDA Application

CPU Code

NVCC

PTX Code

Virtual

Physical

PTX to Target

Compiler

G80

…

GPU

Conclusions • We are done with chp 3 of the book. • We looked at out first CUDA program • What we learned today about CUDA: – – – – – – –

KernelA>(args) cudaMalloc() cudaFree() cudaMemcpy()‫‏‬ gridDim and blockDim threadIdx.x and threadIdx.y dim3