Advanced OpenMP Tutorial - RWTH-Aachen

Advanced OpenMP Tutorial Performance and 4.5 Features Christian Terboven Michael Klemm Eric Stotzer Bronis R. de Supinski

1

Advanced OpenMP Tutorial – OpenMP Overview Christian Terboven

Agenda  OpenMP Overview (60 min.)  Techniques to Obtain High Performance with OpenMP: Memory Access (30 min.)

 Techniques to Obtain High Performance with OpenMP: Vectorization (45 min.)

updated slides:

 OpenMP for Attached Compute Accelerators (60 min.)  Future OpenMP Directions (15 min.) 2


http://bit.ly/isc16-openmp

Advanced OpenMP Tutorial OpenMP Overview Christian Terboven Michael Klemm Eric Stotzer Bronis R. de Supinski

3


OpenMP Overview - Topics  Core Concepts  Synchronization  Tasking  Data Scoping with Tasking  Cancellation  Misc. OpenMP 4.0/4.5 Features

4


Core Concepts

5


What is OpenMP?  De-facto standard Application Programming Interface (API) to write shared memory parallel applications in C, C++, and Fortran  Consists of Compiler Directives, Runtime routines and Environment variables  Version 4.5 has been released July 2015 6


OpenMP Platform Features Cluster

Group of computers communicating through fast interconnect

Coprocessors/Accelerators

Node

Group of processors communicating through shared memory

Socket

Group of cores communicating through shared cache

Core

7

Special compute devices attached to the local node through special interconnect

Group of functional units communicating through registers

Hyper-Threads

Group of thread contexts sharing functional units

Superscalar

Group of instructions sharing functional units

Pipeline

Sequence of instructions sharing functional units

Vector

Single instruction using multiple functional units

Overview of OpenMP Michael Klemm

Defining Parallelism in OpenMP A parallel region is a block of code executed by all threads in the team #pragma omp parallel [clause[[,] clause] ...] { "this code is executed in parallel" } // End of parallel section

(note: implied barrier)

!$omp parallel [clause[[,] clause] ...] "this code is executed in parallel" !$omp end parallel

8


(note: implied barrier)

The OpenMP Execution Model Master Thread Parallel region

Fork and Join Model

Worker Threads Synchronization

Parallel region

Worker Threads

Synchronization

9


Nested Parallelism Master Thread

3-way parallel

Outer parallel region

9-way parallel

Nested parallel region

3-way parallel

Note: Nesting level can be arbitrarily deep 10


Outer parallel region

The Worksharing Constructs #pragma omp for { .... }

#pragma omp sections { .... }

#pragma omp single { .... }

!$OMP DO .... !$OMP END DO

!$OMP SECTIONS .... !$OMP END SECTIONS

!$OMP SINGLE .... !$OMP END SINGLE

      11

The work is distributed over the threads Must be enclosed in a parallel region Must be encountered by all threads in the team, or none at all No implied barrier on entry Implied barrier on exit (unless the nowait clause is specified) A work-sharing construct does not launch any new threads Advanced OpenMP Tutorial – OpenMP Overview Christian Terboven

The Single Directive Only one thread in the team executes the code enclosed #pragma omp single [private][firstprivate] \ [copyprivate][nowait] { } !$omp single [private][firstprivate] !$omp end single [copyprivate][nowait]

12


Additional Directives - Misc #pragma omp master {} !$omp master !$omp end master

There is no implied barrier on entry or exit !

#pragma omp flush [(list)] Don’t use the list

!$omp flush [(list)] #pragma omp ordered {} !$omp ordered !$omp end ordered 13


 Use for debugging only

Additional Directives - Updates #pragma omp critical [(name)] {} !$omp critical [(name)] !$omp end critical [(name)] #pragma omp atomic [clause] !$omp atomic [clause]

Very useful to avoid data races

Useful for simple updates

Clauses on the atomic: read, write, update, capture, seq_cst

14


Synchronization

15


The OpenMP Memory Model private memory

private memory

T

T T

Shared Memory

private memory

private memory

T

T private memory

16


 All threads have access

to the same, globally shared memory  Data in private memory is only accessible by the thread owning this memory  No other thread sees the change(s) in private memory  Data transfer is through shared memory and is 100% transparent to the application

Gotcha’s  Need to get this right Part of the learning curve

 Private data is undefined on entry and exit Can use firstprivate and lastprivate to address this

 Each thread has its own temporary view on the data Applicable to shared data only Means different threads may temporarily not see the same value for the same variable ... Let me explain

17


The Flush Directive Thread A

Thread B

X = 0

X = 1

while (X == 0) { “wait” }

If shared variable X is kept within a register, the modification may not be made visible to the other thread(s) 18


About The Flush  Strongly recommended: do not use this directive … unless really necessary. Really . Could give very subtle interactions with compilers If you insist on still doing so, be prepared to face the OpenMP language lawyers

 Implied on many constructs A good thing This is your safety net

19


The OpenMP Barrier  Several constructs have an implied barrier This is another safety net (has implied flush by the way)

 In some cases, the implied barrier can be left out through the “nowait” clause  This can help fine tuning the application But you’d better know what you’re doing

 The explicit barrier comes in quite handy then #pragma omp barrier

20


!$omp barrier

The Nowait Clause  To minimize synchronization, some directives support the optional nowait clause If present, threads do not synchronize/wait at the end of that particular construct

 In C, it is one of the clauses on the pragma  In Fortran, it is appended at the closing part of the construct #pragma omp for nowait !$omp do { : : : } !$omp end do nowait 21


Tasking

22


Sudoko for Lazy Computer Scientists  Lets solve Sudoku puzzles with brute multi-core force (1) Find an empty field (2) Insert a number (3) Check Sudoku (4 a) If invalid: Delete number, Insert next number (4 b) If valid: Go to next field

23


The OpenMP Task Construct C/C++

Fortran

#pragma omp task [clause] ... structured block ...

!$omp task [clause] ... structured block ... !$omp end task

 Each encountering thread/task creates a new task Code and data is packaged up Tasks can be nested Into another task directive Into a worksharing construct

 Data scoping clauses:  shared(list)  private(list)

firstprivate(list)

 default(shared 24

| none)


Barrier and Taskwait Constructs  OpenMP barrier (implicit or explicit) All tasks created by any thread of the current Team are guaranteed to be completed at barrier exit C/C++

#pragma omp barrier

 Task barrier: taskwait Encountering task is suspended until child tasks are complete Applies only to direct childs, not descendants! C/C++ #pragma omp taskwait 25


Parallel Brute-force Sudoku  This parallel algorithm finds all valid solutions first call contained in a #pragma omp parallel #pragma omp single such that one tasks starts the execution of the algorithm

(1) Search an empty field (2) Insert a number (3) Check Sudoku (4 a) If invalid: Delete number, Insert next number

#pragma omp task needs to work on a new copy of the Sudoku board

26

#pragma omp taskwait Advanced OpenMP Tutorial – OpenMP wait for allOverview child tasks Christian Terboven

(4 b) If valid: Go to next field Wait for completion

Performance Evaluation Sudoku on 2x Intel Xeon E5-2650 @2.0 GHz speedup: Intel C++ 13.1, scatter binding

8

4,0

7

3,5

6

3,0

5

2,5

Is this the best we can can do?

4 3

1,5

2

1,0

1

0,5

0

0,0

1

2

3

4

5

6

7

8

#threads

29

2,0


9

10

11

12

16

24

32

Speedup

Runtime [sec] for 16x16

Intel C++ 13.1, scatter binding

Performance Analysis Event-based profiling gives a good overview :

Tracing gives more details: lvl 6 Duration: 0.16 sec lvl 12

Every thread is executing ~1.3m tasks…

Duration: 0.047 sec lvl 48 Duration: 0.001 sec

lvl 82 Duration: 2.2 μs

… in ~5.7 seconds. => average duration of a task is ~4.4 μs 30


Tasks get much smaller down the call-stack.

Performance Analysis Event-based profiling gives a good overview :

Tracing gives more details: lvl 6

Performance and Scalability Tuning Idea: If you have created Duration: 0.16 sec sufficiently many tasks to make you cores busy, stop creating more tasks! lvl 12 • if-clause Every thread is executing ~1.3m tasks… Duration: 0.047 sec • final-clause, mergeable-clause lvl 48 • natively in your program code Duration: 0.001 sec

Example: stop recursion

lvl 82 Duration: 2.2 μs

… in ~5.7 seconds. => average duration of a task is ~4.4 μs 31


Tasks get much smaller down the call-stack.

Performance Evaluation Intel C++ 13.1, scatter binding

Intel C++ 13.1, scatter binding, cutoff

speedup: Intel C++ 13.1, scatter binding

speedup: Intel C++ 13.1, scatter binding, cutoff

8

18

7

16 14

6

12

5

10

4 8

3

6

2

4

1

2

0

0

1

2

3

4

5

6

7

8

#threads

32


9

10

11

12

16

24

32

Speedup

Runtime [sec] for 16x16

Sudoku on 2x Intel Xeon E5-2650 @2.0 GHz

Data Scoping with Tasking

33


Tasks in OpenMP: Data Scoping  Some rules from Parallel Regions apply: Static and global variables are shared Automatic storage (local) variables are private

 If shared scoping is not inherited: Orphaned task variables are firstprivate by default! Non-orphaned task variables inherit the shared attribute! → Variables are firstprivate unless shared in the enclosing context

34


Data Scoping Example (1/7) int a = 1; void foo() { int b = 2, c = 3; #pragma omp parallel shared(b) #pragma omp parallel private(b) { int d = 4; #pragma omp task { int e = 5; // // // // //

Scope Scope Scope Scope Scope

of of of of of

a: b: c: d: e:

} } } 35




of of of of of

a: shared b: c: d: e:

} } } 36




of of of of of

a: shared b: firstprivate c: d: e:

} } } 37




of of of of of

a: shared b: firstprivate c: shared d: e:

} } } 38




of of of of of

a: b: c: d: e:

} } } 39


shared firstprivate shared firstprivate



of of of of of

a: b: c: d: e:

} } } 40


shared firstprivate shared firstprivate private



of of of of of

a: b: c: d: e:

} } } 41


shared, firstprivate, shared, firstprivate, private,

value value value value value

of of of of of

a: b: c: d: e:

1 0 / undefined 3 4 5

Use default(none)! int a = 1; Hint: Use default(none) to be void foo() forced to think about every { variable if you do not see clearly. int b = 2, c = 3; #pragma omp parallel shared(b) default(none) #pragma omp parallel private(b) default(none) { int d = 4; #pragma omp task { int e = 5; // // // // //


of of of of of

a: b: c: d: e:

} } } 42


shared firstprivate shared firstprivate private

Tasks in OpenMP: Scheduling  Default: Tasks are tied to the thread that first executes them → not neccessarily their creator (the generating thread). Scheduling constraints: Only the thread to which a task is tied can execute it A task can only be suspended at task scheduling points Task creation, task finish, taskwait, barrier, taskyield

If task is not suspended in a barrier, executing thread can only switch to a direct descendant of all tasks tied to the thread

 Tasks created with the untied clause are never tied Resume at task scheduling points possibly by different thread No scheduling restrictions, e.g., can be suspended at any point But: More freedom to the implementation, e.g., load balancing 43


if Clause  If the expression of an if clause on a task evaluates to false The encountering task is suspended The new task is executed immediately The parent task resumes when new tasks finishes → Used for optimization, e.g., avoid creation of small tasks

44


The taskyield Directive  The taskyield directive specifies that the current task can be suspended in favor of execution of a different task. Hint to the runtime for optimization and/or deadlock prevention

45

C/C++

Fortran

#pragma omp taskyield

!$omp taskyield


taskyield Example (1/2) #include void something_useful(); void something_critical(); void foo(omp_lock_t * lock, int n) { for(int i = 0; i < n; i++) #pragma omp task { something_useful(); while( !omp_test_lock(lock) ) { #pragma omp taskyield } something_critical(); omp_unset_lock(lock); } } 46


taskyield Example (2/2) #include void something_useful(); void something_critical(); void foo(omp_lock_t * lock, int n) { for(int i = 0; i < n; i++) #pragma omp task { something_useful(); while( !omp_test_lock(lock) ) { #pragma omp taskyield } something_critical(); omp_unset_lock(lock); } } 47


The waiting task may be suspended here to allow the executing thread to perform other work. This may also avoid deadlock situations.

Task Dependencies

48


The depend Clause C/C++ #pragma omp task depend(dependency-type: list) ... structured block ...

 The task dependence is fulfilled when the predecessor task has completed in dependency-type: the generated task will be a dependent task of all previously generated sibling tasks that reference at least one of the list items in an out or inout clause. out and inout dependency-type: The generated task will be a dependent task of all previously generated sibling tasks that reference at least one of the list items in an in, out, or inout clause. The list items in a depend clause may include array sections. 49


Concurrent Execution w/ Dep. Degree parallism in exploitable in this concrete example:  Note:ofvariables the depend clause do not necessarily T2 have and T3to(2indicate tasks), T1 next flow iteration has to wait for them theofdata T1 has to be completed void process_in_parallel() { before T2 and T3 can be #pragma omp parallel executed. #pragma omp single {

T2 and T3 can be executed in parallel.

int x = 1; ... for (int i = 0; i < T; ++i) { #pragma omp task shared(x, ...) depend(out: x) preprocess_some_data(...); #pragma omp task shared(x, ...) depend(in: x) do_something_with_data(...); #pragma omp task shared(x, ...) depend(in: x) do_something_independent_with_data(...); } } // end omp single, omp parallel } 50


// T1 // T2 // T3

The taskgroup Construct

54


The taskgroup Construct C/C++

Fortran

#pragma omp taskgroup ... structured block ...

!$omp taskgroup ... structured block ... !$omp end task

 Specifies a wait on completion of child tasks and their descendent tasks „deeper“ sychronization than taskwait, but with the option to restrict to a subset of all tasks (as opposed to a barrier)

 Main use case for now in OpenMP 4.0: Cancellation… 55


Cancellation

56


OpenMP 3.1 Parallel Abort  Parallel execution cannot be aborted in OpenMP 3.1 Code regions must always run to completion (or not start at all)

 Cancellation in OpenMP 4.0 provides a best-effort approach to terminate OpenMP regions Best-effort: not guaranteed to trigger termination immediately Triggered “as soon as” possible

57


Cancellation Constructs  Two constructs: Activate cancellation: C/C++: Fortran:

#pragma omp cancel !$omp cancel

Check for cancellation: C/C++: Fortran:

#pragma omp cancellation point !$omp cancellation point

 Check for cancellation only a certain points Avoid unnecessary overheads Programmers need to reason about cancellation Cleanup code needs to be added manually

58


Cancellation Semantics Thread A

Thread B

Thread C

parallel region

59



Thread B

Thread C

parallel region

60



Thread B

Thread C

parallel region

61



Thread B

Thread C

parallel region

62


cancel Construct  Syntax: #pragma omp cancel construct-type-clause [ [, ]if-clause] !$omp cancel construct-type-clause [ [, ]if-clause]

 Clauses: parallel sections for (C/C++) do (Fortran) taskgroup if (scalar-expression)

 Semantics Requests cancellation of the inner-most OpenMP region of the type specified in construct-type-clause Allows the encountering thread/task proceed to the end of the canceled region 63


cancellation point Construct  Syntax: #pragma omp cancellation point construct-type-clause !$omp cancellation point construct-type-clause

 Clauses: parallel sections for (C/C++) do (Fortran) taskgroup

 Semantics Introduces a user-defined cancellation point Pre-defined cancellation points: implicit/explicit barriers regions cancel regions 64


Cancellation Example subroutine example(n, dim) integer, intent(in) :: n, dim(n) integer :: i, s, err real, allocatable :: B(:) err = 0 !$omp parallel shared(err) ... !$omp do private(s, B) do i=1, n allocate(B(dim(i)), stat=s) !$omp cancellation point if (s .gt. 0) then !$omp atomic write err = s !$omp cancel do endif ...

65


! ... example continued ! deallocate private array B in ! normal condition deallocate(B) enddo ! deallcoate any private array that ! has already been allocated if (err .gt. 0) then if (allocated(B)) then deallocate(B) endif endif !$omp end parallel end subroutine

Cancellation of OpenMP Tasks  Cancellation only acts on tasks grouped by the taskgroup construct The encountering tasks jumps to the end of its task region Any executing task will run to completion (or until they reach a cancellation point region) Any task that has not yet begun execution may be discarded (and is considered completed)

 Tasks cancellation also occurs, if a parallel region is canceled But not if cancellation affects a worksharing construct.

66


Task Cancellation Example binary_tree_t* search_tree_parallel(binary_tree_t* tree, int value) { binary_tree_t* found = NULL; #pragma omp parallel shared(found,tree,value) { #pragma omp master { #pragma omp taskgroup { found = search_tree(tree, value); } } } return found; }

67


Task Cancellation Example binary_tree_t* search_tree( binary_tree_t* tree, int value, int level) { binary_tree_t* found = NULL; if (tree) { if (tree->value == value) { found = tree; } else { #pragma omp task shared(found) { binary_tree_t* found_left; found_left = search_tree(tree->left, value); if (found_left) { #pragma omp atomic write found = found_left; #pragma omp cancel taskgroup } }

68


#pragma omp task shared(found) { binary_tree_t* found_right; found_right = search_tree(tree->right, value); if (found_right) { #pragma omp atomic write found = found_right; #pragma omp cancel taskgroup } } #pragma omp taskwait } } return found; }

Misc. OpenMP 4.0/4.1 Features

69


User Defined Reductions

70

Advanced Advanced OpenMP OpenMP Tutorial Tutorial –– TITLE OpenMP OF YOUR Overview TALK YOUR Christian NAME Terboven

User Defined Reductions (UDRs) expand OpenMP’s usability  Use declare reduction directive to define operators  Operators used in reduction clause like predefined ops #pragma omp declare reduction (reduction-identifier : typename-list : combiner) [initializer(initializer-expr)]

 reduction-identifier gives a name to the operator Can be overloaded for different types Can be redefined in inner scopes

 typename-list is a list of types to which it applies  combiner expression specifies how to combine values  initializer specifies the operator’s identity value initializer-expression is an expression or a function 71


A simple UDR example  Declare the reduction operator #pragma omp declare reduction (merge : std::vector : omp_out.insert(omp_out.end(), omp_in.begin(), omp_in.end()))

 Use the reduction operator in a reduction clause void schedule (std::vector &v, std::vector &filtered) { #pragma omp parallel for reduction (merge : filtered) for (std:vector::iterator it = v.begin(); it < v.end(); it++) if ( filter(*it) ) filtered.push_back(*it); }

 Private copies created for a reduction are initialized to the identity that was specified for the operator and type  Default identity defined if identity clause not present

 Compiler uses combiner to combine private copies  omp_out refers to private copy that holds combined value  omp_in refers to the other private copy 72


Atomics

73

Advanced Advanced OpenMP OpenMP Tutorial Tutorial –– TITLE OpenMP OF YOUR Overview TALK YOUR Christian NAME Terboven

The atomic construct supports effcient parallel accesses  Use atomic construct for mutually exclusive access to a single memory location #pragma omp atomic [read|write|update|capture] [seq_cst] expression-stmt

 expression-stmt restricted based on type of atomic  update, the default behavior, reads and writes the single memory location atomically  read reads location atomically  write writes location atomically  capture updates or writes location and captures its value (before or after update) into a private variable 74


3.1 atomic operation additions address an obvious deficiency  Previously could not capture a value atomically int schedule (int upper) { static int iter = 0; int ret; ret = iter; #pragma omp atomic iter++; if (ret size; i++) { c->data[i] = a->data[i] * b->data[i]; } }

16

Advanced OpenMP Tutorial – Performance: Vectorization Michael Klemm

In a Time Before OpenMP 4.0  Support required vendor-specific extensions Programming models (e.g., Intel® Cilk Plus) Compiler pragmas (e.g., #pragma vector) Low-level constructs (e.g., _mm_add_pd()) #pragma omp parallel for #pragma vector always #pragma ivdep for (int i = 0; i < N; i++) { a[i] = b[i] + ...; }

17


You need to trust your compiler to do the “right” thing.

SIMD Loop Construct  Vectorize a loop nest Cut loop into chunks that fit a SIMD vector register No parallelization of the loop body

 Syntax (C/C++) #pragma omp simd [clause[[,] clause],…] for-loops

 Syntax (Fortran) !$omp simd [clause[[,] clause],…] do-loops [!$omp end simd] 18


Example void sprod(float *a, float *b, int n) { float sum = 0.0f; #pragma omp simd reduction(+:sum) for (int k=0; k