SOLUTIONS MANUAL for Multicore and GPU Programming An Integrated Approach 2nd Edition by Gerassimos

Multicore & GPU Programming : An Integrated Approach, 2e Instructor’s Manual Gerassimos Barlas June 15, 2022

Contents Contents

2

1 Introduction

5

2 Multicore and Parallel Program Design

9

3 Threads and Concurrency in standard C++

13

4 Parallel data structures

57

5 Distributed memory programming

61

6 GPU Programming

117

7 GPU and Accelerator Programming : OpenCL

143

8 Shared-memory programming : OpenMP

169

9 The Thrust Template Library

183

10 High-level multi-threaded programming with the Qt library

199

11 Load Balancing

205

3

for more solution manuals,visit Library Genesis: libgen.is, libgen.st, libgen.rs, and forum.mhut.org

Chapter 1

Introduction Exercises 1. Study one of the top 10 most powerful supercomputers in the world. Discover:

What kind of operating system does it run?

How many CPUs/GPUs is it made of?

What is its total memory capacity?

What kind of software tools can be used to program it?

Answer Students should research the answer by visiting the Top 500 site and -if available- the site of one of the reported systems.

2. How many cores are inside the top GPU offerings from NVidia and AMD? What is the GFlop rating of these chips?

Answer N/A. 3. The performance of the most powerful supercomputers in the world is usually reported as two numbers Rpeak and Rmax, both in TFlops (tera floating point operations per second) units. Why is this done? What are the factors reducing performance from Rpeak to Rmax? Would it be possible to ever achieve Rpeak?

Answer This is done because the peak performance is unattainable. Sustained, measured performance on specific benchmarks, is a better indicator of the true machine potential.

The reason these are different is communication overhead.

Rpeak and Rmax could never be equal. Extremely compute-heavy applications, that have no inter-node communications, could asymptotically approach Rpeak if they were to run for a very long time. A very long execution time is required to diminish the influence of the start-up costs.

5

6

CHAPTER 1. INTRODUCTION 4. A sequential application with a 20% part that must be executed sequentially, is required to be accelerated five-fold. How many CPUs are required for this task?

Answer This requires the application of Amdahl’s law. The part that can be parallelized is α = 1 − 20% = 80%. The speedup predicted by Amdahl’s 1 law is speedup = 1−α+ α .

N

Achieving a three-fold speedup requires that: 1 1 0.8 1 0.8 = − 0.2 ⇒ N = 1 =3⇒ =6 α =3⇒ 1−α+ N N 3 0.2 + 0.8 − 0.2 N 3 (1.1) Achieving a 5-fold speedup requires that: 1 0.8 0.8 =∞ = 0.8 = 5 ⇒ N = 1 0 0.2 + N 5 − 0.2

(1.2)

So, it is impossible to achieve a 5-fold speedup, according to Amdahl’s law.

5. A parallel application running on 5 identical machines, has a 10% sequential part. What is the speedup relative to a sequential execution on one of the machines? If we would like to double that speedup, how many CPU would be required?

Answer This requires the application of Gustafson-Barsis’ law as the information relates to a parallel application. The parallel part is α = 1 − 10% = 90%.

The speedup over a single machine is speedup = 1−α+N ·α = .1+5·0.9 = 4.6.

9.1 = 10.1 Doubling the speedup would require .1 + N · 0.9 = 9.2 ⇒ N = 0.9 machines. As N has to be an integer, we have to round-up to the closest integer, i.e. N = 11.

6. An application with a 5% non-parallelizable part, is to be modified for parallel execution. Currently on the market there are two parallel machines available: machine X with 4 CPUs, each CPU capable of executing the application in 1hr on its own, and, machine Y with 16 CPUs, with each CPU capable of executing the application in 2hr on its own. Which is the machine you should buy, if the minimum execution time is required?

Answer As the information provided relates to a sequential application, we have to apply Amdahl’s law. The execution time for machine X is: tX = (1 − α)T +

0.95 · 1hr αT = 0.05 ∗ 1hr + = 0.2875hr N 4

(1.3)

The execution time for machine Y is: tY = (1 − α)T +

αT 0.95 · 2hr = 0.05 ∗ 2hr + = 0.21875hr N 16

(1.4)

7 So we should buy machine Y. 7. Create a simple sorting application that uses the mergesort algorithm to sort a large collection (e.g. 107 ) of 32-bit integers. The input data and output results should be stored in files, and the I/O operations should be considered a sequential part of the application. Mergesort is an algorithm that is considered appropriate for parallel execution, although it cannot be equally divided between an arbitrary number of processors, as Amdahl’s and Gustafson-Barsis’ laws require.

Assuming that this equal division is possible, estimate α, i.e. the part of the program that can be parallelized, by using a profiler like gprof or valgrind to measure the duration of mergesort’s execution relative to the overall execution time. Use this number to estimate the predicted speedup for your program.

Does α depend on the size of the input? If it does, how should you modify your predictions and their graphical illustration?

Answer N/A 8. A parallel application running on 10 CPUs, spends 15% of its total time, in sequential execution. What kind of CPU (how much faster) would we need to run this application completely sequentially, while keeping the same total time?

Answer This is an application of Gustafson-Barsis’ law. If T is the parallel execution time on the 10 CPUs, the sequential execution time on a single one would be Ts = (1 − α)T + N · α · T . The fast CPU should match the parallel time, i.e. Tf = T which means if should be TTfs = (1 − α) + N · α = 0.15 + 10 · 0.85 = 8.65 times faster.

Chapter 2

Multicore and Parallel Program Design 1. Perform a 2D agglomeration step for the image convolution problem of Section 2.2. What is the resulting number of communication operations?

Answer Assuming that we target a grid of task groups forming K rows x M columns and that K and M divide the corresponding dimensions evenly, each group will hold IMMGX IMKGY tasks.

The number of communication operations are:

Four for “internal” groups. There are (K −1)(M −1) internal groups.

Each group sends IMMGX pixel values to its top and bottom neighbors, and IMKGY pixels to its left and right neighbors.

Two for corner groups. There are four corner groups.

Three for “boundary” groups. There are 2(K−2)+2(M −2) boundary groups.

The total data volume communicated is : totalComm = (2

IM GY IM GX +2 )(K − 1)(M − 1) + M K IM GY IM GX + )2(M − 2) + (2 M K IM GX IM GY ( +2 )2(K − 2) + M K IM GX IM GY ( + )4 M K

(2.1) (2.2) (2.3) (2.4)

2. Perform the comparison between the 1D and 2D decompositions of the heat diffusion example in Section 2.3.3, by assuming that (a) half-duplex communication links are available and (b) n-port communications are possible, i.e. all communications can take place at the same time over all the links.

Answer 9

10

CHAPTER 2. MULTICORE AND PARALLEL PROGRAM DESIGN (a) If half-duplex communication links where used, then the time spend on communication would be doubled: comp1D + comm1D =

comp2D + comm2D =

N2 · tcomp + 4 · (tstart + tcomm N ) P

N N2 tcomp + 8 tstart + tcomm · √ P P

(2.5)

which means that comm1D + comp1D < comm2D + comp2D ⇒ tstart > tcomm N

2 1− √ P

(2.6)

So there is no change in the condition that favors 1D over 2D. (b) In the n-port case, the communication time per time step will be: comm1D = tstart + tcomm N N comm2D = tstart + tcomm · √ P so the comparison between the two decompositions would be based on comm1D + comp1D < comm2D + comp2D ⇒

√ P <1

which is obviously false. So, 2D is always better than 1D. 3. How would communication costs affect the pipeline performance? Derive variations of Equations 2.16 to 2.18 that take into account a constant communication overhead between the pipeline stages.

Answer Let’s assume that communication between the stages costs a fixed amount of time tc . Then we would have an additional overall time of tc (N + M ), assuming that the last stage also sends data back to whoever is controlling the execution. Thus: ttotal =

l−1 X j=0

tj + N · tl +

M −1 X

tj + tc (N + M )

j=l+1

The processing rate of the pipeline is: rate = Pl−1

j=0 tj + N · tl +

The latency of the pipeline is: latency =

N PM −1

M −2 X j=0

j=l+1 tj + tc (N + M )

tj + tc M

11 4. The total number of tasks calculated in Section 2.4.5 for the parallel quicksort of Listing 2.8, is based on the best-case assumption that the input is split in equal halves by every call to the PartitionData function. What would be the result if the worst-case (i.e. one part gets N − 1 elements and the other part 0) were considered?

Answer If we assume that the PartitionData function produces a zero sized part and a part with N − 1 elements, then: T (N ) =

0 1 + T (N − 1)

if N ≤ T HRES if N > T HRES

as one of the calls to QuickSort would always have no input.

Backward substitution can provide the answer: T (N ) = 1 + T (N − 1) = 1 + 1 + T (N − 2) = k + T (N − k) T (N − k) can be eliminated when N − k = T HRES ⇒ k = N − T HRES. Substituting this value of k in the previous equation yields: T (N ) = N − T HRES + T (T HRES) = N − T HRES as T (T HRES) = 0. This is obviously a poor result. What this formula does not convey is that parallelism is also eliminated as a result: there is effectively no overlap between the generated tasks.

5. Use a simple problem instance (e.g. a small array of integers) to trace the execution of the parallel quicksort of Listing 2.8. Create a Gantt graph for the tasks generated, assuming an infinite number of compute nodes is available for executing them. Can you calculate an upper bound for the speedup that can be achieved?

Answer Given N input elements, PartitionData executes between N −1 and N +1 key comparisons. For simplicity we will assume that N comparisons are done and that the array is evenly split into parts equal in size to N 2−1 (taking out the pivot element).

The first initial task that starts the sorting operation, will perform N comparisons, before spawning a task for N 2−1 elements and continuing with the remaining N 2−1 . The comparisons are done exclusively inside the PartitionData function, which means we have the following sequence of task executions and corresponding comparison steps:

1 task, N steps 2 tasks, N 2−1 steps each 2

+1 22 tasks, N −2 steps each 22 3

+1 23 tasks, N −2 steps each 23

And so on...

12

CHAPTER 2. MULTICORE AND PARALLEL PROGRAM DESIGN Size of data block to be sorted N −3 N −1 N 2 4

Threads

0 2 1 3 Time

Figure 2.1: The first three steps (assuming an ideal scenario of a perfect partition each time) in the parallel quicksort algorithm of Listing 2.8.

Figure 2.1 illustrates this process.

So, the overall duration in number of comparisons is : totalComp =

L−1 X

N − 2i + 1 + T HRES 2i i=0

where L is associated with when the size become smaller or equal to T HRES and no more tasks are spawned.

The value of L is: N − 2L + 1 = T HRES ⇒ N − 2L + 1 = 2L T HRES ⇒ 2L N +1 N +1 2L = ⇒ L = lg( ) T HRES + 1 T HRES + 1 Thus: totalComp =

L−1 X

N − 2i + 1 + T HRES = 2i i=0 L−1 X

L−1 1 i X ( ) − (N + 1) 1 + T HRES = 2 i=0 i=0

( 1 )L − 1 − L + T HRES = (N + 1) 21 2 −1 T HRES + 1 2(N + 1)(1 − ) − L + T HRES = N +1 2(N + 1) − 2(T HRES + 1) − L + T HRES = 2N − T HRES − L ≈ 2N So the maximum speedup that could be ever achieved, given that the sequential quicksort performs N lgN comparisons, is: speedupmax =

N lgN lgN = 2N 2

Chapter 3

Threads and Concurrency in standard C++ Exercises 1. Enumerate and create the other timing diagrams that show the alternatives of Figure 3.4, when it comes to the final balance of the bank account.

Answer All the possible permutations of the four events are allowed, as long as 1 proceeds 3 and 2 proceeds 4. So we have:

1, 2, 3, 4 : produces wrong result 1, 3, 2, 4 : produces correct result 1, 2, 4, 3 : produces wrong result 2, 1, 3, 4 : produces wrong result 2, 1, 4, 3 : produces wrong result 2, 4, 1, 3 : produces correct result 2. Research the term “fork bomb” and write a program that performs as such.

Answer #i n c l u d e < s t d l i b . h> #i n c l u d e <u n i s t d . h> #i n c l u d e < l i m i t s . h> i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t N = atoi ( argv [ 1 ] ) ; f o r ( i n t i =0; i<N ; i++) pid_t cID = fork ( ) ; sleep ( INT_MAX ) ; return 0; }

13

14 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ 3. Modify the producer-consumer example shown in Listing 3.11, so that the threads terminate after the number 100 is generated.

Answer The only modification required affects the consume method, where the detection of the exit condition is done. It should become: bool consume ( int i ) { // t o be implemented c o u t << ”@” ; // j u s t t o show s o m e t h i n g i s h a p p e n i n g i f ( i == 1 0 0 ) r e t u r n t r u e ; e l s e return f a l s e ; }

4. Suggest a modification to the program of Listing 3.12 so that the IntegrCalc threads can use any function that return a double and takes a double as a parameter.

Answer A pointer to such a function needs to be passed to the initClass method, so that it can be called by the threads. The modifications are highlighted in the following code: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

. . . //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // a c t s a s a consumer c l a s s IntegrCalc : public QThread { private : i n t ID ; s t a t i c QSemaphore * slotsAvail ; s t a t i c QSemaphore * resAvail ; s t a t i c QMutex l2 ; s t a t i c QMutex resLock ; s t a t i c Slice * buffer ; s t a t i c i n t out ; s t a t i c double * result ; s t a t i c double (* f ) ( double ) ; // <======= s t a t i c QSemaphore numProducts ; public : s t a t i c v o i d i n i t C l a s s ( Q S e m a p h o r e * s , Q S e m a p h o r e * a , S l i c e * b , ←֓ double *r , double (* f ) ( double ) ) ;

18 19 20 21 22 23 24 25

I n t e g r C a l c ( i n t i ) : ID ( i ) { }; void run ( ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− . . . double (* IntegrCalc : : f ) ( double ) ; // <=======

26 27

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

28 29

30 31 32 33 34 35 36 37

v o i d I n t e g r C a l c : : i n i t C l a s s ( Q S e m a p h o r e * s , Q S e m a p h o r e * a , S l i c e * b ←֓ , d o u b l e * res , d o u b l e ( * g ) ( d o u b l e ) ) { // <======= slotsAvail = s ; resAvail = a ; buffer = b ; result = res ; * result = 0; f = g ; // <======= } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

38 39 40 41 42 43 44

void IntegrCalc : : run ( ) { while (1) { r e s A v a i l −>a c q u i r e ( ) ; // w a i t f o r an a v a i l a b l e i t e m l2 . lock ( ) ; i n t tmpOut = out ; o u t = ( o u t + 1 ) % B U F F S I Z E ; // update t h e o u t i n d e x

15 l2 . unlock ( ) ;

45 46

// t a k e t h e i t e m o u t double st = buffer [ tmpOut ] . start ; double en = buffer [ tmpOut ] . end ; double div = buffer [ tmpOut ] . divisions ;

47 48 49 50 51

s l o t s A v a i l −>r e l e a s e ( ) ; // s i g n a l

52

f o r a new empty s l o t

53

if

54

( d i v == 0 ) b r e a k ; // e x i t

55

// c a l c u l a t e a r e a double localRes = 0 ; double step = ( en − st ) / div ; double x ; x = st ; l o c a l R e s = f ( st ) + f ( en ) ; // <======= l o c a l R e s /= 2 ; f o r ( i n t i =1; i< d i v ; i++) { x += s t e p ; l o c a l R e s += f ( x ) ; // <======= } l o c a l R e s *= s t e p ;

56 57 58 59 60 61 62 63 64 65 66 67 68

// add i t t o r e s u l t resLock . lock () ; * r e s u l t += l o c a l R e s ; resLock . unlock () ;

69 70 71 72 73 74 75

} } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

76 77 78 79 80 81 82 83 84 85 86 87

i n t main ( i n t argc , char * argv [ ] ) { i f ( a r g c == 1 ) { c e r r << ” Usage ” << a r g v [ 0 ] << ” #t h r e a d s #j o b s \n” ; exit (1) ; } i n t N = atoi ( argv [ 1 ] ) ; i n t J = atoi ( argv [ 2 ] ) ; S l i c e * b u f f e r = new S l i c e [ B U F F S I Z E ] ; QSemaphore avail , buffSlots ( BUFFSIZE ) ; i n t in = 0 ; double result ;

88

I n t e g r C a l c : : i n i t C l a s s (& b u f f S l o t s , &a v a i l , b u f f e r , &r e s u l t , ←֓ func ) ; // <=======

89

90 91

.

.

.

5. In a remote region of Siberia there are single tracks joining railroad stations. Obviously only one train can use a piece of track between two stations. The other trains can wait at the stations before they do their crossings. The following graph indicates the track and station layout: A

B

C

D

E

F

G

Write a Qt program that simulates the journey of 3 trains with the following schedules:

A→B→E→C

D→B→E→G

C→E→B→D→F

16 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ As each trains arrives at a station display a relative message. You can assume that a station can hold any number of trains waiting.

Answer The Train class constructor expects two vectors : one containing the names of the stations that will be traversed, and one containing references to the binary semaphores controlling access to the lines connecting the stations.

1 2 3 4 5

#i n c l u d e <QThread> #i n c l u d e <QMutex> #i n c l u d e <i o s t r e a m > #i n c l u d e <s t r i n g > #i n c l u d e <v e c t o r >

6 7

u s i n g namespace s t d ;

8 9 10 11 12 13 14 15 16

17 18

c l a s s Train : public QThread { private : i n t ID ; v e c t o r <s t r i n g > * s t a t i o n s ; v e c t o r <Q M u t e x *> * l i n e L o c k s ; public : T r a i n ( i n t i , v e c t o r <s t r i n g > * s , v e c t o r <Q M u t e x *> * l ) : I D ( i ) , ←֓ s t a t i o n s ( s ) , l i n e L o c k s ( l ) {} void run ( ) ; };

19 20 21 22 23 24 25

26 27

28 29 30

31

void Train : : run ( ) { int i ; f o r ( i =0; i<l i n e L o c k s −>s i z e ( ) ; i++) { c o u t << ” T r a i n ” << I D << ” i s i n s t a t i o n ” << s t a t i o n s −>←֓ a t ( i ) << e n d l ; l i n e L o c k s −>a t ( i )−>l o c k ( ) ; c o u t << ” T r a i n ” << I D << ” i s t r a v e l i n g t o s t a t i o n ” << ←֓ s t a t i o n s −>a t ( i +1) << e n d l ; l i n e L o c k s −>a t ( i )−>u n l o c k ( ) ; } c o u t << ” T r a i n ” << I D << ” a r r i v e d i n s t a t i o n ” << s t a t i o n s ←֓ −>a t ( i ) << e n d l ; }

32 33 34 35 36 37 38 39

i n t main ( i n t argc , char * argv [ ] ) { Q M u t e x AB , BD , DF , BE , CE , E G ; // m u t i c e s f o r t h e l i n e s // one f o r e a c h t r a i n v e c t o r <Q M u t e x *> r o u t e s [ 3 ] ; v e c t o r <s t r i n g > s t [ 3 ] ;

40 41 42 43 44 45 46 47

r o u t e s [ 0 ] . p u s h _ b a c k (& A B ) ; r o u t e s [ 0 ] . p u s h _ b a c k (& B E ) ; r o u t e s [ 0 ] . p u s h _ b a c k (& C E ) ; s t [ 0 ] . p u s h _ b a c k ( ”A” ) ; s t [ 0 ] . p u s h _ b a c k ( ”B” ) ; s t [ 0 ] . p u s h _ b a c k ( ”E” ) ; s t [ 0 ] . p u s h _ b a c k ( ”C” ) ;

48 49 50 51 52 53 54 55

r o u t e s [ 1 ] . p u s h _ b a c k (& B D ) ; r o u t e s [ 1 ] . p u s h _ b a c k (& B E ) ; r o u t e s [ 1 ] . p u s h _ b a c k (& E G ) ; s t [ 1 ] . p u s h _ b a c k ( ”D” ) ; s t [ 1 ] . p u s h _ b a c k ( ”B” ) ; s t [ 1 ] . p u s h _ b a c k ( ”E” ) ; s t [ 1 ] . p u s h _ b a c k ( ”G” ) ;

56 57 58 59

r o u t e s [ 2 ] . p u s h _ b a c k (& C E ) ; r o u t e s [ 2 ] . p u s h _ b a c k (& B E ) ; r o u t e s [ 2 ] . p u s h _ b a c k (& B D ) ;

17 r o u t e s [ 2 ] . p u s h _ b a c k (& D F ) ; s t [ 2 ] . p u s h _ b a c k ( ”C” ) ; s t [ 2 ] . p u s h _ b a c k ( ”E” ) ; s t [ 2 ] . p u s h _ b a c k ( ”B” ) ; s t [ 2 ] . p u s h _ b a c k ( ”D” ) ; s t [ 2 ] . p u s h _ b a c k ( ”F” ) ;

60 61 62 63 64 65 66

// t h r e a d spawning Train *t [ 3 ] ; f o r ( i n t i =0; i <3; i++) { t [ i ] = new T r a i n ( i , &s t [ i ] , &r o u t e s [ i ] ) ; t [ i]−> s t a r t ( ) ; }

67 68 69 70 71 72 73 74

f o r ( i n t i =0; i <3; i++) t [ i]−> w a i t ( ) ; return 0;

75 76 77 78

}

6. Modify the program of the previous exercise so that each station can hold only 2 trains. Can this lead to deadlocks?

If you have not done so already, make sure that your program uses only one thread class.

Answer This cannot lead to deadlocks as long as we have three or less trains.

For four or more we could have pairs of trains trying to cross opposite directions of the same line, leading to deadlock.

The only required changes involves controlling access to the stations via a set of general semaphores.

1 2 3 4 5 6

#i n c l u d e <QThread> #i n c l u d e <QMutex> #i n c l u d e <QSemaphore> #i n c l u d e <i o s t r e a m > #i n c l u d e <s t r i n g > #i n c l u d e <v e c t o r >

7 8

u s i n g namespace s t d ;

9 10 11 12 13 14 15 16 17 18

19 20

c l a s s Train : public QThread { private : i n t ID ; v e c t o r <s t r i n g > * s t a t i o n s ; v e c t o r <Q S e m a p h o r e *> * s t a t i o n s C t r l ; v e c t o r <Q M u t e x *> * l i n e L o c k s ; public : T r a i n ( i n t i , v e c t o r <s t r i n g > * s , v e c t o r <Q M u t e x *> * l , v e c t o r <←֓ Q S e m a p h o r e *> * c ) : I D ( i ) , s t a t i o n s ( s ) , l i n e L o c k s ( l ) , ←֓ s t a t i o n s C t r l ( c ) {} void run ( ) ; };

21 22 23 24 25 26 27 28

29

void Train : : run ( ) { int i ; s t a t i o n s C t r l −>a t ( 0 )−>a c q u i r e ( ) ; f o r ( i =0; i<l i n e L o c k s −>s i z e ( ) ; i++) { c o u t << ” T r a i n ” << I D << ” i s i n s t a t i o n ” << s t a t i o n s −>←֓ a t ( i ) << e n d l ; s t a t i o n s C t r l −>a t ( i +1)−>a c q u i r e ( ) ; // r e s e r v e s t a t i o n ←֓ b e f o r e l o c k i n g the l i n e

30 31

l i n e L o c k s −>a t ( i )−>l o c k ( ) ;

18 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ c o u t << ” T r a i n ” << I D << ” i s t r a v e l i n g t o s t a t i o n ” << ←֓ s t a t i o n s −>a t ( i +1) << e n d l ; l i n e L o c k s −>a t ( i )−>u n l o c k ( ) ;

32

33 34

s t a t i o n s C t r l −>a t ( i )−>r e l e a s e ( ) ; // r e l e a s e p r e v i o u s ←֓ station reservation c o u t<<I D <<” ”<< s t a t i o n s C t r l −>a t ( i )−>a v a i l a b l e ( ) << e n d l ←֓ ;

35

36

} c o u t << ” T r a i n ” << I D << ” a r r i v e d i n s t a t i o n ” << s t a t i o n s ←֓ −>a t ( i ) << e n d l ;

37 38

39

}

40 41 42 43 44 45 46 47 48 49

i n t main ( i n t argc , char * argv [ ] ) { Q M u t e x AB , BD , DF , BE , CE , E G ; QSemaphore A (2) , B (2) , C (2) , D (2) , E (2) , F (2) , G (2) ; // one f o r e a c h t r a i n v e c t o r <Q M u t e x *> r o u t e s [ 3 ] ; v e c t o r <Q S e m a p h o r e *> s t C a p a c [ 3 ] ; v e c t o r <s t r i n g > s t [ 3 ] ;

50

r o u t e s [ 0 ] . p u s h _ b a c k (& A B ) ; r o u t e s [ 0 ] . p u s h _ b a c k (& B E ) ; r o u t e s [ 0 ] . p u s h _ b a c k (& C E ) ; s t [ 0 ] . p u s h _ b a c k ( ”A” ) ; s t [ 0 ] . p u s h _ b a c k ( ”B” ) ; s t [ 0 ] . p u s h _ b a c k ( ”E” ) ; s t [ 0 ] . p u s h _ b a c k ( ”C” ) ; s t C a p a c [ 0 ] . p u s h _ b a c k (&A ) ; s t C a p a c [ 0 ] . p u s h _ b a c k (&B ) ; s t C a p a c [ 0 ] . p u s h _ b a c k (&E ) ; s t C a p a c [ 0 ] . p u s h _ b a c k (&C ) ;

51 52 53 54 55 56 57 58 59 60 61 62

r o u t e s [ 1 ] . p u s h _ b a c k (& B D ) ; r o u t e s [ 1 ] . p u s h _ b a c k (& B E ) ; r o u t e s [ 1 ] . p u s h _ b a c k (& E G ) ; s t [ 1 ] . p u s h _ b a c k ( ”D” ) ; s t [ 1 ] . p u s h _ b a c k ( ”B” ) ; s t [ 1 ] . p u s h _ b a c k ( ”E” ) ; s t [ 1 ] . p u s h _ b a c k ( ”G” ) ; s t C a p a c [ 1 ] . p u s h _ b a c k (&D ) ; s t C a p a c [ 1 ] . p u s h _ b a c k (&B ) ; s t C a p a c [ 1 ] . p u s h _ b a c k (&E ) ; s t C a p a c [ 1 ] . p u s h _ b a c k (&G ) ;

63 64 65 66 67 68 69 70 71 72 73 74

r o u t e s [ 2 ] . p u s h _ b a c k (& C E ) ; r o u t e s [ 2 ] . p u s h _ b a c k (& B E ) ; r o u t e s [ 2 ] . p u s h _ b a c k (& B D ) ; r o u t e s [ 2 ] . p u s h _ b a c k (& D F ) ; s t [ 2 ] . p u s h _ b a c k ( ”C” ) ; s t [ 2 ] . p u s h _ b a c k ( ”E” ) ; s t [ 2 ] . p u s h _ b a c k ( ”B” ) ; s t [ 2 ] . p u s h _ b a c k ( ”D” ) ; s t [ 2 ] . p u s h _ b a c k ( ”F” ) ; s t C a p a c [ 2 ] . p u s h _ b a c k (&C ) ; s t C a p a c [ 2 ] . p u s h _ b a c k (&E ) ; s t C a p a c [ 2 ] . p u s h _ b a c k (&B ) ; s t C a p a c [ 2 ] . p u s h _ b a c k (&D ) ; s t C a p a c [ 2 ] . p u s h _ b a c k (&F ) ;

75 76 77 78 79 80 81 82 83 84 85 86 87 88 89

Train *t [ 3 ] ; f o r ( i n t i =0; i <3; i++) { t [ i ] = new T r a i n ( i , &s t [ i ] , &r o u t e s [ i ] , &s t C a p a c [ i ] ) ; t [ i]−> s t a r t ( ) ; }

90 91 92 93 94 95 96

f o r ( i n t i =0; i <3; i++) t [ i]−> w a i t ( ) ; return 0;

97 98 99 100

}

19 7. A desktop publishing (like PageMaker) application has two threads running: one for running the GUI and one for doing background work. Simulate this application in Qt. Your implementation should have the thread corresponding to the GUI, send requests to the other thread to run tasks on its behalf. The tasks should be (obviously just printing a message is enough for the simulation):

Printing Mail merging PDF generation After performing each requested task, the second thread should wait for a new request to be send to it. Make sure that the first thread does not have to wait for the second thread to finish before making new requests.

Answer The following solution is a producer-consumer derivative, where just two counting semaphores are needed for each pair of interacting threads. There are two buffers, one for handling the interaction between GUI and mailmerge and one for the interaction between GUI and PDF generation.

1 2 3 4 5

#i n c l u d e <QSemaphore> #i n c l u d e <QThread> #i n c l u d e < s t d l i b . h> #i n c l u d e <u n i s t d . h> #i n c l u d e <i o s t r e a m >

6 7

u s i n g namespace s t d ;

8 9 10 11

c o n s t i n t T E R M I N F L A G = −1; const i n t RUNS = 20; const int BUFFSIZE = 5;

12 13 14 15 16 17

// t h e f o l l o w i n g s h o u l d n o t be g l o b a l v a r i a b l e s // but t h e y a r e s o t o r e d u c e t h e l e n g t h o f t h e c o d e QSemaphore mailReq (0) , mailSpace ( BUFFSIZE ) ; int mailJob [ BUFFSIZE ] ; // b u f f e r i n t m_in = 0 , mout = 0;

18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

QSemaphore pdfReq (0) , pdfSpace ( BUFFSIZE ) ; int pdfJob [ BUFFSIZE ] ; // b u f f e r i n t pin = 0 , pout = 0 ; //============================================= c l a s s GUI : p u b l i c QThread { public : void run ( ) ; }; //−−−−−−−−−−−−−− void GUI : : run ( ) { f o r ( i n t i = 0 ; i < R U N S ; i++) { i n t choice = rand () % 2; i f ( choice ) // m a i l merge { mailSpace . acquire () ; mailJob [ m_in ] = i ; m_in = ( m_in + 1) % BUFFSIZE ; mailReq . release () ; } else // PDF g e n e r a t i o n { pdfSpace . acquire () ; pdfJob [ pin ] = i ; pin = ( pin + 1) % BUFFSIZE ;

20 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ pdfReq . release () ;

46

}

47

} // t e r m i n a t i o n mailSpace . acquire () ; mailJob [ m_in ] = TERMINFLAG ; mailReq . release () ;

48 49 50 51 52 53

pdfSpace . acquire () ; pdfJob [ pin ] = TERMINFLAG ; pdfReq . release () ;

54 55 56 57

}

58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78

//============================================= c l a s s MailMerge : public QThread { public : void run ( ) ; }; //−−−−−−−−−−−−−− void MailMerge : : run ( ) { while (1) { mailReq . acquire () ; i n t job = mailJob [ mout ] ; mout = ( mout + 1) % BUFFSIZE ; i f ( j o b == T E R M I N F L A G ) break ; c o u t << ” Mail merge j o b #” << j o b << e n d l ; mailSpace . release () ; } }

79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

//============================================= c l a s s PDFGen : public QThread { public : void run ( ) ; }; //−−−−−−−−−−−−−− void PDFGen : : run ( ) { while (1) { pdfReq . acquire () ; i n t job = pdfJob [ pout ] ; pout = ( pout + 1) % BUFFSIZE ; i f ( j o b == T E R M I N F L A G ) break ; c o u t << ”PDF g e n e r a t i o n j o b #” << j o b << e n d l ; pdfSpace . release () ; } }

100 101 102 103 104 105 106 107 108 109

//============================================= i n t main ( i n t argc , char * argv [ ] ) { GUI g ; PDFGen p ; MailMerge m ; g . start () ; p . start () ; m . run ( ) ; // m. s t a r t ( ) ; // can u s e run ←֓ i n s t e a d o f s t a r t t o a v o i d s t a r t i n g an e x t r a t h r e a d

110

g . wait () ; p . wait () ; // m. w a i t ( ) ; // s e e above return 0;

111 112 113 114 115

}

8. A popular bakery has a baker that cooks a loaf of bread at a time and

21 deposits it on a counter. Incoming customers pick up a loaf from the counter and exit the bakery. The counter can hold 20 loafs. If it is full the baker stops baking bread. If it is empty, a customer waits. Use semaphores to solve the coordination problem between the baker and the customers.

Answer This is an instance of the producer-consumers problem. Because the baker thread is producing just simple integers in sequence, we can use a QAtomicInt variable for retrieving loaf IDs on the customers’ side.

1 2 3 4 5

#i n c l u d e <QThread> #i n c l u d e <QSemaphore> #i n c l u d e <QAtomicInt> #i n c l u d e <i o s t r e a m > #i n c l u d e < s t d l i b . h>

6 7

u s i n g namespace s t d ;

8 9

c o n s t i n t S P A C E =20;

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

48 49 50 51 52 53 54 55 56 57 58 59 60

QSemaphore loafs (0) , counterSpace ( SPACE ) ; QAtomicInt loafID ; //============================ c l a s s Baker : public QThread { private : int totalBread ; public : B a k e r ( i n t t ) : t o t a l B r e a d ( t ) {} void run ( ) ; }; //−−−−−−−−−−−−−−−−−− void Baker : : run ( ) { f o r ( i n t i =0; i<t o t a l B r e a d ; i++) { counterSpace . acquire () ; c o u t << ” Baker baked # ” << i << e n d l ; loafs . release () ; } } //============================ c l a s s Customer : public QThread { private : i n t ID ; public : C u s t o m e r ( i n t i ) : I D ( i ) {} void run ( ) ; }; //−−−−−−−−−−−−−−−−−− void Customer : : run ( ) { loafs . acquire () ; int myBread = loafID . fetchAndAddOrdered (1) ; counterSpace . release () ; c o u t << ” Customer ” << I D << ” g o t b r e a d #” << m y B r e a d << e n d l ←֓ ; } //============================ i n t main ( i n t argc , char * argv [ ] ) { i n t totalCustomers = atoi ( argv [ 1 ] ) ; Customer *c [ totalCustomers ] ; f o r ( i n t i =0; i<t o t a l C u s t o m e r s ; i++) { c [ i ]=new C u s t o m e r ( i ) ; c [ i]−> s t a r t ( ) ; } Baker b ( totalCustomers ) ; b . run ( ) ;

22 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ 61

// w a i t f o r t e r m i n a t i o n f o r ( i n t i =0; i<t o t a l C u s t o m e r s ; i++) c [ i]−> w a i t ( ) ;

62 63 64 65

return 0;

66 67

}

9. Because of customer demand, the bakery owner is considering the following enhancements to his shop: (a) Increase the capacity of the counter to 1000 (b) Hire 3 more bakers Modify the solution of the previous exercise to accommodate these changes.

Which is the easiest to implement?

Answer (a) In this case, the only change required is to modify line 9 in the previous listing to: c o n s t i n t S P A C E =1000;

(b) The complication in this case is the termination of the baker threads.

For this purpose, the totalCustomers variable is used to initialize a general semaphore that is tested and decremented prior to the execution of each iteration of a baker thread (line 27): 1 2 3 4 5

#i n c l u d e <QThread> #i n c l u d e <QSemaphore> #i n c l u d e <QAtomicInt> #i n c l u d e <i o s t r e a m > #i n c l u d e < s t d l i b . h>

6 7

u s i n g namespace s t d ;

8 9

c o n s t i n t S P A C E =20;

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

34 35 36 37 38

QSemaphore loafs (0) , counterSpace ( SPACE ) ; QSemaphore moreLoafs (0) ; QAtomicInt loafID ; //============================ c l a s s Baker : public QThread { private : i n t ID ; public : B a k e r ( i n t i ) : I D ( i ) {} void run ( ) ; }; //−−−−−−−−−−−−−−−−−− void Baker : : run ( ) { i n t m y P r o d =0; while ( moreLoafs . tryAcquire () ) { counterSpace . acquire () ; loafs . release () ; m y P r o d ++; } c o u t << ” Baker #” << I D << ” baked a t o t a l ” b r e a d s ” << e n d l ; } //============================ c l a s s Customer : public QThread { private :

o f ” << m y P r o d << ←֓

23 39 40 41 42 43 44 45 46 47 48 49 50

51 52 53 54 55 56 57 58 59 60 61 62

i n t ID ; public : C u s t o m e r ( i n t i ) : I D ( i ) {} void run ( ) ; }; //−−−−−−−−−−−−−−−−−− void Customer : : run ( ) { loafs . acquire () ; int myBread = loafID . fetchAndAddOrdered (1) ; counterSpace . release () ; c o u t << ” Customer ” << I D << ” g o t b r e a d #” << m y B r e a d << ←֓ endl ; } //============================ i n t main ( i n t argc , char * argv [ ] ) { i n t totalCustomers = atoi ( argv [ 1 ] ) ; Customer *c [ totalCustomers ] ; moreLoafs . release ( totalCustomers ) ; f o r ( i n t i =0; i<t o t a l C u s t o m e r s ; i++) { c [ i ]=new C u s t o m e r ( i ) ; c [ i]−> s t a r t ( ) ; }

63

Baker *b [ 4 ] ; f o r ( i n t i =0; i <4; i++) { b [ i ] = new B a k e r ( i ) ; b [ i ] −> s t a r t ( ) ; }

64 65 66 67 68 69 70 71

// w a i t f o r t e r m i n a t i o n f o r ( i n t i =0; i<t o t a l C u s t o m e r s ; i++) c [ i]−> w a i t ( ) ; f o r ( i n t i =0; i <4; i++) b [ i]−> w a i t ( ) ;

72 73 74 75 76 77

return 0;

78 79

}

10. A bank account class is defined as follows: c l a s s BankAccount { protected : double balance ; string holderName ; public : double getBalance ( ) ; void deposit ( double ) ; v o i d w i t h d r a w ( d o u b l e , i n t ) ; // t h e h i g h e s t t h e s e c o n d ←֓ argument , t h e h i g h e r t h e p r i o r i t y o f t h e r e q u e s t };

Write the implementation of the three methods given above, so that withdraw operations are prioritized: if there are not enough funds in the account for all, the withdrawals must be done in order of priority regardless if there are some that can be performed with the available funds. You can assume that the priority level in the withdraw method is by default equal to 0, and that it can is upper bounded by a fixed constant MAXPRIORITY. Answer

for more solution manuals,visit libgen.is, libgen.st, libgen.rs, and forum.mhut.org

Solution is monitor based, using a separate wait condition for each level of priority.

1 2

// Monitor−b a s e d s o l u t i o n #i n c l u d e <QThread>

24 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ 3 4 5 6 7 8

#i n c l u d e <QMutexLocker> #i n c l u d e <QWaitCondition> #i n c l u d e <c s t r i n g > #i n c l u d e <u n i s t d . h> #i n c l u d e <s t r i n g > #i n c l u d e <i o s t r e a m >

9 10

u s i n g namespace s t d ;

11 12 13

const int MAXPRIORITY = 10; const int NUMAGENTS = 20;

14 15 16 17 18 19 20 21 22 23

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s BankAccount { protected : double balance ; string holderName ; QMutex l ; QWaitCondition priCond [ MAXPRIORITY ] ; int waitingCount [ MAXPRIORITY ] ;

24 25 26 27 28 29

30 31 32 33 34 35 36 37

public : B a n k A c c o u n t ( string name , double init ) ; double getBalance ( ) ; void deposit ( double ) ; void w i t h d r a w ( double , i n t ) ; // t h e h i g h e s t t h e s e c o n d ←֓ argument , t h e h i g h e r t h e p r i o r i t y o f t h e r e q u e s t }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− B a n k A c c o u n t : : B a n k A c c o u n t ( string name , double init ) { holderName = name ; balance = init ; memset ( waitingCount , 0 , MAXPRIORITY * s i z e o f ( int ) ) ; }

38 39 40 41 42 43

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− double BankAccount : : getBalance ( ) { return balance ; }

44 45 46 47 48 49 50

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void BankAccount : : deposit ( double x ) { Q M u t e x L o c k e r m l (&l ) ; b a l a n c e += x ; c e r r << ” D e p o s i t e d ” << x << e n d l ;

51

// now d e t e r m i n e i f any t h r e a d i s w a i t i n g t o withdraw int priLvl = MAXPRIORITY − 1; w h i l e ( p r i L v l >= 0 ) { i f ( waitingCount [ priLvl ] > 0) { c e r r << ”Waking l e v e l ” << p r i L v l << e n d l ; priCond [ priLvl ] . wakeAll () ; // wake up a l l t h e t h r e a d s ←֓ at that p r i o r i t y l e v e l return ; } p r i L v l −−; }

52 53 54 55 56 57 58 59

60 61 62 63 64

}

65 66 67

68 69 70 71

72

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // t h e h i g h e s t t h e s e c o n d argument , t h e h i g h e r t h e p r i o r i t y o f ←֓ the request void BankAccount : : withdraw ( double x , i n t lvl ) { Q M u t e x L o c k e r m l (&l ) ; l v l = ( l v l >= M A X P R I O R I T Y ) ? M A X P R I O R I T Y − 1 : l v l ; //make ←֓ s u r e p r i o r i t y i s not too high

25 // d e t e r m i n e i f o t h e r s a t t h e same o r h i g h e r waiting int othersWaiting = 0; int priLvl = MAXPRIORITY − 1; w h i l e ( p r i L v l >= l v l ) { o t h e r s W a i t i n g += w a i t i n g C o u n t [ p r i L v l ] ; p r i L v l −−; }

73

74 75 76 77 78 79 80

priority

a r e ←֓

81

// i f t h e y a r e o r t h e f u n d s a r e n o t enough w a i t a l s o i f ( othersWaiting > 0 | | balance < x ) { w a i t i n g C o u n t [ l v l ]++; while ( x > balance ) p r i C o n d [ l v l ] . w a i t (&l ) ;

82 83 84 85 86 87 88

b a l a n c e −= x ; w a i t i n g C o u n t [ l v l ]−−; i f ( w a i t i n g C o u n t [ l v l ] == 0 ) { priLvl = lvl ; w h i l e ( p r i L v l >= 0 && w a i t i n g C o u n t [ p r i L v l ] == 0 ) p r i L v l −−; i f ( p r i L v l >= 0 ) priCond [ priLvl ] . wakeAll () ; }

89 90 91 92 93 94 95 96 97 98

} else b a l a n c e −= x ;

99 100 101 102

c e r r << ” Withdrew ” << x << ” w i t h p r i ” << l v l << e n d l ;

103 104

}

105 106 107 108 109 110 111 112 113 114 115

116 117

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // c l a s s f o r t e s t i n g p u r p o s e s c l a s s Agent : public QThread { private : B a n k A c c o u n t * ba ; double amount ; int priority ; public : Agent ( BankAccount * b , double x , p r i o r i t y ( p r i ) {} void run ( ) ; };

i n t p r i ) : b a ( b ) , a m o u n t ( x ) , ←֓

118 119 120 121 122 123 124 125 126

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void Agent : : run ( ) { i f ( amount > 0) ba−>d e p o s i t ( a m o u n t ) ; else ba−>w i t h d r a w (− a m o u n t , p r i o r i t y ) ; }

127 128 129 130 131

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t main ( i n t argc , char * argv [ ] ) { B a n k A c c o u n t b ( ” John Doe” , 0 ) ;

132 133 134 135 136 137 138 139 140

141 142 143

Agent * ag [ N U M A G E N T S ] ; f o r ( i n t i = 0 ; i < N U M A G E N T S ; i++) { a g [ i ] = new A g e n t (&b , −r a n d ( ) % 2 0 , i / 2 ) ; a g [ i]−> s t a r t ( ) ; } sleep (1) ; b . d e p o s i t ( N U M A G E N T S * 2 0 ) ; // make s u r e enough f u n d s a r e ←֓ available f o r ( i n t i = 0 ; i < N U M A G E N T S ; i++) a g [ i]−> w a i t ( ) ;

26 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ return 0;

144 145

}

11. The IT department of a big corporation is equipped with 5 high speed printers that are used by a multitude of threads. The threads are part of the same accounting process. Each of the threads is supposed to perform the following (pseudocode) sequence in order to printout any material: ... printerID = get_available_printer () ; // p r i n t t o p r i n t e r I D p r i n t e r releasePrinter ( printerID ) ; ...

Write an appropriate implementation for the two functions listed above using semaphores. You can assume that the available printer IDs are stored in a shared buffer.

Answer Solution is based on the producers-consumers pattern. A buffer with the available printer IDs is setup and managed. Function get available printer() operates as a consumer and function releasePrinter() operates as a producer. Only one counting semaphore is required as the buffer is as big as the overall number of printers.

1 2 3 4 5

#i n c l u d e <QThread> #i n c l u d e <QMutex> #i n c l u d e <QSemaphore> #i n c l u d e <u n i s t d . h> #i n c l u d e <i o s t r e a m >

6 7

u s i n g namespace s t d ;

8 9 10

const int PRINTERS = 5; const int NUMAGENTS = 20;

11 12 13 14 15

int printIDbuffer [ PRINTERS ]={0 ,1 ,2 ,3 ,4}; QSemaphore prnAvail ( PRINTERS ) ; i n t i n =0 , o u t =0; Q M u t e x l1 , l 2 ;

16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− int get_available_printer () { prnAvail . acquire () ; l1 . lock ( ) ; i n t tmp = printIDbuffer [ out ] ; o u t = ( o u t +1)%P R I N T E R S ; l1 . unlock ( ) ; return tmp ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void releasePrinter ( i n t printerID ) { l2 . lock ( ) ; p r i n t I D b u f f e r [ i n ]= p r i n t e r I D ; i n = ( i n +1)%P R I N T E R S ; l2 . unlock ( ) ; prnAvail . release () ; }

36 37 38 39 40 41 42

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // c l a s s f o r t e s t i n g p u r p o s e s c l a s s Agent : public QThread { private :

27 43 44 45 46 47

i n t ID ; public : A g e n t ( i n t i ) : I D ( i ) {} void run ( ) ; };

48 49 50 51 52 53

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void Agent : : run ( ) { s l e e p ( r a n d ( ) %3) ; int printerID = get_available_printer () ;

54

// p r i n t t o p r i n t e r I D p r i n t e r c o u t << I D << ” i s p r i n t i n g t o ” << p r i n t e r I D << e n d l ; releasePrinter ( printerID ) ;

55 56 57 58

}

59 60 61 62 63 64 65 66 67 68

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t main ( i n t argc , char * argv [ ] ) { Agent * ag [ N U M A G E N T S ] ; f o r ( i n t i = 0 ; i < N U M A G E N T S ; i++) { a g [ i ] = new A g e n t ( i ) ; a g [ i]−> s t a r t ( ) ; }

69

f o r ( i n t i = 0 ; i < N U M A G E N T S ; i++) a g [ i]−> w a i t ( ) ;

70 71 72

return 0;

73 74

}

12. Create 3 threads, each printing out the letters ’A’, ’B’ and ’C’. The printing must adhere to these rules:

The total number of ’B’s and ’C’s that have been output at any point in the output string cannot exceed the total number of ’A’s that have been output at that point.

After a ’C’ has been output, another ’C’ cannot be output until one or more ’B’s have been output.

Use semaphores to solve the problem.

Answer 1 2 3 4 5

#i n c l u d e <QThread> #i n c l u d e <QMutex> #i n c l u d e <QSemaphore> #i n c l u d e <i o s t r e a m > #i n c l u d e <u n i s t d . h>

6 7

u s i n g namespace s t d ;

8 9 10 11

QSemaphore permit (0) ; QMutex allowC ; v o l a t i l e b o o l B f l a g=t r u e ;

12 13 14 15 16 17 18 19 20 21 22 23

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrA : public QThread { public : void run ( ) ; }; //−−−−−−−−−−−−−−− void ThrA : : run ( ) { while (1) {

28 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ c o u t << ’A ’ ; permit . release () ;

24 25

}

26 27

}

28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrB : public QThread { public : void run ( ) ; }; //−−−−−−−−−−−−−−− void ThrB : : run ( ) { while (1) { permit . acquire () ; c o u t << ’B ’ ; i f ( B f l a g==f a l s e ) { allowC . unlock () ; B f l a g=t r u e ; } } } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrC : public QThread { public : void run ( ) ; }; //−−−−−−−−−−−−−−− void ThrC : : run ( ) { while (1) { allowC . lock () ; permit . acquire () ; c o u t << ’C ’ ; B f l a g= f a l s e ; } } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t main ( i n t argc , char * argv [ ] ) { ThrA a ; ThrB b ; ThrC c ;

72

a . start () ; b . start () ; c . run ( ) ;

73 74 75 76

return 0;

77 78

}

13. Modify the previous exercise so that the printing is governed by this set of rules:

One ’C’ must be output after two ’A’s and three ’B’s are output.

While there is no restriction on the order of printing ’A’ and ’B’, the corresponding threads must wait for a ’C’ to be printed when the previous condition is met.

Use a monitor to solve the problem.

Answer The following solution uses the “critical section outside the monitor” approach.

29 1 2 3 4 5 6

#i n c l u d e <QThread> #i n c l u d e <QMutex> #i n c l u d e <QMutexLocker> #i n c l u d e <QWaitCondition> #i n c l u d e <i o s t r e a m > #i n c l u d e <u n i s t d . h>

7 8

u s i n g namespace s t d ;

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74

c l a s s Monitor { private : i n t cntA , cntB ; QMutex l ; Q W a i t C o n d i t i o n cAB , c C ; public : M o n i t o r ( ) : c n t A ( 0 ) , c n t B ( 0 ) {} void allowA ( ) ; void allowB ( ) ; void allowC ( ) ; void doneA ( ) ; void doneB ( ) ; void doneC ( ) ; }; //−−−−−−−−−−−−−−− void Monitor : : allowA ( ) { Q M u t e x L o c k e r m l (&l ) ; w h i l e ( c n t A == 3 ) c A B . w a i t (&l ) ; } //−−−−−−−−−−−−−−− void Monitor : : allowB ( ) { Q M u t e x L o c k e r m l (&l ) ; w h i l e ( c n t B == 2 ) c A B . w a i t (&l ) ; } //−−−−−−−−−−−−−−− void Monitor : : allowC ( ) { Q M u t e x L o c k e r m l (&l ) ; w h i l e ( c n t A != 3 | | c n t B ! = 2 ) c C . w a i t (&l ) ; } //−−−−−−−−−−−−−−− void Monitor : : doneA ( ) { Q M u t e x L o c k e r m l (&l ) ; c n t A ++; i f ( c n t A ==3) cC . w a k e O n e ( ) ; } //−−−−−−−−−−−−−−− void Monitor : : doneB ( ) { Q M u t e x L o c k e r m l (&l ) ; c n t B ++; i f ( c n t B ==2) cC . w a k e O n e ( ) ; } //−−−−−−−−−−−−−−− void Monitor : : doneC ( ) { Q M u t e x L o c k e r m l (&l ) ; cntA = cntB = 0; cAB . wakeAll ( ) ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrA : public QThread { private : Monitor *m ; public :

30 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ T h r A ( M o n i t o r * x ) : m ( x ) {} void run ( ) ;

75 76 77 78 79 80 81 82 83 84 85 86 87

}; //−−−−−−−−−−−−−−− void ThrA : : run ( ) { while (1) { m−>a l l o w A ( ) ; c o u t<< ’A ’ ; m−>d o n e A ( ) ; } }

88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrB : public QThread { private : Monitor *m ; public : T h r B ( M o n i t o r * x ) : m ( x ) {} void run ( ) ; }; //−−−−−−−−−−−−−−− void ThrB : : run ( ) { while (1) { m−>a l l o w B ( ) ; c o u t<< ’B ’ ; m−>d o n e B ( ) ; } } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrC : public QThread { private : Monitor *m ; public : T h r C ( M o n i t o r * x ) : m ( x ) {} void run ( ) ; }; //−−−−−−−−−−−−−−− void ThrC : : run ( ) { while (1) { m−>a l l o w C ( ) ; c o u t<< ’C ’ ; m−>d o n e C ( ) ; } } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t main ( i n t argc , char * argv [ ] ) { Monitor m ; T h r A a(&m ) ; T h r B b(&m ) ; T h r C c(&m ) ;

134

a . start () ; b . start () ; c . run ( ) ;

135 136 137 138

return 0;

139 140

}

14. Address the termination problem in the previous exercise. How can the three threads terminate after e.g. a fixed number of As has been output?

Or when a fixed total number of character has been output?

Answer

31 The following code solves the termination problem for a fixed number of total characters. A character is reserved at the output before the termination of the allow?() methods, which compared to the previous exercise, return a boolean to indicate whether to continue execution or not. If the maximum number of characters has been reached, appropriate signals wake up any waiting threads.

1 2 3 4 5 6

#i n c l u d e <QThread> #i n c l u d e <QMutex> #i n c l u d e <QMutexLocker> #i n c l u d e <QWaitCondition> #i n c l u d e <i o s t r e a m > #i n c l u d e <u n i s t d . h>

7 8

u s i n g namespace s t d ;

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

c l a s s Monitor { private : i n t cntA , cntB , total ; QMutex l ; Q W a i t C o n d i t i o n cAB , c C ; public : M o n i t o r ( i n t t ) : c n t A ( 0 ) , c n t B ( 0 ) , t o t a l ( t ) {} bool allowA () ; bool allowB () ; bool allowC () ; void doneA ( ) ; void doneB ( ) ; void doneC ( ) ; };

25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

//−−−−−−−−−−−−−−− bool Monitor : : allowA () { Q M u t e x L o c k e r m l (&l ) ; w h i l e ( c n t A == 3 && t o t a l > 0 ) c A B . w a i t (&l ) ; i f ( total ) { t o t a l −−; return true ; } else { cAB . wakeOne ( ) ; cC . w a k e O n e ( ) ; return f a l s e ; } }

44 45 46 47 48 49 50

//−−−−−−−−−−−−−−− bool Monitor : : allowB () { Q M u t e x L o c k e r m l (&l ) ; w h i l e ( c n t B == 2 && t o t a l > 0 ) c A B . w a i t (&l ) ;

51

if

52

( total ) {

53

t o t a l −−; return true ;

54 55

} else {

56 57 58

cAB . wakeOne ( ) ; cC . w a k e O n e ( ) ; return f a l s e ;

59 60 61

}

62 63

}

64 65

//−−−−−−−−−−−−−−−

32 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ 66 67 68 69 70

bool Monitor : : allowC () { Q M u t e x L o c k e r m l (&l ) ; w h i l e ( ( c n t A != 3 | | c n t B != 2 ) && ( t o t a l > 0 ) ) c C . w a i t (&l ) ;

71

if

72

( total ) {

73

t o t a l −−; return true ;

74 75

} else {

76 77 78

cAB . wakeOne ( ) ; cC . w a k e O n e ( ) ; return f a l s e ;

79 80 81

}

82 83

}

84 85 86 87 88 89 90 91 92

//−−−−−−−−−−−−−−− void Monitor : : doneA ( ) { Q M u t e x L o c k e r m l (&l ) ; c n t A ++; i f ( c n t A == 3 ) cC . w a k e O n e ( ) ; }

93 94 95 96 97 98 99 100 101

//−−−−−−−−−−−−−−− void Monitor : : doneB ( ) { Q M u t e x L o c k e r m l (&l ) ; c n t B ++; i f ( c n t B == 2 ) cC . w a k e O n e ( ) ; }

102 103 104 105 106 107 108 109

//−−−−−−−−−−−−−−− void Monitor : : doneC ( ) { Q M u t e x L o c k e r m l (&l ) ; cntA = cntB = 0; cAB . wakeAll ( ) ; }

110 111 112 113 114 115 116 117 118 119 120 121

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrA : public QThread { private : Monitor * m ; public : ThrA ( Monitor * x ) : m ( x ) { } void run ( ) ; };

122 123 124 125 126 127 128 129 130 131

//−−−−−−−−−−−−−−− void ThrA : : run ( ) { w h i l e ( m−>a l l o w A ( ) ) { c o u t << ’A ’ ; m−>d o n e A ( ) ; } }

132 133 134 135 136 137 138 139

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrB : public QThread { private : Monitor * m ; public : ThrB ( Monitor * x ) : m ( x )

33 { } void run ( ) ;

140 141 142 143

};

144 145 146 147 148 149 150 151 152 153

//−−−−−−−−−−−−−−− void ThrB : : run ( ) { w h i l e ( m−>a l l o w B ( ) ) { c o u t << ’B ’ ; m−>d o n e B ( ) ; } }

154 155 156 157 158 159 160 161 162 163 164 165

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrC : public QThread { private : Monitor * m ; public : ThrC ( Monitor * x ) : m ( x ) { } void run ( ) ; };

166 167 168 169 170 171 172 173 174 175

//−−−−−−−−−−−−−−− void ThrC : : run ( ) { w h i l e ( m−>a l l o w C ( ) ) { c o u t << ’C ’ ; m−>d o n e C ( ) ; } }

176 177 178 179 180 181 182 183

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t main ( i n t argc , char * argv [ ] ) { Monitor m (100) ; T h r A a (&m ) ; T h r B b (&m ) ; T h r C c (&m ) ;

184

a . start () ; b . start () ; c . run ( ) ;

185 186 187 188

a . wait () ; b . wait () ; return 0;

189 190 191 192

}

15. Create 4 threads, each printing out the letters ’A’, ’B’, ’C’, and ’D’. The printing must adhere to these rules:

The total number of ’A’s and ’B’s that have been output at any point in the output string cannot exceed the total number of ’C’s and ’D’s that have been output at that point.

The total number of ’A’s that have been output at any point in the output string cannot exceed twice the number of ’B’s that have been output at that point.

After a ’C’ has been output, another ’C’ cannot be output until one or more ’D’s have been output.

Solve the problem using (a) semaphores and (b) a monitor.

34 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ Answer (a) 1 2 3 4 5

#i n c l u d e <Q T h r e a d > #i n c l u d e <QMutex> #i n c l u d e <QSemaphore> #i n c l u d e <i o s t r e a m > #i n c l u d e <u n i s t d . h>

6 7

u s i n g namespace s t d ;

8 9 10 11

QSemaphore allowAB (0) , permitA (0) , total (100) ; QMutex permitC ; bool DisOut = true ;

12 13 14 15 16 17 18

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrA : public QThread { public : void run ( ) ; };

19 20 21 22 23 24 25 26 27 28 29 30 31 32

//−−−−−−−−−−−−−−− void ThrA : : run ( ) { while (1) { permitA . acquire () ; // f o r allowAB . acquire () ; // f o r i f ( ! total . tryAcquire () ) break ; c o u t << ’A ’ ; usleep (1) ; } }

r u l e #2 r u l e #1

33 34 35 36 37 38 39

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrB : public QThread { public : void run ( ) ; };

40 41 42 43 44 45 46 47 48 49 50 51 52 53

//−−−−−−−−−−−−−−− void ThrB : : run ( ) { while (1) { a l l o w A B . a c q u i r e ( ) ; // f o r r u l e #1 i f ( ! total . tryAcquire () ) break ; c o u t << ’B ’ ; p e r m i t A . r e l e a s e ( 2 ) ; // f o r r u l e #2 usleep (1) ; } }

54 55 56 57 58 59 60

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrC : public QThread { public : void run ( ) ; };

61 62 63 64 65 66 67 68 69 70

//−−−−−−−−−−−−−−− void ThrC : : run ( ) { while (1) { p e r m i t C . l o c k ( ) ; // f o r r u l e #3 i f ( ! total . tryAcquire () ) break ; c o u t << ’C ’ ;

35 DisOut = f a l s e ; a l l o w A B . r e l e a s e ( ) ; // f o r usleep (1) ;

71 72 73

}

74 75

r u l e #1

}

76 77 78 79 80 81 82

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrD : public QThread { public : void run ( ) ; };

83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

//−−−−−−−−−−−−−−− void ThrD : : run ( ) { while (1) { c o u t << ’D ’ ; i f ( ! total . tryAcquire () ) break ; a l l o w A B . r e l e a s e ( ) ; // f o r r u l e #1 i f ( D i s O u t == f a l s e ) // f o r r u l e #3 { DisOut = true ; permitC . unlock () ; } usleep (1) ; }

100

// Make s u r e a l l t e r m i n a t e // T h i s i s t h e most a p p r o p r i a t e t h r e a d t o do t h i s , a s ThrD d o e s ←֓ n o t w a i t f o r any o t h e r allowAB . release (2) ; permitA . release () ; i f ( D i s O u t == f a l s e ) { permitC . unlock () ; }

101 102

103 104 105 106 107 108 109 110

}

111 112 113 114 115 116 117 118

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t main ( i n t argc , char * argv [ ] ) { ThrA a ; ThrB b ; ThrC c ; ThrD d ;

119

a . start () ; b . start () ; c . start () ; d . run ( ) ;

120 121 122 123 124

a . wait b . wait c . wait return

125 126 127 128 129

() ; () ; () ; 0;

}

(b) In this case, the Monitor class could be equipped with generic allow() and done() methods, that would accept as a parameter the character that is supposed to be printed, i.e. the type of thread calling. In the following code we stick with separate pairs of Monitor methods for each type of thread, to make the program easier to read.

1 2 3 4 5

#i n c l u d e <QThread> #i n c l u d e <QMutex> #i n c l u d e <QMutexLocker> #i n c l u d e <QWaitCondition> #i n c l u d e <i o s t r e a m >

36 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ 6

#i n c l u d e <u n i s t d . h>

7 8

u s i n g namespace s t d ;

9 10 11 12 13 14 15 16

17 18

19 20 21 22 23 24 25 26 27 28 29

c l a s s Monitor { private : i n t allowAB , permitA , total ; bool permitC ; QMutex l ; Q W a i t C o n d i t i o n cAB , cA , c C ; // one c o n d i t i o n f o r e a c h o f t h e ←֓ three rules public : M o n i t o r ( i n t t ) : a l l o w A B ( 0 ) , p e r m i t A ( 0 ) , t o t a l ( t ) , p e r m i t C ←֓ ( true ) { } bool allowA () ; bool allowB () ; bool allowC () ; bool allowD () ; void doneA ( ) ; void doneB ( ) ; void doneC ( ) ; void doneD ( ) ; };

30 31 32 33 34 35 36

//−−−−−−−−−−−−−−− bool Monitor : : allowA () { Q M u t e x L o c k e r m l (&l ) ; w h i l e ( p e r m i t A == 0 && t o t a l > 0 ) c A . w a i t (&l ) ;

37

w h i l e ( a l l o w A B == 0 && t o t a l > 0 ) c A B . w a i t (&l ) ;

38 39 40

a l l o w A B −−; p e r m i t A −−; i f ( total ) { t o t a l −−; return true ; } else { cAB . wakeOne ( ) ; cC . w a k e O n e ( ) ; return f a l s e ; }

41 42 43 44 45 46 47 48 49 50 51 52 53 54

}

55 56 57 58 59 60 61

//−−−−−−−−−−−−−−− bool Monitor : : allowB () { Q M u t e x L o c k e r m l (&l ) ; w h i l e ( a l l o w A B == 0 && t o t a l > 0 ) c A B . w a i t (&l ) ;

62

a l l o w A B −−; i f ( total ) { t o t a l −−; return true ; } else { cAB . wakeOne ( ) ; cC . w a k e O n e ( ) ; return f a l s e ; }

63 64 65 66 67 68 69 70 71 72 73 74 75

}

76 77

//−−−−−−−−−−−−−−−

37 78 79 80 81 82

bool Monitor : : allowC () { Q M u t e x L o c k e r m l (&l ) ; w h i l e ( p e r m i t C == f a l s e && t o t a l > 0 ) c C . w a i t (&l ) ;

83

if

84

( total ) {

85

t o t a l −−; return true ;

86 87

} else {

88 89 90

cAB . wakeAll ( ) ; cA . w a k e O n e ( ) ; return f a l s e ;

91 92 93

}

94 95

}

96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113

//−−−−−−−−−−−−−−− bool Monitor : : allowD () { Q M u t e x L o c k e r m l (&l ) ; i f ( total ) { t o t a l −−; return true ; } else { cAB . wakeAll ( ) ; cA . w a k e O n e ( ) ; cC . w a k e O n e ( ) ; return f a l s e ; } }

114 115 116 117 118 119

//−−−−−−−−−−−−−−− void Monitor : : doneA ( ) { // n o t h i n g t o be done h e r e }

120 121 122 123 124 125 126 127

//−−−−−−−−−−−−−−− void Monitor : : doneB ( ) { Q M u t e x L o c k e r m l (&l ) ; p e r m i t A += 2 ; cA . w a k e O n e ( ) ; }

128 129 130 131 132 133 134 135 136

//−−−−−−−−−−−−−−− void Monitor : : doneC ( ) { Q M u t e x L o c k e r m l (&l ) ; a l l o w A B ++; permitC = f a l s e ; cAB . wakeAll ( ) ; }

137 138 139 140 141 142 143 144 145 146 147 148 149

//−−−−−−−−−−−−−−− void Monitor : : doneD ( ) { Q M u t e x L o c k e r m l (&l ) ; i f ( p e r m i t C == f a l s e ) { permitC = true ; cC . w a k e O n e ( ) ; } a l l o w A B ++; cAB . wakeAll ( ) ; }

150 151

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

38 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ 152 153 154 155 156 157 158 159 160 161

c l a s s ThrA : public QThread { private : Monitor * m ; public : ThrA ( Monitor * x ) : m ( x ) { } void run ( ) ; };

162 163 164 165 166 167 168 169 170 171 172

//−−−−−−−−−−−−−−− void ThrA : : run ( ) { w h i l e ( m−>a l l o w A ( ) ) { c o u t << ’A ’ ; m−>d o n e A ( ) ; usleep (1) ; } }

173 174 175 176 177 178 179 180 181 182 183 184

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrB : public QThread { private : Monitor * m ; public : ThrB ( Monitor * x ) : m ( x ) { } void run ( ) ; };

185 186 187 188 189 190 191 192 193 194 195

//−−−−−−−−−−−−−−− void ThrB : : run ( ) { w h i l e ( m−>a l l o w B ( ) ) { c o u t << ’B ’ ; m−>d o n e B ( ) ; usleep (1) ; } }

196 197 198 199 200 201 202 203 204 205 206 207

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrC : public QThread { private : Monitor * m ; public : ThrC ( Monitor * x ) : m ( x ) { } void run ( ) ; };

208 209 210 211 212 213 214 215 216 217 218

//−−−−−−−−−−−−−−− void ThrC : : run ( ) { w h i l e ( m−>a l l o w C ( ) ) { c o u t << ’C ’ ; m−>d o n e C ( ) ; usleep (1) ; } }

219 220 221 222 223 224 225

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− c l a s s ThrD : public QThread { private : Monitor * m ; public :

39 ThrD ( Monitor * x ) : m ( x ) { } void run ( ) ;

226 227 228 229 230

};

231 232 233 234 235 236 237 238 239 240 241

//−−−−−−−−−−−−−−− void ThrD : : run ( ) { w h i l e ( m−>a l l o w D ( ) ) { c o u t << ’D ’ ; m−>d o n e D ( ) ; usleep (1) ; } }

242 243 244 245 246 247 248 249 250

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t main ( i n t argc , char * argv [ ] ) { Monitor m (100) ; T h r A a (&m ) ; T h r B b (&m ) ; T h r C c (&m ) ; T h r D d (&m ) ;

251

a . start () ; b . start () ; c . start () ; d . run ( ) ;

252 253 254 255 256

a . wait b . wait c . wait return

257 258 259 260 261

() ; () ; () ; 0;

}

16. Use semaphores to solve the typical cigarette smokers problem, where the agent signals directly the smoker missing the two ingredients placed on the table.

Answer 1 2 3 4 5

#i n c l u d e <QThread> #i n c l u d e <QMutex> #i n c l u d e <QSemaphore> #i n c l u d e <i o s t r e a m > #i n c l u d e < s t d l i b . h>

6 7

u s i n g namespace s t d ;

8 9 10 11 12

#d e f i n e #d e f i n e #d e f i n e #d e f i n e

MAXSLEEP 1000 TOBACCO PAPER 0 TOBACCO MATHCES 1 PAPER MATHCES 2

13 14 15 16 17

18 19 20 21 22 23 24 25 26 27 28

QSemaphore missingIngr [ 3 ] ; QSemaphore wakeAgent (0) ; b o o l t e r m F l a g= f a l s e ; c o n s t c h a r * m s g [ ] = { ” h a v i n g matches ” , ” h a v i n g p a p e r ” , ” h a v i n g ←֓ tobacco ” }; // *************************************************** c l a s s Smoker : public QThread { private : int missing ; public : Smoker ( int ) ; void run ( ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− S m o k e r : : S m o k e r ( i n t m ) : m i s s i n g ( m ) {}

40 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ 29 30 31 32 33 34

35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void Smoker : : run ( ) { while (1) { missingIngr [ missing ] . acquire () ; // w a i t f o r a g e n t t o s e n d ←֓ signal i f ( termFlag ) break ; // t e r m i n a t i o n c h e c k c o u t << ” Smoker ” << m s g [ m i s s i n g ] << ” i s smoking \n” ; msleep ( rand () % MAXSLEEP ) ; wakeAgent . release () ; // wake up a g e n t } } // *************************************************** c l a s s Agent : public QThread { private : i n t runs ; public : Agent ( int ) ; void run ( ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− A g e n t : : A g e n t ( i n t r ) : r u n s ( r ) {} //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void Agent : : run ( ) { f o r ( i n t i =0; i<r u n s ; i++) { i n t ingreds = rand () % 3; missingIngr [ ingreds ] . release () ; wakeAgent . acquire () ; } // s e t t e r m i n a t i o n f l a g and wake up a l l t h r e a d s t e r m F l a g=t r u e ; missingIngr [ 0 ] . release () ; missingIngr [ 1 ] . release () ; missingIngr [ 2 ] . release () ; } // *************************************************** i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { Smoker *s [ 3 ] ; f o r ( i n t i =0; i <3; i++) { s [ i ] = new S m o k e r ( i ) ; s [ i]−> s t a r t ( ) ; } Agent a ( atoi ( argv [ 1 ] ) ) ; a . run ( ) ;

78

f o r ( i n t i =0; i <3; i++) s [ i]−> w a i t ( ) ;

79 80 81

return EXIT_SUCCESS ;

82 83

}

17. Solve the cigarette smokers problem as described in Section 3.6.2, using semaphores.

Answer In the following solution all smoker threads wake up and check the table contents.

1 2 3 4 5

#i n c l u d e <QThread> #i n c l u d e <QMutex> #i n c l u d e <QSemaphore> #i n c l u d e <i o s t r e a m > #i n c l u d e < s t d l i b . h>

6 7

u s i n g namespace s t d ;

41 8 9 10 11 12

#d e f i n e #d e f i n e #d e f i n e #d e f i n e

MAXSLEEP 10 TOBACCO PAPER 0 TOBACCO MATHCES 1 PAPER MATHCES 2

13 14 15 16 17 18

19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

QSemaphore wakeUpSmoker [ 3 ] ; QSemaphore wakeAgent (0) ; v o l a t i l e b o o l t e r m F l a g= f a l s e ; int table ; c o n s t c h a r * m s g [ ] = { ” h a v i n g matches ” , ” h a v i n g p a p e r ” , ” h a v i n g ←֓ tobacco ” }; // *************************************************** c l a s s Smoker : public QThread { private : int missing ; public : Smoker ( int ) ; void run ( ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− S m o k e r : : S m o k e r ( i n t m ) : m i s s i n g ( m ) {} //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void Smoker : : run ( ) { while (1) { do { wakeUpSmoker [ missing ] . acquire () ; } w h i l e ( t a b l e != m i s s i n g && ! t e r m F l a g ) ;

40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

i f ( termFlag ) break ; // t e r m i n a t i o n c h e c k c o u t << ” Smoker ” << m s g [ m i s s i n g ] << ” i s smoking \n” ; msleep ( rand () % MAXSLEEP ) ; wakeAgent . release () ; // wake up a g e n t } } // *************************************************** c l a s s Agent : public QThread { private : i n t runs ; public : Agent ( int ) ; void run ( ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− A g e n t : : A g e n t ( i n t r ) : r u n s ( r ) {} //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void Agent : : run ( ) { f o r ( i n t i =0; i<r u n s ; i++) { table = rand () % 3; wakeUpSmoker [ 0 ] . release () ; wakeUpSmoker [ 1 ] . release () ; wakeUpSmoker [ 2 ] . release () ; wakeAgent . acquire () ; } // s e t t e r m i n a t i o n f l a g and wake up a l l t h r e a d s t e r m F l a g=t r u e ; wakeUpSmoker [ 0 ] . release () ; wakeUpSmoker [ 1 ] . release () ; wakeUpSmoker [ 2 ] . release () ; } // *************************************************** i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { Smoker *s [ 3 ] ; f o r ( i n t i =0; i <3; i++) {

42 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ s [ i ] = new S m o k e r ( i ) ; s [ i]−> s t a r t ( ) ;

81 82

} Agent a ( atoi ( argv [ 1 ] ) ) ; a . run ( ) ;

83 84 85 86

f o r ( i n t i =0; i <3; i++) s [ i]−> w a i t ( ) ;

87 88 89

return EXIT_SUCCESS ;

90 91

}

18. Model the movie-going process at a multiplex cinema using a monitor.

Assume the following conditions:

There are 3 different movies played at the same time in 3 halls. The capacity of each hall is 4, 5 and 7 respectively.

One hundred customers are waiting to see a randomly chosen movie.

A cashier issues the tickets.

If a hall is full a movie begins to play.

A customer cannot enter a hall while a movie is playing or while the previous viewers are exiting the hall.

A movie will play for the last customers even if the corresponding hall is not full.

Answer 1 2 3 4 5 6

#i n c l u d e <QThread> #i n c l u d e <QMutex> #i n c l u d e <QWaitCondition> #i n c l u d e <i o s t r e a m > #i n c l u d e <v e c t o r > #i n c l u d e < s t d l i b . h>

7 8

u s i n g namespace s t d ;

9 10 11 12 13 14 15 16 17 18 19 20 21 22

23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

c o n s t i n t N U M C U S T =100; // ***************************************** c l a s s Monitor { private : int numHalls ; v e c t o r <i n t > &c a p a c i t y ; int * inside ; i n t * left ; bool * finished ; int numCustomersRemain ; QMutex l ; Q W a i t C o n d i t i o n * c o n d E ; // ’ e n t e r ’ a r r a y o f c o n d i t i o n s . One ←֓ f o r each h a l l Q W a i t C o n d i t i o n * c o n d L ; // ’ l e a v e ’ a r r a y o f c o n d i t i o n s public : M o n i t o r ( v e c t o r <i n t > &cap , i n t n u m C u s t ) ; ˜ Monitor () ; void enterTheater ( i n t movie ) ; void leaveTheater ( i n t movie ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− M o n i t o r : : M o n i t o r ( v e c t o r <i n t > &cap , i n t n u m C u s t ) : c a p a c i t y ( c a p ) { n u m H a l l s=c a p . s i z e ( ) ; numCustomersRemain = numCust ; i n s i d e = new i n t [ n u m H a l l s ] ; f o r ( i n t i =0; i<n u m H a l l s ; i++) i n s i d e [ i ] = 0 ; l e f t = new i n t [ n u m H a l l s ] ;

43 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

f o r ( i n t i =0; i<n u m H a l l s ; i++) l e f t [ i ] = 0 ; f i n i s h e d = new b o o l [ n u m H a l l s ] ; f o r ( i n t i =0; i<n u m H a l l s ; i++) f i n i s h e d [ i ]= f a l s e ; c o n d E = new Q W a i t C o n d i t i o n [ n u m H a l l s ] ; c o n d L = new Q W a i t C o n d i t i o n [ n u m H a l l s ] ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− Monitor : : ˜ Monitor () { delete [ ] inside ; d e l e t e [ ] left ; delete [ ] finished ; d e l e t e [ ] condE ; d e l e t e [ ] condL ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void Monitor : : enterTheater ( i n t m ) { Q M u t e x L o c k e r m l (&l ) ; w h i l e ( i n s i d e [ m ] == c a p a c i t y [ m ] ) c o n d E [ m ] . w a i t (&l ) ; i n s i d e [ m ]++; i f ( i n s i d e [ m ] == c a p a c i t y [ m ] ) // i s h a l l f u l l ? { c o u t << ” Movie ” << m << ” s t a r t e d \n” ; f i n i s h e d [ m ]= t r u e ; // s e t end−o f −movie f l a g left [ m ]=0; condL [ m ] . wakeAll () ; // l e t a l l c u s t o m e r s l e a v e }

67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104

105 106 107 108 109 110

n u m C u s t o m e r s R e m a i n −−; i f ( n u m C u s t o m e r s R e m a i n ==0) // s t a r t t h e r e m a i n i n g shows { f o r ( i n t i =0; i<n u m H a l l s ; i++) i f ( i n s i d e [ i ] >0) { c o u t << ” Movie ” << i << ” s t a r t e d \n” ; f i n i s h e d [ i ]= t r u e ; left [ i ]=0; condL [ i ] . wakeAll () ; } } } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void Monitor : : leaveTheater ( i n t m ) { Q M u t e x L o c k e r m l (&l ) ; w h i l e ( f i n i s h e d [ m]== f a l s e ) c o n d L [ m ] . w a i t (&l ) ; l e f t [ m ]++; i f ( l e f t [ m ] == c a p a c i t y [ m ] ) // i s h a l l empty { f i n i s h e d [ m ]= f a l s e ; inside [ m ]=0; left [ m ]=0; condE [ m ] . wakeAll () ; } } // ***************************************** c l a s s Customer : public QThread { private : i n t ID ; int movieID ; Monitor *m ; public : C u s t o m e r ( i n t i , M o n i t o r * c , i n t m I D ) : I D ( i ) , m o v i e I D ( m I D ) , m←֓ ( c ) {}; void run ( ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void Customer : : run ( ) { c o u t << ” Customer ” << I D << ” wants t o s e e ” << m o v i e I D << ”←֓

44 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ \n” ; m−>e n t e r T h e a t e r ( m o v i e I D ) ; c o u t << ” Customer ” << I D << ” e n t e r e d t h e t h e a t e r \n” ; m−>l e a v e T h e a t e r ( m o v i e I D ) ; c o u t << ” Customer ” << I D << ” l e f t t h e m o v i e s \n” ;

111 112 113 114 115 116 117 118 119 120 121 122

} // ***************************************** i n t main ( i n t argc , char * argv [ ] ) { v e c t o r <i n t > c ; c . push_back (4) ; c . push_back (5) ; c . push_back (7) ;

123

Monitor m (c , NUMCUST ) ; Customer * cust [ NUMCUST ] ; f o r ( i n t i =0; i<N U M C U S T ; i++) { c u s t [ i ] = new C u s t o m e r ( i , &m , r a n d ( ) %3) ; c u s t [ i]−> s t a r t ( ) ; }

124 125 126 127 128 129 130 131

f o r ( i n t i =0; i<N U M C U S T ; i++) c u s t [ i]−> w a i t ( ) ;

132 133 134

return 0;

135 136

}

19. Write a multi-threaded password cracker based on the Producer-Consumer paradigm. The producer should generate plain-text passwords according to a set of rules and the consumers should be hashing each password and checking whether it matches a target signature. All the threads should terminate upon the discovery of a matching password. You can use the MD5 cryptographic hash function for this exercise.

Answer The following listing shows a monitor based solution to the problem. There are two classes whose instances act as a producer (class PassGenerator) and consumers (class PassChecker).

The main program is expecting two command-line parameters : the number of password checkers and the maximum length of the brute-force generated passwords to be examined. Once a password with the same MD5 hash value is found, the finishUp method is called, which in turn terminates all remaining threads upon their next interaction with the Monitor.

1 2 3 4 5 6 7 8

#i n c l u d e <i o s t r e a m > #i n c l u d e <s t r i n g > #i n c l u d e < s t r i n g . h> #i n c l u d e ”md5 . h” #i n c l u d e <QThread> #i n c l u d e <QMutex> #i n c l u d e <QMutexLocker> #i n c l u d e <QWaitCondition>

9 10

u s i n g namespace s t d ;

11 12 13 14 15 16 17 18 19 20 21

c h a r * a l p h a b e t=” 1 2 3 4 5 6 7 8 9 0 a b c d e f g h i j k l m n o p q r s t u v w x y z ” ; s t r i n g t a r g e t ( ” 5912 d 7 b f d 1 0 f 6 3 1 f 1 7 1 5 b f 8 5 b b b 7 2 d 9 7 ” ) ; // c i p h e r t e x t c o n s t i n t P A S S S T O R E S I Z E =100; // s i z e o f p as s word b u f f e r // ************************************************* c l a s s Monitor { private : QMutex l ; QWaitCondition full ; QWaitCondition empty ;

45 22 23 24 25 26 27 28 29

30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79

char * buff [ PASSSTORESIZE ] ; i n t in , out , N , N c o n s ; int total ; bool found ; public : Monitor ( i n t consumers , i n t maxLen ) ; ˜ Monitor () ; bool putCandidate ( char *) ; // both r e t u r n f a l s e when i t time to stop bool getCandidate ( char *) ; void finishUp ( ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− Monitor : : Monitor ( i n t consumers , i n t maxLen ) { Ncons = consumers ; b u f f [ 0 ] = new c h a r [ ( m a x L e n +1) * P A S S S T O R E S I Z E ] ; f o r ( i n t i =1; i<P A S S S T O R E S I Z E ; i++) b u f f [ i ]= b u f f [ 0 ] + ( m a x L e n +1) * i ; i n =0; o u t =0; N =0; t o t a l =0; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− Monitor : : ˜ Monitor () { d e l e t e [ ] buff [ 0 ] ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void Monitor : : finishUp ( ) { Q M u t e x L o c k e r m l (&l ) ; f o u n d=t r u e ; empty . wakeAll () ; full . wakeOne () ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− bool Monitor : : putCandidate ( char *s ) { Q M u t e x L o c k e r m l (&l ) ; w h i l e ( N==P A S S S T O R E S I Z E && ! f o u n d ) f u l l . w a i t (&l ) ; strcpy ( buff [ in ] , s ) ; i n =( i n +1)%P A S S S T O R E S I Z E ; N++; t o t a l ++; i f ( t o t a l %1000000==0) c o u t << ” Checking #” << t o t a l << ” : ” << s << e n d l ; empty . wakeOne () ; return ! found ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− bool Monitor : : getCandidate ( char *s ) { Q M u t e x L o c k e r m l (&l ) ; w h i l e ( N==0 && ! f o u n d ) e m p t y . w a i t (&l ) ; i f ( found ) return f a l s e ;

80 81 82 83 84 85 86 87 88 89 90 91 92 93 94

strcpy ( s , buff [ out ] ) ; o u t =( o u t +1)%P A S S S T O R E S I Z E ; N−−; full . wakeOne () ; return ! found ; } // ************************************************* c l a s s PassGenerator : public QThread { private : char * candidate ; v o i d r e c G e n ( i n t pos , i n t r e m a i n ) ; Monitor *m ; int maxLen ;

i s ←֓

46 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ 95

bool stopFlag ;

96 97 98 99 100 101 102

103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161

public : P a s s G e n e r a t o r ( Monitor *x , i n t l ) ; void run ( ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− P a s s G e n e r a t o r : : P a s s G e n e r a t o r ( M o n i t o r * x , i n t l ) : m ( x ) , m a x L e n ( l ) ←֓ , stopFlag ( f a l s e ) { c a n d i d a t e = new c h a r [ m a x L e n + 1 ] ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− v o i d P a s s G e n e r a t o r : : r e c G e n ( i n t pos , i n t r e m a i n ) { i f ( r e m a i n ==0) { candidate [ pos ]=0; s t o p F l a g = ! m−>p u t C a n d i d a t e ( c a n d i d a t e ) ; } else { f o r ( u n s i g n e d i n t i =0; i<s t r l e n ( a l p h a b e t ) && ! s t o p F l a g ; i++) { candidate [ pos ] = alphabet [ i ] ; r e c G e n ( p o s +1 , r e m a i n −1) ; } } } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void PassGenerator : : run ( ) { i n t i =6; w h i l e ( s t o p F l a g==f a l s e && i<=m a x L e n ) r e c G e n ( 0 , i++) ; } // ************************************************* c l a s s PassChecker : public QThread { private : Monitor *m ; int maxLen ; public : P a s s C h e c k e r ( Monitor *x , i n t L ) : m ( x ) , maxLen ( L ) { } ; void run ( ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void PassChecker : : run ( ) { char tmp [ maxLen +1]; w h i l e ( m−>g e t C a n d i d a t e ( t m p ) ) { i f ( t a r g e t == m d 5 ( t m p ) ) { m−>f i n i s h U p ( ) ; c o u t << ” Password : ” << t m p << e n d l ; break ; } } } // ************************************************* i n t main ( i n t argc , char * argv [ ] ) { i n t Nchk = atoi ( argv [ 1 ] ) ; int L = atoi ( argv [ 2 ] ) ; M o n i t o r m ( Nchk , L ) ; PassChecker * chk [ Nchk ] ; P a s s G e n e r a t o r g e n (&m , L ) ;

162 163 164 165 166 167

gen . start ( ) ; f o r ( i n t i =0; i<N c h k ; i++) { c h k [ i ] = new P a s s C h e c k e r (&m , L ) ; c h k [ i]−> s t a r t ( ) ;

47 }

168 169

gen . wait ( ) ; f o r ( i n t i =0; i<N c h k ; i++) c h k [ i]−> w a i t ( ) ;

170 171 172 173

return 0;

174 175

}

20. Write a multi-threaded program for finding the prime numbers in a usersupplied range of numbers. Compare the following design approaches: (a) Split the range in equal pieces and assign each one to a thread. (b) Have a shared QAtomicInt variable that holds the next number to be checked. Threads should read and increment this number before testing it. (c) Have a shared “monitor” object that returns upon request, a range of numbers to be tested. This can be considered a generalization of the previous design.

Which of the designs is more efficient? Explain your findings.

Answer (a) In the following listing, the Repository class is used to provide thread-safe storage for the discovered prime numbers. The PrimeChecker class is used to create threads that will check all the odd numbers in the range that is specified in their constructor. The main thread actually runs one of the PrimeChecker objects to avoid having to idle while they are doing useful work.

1 2 3 4 5 6 7

#i n c l u d e <QThread> #i n c l u d e <QMutex> #i n c l u d e <QMutexLocker> #i n c l u d e <math . h> #i n c l u d e < s t d l i b . h> #i n c l u d e <i o s t r e a m > #i n c l u d e <v e c t o r >

8 9

u s i n g namespace s t d ;

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

//=========================================== // u s e d f o r s t o r i n g t h e r e s u l t s i n a t h r e a d −s a f e manner c l a s s Repository { private : v e c t o r <i n t > r e s u l t ; QMutex l ; public : void store ( i n t i ) ; v e c t o r <i n t > * g e t R e s u l t ( ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void Repository : : store ( i n t i ) { Q M u t e x L o c k e r m l (&l ) ; result . push_back ( i ) ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− v e c t o r <i n t > * R e p o s i t o r y : : g e t R e s u l t ( ) { r e t u r n &r e s u l t ; } //=========================================== c l a s s PrimeChecker : public QThread

48 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ 35 36 37 38

{ private : i n t first , last ; Repository * repo ;

39

public : PrimeChecker ( int f , int l , l a s t ( l ) , r e p o ( r ) {} void run ( ) ;

40 41

42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61

R e p o s i t o r y * r ) : f i r s t ( f ) , ←֓

}; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void PrimeChecker : : run ( ) { f o r ( i n t i=f i r s t ; i<=l a s t ; i+=2) { b o o l i s P r i m e=t r u e ; i n t j =3; i n t limit = sqrt ( i ) ; w h i l e ( j<= l i m i t && i s P r i m e ) { i f ( i % j == 0 ) i s P r i m e= f a l s e ; j+=2; } i f ( isPrime ) r e p o−>s t o r e ( i ) ; } }

62 63 64 65 66 67 68 69 70

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t main ( i n t argc , char * argv [ ] ) { int numThreads ; int a , b ; a=a t o i ( a r g v [ 1 ] ) ; b=a t o i ( a r g v [ 2 ] ) ; n u m T h r e a d s=a t o i ( a r g v [ 3 ] ) ;

71

// make s u r e a and b a r e odd a = ( a % 2 == 0 ) ? a+1 : a ; b = ( b % 2 == 0 ) ? b−1 : b ;

72 73 74 75

int rangePerThread = ( b − a ) * 1.0 / numThreads ; // make s u r e r a n g e P e r T h r e a d i s even r a n g e P e r T h r e a d = ( r a n g e P e r T h r e a d % 2 == 0 ) ? ←֓ rangePerThread : rangePerThread + 1;

76 77 78

79

Repository rep ; P r i m e C h e c k e r * pc [ n u m T h r e a d s ] ; i n t f i r s t , l a s t=a −2; f o r ( i n t i =0; i<n u m T h r e a d s −1; i++) { f i r s t=l a s t +2; last = first + rangePerThread ; p c [ i ]= new P r i m e C h e c k e r ( f i r s t , l a s t , & r e p ) ; p c [ i]−> s t a r t ( ) ; } p c [ n u m T h r e a d s −1] = new P r i m e C h e c k e r ( l a s t +2 , b , & r e p ) ; p c [ n u m T h r e a d s −1]−>r u n ( ) ;

80 81 82 83 84 85 86 87 88 89 90 91 92

f o r ( i n t i =0; i<n u m T h r e a d s −1; i++) p c [ i]−> w a i t ( ) ;

93 94 95

c o u t << ” T o t a l p r i m e s f o u n d ” << r e p . g e t R e s u l t ( )−>s i z e ( ) ←֓ << e n d l ; return 0;

96

97 98

}

To run the above program in order to find all the prime numbers in the range [103 , 108 ] using eight threads, one has to use the following command-line (the same conventions are used by the next versions as well):

49 $

p r i m e C h e c k e r _ V 3 1000 1 0 0 0 0 0 0 0 0 8

(b) In this version, a shared QAtomicInt object is used to fetch odd number to check from primality. The Repository class serves the same purpose as above.

1 2 3 4 5 6 7 8

#i n c l u d e <QThread> #i n c l u d e <QMutex> #i n c l u d e <QMutexLocker> #i n c l u d e <QAtomicInt> #i n c l u d e <math . h> #i n c l u d e < s t d l i b . h> #i n c l u d e <i o s t r e a m > #i n c l u d e <v e c t o r >

9 10

u s i n g namespace s t d ;

11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

//=========================================== // u s e d f o r s t o r i n g t h e r e s u l t s i n a t h r e a d −s a f e manner c l a s s Repository { private : v e c t o r <i n t > r e s u l t ; QMutex l ; public : void store ( i n t i ) ; v e c t o r <i n t > * g e t R e s u l t ( ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void Repository : : store ( i n t i ) { Q M u t e x L o c k e r m l (&l ) ; result . push_back ( i ) ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− v e c t o r <i n t > * R e p o s i t o r y : : g e t R e s u l t ( ) { r e t u r n &r e s u l t ; } //=========================================== c l a s s PrimeChecker : public QThread { private : i n t last ; Repository * repo ; QAtomicInt * next ;

41 42 43

44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64

public : P r i m e C h e c k e r ( QAtomicInt *n , i n t l , ( l ) , r e p o ( r ) , n e x t ( n ) {} void run ( ) ;

R e p o s i t o r y * r ) : l a s t ←֓

}; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void PrimeChecker : : run ( ) { int candidate ; w h i l e ( ( c a n d i d a t e = n e x t−>f e t c h A n d A d d O r d e r e d ( 2 ) ) <= l a s t ) { b o o l i s P r i m e=t r u e ; i n t j =3; i n t limit = sqrt ( candidate ) ; w h i l e ( j<= l i m i t && i s P r i m e ) { i f ( c a n d i d a t e % j == 0 ) i s P r i m e= f a l s e ; j+=2; } i f ( isPrime ) r e p o−>s t o r e ( c a n d i d a t e ) ; } }

65 66

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

50 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ 67 68 69 70 71 72 73

i n t main ( i n t argc , char * argv [ ] ) { int numThreads ; int a , b ; a=a t o i ( a r g v [ 1 ] ) ; b=a t o i ( a r g v [ 2 ] ) ; n u m T h r e a d s=a t o i ( a r g v [ 3 ] ) ;

74

// make s u r e a and b a r e odd a = ( a % 2 == 0 ) ? a+1 : a ; b = ( b % 2 == 0 ) ? b−1 : b ;

75 76 77 78

QAtomicInt first ( a ) ;

79 80

Repository rep ; P r i m e C h e c k e r * pc [ n u m T h r e a d s ] ; f o r ( i n t i =0; i<n u m T h r e a d s −1; i++) { p c [ i ]= new P r i m e C h e c k e r (& f i r s t , b , & r e p ) ; p c [ i]−> s t a r t ( ) ; } p c [ n u m T h r e a d s −1] = new P r i m e C h e c k e r (& f i r s t , b , & r e p ) ; p c [ n u m T h r e a d s −1]−>r u n ( ) ;

81 82 83 84 85 86 87 88 89 90

f o r ( i n t i =0; i<n u m T h r e a d s −1; i++) p c [ i]−> w a i t ( ) ;

91 92 93

c o u t << ” T o t a l p r i m e s f o u n d ” << r e p . g e t R e s u l t ( )−>s i z e ( ) ←֓ << e n d l ; return 0;

94

95 96

}

The overall load balancing achieved by this version is substantially better: As the numbers get larger, the checking procedure becomes more time consuming. This makes the thread assigned the last portion of the range in the previous version, dominate the execution time. (c) This version shares the load balancing efficiency of the second version, while reducing the cost of performing atomic operations. The Monitor effectively returns sets of 50 numbers to check, instead of the single one of the second version.

1 2 3 4 5 6 7 8

#i n c l u d e <QThread> #i n c l u d e <QMutex> #i n c l u d e <QMutexLocker> #i n c l u d e <QAtomicInt> #i n c l u d e <math . h> #i n c l u d e < s t d l i b . h> #i n c l u d e <i o s t r e a m > #i n c l u d e <v e c t o r >

9 10

u s i n g namespace s t d ;

11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

c o n s t i n t P A R T L E N =100; //=========================================== // u s e d f o r s t o r i n g t h e r e s u l t s i n a t h r e a d −s a f e manner c l a s s Repository { private : v e c t o r <i n t > r e s u l t ; QMutex l ; public : void store ( i n t i ) ; v e c t o r <i n t > * g e t R e s u l t ( ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void Repository : : store ( i n t i ) { Q M u t e x L o c k e r m l (&l ) ; result . push_back ( i ) ;

51 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

} //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− v e c t o r <i n t > * R e p o s i t o r y : : g e t R e s u l t ( ) { r e t u r n &r e s u l t ; } //=========================================== c l a s s Monitor { private : i n t first , last ; int currentAssign ; int assignRange ; QMutex l ; public : M o n i t o r ( i n t a , i n t b , i n t r ) : f i r s t ( a ) , l a s t ( b ) , ←֓ assignRange ( r ){ currentAssign = a ;} void g e t N e x t R a n g e ( i n t *a , i n t *b ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void Monitor : : g e t N e x t R a n g e ( i n t *a , i n t *b ) { Q M u t e x L o c k e r m l (&l ) ; i f ( c u r r e n t A s s i g n >= l a s t ) // t e r m i n a t i n g c h e c k { * a = −1; return ; } *a = currentAssign ; *b = currentAssign + assignRange ; i f (* b > last ) *b = last ; currentAssign = *b + 2; }

61 62 63 64 65 66 67

//=========================================== c l a s s PrimeChecker : public QThread { private : Repository * repo ; Monitor * mon ;

68 69 70

71 72 73 74 75 76 77 78 79 80

81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96

public : P r i m e C h e c k e r ( M o n i t o r * m , R e p o s i t o r y * r ) : r e p o ( r ) , m o n ( m←֓ ) {} void run ( ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void PrimeChecker : : run ( ) { i n t first , last ; while (1) { mon−>g e t N e x t R a n g e (& f i r s t , &l a s t ) ; i f ( f i r s t == −1) b r e a k ; // i f t h e Monitor r e t u r n s −1←֓ for first , exit f o r ( i n t i=f i r s t ; i<=l a s t ; i+=2) { b o o l i s P r i m e=t r u e ; i n t j =3; i n t limit = sqrt ( i ) ; w h i l e ( j<= l i m i t && i s P r i m e ) { i f ( i % j == 0 ) i s P r i m e= f a l s e ; j+=2; } i f ( isPrime ) r e p o−>s t o r e ( i ) ; } } }

97 98 99

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t main ( i n t argc , char * argv [ ] )

52 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++

Table 3.1: Average execution times (in sec) over 10 runs, of the three prime checker variations. The best times in each case, are highlighted.

100

{ int numThreads ; int a , b ; a=a t o i ( a r g v [ 1 ] ) ; b=a t o i ( a r g v [ 2 ] ) ; n u m T h r e a d s=a t o i ( a r g v [ 3 ] ) ;

101 102 103 104 105 106

// make s u r e a and b a r e odd a = ( a % 2 == 0 ) ? a+1 : a ; b = ( b % 2 == 0 ) ? b−1 : b ;

107 108 109 110

Monitor mon ( a , b , PARTLEN ) ; // r a n g e o f PARTLEN numbers ←֓ , o r PARTLEN/2 odd o n e s Repository rep ; P r i m e C h e c k e r * pc [ n u m T h r e a d s ] ; f o r ( i n t i =0; i<n u m T h r e a d s −1; i++) { p c [ i ]= new P r i m e C h e c k e r (& mon , & r e p ) ; p c [ i]−> s t a r t ( ) ; } p c [ n u m T h r e a d s −1] = new P r i m e C h e c k e r (& mon , & r e p ) ; p c [ n u m T h r e a d s −1]−>r u n ( ) ;

111

112 113 114 115 116 117 118 119 120 121

f o r ( i n t i =0; i<n u m T h r e a d s −1; i++) p c [ i]−> w a i t ( ) ;

122 123 124

c o u t << ” T o t a l p r i m e s f o u n d ” << r e p . g e t R e s u l t ( )−>s i z e ( ) ←֓ << e n d l ; return 0;

125

126 127

}

The exact performance figures depend on the execution platform. On a Intel i7 3770K CPU, running under Linux and compiled with the GCC 4.8.2 compiler and -O2 optimization, we were able to measure the execution times shown in Table 3.1, for checking the range [103 , 107 ]. With the exception of the single thread case, where the no-coordination-overheads version #1 wins, in all other cases the last, monitor-based version is superior.

21. Modify Listing ?? so that termination of the producer and consumer threads is accomplished via atomic integers. Answer This can be done by definining one atomic integer for each class: 1 2 3 4 5 6 7 8 9 10 11

// F i l e : m o n i t o r 1 P r o d C o n s E x e r c i s e A t o m i c . cpp . . . t e m p l a t e <typename T> c l a s s Producer { private : s t a t i c a t o m i c <i n t > n u m P r o d u c t s ; i n t ID ; s t a t i c M o n i t o r <T> * m o n ; public : s t a t i c T (* produce ) ( ) ; s t a t i c v o i d i n i t C l a s s ( i n t n u m P , M o n i t o r <T> * m , T ( * p r o d ) ( ) ) ;

53 12

P r o d u c e r <T >( i n t i ) : I D ( i ) {} void operator ( ) ( ) ;

13 14 15 16 17 18 19 20 21 22 23 24 25 26

}; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− . . . t e m p l a t e <typename T> v o i d P r o d u c e r <T > : : o p e r a t o r ( ) ( ) { while ( true ) { i n t temp = numProducts . load () ; w h i l e ( ! n u m P r o d u c t s . c o m p a r e _ e x c h a n g e _ w e a k ( t e m p , t e m p −1) ) temp = numProducts . load () ; i f ( t e m p <=0) return ;

27

T item = (* produce ) ( ) ; mon−>p u t ( i t e m ) ;

28 29 30 31 32

} } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

33 34 35 36 37 38 39 40 41 42

t e m p l a t e <typename T> c l a s s Consumer { private : i n t ID ; s t a t i c M o n i t o r <T> * m o n ; s t a t i c a t o m i c <i n t > n u m P r o d u c t s ; public : s t a t i c void (* consume ) ( T i ) ; s t a t i c v o i d i n i t C l a s s ( i n t n u m P , M o n i t o r <T> * m , v o i d ( * c o n s ) ( T←֓ ));

43

C o n s u m e r <T >( i n t i ) : I D ( i ) {} void operator ( ) ( ) ;

44 45 46 47 48 49 50 51 52 53 54 55 56

}; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− . . . t e m p l a t e <typename T> v o i d C o n s u m e r <T > : : o p e r a t o r ( ) ( ) { while ( true ) { i n t temp = numProducts . load () ; w h i l e ( ! n u m P r o d u c t s . c o m p a r e _ e x c h a n g e _ w e a k ( t e m p , t e m p −1) ) temp = numProducts . load () ; i f ( t e m p <=0) return ;

57 58 59

}

60 61

T i t e m = mon−>g e t ( ) ; // t a k e t h e i t e m o u t (* consume ) ( item ) ;

}

22. Which type of thread is favored by the solution to the readers-writers problem in Listing ??? If either of the readers or writers are favored, modify the code to produce a fair solution.

Answer The readers are favored as once the shared mutex is locked in shared mode, there is no restriction to how many readers can enter their critical section. A fair solution is shown below: 1 2

#i n c l u d e <s h a r e d m u t e x> #i n c l u d e <mutex> // For s t d : : u n i q u e l o c k

3 4 5

shared_mutex rwlock ; mutex fair ;

6 7 8 9 10 11

void Reader : : operator ( ) ( ) { fair . lock () ; s h a r e d _ l o c k r e a d G u a r d ( r w l o c k ) ; // s h a r e d l o c k fair . unlock () ;

for

readers

54 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++ 12

// r e a d e r

13 14

critical

section

}

15 16 17 18 19 20

void Writer : : operator ( ) ( ) { fair . lock () ; u n i q u e _ l o c k w r i t e G u a r d ( r w l o c k ) ; // e x c l u s i v e l o c k fair . unlock () ;

for

writers

21

// w r i t e r

22 23

critical

section

}

23. Justify the statement at the end of Section ?? about the use of the yield and sleep for methods in Listing ??. Answer The use of this thread::yield and this thread::sleep for methods results in calling the kernel and switching to another thread, after changing the state of the executing thread to ready and blocked respectively.

Switching to another fiber requires that the fiber library is called. And this can only happen if this fiber::yield and this fiber::sleep for are called in our case.

24. Devise a benchmark for testing how much faster spawing a fiber is compared to spawning a thread. Answer The folowing code: 1 2 3 4 5 6

#i n c l u d e <b o o s t / f i b e r / a l l . hpp> #i n c l u d e <t h r e a d > #i n c l u d e <mutex> #i n c l u d e <i o s t r e a m > #i n c l u d e <chrono> #i n c l u d e ” semaphore . cpp ”

7 8 9

u s i n g namespace s t d ; u s i n g namespace s t d : : c h r o n o ;

10 11 12 13 14

// ****************************************** i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t numTests = atoi ( argv [ 1 ] ) ;

15 16

auto t1 = h i g h _ r e s o l u t i o n _ c l o c k : : now ( ) ;

17 18 19 20 21 22 23 24 25 26 27

f o r ( i n t i = 0 ; i < n u m T e s t s ; i++) { thread t ( [ ] ( ) { return 0; } ); t . join () ; }

28 29 30 31

auto t2 = h i g h _ r e s o l u t i o n _ c l o c k : : now ( ) ; auto dur = t2 − t1 ; c o u t << ”Time p e r t h r e a d : ” << d u r a t i o n _ c a s t <m i l l i s e c o n d s >( d u r ←֓ ) . c o u n t ( ) * 1 . 0 L / n u m T e s t s << e n d l ;

32 33 34 35 36 37 38 39 40 41 42

t1 = h i g h _ r e s o l u t i o n _ c l o c k : : now ( ) ; // c r e a t e t h e f i b e r s f o r ( i n t i = 0 ; i < n u m T e s t s ; i++) { boost : : fibers : : fiber f ( [ ] ( ) { return 0; } );

55 f . join () ; } t2 = h i g h _ r e s o l u t i o n _ c l o c k : : now ( ) ; dur = t2 − t1 ; c o u t << ”Time p e r f i b e r : ” << d u r a t i o n _ c a s t <m i l l i s e c o n d s >( d u r ) ←֓ . c o u n t ( ) * 1 . 0 L / n u m T e s t s << e n d l ; return 0;

43 44 45 46 47

48 49

}

Produces this output on an AMD Ryzen 3900x (times in ms): . / f i b e r _ b e n c h 1000000 Time per thread : 0.015655 Time per fiber : 0.001067

56 CHAPTER 3. THREADS AND CONCURRENCY IN STANDARD C++

Chapter 4

Parallel data structures Exercises 1. Modify the queue structure of Listing 4.3, so that a thread trying to deque from an empty container will wait for a signal, instead of throwing an exception. Answer 2. In Figure 4.5 a possible scenario of list corruption is described if validation is not performed. Suggest other possible ways the list could be corrupted.

Answer For two threads any of the insert/erase combos that affect different neighboring nodes could be tested. For example, if Thread2 was erasing 7 instead of 11, then 9 would not be inserted in the list.

If Thread1 was inserting 8, then whem Thread1 completed its insertion, 9 would be removed from the list.

3. Someone suggests that in the lazy synchronization list, we could replace the marked flag by setting the next field to mullptr. Could this modification be a viable alternative? Answer This is not a viable alternative because another thread may reach the deleted node during the list traversal, and try to dereference the next pointer causing a protection fault.

4. Modify the stack implementation of Listing 4.16 to support the empty and size methods. Answer We need to add an atomic integer to count the number of items in the stack: 1 2 3 4 5 6

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− t e m p l a t e < typename T > c l a s s s t a c k _ l o c k _ f r e e { private : . . . std : : atomic < i n t > numItems {0};

7 8 9 10 11

public : bool empty () ; . . . };

12 13 14 15 16 17 18

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− t e m p l a t e < typename T > b o o l s t a c k _ l o c k _ f r e e < T > : : e m p t y ( ) { r e t u r n n u m I t e m s . l o a d ( ) ==0; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

57

58

CHAPTER 4. PARALLEL DATA STRUCTURES

19 20 21 22 23 24

25 26 27 28 29 30 31

t e m p l a t e < typename T > i n t s t a c k _ l o c k _ f r e e < T > : : s i z e ( ) { return numItems . load () ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− t e m p l a t e < typename T > v o i d s t a c k _ l o c k _ f r e e < T > : : p u s h ( c o n s t T←֓ & i) { Node < T > * newNode = getFree () ; n e w N o d e −>d a t a = i ; n e w N o d e −>n e x t = h e a d . l o a d ( ) ; w h i l e ( ! h e a d . c o m p a r e _ e x c h a n g e _ w e a k ( n e w N o d e −>n e x t , n e w N o d e ) ) ; n u m I t e m s ++; }

32 33 34 35 36 37

38 39 40 41 42 43 44

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− t e m p l a t e < typename T > b o o l s t a c k _ l o c k _ f r e e < T > : : p o p ( T &i ) { Node < T > * candidate = head . load () ; w h i l e ( c a n d i d a t e != n u l l p t r && ! h e a d . c o m p a r e _ e x c h a n g e _ w e a k ( ←֓ c a n d i d a t e , c a n d i d a t e −>n e x t ) ) ; i f ( c a n d i d a t e == n u l l p t r ) return f a l s e ; i = c a n d i d a t e −>d a t a ; releaseNode ( candidate ) ; n u m I t e m s −−; return true ; }

5. In Listing 4.16 we have a stack that manages node memory allocation, one node at a time. Modify the code so that a bunch of Node instances are created initially as an array, and kept in the freeNodes list. What are the advantages and disadvantages of this solution? Answer We just need to modify the constructor and destructor of the class. The rest of the code can remain identical. The destructor needs to be modified because the initial nodes are allocated as an array and they cannot be deallocated as individual nodes.

1 2 3 4 5 6 7 8 9

/* //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− t e m p l a t e < typename T > c l a s s s t a c k l o c k f r e e { private : . . .

Node<T> * i n i t S t o r e ; int initNodes ; . . .

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

public : s t a c k l o c k f r e e ( i n t i n i t N o d e s =100) { head . s t o r e ( n u l l p t r ) ; f r e e N o d e s . s t o r e ( new Node<T>[ i n i t N o d e s ] ) ; Node<T> * tmp = f r e e N o d e s . l o a d ( ) ; f o r ( i n t i =0; i <i n i t N o d e s −1; i ++) tmp [ i ] . n e x t = &(tmp [ i + 1 ] ) ; tmp [ i n i t N o d e s − 1 ] . n e x t=n u l l p t r ; i n i t S t o r e = tmp ; t h i s −>i n i t N o d e s = i n i t N o d e s ; } ˜ stack lock free () ; s t a c k l o c k f r e e ( c o n s t s t a c k l o c k f r e e &) = d e l e t e ; s t a c k l o c k f r e e & o p e r a t o r= ( c o n s t s t a c k l o c k f r e e &) = d e l e t e ; };

27 28 29

30 31

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− t e m p l a t e < typename T > s t a c k l o c k f r e e < T > : : ˜ s t a c k l o c k f r e e ←֓ () { Node < T > * tmp , * aux ;

59 tmp = f r e e N o d e s ; w h i l e ( tmp != n u l l p t r ) { aux = tmp−>n e x t ; i f ( tmp< i n i t S t o r e | | tmp>i n i t S t o r e+i n i t N o d e s ) // o n l y not part o f i n i t S t o r e c a l l d e l e t e d e l e t e tmp ; tmp = aux ; } tmp = head ; w h i l e ( tmp != n u l l p t r ) { aux = tmp−>n e x t ; i f ( tmp< i n i t S t o r e | | tmp>i n i t S t o r e+i n i t N o d e s ) // o n l y not part o f i n i t S t o r e c a l l d e l e t e d e l e t e tmp ; tmp = aux ; } delete [ ] initStore ;

32 33 34 35 36

37 38 39 40 41 42 43 44

45 46 47 48 49

i f ←֓

i f ←֓

}

The advantage is that we can allocate a bunch of Node instances in one go, making the initial operation of the stack faster.

6. What would happen in the lazy synchronization list of Section ??, if an item was inserted to the list after it was erased by another thread? Answer When the thread calls the findLoc method as part of the insert, any nodes marked as deleted will be removed. As such, a new Node will be created to hold the item.

60

CHAPTER 4. PARALLEL DATA STRUCTURES

Chapter 5

Distributed memory programming Exercises 1. Write a MPMD version of the “Hello World” program of Figure 5.2, in effect eliminating the if/else structure around which the program is built. You may use the C or C++ bindings.

Answer Two source code files are needed: // F i l e : h e l l o n o d e 0 . c #i n c l u d e <mpi . h> #i n c l u d e <s t r i n g . h> #i n c l u d e <s t d i o . h> #d e f i n e MESSTAG 0 #d e f i n e MAXLEN 100 i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { M P I _ I n i t (& a r g c , &a r g v ) ; i n t r a n k , num , i ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &n u m ) ; c h a r m e s s [ ] = ” H e l l o World ” ; f o r ( i = 1 ; i < n u m ; i++) M P I _ S e n d ( m e s s , s t r l e n ( m e s s ) + 1 , M P I _ C H A R , i , M E S S T A G , ←֓ MPI_COMM_WORLD ) ; MPI_Finalize () ; return 0; } // F i l e : h e l l o n o d e i . c #i n c l u d e <mpi . h> #i n c l u d e <s t r i n g . h> #i n c l u d e <s t d i o . h> #d e f i n e MESSTAG 0 #d e f i n e MAXLEN 100 i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { M P I _ I n i t (& a r g c , &a r g v ) ; i n t r a n k , num , i ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; char mess [ MAXLEN ] ;

61

62

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING MPI_Status status ; M P I _ R e c v ( m e s s , M A X L E N , M P I _ C H A R , 0 , M E S S T A G , M P I _ C O M M _ W O R L D , &←֓ status ) ; p r i n t f ( ”%i r e c e i v e d %s \n” , r a n k , m e s s ) ; MPI_Finalize () ; return 0; }

The program can run with the following appfile: # File : hello app −h o s t l o c a l h o s t −n p 1 h e l l o _ n o d e 0 −h o s t l o c a l h o s t −n p 3 h e l l o _ n o d e _ i

and via the command sequence: $

$

$

m p i c c h e l l o _ n o d e 0 . c −o h e l l o _ n o d e 0 m p i c c h e l l o _ n o d e _ i . c −o h e l l o _ n o d e _ i m p i r u n −a p p h e l l o _ a p p

2. Write a SPMD version of the two programs shown in Listing 5.5.

Answer The only requirement is the inclusion of an if-else control logic.

1 2 3 4

#i n c l u d e <mpi . h> #i n c l u d e <s t d i o . h> #i n c l u d e < s t r i n g . h> #i n c l u d e < s t d l i b . h>

5 6 7

#d e f i n e MAXLEN 100 c h a r * g r e e t i n g s [ ] = { ” H e l l o ” , ” Hi ” , ” A w a i t i n g your command” } ;

8 9

char buff [ MAXLEN ] ;

10 11 12 13 14 15

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { M P I _ S t a t u s st ; int procNum ; i n t rank ;

16

M P I _ I n i t (& a r g c , &a r g v ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &p r o c N u m ) ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; i f ( r a n k == 0 ) { w h i l e (−− p r o c N u m ) { M P I _ R e c v ( b u f f , M A X L E N , M P I _ C H A R , M P I _ A N Y _ S O U R C E , ←֓ M P I _ A N Y _ T A G , M P I _ C O M M _ W O R L D , &s t ) ; i n t aux ; M P I _ G e t _ c o u n t (&st , M P I _ C H A R , &a u x ) ; buff [ aux ] = 0 ; p r i n t f ( ”%s \n” , b u f f ) ; } } else { srand ( time (0) ) ; i n t grID = rand () % 3; s p r i n t f ( b u f f , ”Node %i s a y s %s ” , r a n k , g r e e t i n g s [ g r I D ] ) ; M P I _ S e n d ( b u f f , s t r l e n ( b u f f ) , M P I _ C H A R , 0 , 0 , ←֓ MPI_COMM_WORLD ) ; }

17 18 19 20 21 22 23 24

25 26 27 28 29 30 31 32 33 34 35 36

37 38

MPI_Finalize () ;

39 40

}

63 3. Modify the program shown in Listing 5.5 so that the master node prints out a list of the processes IDs for which the message has not been read yet. Your output should be similar to: 1 2 3 4

m p i r u n −n p 1 m a s t e r : −n p 3 w o r k e r Node 2 says Hi . A w a i t i n g nodes : 1 3 Node 3 says Hi . A w a i t i n g nodes : 1 Node 1 says Hi . A w a i t i n g nodes :

$

Answer visit Library Genesis for more solution manuals: libgen.is, libgen.st, libgen.rs, forum.mhut.org An STL set<int> container can be used to hold the ID’s of the processes still pending: 1 2 3 4 5

#i n c l u d e <mpi . h> #i n c l u d e <s t d i o . h> #i n c l u d e < s t r i n g . h> #i n c l u d e <s e t > #i n c l u d e <i o s t r e a m >

6 7

u s i n g namespace s t d ;

8 9 10

#d e f i n e MAXLEN 100 char buff [ MAXLEN ] ;

11 12 13 14 15

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { M P I _ S t a t u s st ; int procNum ;

16

M P I _ I n i t (& a r g c , &a r g v ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &p r o c N u m ) ; set<i n t > l e f t O v e r ; f o r ( i n t i =1; i<p r o c N u m ; i++) leftOver . insert ( i ) ;

17 18 19 20 21 22

w h i l e (−− p r o c N u m ) { M P I _ R e c v ( b u f f , M A X L E N , M P I _ C H A R , M P I _ A N Y _ S O U R C E , ←֓ MPI_ANY_TAG , M P I _ C O M M _ W O R L D , &s t ) ; i n t aux ; M P I _ G e t _ c o u n t (&st , M P I _ C H A R , &a u x ) ; l e f t O v e r . erase ( st . M P I _ S O U R C E ) ;

23 24 25

26 27 28 29 30

buff [ aux ] = 0 ; p r i n t f ( ”%s . A w a i t i n g n o d e s ” , b u f f ) ; f o r ( set<i n t > : : i t e r a t o r i t=l e f t O v e r . b e g i n ( ) ; i t != l e f t O v e r . ←֓ e n d ( ) ; ++i t ) c o u t << ’ ’ << * i t ; c o u t << ’ \n ’ ;

31 32 33

34 35 36

} MPI_Finalize () ;

37 38 39

}

4. Create a model of the characteristics of the communication link joining two processes running on two different machines, i.e. calculate the start-up latency and communication rate, by implementing and testing a ping-pong benchmark program. A ping-pong program measures the time elapsed between sending a message, having it bounce at its destination, and receiving it back at its origin. By varying the message size, you can use statistical methods (least-squares) to estimate the start-up latency and rate as the intercept and slope respectively, of the line fitted to the experimental data.

Answer

64

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING The following code will calculate the average communication time over REP number of tests between two processes, and for message sizes ranging from 0 to MAX MESG in increments of 1000.

1 2 3

#i n c l u d e <s t d i o . h> #i n c l u d e < s t d l i b . h> #i n c l u d e <mpi . h>

4 5

u s i n g namespace s t d ;

6 7 8 9

#d e f i n e MESG TAG 0 #d e f i n e END TAG 1 #d e f i n e MAX MESG 1000000

10 11

const i n t REP = 10;

12 13 14 15 16 17 18 19 20 21 22

i n t main ( i n t argc , char * argv [ ] ) { i n t size , rank ; int namelen ; char processor_name [ MPI_MAX_PROCESSOR_NAME ] ; int mesg_size ; i n t tag ; char * buffer ; double start_time , end_time ; MPI_Status status ;

23 24

b u f f e r = new c h a r [ M A X _ M E S G ] ;

25 26 27 28 29

M P I _ I n i t (& a r g c , &a r g v ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &s i z e ) ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ G e t _ p r o c e s s o r _ n a m e ( p r o c e s s o r _ n a m e , &n a m e l e n ) ;

30 31

32 33 34 35 36

p r i n t f ( ” P r o c e s s %d o f %d on %s \n” , r a n k , s i z e , p r o c e s s o r _ n a m e ) ←֓ ; i f ( size < 2) { p r i n t f ( ” Need more than 1 p r o c e s s o r t o run \n” ) ; exit (1) ; }

37 38 39

if

( r a n k == 0 ) { f o r ( i n t m e s g _ s i z e = 0 ; m e s g _ s i z e <= M A X _ M E S G ; m e s g _ s i z e +=←֓ 1000) { start_time = MPI_Wtime () ; f o r ( i n t i = 0 ; i < R E P ; i++) { M P I _ S e n d ( b u f f e r , m e s g _ s i z e , M P I _ C H A R , 1 , tag , ←֓ MPI_COMM_WORLD ) ; M P I _ R e c v ( b u f f e r , M A X _ M E S G , M P I _ C H A R , 1 , ←֓ M P I _ A N Y _ T A G , M P I _ C O M M _ W O R L D , &s t a t u s ) ; }

40

41 42 43 44 45

46

47 48

end_time = MPI_Wtime () ; p r i n t f ( ”%i % l f \n” , m e s g _ s i z e , ( e n d _ t i m e − s t a r t _ t i m e ) ←֓ / 2 / REP ) ;

49 50

}

51 52

tag = END_TAG ; M P I _ S e n d ( b u f f e r , 0 , M P I _ C H A R , 1 , tag ,

53 54 55 56 57 58 59 60

61 62 63 64

MPI_COMM_WORLD ) ;

} else { while (1) { M P I _ R e c v ( b u f f e r , M A X _ M E S G , M P I _ C H A R , 0 , M P I _ A N Y _ T A G , ←֓ M P I _ C O M M _ W O R L D , &s t a t u s ) ; i f ( s t a t u s . M P I _ T A G == E N D _ T A G ) break ; M P I _ G e t _ c o u n t (& s t a t u s , M P I _ C H A R , &m e s g _ s i z e ) ; M P I _ S e n d ( b u f f e r , m e s g _ s i z e , M P I _ C H A R , 0 , tag , ←֓

65 MPI_COMM_WORLD ) ; }

65

}

66 67

delete [ ] buffer ; MPI_Finalize () ; return 0;

68 69 70 71

}

5. How would we need to modify the broadcasting program of Listing 5.9, if the source of the message was an arbitrary process, and not the one with rank 0?

Answer The solution follows the guidelines of Listing 5.9, i.e. treating the processes as nodes in a hypercube, and proceeding to propagate the message along the hypercube’s dimension one-by-one. However, the source-destination process pairings has to be done using an alternative approach. The key to the pairing calculation is the communication phase during which a node receives data. This is equal to the distance of a process/node from the root on the virtual hypercube, which can be calculated by counting the number of bits in the rank XOR root expression (variable recvPhase, line 35).

The sender node can be found by reversing the bit of a node’s rank in the recvPhase - 1 position (line 51). Once a node receives the message, it can start to send it to all the nodes which are adjacent to it (ranks different by a single bit). The ranks of the receivers can be determined by reversing the bits in a node’s rank from position recvPhase and onwards (lines 58-64).

1 2 3 4

#i n c l u d e <mpi . h> #i n c l u d e <s t r i n g . h> #i n c l u d e <s t d i o . h> #d e f i n e MESSTAG 0

5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

// ***************************************** // R e t u r n s t h e number o f s e t b i t s i n i t s argument int bitCount ( int i ) { int count = 0; w h i l e ( i != 0 ) { int j = i ; i >>= 1 ; i f ( i != j ) c o u n t ++; } return count ; }

20 21

// *****************************************

22 23 24 25

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { M P I _ I n i t (& a r g c , &a r g v ) ;

26 27 28 29 30 31 32

33

i n t r a n k , num , i ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &n u m ) ; MPI_Status status ; i n t root = atoi ( argv [ 1 ] ) ; d o u b l e d a t a = r a n k * 1 . 0 ; // s o m e t h i n g s i m p l e t o e n a b l e ←֓ v e r i f i c a t i o n of correct receipt

66

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING p r i n t f ( ”%i w i l l be s e r v e d d u r i n g p h a s e %i \n” , r a n k , b i t C o u n t ( ←֓ rank ˆ root ) ) ; i n t recvPhase = bitCount ( rank ˆ root ) ; i f ( r a n k == r o o t ) { int phase = 0; i n t d e s t I D = r a n k ˆ ( 1 << p h a s e ) ; while ( destID < num ) { // a s u b s e t o f n o d e s g e t s a ←֓ message p r i n t f ( ”#%i s e n d i n g t o %i \n” , r a n k , d e s t I D ) ; M P I _ S e n d (& d a t a , 1 , M P I _ D O U B L E , d e s t I D , M E S S T A G , ←֓ MPI_COMM_WORLD ) ; p h a s e ++; d e s t I D = r a n k ˆ ( 1 << p h a s e ) ; } } else { int srcID ; s r c I D = r a n k ˆ ( 1 << ( r e c v P h a s e − 1 ) ) ; p r i n t f ( ”SRC o f #%i : %i \n” , r a n k , s r c I D ) ;

34

35 36 37 38 39 40 41

42 43

44 45 46 47 48 49 50 51 52 53

M P I _ R e c v (& d a t a , 1 , M P I _ D O U B L E , s r c I D , M E S S T A G , ←֓ M P I _ C O M M _ W O R L D , &s t a t u s ) ;

54

55

// c a l c u l a t e t h e ID o f t h e node t h a t w i l l r e c e i v e a copy o f ←֓ t h e message i n t d e s t I D = r a n k ˆ ( 1 << r e c v P h a s e ) ; while ( destID < num ) { p r i n t f ( ”#%i s e n d i n t o %i \n” , r a n k , d e s t I D ) ; M P I _ S e n d (& d a t a , 1 , M P I _ D O U B L E , d e s t I D , M E S S T A G , ←֓ MPI_COMM_WORLD ) ; r e c v P h a s e ++; d e s t I D = r a n k ˆ ( 1 << r e c v P h a s e ) ; }

56

57 58 59 60 61

62 63 64

} p r i n t f ( ”Node #%i h as % l f \n” , r a n k , d a t a ) ; MPI_Finalize () ; return 0;

65 66 67 68 69

}

6. Assuming that the execution platform of your program, consists of 4 machines with identical architecture but different CPU clocks: one with 4GHz, one with 3 GHz and 2 with 2 GHz. How should you split the matrix A used in the example of Section 5.11.1, in order to solve the matrix-vector product problem in the smallest possible time?

Answer The heterogeneity of the execution platform calls for an uneven partitioning of the problem’s data, i.e. we should assign a bigger portion of the problem to the faster nodes. If we assume that the computing speed of the machines is linearly dependent on their clock, and we ignore communication overheads, then if T is the time to complete the multiplication on a 1GHz machine, then on a C times faster-clock machine we would need T time C . If each node i were assigned parti percent of the M rows of the input matrix, then it would need time Ti = parti CTi to complete its part of the computation.

It can be easily proven by contradiction, that the overall execution time is minimized if Ti = Tj for ∀i 6= j, which translates to: Ti = Tj ⇒ parti

T T Ci = partj ⇒ parti = partj Ci Cj Cj

(5.1)

67 As the sum of all parts should be equal to 1, we can derive the value of part0 and subsequently from Eq. 5.1 all the other parts: X ∀i

parti = 1 ⇒ part0

X Ci ∀i

C0

= 1 ⇒ part0 = (

X Ci ∀i

C0

)−1

(5.2)

From the problem description we have C0 = 4GHz, C1 = 3GHz, C2 = 2GHz and C3 = 2GHz, which when substituted in Eq. 5.1 and 5.2 return: 4 3 2 2 4 part0 = ( + + + )−1 = 4 4 4 4 11 part1 =

3 4 3 = 11 4 11

part2 = part3 =

4 2 2 = 11 4 11

The modified code for setting up the scattering of the matrix to the four nodes, is shown below: 1

d o u b l e p a r t [ ] = { 4 . 0 / 1 1 , 3 . 0 / 1 1 , 2 . 0 / 1 1 , 2 . 0 / 11 } ;

2 3 4 5 6 7 8 9 10 11

12 13 14

if

( r a n k == 0 ) { int displs [ N ] ; int sendcnts [ N ] ; f o r ( i n t i = 0 ; i < N ; i++) { i n t rowsForProcess = M * part [ i ] ; sendcnts [ i ] = M * rowsForProcess ; d i s p l s [ i ] = ( i == 0 ) ? 0 : d i s p l s [ i − 1 ] + s e n d c n t s [ i −←֓ 1]; i f ( i == N − 1 ) sendcnts [ i ] = M * M − displs [ i ] ; }

15 16

M P I _ S c a t t e r v ( A , s e n d c n t s , d i s p l s , M P I _ D O U B L E , M P I _ I N _ P L A C E ←֓ , 0 , MPI_DOUBLE , 0 , MPI_COMM_WORLD ) ;

7. Write a program that performs gathering as efficiently as possible, using point-to-point communications, i.e. the equivalent of MPI Gather. What is the time complexity of your algorithm?

Answer The answer is very closely related to the solution of Exercise 5.5. The major thing that needs to be altered, is the reversal of data flow, i.e. replacing MPI Send with MPI Recv and vice-versa. The XOR operator can be still used to calculate the destination and source nodes in the communications.

The other change stems from the need to accumulate the data that are collected. Because the order of collection does not necessarily (and typically does not) match the node ranking, all the parts that are accumulated are prefixed in the same buffer with the rank of the node that contributed them. In this fashion, the designated root of the gathering operation, can reshuffle the parts and produce an outcome compatible with what MPI Gather would produce.

68

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING The following program requires as a command-line parameter, the rank of the process/node that will gather the data.

1 2 3 4 5 6 7

#i n c l u d e <mpi . h> #i n c l u d e <s t r i n g . h> #i n c l u d e <s t d i o . h> #i n c l u d e <s t d l i b . h> #i n c l u d e <math . h> #d e f i n e MESSTAG 0 #d e f i n e MESSSIZE 10

8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

// ***************************************** // R e t u r n s t h e number o f s e t b i t s i n i t s argument int bitCount ( int i ) { int count = 0; w h i l e ( i != 0 ) { int j = i ; i >>= 1 ; i f ( i != j ) c o u n t ++; } return count ; }

23 24

// *****************************************

25 26 27 28

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { M P I _ I n i t (& a r g c , &a r g v ) ;

29 30 31 32 33 34 35 36 37

i n t r a n k , num , i ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &n u m ) ; MPI_Status status ; i n t root = atoi ( argv [ 1 ] ) ; i n t data [ MESSSIZE ] ; f o r ( i n t i = 0 ; i < M E S S S I Z E ; i++) data [ i ] = rank * 1 . 0 ; // s o m e t h i n g s i m p l e t o e n a b l e ←֓ v e r i f i c a t i o n of correct receipt

38 39

i n t t o t a l P h a s e s = c e i l ( l o g 2 ( n u m ) ) ; // t o t a l

c o l l e c t i o n phases

40 41

42

i n t s e n d P h a s e = b i t C o u n t ( r a n k ˆ r o o t ) ; // p h a s e when a node ←֓ forwards i t s part o f the data p r i n t f ( ”%i w i l l s e n t t o r o o t d u r i n g p h a s e %i \n” , r a n k , ←֓ sendPhase ) ;

43 44

45 46 47 48 49 50 51 52 53

// a l l o c a t e t h e temp . b u f f e r f o r h o l d i n g t h e d a t a t o be ←֓ communicated t o w a r d s t h e ” r o o t ” i n t b u f f S p a c e = ( M E S S S I Z E + 1 ) << ( t o t a l P h a s e s − s e n d P h a s e ) ; i f ( r a n k == r o o t ) buffSpace = num * ( MESSSIZE + 1) ; i n t * c o m m B u f f = new i n t [ b u f f S p a c e ] ; int inCommBuff = MESSSIZE + 1; // s e t u p l o c a l p a r t o f t h e comm d a t a and t h e r e l e v a n t ” h e a d e r ” commBuff [ 0 ] = rank ; memcpy ( c o m m B u f f + 1 , data , s i z e o f ( i n t ) * M E S S S I Z E ) ; p r i n t f ( ”#%i h as BUFF %i \n” , r a n k , b u f f S p a c e ) ;

54 55 56 57 58 59 60 61 62 63

64

if

( r a n k == r o o t ) { int phase = totalPhases − 1; i n t s r c I D = r a n k ˆ ( 1 << p h a s e ) ; w h i l e ( p h a s e >= 0 ) { i f ( srcID < num ) { p r i n t f ( ”#%i r e c v from %i SPACE %i \n” , r a n k , s r c I D ←֓ , buffSpace − inCommBuff ) ; M P I _ R e c v ( c o m m B u f f + i n C o m m B u f f , b u f f S p a c e − ←֓ i n C o m m B u f f , M P I _ I N T , s r c I D , M E S S T A G , ←֓ M P I _ C O M M _ W O R L D , &s t a t u s ) ;

69 int recvd ; M P I _ G e t _ c o u n t (& s t a t u s , M P I _ I N T , &r e c v d ) ; i n C o m m B u f f += r e c v d ;

65 66 67

} p h a s e −−; s r c I D = r a n k ˆ ( 1 << p h a s e ) ;

68 69 70

}

71 72

// now r e −a r r a n g e t h e c o l l e c t e d d a t a i n t gatheredData [ MESSSIZE * num ] ; i n t pos = 0 ; f o r ( i n t i = 0 ; i < n u m ; i++) { i n t d e s t P a r t I d x = c o m m B u f f [ p o s ++]; f o r ( i n t j = 0 ; j < M E S S S I Z E ; j++) g a t h e r e d D a t a [ j + d e s t P a r t I d x * M E S S S I Z E ] = c o m m B u f f [ ←֓ p o s ++]; }

73 74 75 76 77 78 79 80

81 82

// p r i n t −o u t f o r v e r i f i c a t i o n p u r p o s e s f o r ( i n t i = 0 ; i < M E S S S I Z E * n u m ; i++) p r i n t f ( ”%i ” , g a t h e r e d D a t a [ i ] ) ; p r i n t f ( ” \n” ) ;

83 84 85 86

} else {

87 88 89

int phase = totalPhases ; // c a l c u l a t e t h e ID o f t h e node t h a t w i l l s e n d i t s p a r t o f ←֓ t h e d a t a t o t h i s node i n t s r c I D = r a n k ˆ ( 1 << ( p h a s e − 1 ) ) ; // c o l l e c t from o t h e r n o d e s f i r s t while ( phase > sendPhase ) { i f ( srcID < num ) { p r i n t f ( ”#%i r e c v from %i \n” , r a n k , s r c I D ) ; M P I _ R e c v ( c o m m B u f f + i n C o m m B u f f , b u f f S p a c e − ←֓ i n C o m m B u f f , M P I _ I N T , s r c I D , M E S S T A G , ←֓ M P I _ C O M M _ W O R L D , &s t a t u s ) ; int recvd ; M P I _ G e t _ c o u n t (& s t a t u s , M P I _ I N T , &r e c v d ) ; i n C o m m B u f f += r e c v d ; } p h a s e −−; s r c I D = r a n k ˆ ( 1 << ( p h a s e − 1 ) ) ; }

90 91

92 93 94 95 96 97 98 99

100 101 102 103 104 105 106 107

// t h e n s e n d t o w a r d s t h e g a t h e r i n g r o o t i n t d e s t I D = r a n k ˆ ( 1 << ( s e n d P h a s e − 1 ) ) ; p r i n t f ( ”DEST o f #%i : %i \n” , r a n k , d e s t I D ) ;

108 109 110 111

M P I _ S e n d ( c o m m B u f f , i n C o m m B u f f , M P I _ I N T , d e s t I D , M E S S T A G , ←֓ MPI_COMM_WORLD ) ;

112

}

113 114

MPI_Finalize () ; return 0;

115 116 117

}

8. Write a program that performs scattering as efficiently as possible, using point-to-point communications, i.e. the equivalent of MPI Scatter. What is the volume of data collectively communicated, if each process is to receive K number of items? Express this number as a function of K and the number of processes N .

Answer The following program requires as a command-line parameter, the rank of the process/node that will scatter the data. The algorithm mirrors the broadcasting algorithm used in Exercise 5.5, as far as the tree of nodes

70

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING

2 3,1,5,7

1,5

1

3

0,4

0 7

4

7

4

6

6

5

5 Figure 5.1: An example of a scattering operation initiated from node 2. The edges are labeled with the parts sent in a single operation. that is implicitly formed is concerned.

The major difference is that the data to be scattered are rearranged in the root node, as the data sent-down a branch of the scatter “tree” are not related to consecutive ranking nodes. Figure 5.1 illustrates a scattering operation initiated from node 2 in an eight-node communicator.

The recursive findOrder() function of lines 31-43, calculates the order with which the parts need to be rearranged, so that parts collectively sent down a tree branch, occupy a contiguous buffer space.

For the example shown in Figure 5.1, the parts residing in the scatteredData array are rearranged accordingly to form the sequence : 2 3 1 5 7 0 4 6.

1 2 3 4 5 6 7

#i n c l u d e <mpi . h> #i n c l u d e <s t r i n g . h> #i n c l u d e <s t d i o . h> #i n c l u d e <s t d l i b . h> #i n c l u d e <i o s t r e a m > #i n c l u d e <v e c t o r > #i n c l u d e <math . h>

8 9

u s i n g namespace s t d ;

10 11 12

#d e f i n e MESSTAG 0 #d e f i n e MESSSIZE 10

13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

// ***************************************** // R e t u r n s t h e number o f s e t b i t s i n i t s argument int bitCount ( int i ) { int count = 0; w h i l e ( i != 0 ) { int j = i ; i >>= 1 ; i f ( i != j ) c o u n t ++; } return count ; }

28 29 30

31 32

// ***************************************** // u s e d t o c a l c u l a t e i n what o r d e r t h e d a t a t o be s c a t t e r e d ←֓ s h o u l d be a r r a n g e d v o i d f i n d O r d e r ( i n t ID , i n t p h a s e , i n t N , v e c t o r < i n t >&v ) {

71 if

( I D >= N ) return ; v . p u s h _ b a c k ( ID ) ; i n t n e x t I D = I D ˆ ( 1 << p h a s e ) ; while ( nextID < N ) { findOrder ( nextID , phase + 1 , N , v ) ; p h a s e ++; n e x t I D = I D ˆ ( 1 << p h a s e ) ; }

33 34 35 36 37 38 39 40 41 42 43

}

44 45

// *****************************************

46 47 48 49

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { M P I _ I n i t (& a r g c , &a r g v ) ;

50 51 52 53 54 55 56

i n t r a n k , num , i ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &n u m ) ; MPI_Status status ; i n t root = atoi ( argv [ 1 ] ) ; i n t * data ; // d a t a t o be s c a t t e r e d

57 58

i n t totalPhases = ceil ( log2 ( num ) ) ; phases

// t o t a l communication ←֓

59 60

i n t s e n d P h a s e = b i t C o u n t ( r a n k ˆ r o o t ) ; // p h a s e when a node ←֓ forwards i t s part o f the data

61 62

63 64 65

// a l l o c a t e t h e temp . b u f f e r f o r h o l d i n g t h e d a t a t o be ←֓ communicated i n t b u f f S p a c e = M E S S S I Z E << ( t o t a l P h a s e s − s e n d P h a s e ) ; i f ( r a n k == r o o t ) buffSpace = num * MESSSIZE ;

66 67

i n t * s c a t t e r e d D a t a = new i n t [ b u f f S p a c e ] ;

68 69 70 71 72 73

if

( r a n k == r o o t ) {

d a t a = new i n t [ M E S S S I Z E * n u m ] ; f o r ( i n t i = 0 ; i < b u f f S p a c e ; i++) data [ i ] = i ; // s o m e t h i n g s i m p l e t o e n a b l e of correct receipt

v e r i f i c a t i o n ←֓

74 75 76

v e c t o r < i n t >r e O r d e r ; f i n d O r d e r ( r o o t , 0 , num , r e O r d e r ) ;

77 78 79 80 81 82

// r e −a r r a n g e t h e d a t a t o be s c a t t e r e d i n t pos = 0 ; f o r ( i n t i = 0 ; i < n u m ; i++) { int destPartIdx = reOrder [ i ] ;

83

f o r ( i n t j = 0 ; j < M E S S S I Z E ; j++) s c a t t e r e d D a t a [ p o s ++] = d a t a [ j + d e s t P a r t I d x * ←֓ MESSSIZE ] ;

84 85

86

}

87 88

89

// keep t h e d a t a t h a t d e s t i n e d f o r t h e r o o t o f t h e s c a t t e r ←֓ tree memcpy ( data , scatteredData , M E S S S I Z E * s i z e o f ( i n t ) ) ;

90 91 92

93 94 95 96

int phase = 0; int buffPos = MESSSIZE ; // s k i p t h e p a r t a s s i g n e d t o t h e ←֓ root w h i l e ( p h a s e <= t o t a l P h a s e s ) { i n t t o S e n d = M E S S S I Z E << ( t o t a l P h a s e s − p h a s e − 1 ) ; i n t d e s t I D = r a n k ˆ ( 1 << p h a s e ) ;

97 98 99

if

( destID < num ) {

72

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING p r i n t f ( ”#%i s e n d s t o %i (% i ) a t %i \n” , r a n k , ←֓ destID , toSend , buffPos ) ; M P I _ S e n d ( s c a t t e r e d D a t a + b u f f P o s , t o S e n d , M P I _ I N T , ←֓ destID , MESSTAG , MPI_COMM_WORLD ) ; b u f f P o s += t o S e n d ;

100

101

102

} p h a s e ++;

103 104

}

105

} else {

106 107 108

d a t a = new i n t [ M E S S S I Z E ] ; // c a l c u l a t e t h e ID o f t h e node t h a t w i l l s e n d t h e d a t a t o ←֓ be s c a t t e r e d i n t s r c I D = r a n k ˆ ( 1 << ( s e n d P h a s e − 1 ) ) ; i n t t o R e c v = M E S S S I Z E << ( t o t a l P h a s e s − s e n d P h a s e ) ;

109 110

111 112 113

p r i n t f ( ”#%i

114

g e t t i n g from %i (% i ) \n” , r a n k , s r c I D , t o R e c v ) ;

115

M P I _ R e c v ( s c a t t e r e d D a t a , t o R e c v , M P I _ I N T , s r c I D , M E S S T A G , ←֓ MPI_COMM_WORLD , MPI_STATUS_IGNORE ) ;

116

117

memcpy ( data , scatteredData , M E S S S I Z E * s i z e o f ( i n t ) ) ; // keep t h e d a t a b e l o n g i n g t o t h e node

118

←֓

119

// s c a t t e r t o o t h e r n o d e s now int phase = sendPhase + 1; i n t d e s t I D = r a n k ˆ ( 1 << ( p h a s e − 1 ) ) ; int buffPos = MESSSIZE ; // s k i p t h e p a r t a s s i g n e d t o t h e ←֓ root i n t t o S e n d = t o R e c v >> 1 ; w h i l e ( p h a s e <= t o t a l P h a s e s ) { i n t d e s t I D = r a n k ˆ ( 1 << p h a s e − 1 ) ; i f ( destID < num ) { p r i n t f ( ”#%i s e n d s t o %i (% i ) \n” , r a n k , d e s t I D , ←֓ toSend ) ; M P I _ S e n d ( s c a t t e r e d D a t a + b u f f P o s , t o S e n d , M P I _ I N T , ←֓ destID , MESSTAG , MPI_COMM_WORLD ) ; b u f f P o s += t o S e n d ; } p h a s e ++; t o S e n d >>= 1 ; }

120 121 122 123

124 125 126 127 128 129 130

131

132 133 134 135 136

} // v e r i f y s c a t t e r i n g p r i n t f ( ”%i f i n i s h e d : ” , r a n k ) ; f o r ( i n t i = 0 ; i < M E S S S I Z E ; i++) p r i n t f ( ”%i ” , d a t a [ i ] ) ; p r i n t f ( ” \n” ) ;

137 138 139 140 141 142 143

MPI_Finalize () ; d e l e t e [ ] data ; delete [ ] scatteredData ; return 0;

144 145 146 147 148

}

The scattering is performed over ⌈lg(N )⌉ number of steps. In each step, the participating nodes grow exponentially, while the communicated parts shrink exponentially (by a factor of 2 in both cases).

If we number the communication phases i = 0, . . . , ⌈lg(N )⌉ − 1, then we have:

Phase 0: participating nodes 20 , data communicated K·N 2 Phase 1: participating nodes 21 , data communicated per node K·N 22 Phase 2: participating nodes 22 , data communicated per node K·N 23 ...

73 Therefore, the overall data volume is: ⌈lg(N )⌉−1

X

2i

i=0

K ·N = 2i+1

⌈lg(N )⌉−1

X i=0

K ·N K ·N = ⌈lg(N )⌉ 2 2

(5.3)

9. The amount of data exchanged during every step of the butterfly pattern in Figure 5.9, double in relation to the previous step. If initially every process had data of size K bytes to exchange, what is the total time required for the operation to complete, if we assume that each message exchange takes time ts + l · V , where ts is the link’s start-up latency, V is the volume of data to be sent and l is the inverse of the communication speed.

Answer The butterfly operation is completed in lg(N ) steps, with each step i = 0, . . . , lg(N ) − 1, requiring the exchange of 2i K data moving in opposite directions. If we assume that there is no delay between the steps, and that communications are bi-directional, then the overall time would be equal to: lg(N )−1

X i=0

ts + l · 2i K = lg(N )−1

lg(N )ts + l · K

X

2i =

i=0 lg(N )

lg(N )ts + l · K(2

− 1) =

lg(N )ts + l · K(N − 1)

(5.4)

10. An alternative parallel bucket sort algorithm, would have the root process of a N -process run, scan the input data and split them into N buckets before scattering the buckets to the corresponding processes. Implement this alternative design and compare its performance with the version presented in Section 5.11.5.

Answer 1 2 3 4

#i n c l u d e <mpi . h> #i n c l u d e <s t d l i b . h> #i n c l u d e <math . h> #i n c l u d e <i o s t r e a m >

5 6

u s i n g namespace s t d ;

7 8 9 10 11 12 13

14

const i n t MIN = 0 ; const i n t MAX = 10000; // ***************************************** i n t comp ( const void *a , const void *b ) { r e t u r n * ( r e i n t e r p r e t c a s t < c o n s t i n t * >( a ) ) −*(←֓ r e i n t e r p r e t c a s t < c o n s t i n t * >( b ) ) ; }

15 16 17 18 19

// ***************************************** v o i d i n i t D a t a ( i n t min , i n t max , i n t * d , i n t M ) { srand ( time (0) ) ;

74

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING f o r ( i n t i = 0 ; i < M ; i++) d [ i ] = ( rand ( ) % ( max − min ) ) + min ;

20 21 22

}

23 24 25 26 27

// ***************************************** i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { M P I _ I n i t (& a r g c , &a r g v ) ;

28 29 30 31 32

i n t rank , N ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &N ) ; MPI_Status status ;

33 34

if

( a r g c == 1 ) {

35

if

( r a n k == 0 ) c e r r << ” Usage ” << a r g v [ 0 ] << ” n u m b e r o f i t e m s \n” ; exit (1) ;

36 37 38

}

39 40 41 42 43 44 45

int int int int int

M = atoi ( argv [ 1 ] ) ; maxItemsPerBucket = M ; deliveredItems ; bucketRange = ceil ( 1 . 0 * ( MAX − MIN ) / N ) ; * d a t a = new i n t [ M ] ;

if

( r a n k == 0 )

46 47 48 49 50 51 52

{ i n i t D a t a ( MIN , MAX , d a t a , M ) ; i n t * b u c k e t s = new i n t [ N * m a x I t e m s P e r B u c k e t ] ; i n t * b u c k e t O f f s e t = new i n t [ N ] ; // where do b u c k e t s b e g i n ? i n t * i n B u c k e t = new i n t [ N ] ; // how many i t e m s i n e a c h ←֓ one ?

53 54 55 56 57 58 59 60

// i n i t i a l i z e b u c k e t c o u n t e r s and o f f s e t s f o r ( i n t i = 0 ; i < N ; i++) { inBucket [ i ] = 0; bucketOffset [ i ] = i * maxItemsPerBucket ; }

61 62 63 64 65 66 67 68 69

// s p l i t i n t o b u c k e t s f o r ( i n t i = 0 ; i < M ; i++) { i n t idx = ( data [ i ] − MIN ) / bucketRange ; i n t off = bucketOffset [ idx ] + inBucket [ idx ] ; buckets [ off ] = data [ i ] ; i n B u c k e t [ i d x ]++; }

70 71

72

73

M P I _ S c a t t e r ( i n B u c k e t , 1 , M P I _ I N T , &d e l i v e r e d I t e m s , 1 , ←֓ MPI_INT , 0 , MPI_COMM_WORLD ) ; M P I _ S c a t t e r v ( b u c k e t s , i n B u c k e t , b u c k e t O f f s e t , M P I _ I N T , ←֓ data , m a x I t e m s P e r B u c k e t , MPI_INT , 0 , M P I _ C O M M _ W O R L D ) ; qsort ( data , d e l i v e r e d I t e m s , s i z e o f ( i n t ) , comp ) ;

74 75

76 77

// c a l c u l a t e t h e o f f s e t s o f t h e b u c k e t s a g a i n , s o t h a t t h e y ←֓ end−up b e i n g c o n t i g u o u s a f t e r t h e g a t h e r i n g f o r ( i n t j = 1 ; j < N ; j++) bucketOffset [ j ] = bucketOffset [ j − 1] + inBucket [ j − 1 ] ;

78 79

M P I _ G a t h e r v ( d a t a , d e l i v e r e d I t e m s , M P I _ I N T , d a t a , i n B u c k e t , ←֓ bucketOffset , MPI_INT , 0 , MPI_COMM_WORLD ) ;

80 81 82 83 84

// p r i n t −o u t f o r v e r i f i c a t i o n f o r ( i n t i = 0 ; i < M ; i++) c o u t << d a t a [ i ] << ” ” ; c o u t << e n d l ;

85 86 87 88

// r e l e a s e memory delete [ ] buckets ; delete [ ] inBucket ;

purposes

75 delete [ ] bucketOffset ;

89

} else {

90 91 92

// g e t t h e s i z e o f t h e d a t a t o be s o r t e d l o c a l l y f i r s t M P I _ S c a t t e r ( N U L L , 1 , M P I _ I N T , &d e l i v e r e d I t e m s , 1 , M P I _ I N T , ←֓ 0 , MPI_COMM_WORLD ) ; M P I _ S c a t t e r v ( N U L L , N U L L , N U L L , M P I _ I N T , d a t a , ←֓ maxItemsPerBucket , MPI_INT , 0 , MPI_COMM_WORLD ) ; qsort ( data , d e l i v e r e d I t e m s , s i z e o f ( i n t ) , comp ) ; M P I _ G a t h e r v ( d a t a , d e l i v e r e d I t e m s , M P I _ I N T , N U L L , N U L L , ←֓ NULL , MPI_INT , 0 , M P I _ C O M M _ W O R L D ) ;

93 94

95

96 97

}

98 99

MPI_Finalize () ; d e l e t e [ ] data ; return 0;

100 101 102 103

}

11. Write a function that could be used for providing multicasting capabilities to a program, i.e. to be able to send a message to a subset of the processes of a communicator. Use an appropriate collective operation for the task.

Answer 1 2 3 4 5

6 7 8

// M u l t i c a s t s N e l e m e n t s o f t y p e t from a d d r e s s d a t a t o t h e l i s t // o f p r o c e s s e s i n c l u d e d i n t h e r a n k s a r r a y // The f i r s t e l e m e n t i n r a n k s i s assumed t o be t h e r o o t f o r // MPI Bcast a s i t a u t o m a t i c a l l y g e t rank 0 i n t h e new commun . v o i d m u l t i c a s t ( v o i d * d a t a , i n t N , M P I _ D a t a t y p e t , i n t * r a n k s , ←֓ i n t rN ) { MPI_Group g , all ; M P I _ C o m m mc ;

9

M P I _ C o m m _ g r o u p ( M P I _ C O M M _ W O R L D , &a l l ) ; M P I _ G r o u p _ i n c l ( all , rN , r a n k s , &g ) ; int localRank ; M P I _ G r o u p _ r a n k ( g , &l o c a l R a n k ) ;

10 11 12 13 14

M P I _ C o m m _ c r e a t e ( M P I _ C O M M _ W O R L D , g , &m c ) ; i f ( l o c a l R a n k != M P I _ U N D E F I N E D ) { M P I _ B c a s t ( data , N , t , 0 , mc ) ; M P I _ C o m m _ f r e e (& m c ) ; }

15 16 17 18 19 20 21

M P I _ G r o u p _ f r e e (&g ) ; M P I _ G r o u p _ f r e e (& a l l ) ;

22 23 24

}

12. Write the equivalent of the ping-pong program using RMA functions, and measure the communication speed achieved versus the size of the message used. Compare your results with the data rates accomplished with pointto-point communications.

Answer The following program prints out the average communication time over REP tests, for messages ranging in size from 0 to MAX MESG bytes, in increments of 1000.

1 2 3 4

#i n c l u d e <s t d i o . h> #i n c l u d e < s t d l i b . h> #i n c l u d e < s t r i n g . h> #i n c l u d e <mpi . h>

5 6 7

u s i n g namespace s t d ;

76

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING 8

#d e f i n e MAX MESG 1000000

9 10

const i n t REP = 10;

11 12 13 14 15 16 17 18 19 20 21 22 23

i n t main ( i n t argc , char * argv [ ] ) { i n t size , rank ; int namelen ; char processor_name [ MPI_MAX_PROCESSOR_NAME ] ; int mesg_size ; i n t tag ; unsigned char * buffer ; double start_time , end_time ; MPI_Status status ; MPI_Win w ; MPI_Group otherProc , all ;

24 25 26 27 28

M P I _ I n i t (& a r g c , &a r g v ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &s i z e ) ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ G e t _ p r o c e s s o r _ n a m e ( p r o c e s s o r _ n a m e , &n a m e l e n ) ;

29 30

31 32 33 34 35

p r i n t f ( ” P r o c e s s %d o f %d on %s \n” , r a n k , s i z e , p r o c e s s o r _ n a m e ) ←֓ ; i f ( s i z e != 2 ) { p r i n t f ( ” R e q u i r e s e x a c t l y 2 p r o c e s s e s t o run \n” ) ; exit (1) ; }

36 37 38

b u f f e r = new u n s i g n e d c h a r [ M A X _ M E S G ] ; memset ( buffer , rank , M A X _ M E S G ) ;

39 40 41 42

M P I _ C o m m _ g r o u p ( M P I _ C O M M _ W O R L D , &a l l ) ; M P I _ G r o u p _ e x c l ( all , 1 , &r a n k , &o t h e r P r o c ) ; M P I _ W i n _ c r e a t e ( b u f f e r , M A X _ M E S G , 1 , M P I _ I N F O _ N U L L , ←֓ M P I _ C O M M _ W O R L D , &w ) ;

43 44 45

if

( r a n k == 0 ) { f o r ( i n t m e s g _ s i z e = 0 ; m e s g _ s i z e <= M A X _ M E S G ; m e s g _ s i z e +=←֓ 1000) { start_time = MPI_Wtime () ; f o r ( i n t i = 0 ; i < R E P ; i++) { // s e n d MPI_Win_start ( otherProc , 0 , w ) ; M P I _ P u t ( b u f f e r , m e s g _ s i z e , M P I _ U N S I G N E D _ C H A R , 1 , ←֓ 0 , mesg_size , MPI_CHAR , w ) ; MPI_Win_complete ( w ) ;

46

47 48 49 50 51 52 53

54 55

// w a i t f o r r e s p o n s e MPI_Win_post ( otherProc , 0 , w ) ; MPI_Win_wait ( w ) ;

56 57 58

}

59 60

end_time = MPI_Wtime () ; p r i n t f ( ”%i % l f \n” , m e s g _ s i z e , ( e n d _ t i m e − s t a r t _ t i m e ) ←֓ / 2 / REP ) ;

61 62

}

63 64 65 66 67 68

69 70 71 72

} else { // o t h e r p r o c e s s m i r r o r s what rank 0 i s d o i n g f o r ( i n t m e s g _ s i z e = 0 ; m e s g _ s i z e <= M A X _ M E S G ; 1000) f o r ( i n t i = 0 ; i < R E P ; i++) { MPI_Win_post ( otherProc , 0 , w ) ; MPI_Win_wait ( w ) ;

m e s g _ s i z e +=←֓

73 74 75

MPI_Win_start ( otherProc , 0 , w ) ; M P I _ P u t ( b u f f e r , m e s g _ s i z e , M P I _ U N S I G N E D _ C H A R , 0 , 0 , ←֓

77 mesg_size , MPI_CHAR , w ) ; MPI_Win_complete ( w ) ;

76

}

77

}

78 79

delete [ ] buffer ; M P I _ W i n _ f r e e (&w ) ; M P I _ G r o u p _ f r e e (& a l l ) ; M P I _ G r o u p _ f r e e (& o t h e r P r o c ) ; MPI_Finalize () ; return 0;

80 81 82 83 84 85 86

}

13. Modify the program of Listing 5.7, so that the partitioning of the range depends on the relative speed of the participating nodes. One easy approach is to make the ranges proportional to the CPU operating frequency, or the calculated bogomips. Both numbers are available in the /proc/cpuinfo pseudo-file. The master can collect the numbers and reply back to the worker nodes with the calculated ranges. If we represent as mi the inode’s bogomips, the percent of the range αi that should be assigned to node i can be calculated as αi = P mimk ∀k

Answer The master node has to gather the bogomips of the execution platform machines, and calculate the appropriate ranges before sending them back to the nodes. The bogomips can be read from the /proc/cpuinfo pseudofile using a pipe as shown in line 23. The bogomips numbers are gathered by the master node (line 31) before calculating the parts αi (lines 35-41, identified as part[] in the code) and the ranges that should be assigned to each node (lines 43-51). The remaining code is identical to Listing 5.7.

1 2 3 4

#i n c l u d e <mpi . h> #i n c l u d e <s t d i o . h> #i n c l u d e <s t d l i b . h> #i n c l u d e <s t r i n g . h>

5 6 7 8

#d e f i n e RANGEMIN 0 #d e f i n e RANGEMAX 1000 #d e f i n e MSGTAG 0

9 10 11 12 13 14 15 16 17 18

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t r a n k , num , i ; int range [ 2 ] ; M P I _ I n i t (& a r g c , &a r g v ) ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &n u m ) ; MPI_Status status ;

19 20 21

f l o a t bogoMIPS ; f l o a t * allBogo = NULL ;

22 23

24 25

FILE *f = popen ( ”/ bin / cat / proc / cpuinfo | / b i n / g r e p bogo | /←֓ u s r / b i n / head −n 1 | / u s r / b i n /gawk −F ’ : ’ ’ { p r i n t $ 2 } ’ ” , ”←֓ r”) ; f s c a n f ( f , ”%f ” , &b o g o M I P S ) ; pclose ( f ) ;

26 27 28 29

// a r r a y f o r g a t h e r i n g i s o n l y need i n rank 0 i f ( r a n k == 0 ) a l l B o g o = new f l o a t [ n u m ] ;

30 31

32

M P I _ G a t h e r (& b o g o M I P S , 1 , M P I _ F L O A T , a l l B o g o , 1 , M P I _ F L O A T , 0 , ←֓ MPI_COMM_WORLD ) ;

78

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING if

33

( r a n k == 0 ) {

34

f l o a t sumBogo = allBogo [ 0 ] ; f o r ( i = 1 ; i < n u m ; i++) s u m B o g o += a l l B o g o [ i ] ;

35 36 37 38

f l o a t part [ num ] ; f o r ( i = 0 ; i < n u m ; i++) part [ i ] = allBogo [ i ] / sumBogo ;

39 40 41 42

i n t rng [ 2 * num ] ; int width = RANGEMAX − RANGEMIN ; rng [ 0 ] = RANGEMIN ; // l e f t l i m i t rng [ 1 ] = rng [ 0 ] + width * part [ 0 ] − 1 ; // r i g h t l i m i t f o r ( i = 1 ; i < n u m ; i++) { rng [ i * 2 ] = rng [ i * 2 − 1 ] + 1 ; r n g [ i * 2 + 1 ] = ( i == n u m − 1 ) ? R A N G E M A X : r n g [ i * 2 ] ←֓ + width * part [ i ] − 1; }

43 44 45 46 47 48 49 50

51 52

M P I _ R e q u e s t rq [ num − 1 ] ; f o r ( i = 1 ; i < n u m ; i++) M P I _ I s e n d ( r n g + i * 2 , 2 , M P I _ I N T , i , M S G T A G , ←֓ M P I _ C O M M _ W O R L D , &( r q [ i − 1 ] ) ) ;

53 54 55

56

f o r ( i = 1 ; i < n u m ; i++) M P I _ W a i t (&( r q [ i − 1 ] ) , &s t a t u s ) ;

57 58 59

range [ 0 ] = rng [ 0 ] ; range [ 1 ] = rng [ 1 ] ;

60 61

// m a s t e r ’ s l i m i t s

62

delete [ ] allBogo ;

63

} else {

64 65 66

M P I _ R e q u e s t rq ; M P I _ I r e c v ( r a n g e , 2 , M P I _ I N T , 0 , M S G T A G , M P I _ C O M M _ W O R L D , &←֓ rq ) ; M P I _ W a i t (&rq , &s t a t u s ) ;

67 68

69

}

70 71

p r i n t f ( ”Node %i ’ s r a n g e : ( %i , %i ) \n” , r a n k , r a n g e [ 0 ] , [1]) ;

72

r a n g e ←֓

73

MPI_Finalize () ; return 0;

74 75 76

}

14. The butterfly communication scheme that is outlined in Section 5.11.4, is only one of the possible strategies for an all-to-all, or all-reduce data exchange. A different approach would be mandated if the underlying communication infrastructure did not provide the required links, making the procedure inefficient. An example of such an architecture is the ring, where each node is directly connected to just two others. (a) Write an MPI program that would implement an efficient all-to-all exchange of data on a ring of machines. (b) How many steps would be required in comparison to a butterfly scheme, if the number of nodes/processes were N = 2k ? (c) If we assume that time taken to send V bytes over a communication link is given by l · V , where l is the (inverse of the) link speed in sec/byte, how does your algorithm compare against the butterfly scheme in terms of overall communication time?

Answer

79 (a) The following program contains an allToAll function that is equivalent to MPI Alltoall. Assuming that N processes are organized in a ring, each process performs N −1 rounds/steps of forwarding its data to the next process (identified by destID) and receiving data from the previous process (identified by srcID). The data are rearranged so that the first block of NperNode items received, should reside with the process. The remaining N - round - 1 blocks are sent to the next process and so on. The data rearrangement is performed before commencing any communications in lines 24-26. The target data layout depends on the rank of the process.

The MPI datatype of the communicated data is needed so that heterogeneous platforms can be supported, while the native type size (parameter typeSize) is needed so that a properly-sized data buffer is allocated for handling the communication traffic.

The communication pattern for the allToAllGather function is made up from the same number of steps, but with a reverse flow of communications and each communication carries NperNode elements.

1 2 3 4 5 6 7

// A l l −to− a l l s c a t t e r i n g on a r i n g // Number o f p r o c e s s e s can be a r b i t r a r y #i n c l u d e <mpi . h> #i n c l u d e <s t r i n g . h> #i n c l u d e <u n i s t d . h> #i n c l u d e <math . h> #i n c l u d e <i o s t r e a m >

8 9 10 11

const int K = 10; const int ALLGATHERTAG = 0; const int ALLSCATTERTAG = 0;

12 13

u s i n g namespace s t d ;

14 15 16

17

18 19 20 21

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // a l l −to− a l l s c a t t e r i n g . o u t B u f f e r / i n B u f f e r a r e s u p p o s e d t o ←֓ have N * NperNode e l e m e n t s / s p a c e v o i d a l l T o A l l ( v o i d * o u t B u f f e r , i n t N p e r N o d e , i n t t y p e S i z e , ←֓ MPI_Datatype t , void * inBuffer ) { i n t rank , N ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &N ) ;

22 23 24

// l o c a l −d e s t i n e d p a r t o f d a t a c o p i e d f i r s t m e m c p y ( i n B u f f e r + N p e r N o d e * t y p e S i z e * r a n k , o u t B u f f e r +←֓ N p e r N o d e * t y p e S i z e * rank , N p e r N o d e * t y p e S i z e ) ;

25 26 27

28

29

// r e a r r a n g e l o c a l d a t a b e f o r e s t a r t i n g t o s e n t u n s i g n e d c h a r * t e m p B u f f = new u n s i g n e d c h a r [ ( N − 1 ) * ←֓ NperNode * typeSize ] ; m e m c p y ( t e m p B u f f , o u t B u f f e r + N p e r N o d e * t y p e S i z e * ( r a n k ←֓ + 1) , ( N − rank − 1) * NperNode * typeSize ) ; m e m c p y ( t e m p B u f f + ( N − r a n k − 1 ) * N p e r N o d e * t y p e S i z e , ←֓ outBuffer , rank * NperNode * typeSize ) ;

30 31 32 33 34 35 36 37 38

39

40

int round = 1; i n t toSend = ( N − 1) * NperNode ; i n t s r c I D = ( r a n k == 0 ) ? N − 1 : r a n k − 1 ; i n t destID = ( rank + 1) % N ; void * ptr1 = tempBuff , * ptr2 = outBuffer , * aux ; while ( round < N ) { M P I _ S e n d ( p t r 1 , t o S e n d , t , d e s t I D , A L L S C A T T E R T A G , ←֓ MPI_COMM_WORLD ) ; M P I _ R e c v ( p t r 2 , t o S e n d , t , s r c I D , A L L S C A T T E R T A G , ←֓ MPI_COMM_WORLD , MPI_STATUS_IGNORE ) ;

80

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING // keep l o c a l −d e s t i n e d p a r t and s e n t t h e r e s t i n t o f f s e t = ( ( r a n k − r o u n d + N ) % N ) * N p e r N o d e * ←֓ typeSize ; memcpy ( i n B u f f e r + offset , ptr2 , N p e r N o d e * t y p e S i z e ) ;

41 42

43 44

// s w i t c h b u f f e r p o i n t e r s aux = ptr1 ; ptr1 = ptr2 ; ptr2 = aux ; p t r 1 += N p e r N o d e * t y p e S i z e ; o f b u f f e r s to send / r e c e i v e p t r 2 += N p e r N o d e * t y p e S i z e ; t o S e n d −= N p e r N o d e ; r o u n d ++;

45 46 47 48 49

50 51 52

b e g i n n i n g ←֓

} delete [ ] tempBuff ;

53 54 55

// s h i f t

}

56 57 58 59 60 61

62 63 64 65

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // a l l −to− a l l g a t h e r i n g . o u t B u f f e r i s s u p p o s e d t o have // NperNode e l e m e n t s , w h i l e i n B u f f e r h as enough s p a c e // f o r N * NperNode v o i d a l l T o A l l G a t h e r ( v o i d * o u t B u f f e r , i n t N p e r N o d e , i n t ←֓ typeSize , MPI_Datatype t , void * inBuffer ) { i n t rank , N ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &N ) ;

66

// l o c a l −d e s t i n e d p a r t o f d a t a c o p i e d f i r s t m e m c p y ( i n B u f f e r + N p e r N o d e * t y p e S i z e * r a n k , o u t B u f f e r , ←֓ NperNode * typeSize ) ;

67 68

69 70

int round = 1; // r e v e r s e r i n g t r a v e r s a l than a l l T o A l l i n t srcID = ( rank + 1) % N ; i n t d e s t I D = ( r a n k == 0 ) ? N − 1 : r a n k − 1 ; void * ptr1 = inBuffer + NperNode * typeSize * rank ; v o i d * p t r 2 = ( r a n k == N − 1 ) ? i n B u f f e r : p t r 1 + N p e r N o d e ←֓ * typeSize ; void * aux ; while ( round < N ) { M P I _ S e n d ( p t r 1 , N p e r N o d e , t , d e s t I D , A L L G A T H E R T A G , ←֓ MPI_COMM_WORLD ) ; M P I _ R e c v ( p t r 2 , N p e r N o d e , t , s r c I D , A L L G A T H E R T A G , ←֓ MPI_COMM_WORLD , MPI_STATUS_IGNORE ) ;

71 72 73 74 75 76

77 78 79 80

81

82

// advance b u f f e r p o i n t e r s aux = ptr2 ; ptr1 = ptr2 ; p t r 2 = ( ( r a n k + r o u n d + 1 ) % N == 0 ) ? i n B u f f e r : a u x ←֓ + NperNode * typeSize ; r o u n d ++;

83 84 85 86

87

}

88 89

}

90 91 92 93 94

// ***************************************** i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { M P I _ I n i t (& a r g c , &a r g v ) ;

95 96 97 98 99

i n t rank , N ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &N ) ; MPI_Status status ;

100 101 102 103

d o u b l e * l o c a l P a r t = new d o u b l e [ K * N ] ; d o u b l e * a l l P a r t s = new d o u b l e [ K * N ] ;

104 105 106

if

( r a n k == 0 ) c o u t << ” T e s t f o r

a l l t o A l l \n” ;

81 f o r ( i n t i = 0 ; i < K * N ; i++) localPart [ i ] = pow (10 , rank ) + i ;

107 108 109

allToAll ( localPart , K , allParts ) ;

110

s i z e o f ( d o u b l e ) , M P I _ D O U B L E , ←֓

111

sleep ( rank ) ; c o u t << r a n k << ” : ” ; f o r ( i n t i = 0 ; i < K * N ; i++) c o u t << a l l P a r t s [ i ] << ” ” ; c o u t << e n d l ;

112 113 114 115 116 117

MPI_Barrier ( MPI_COMM_WORLD ) ;

118 119

if

( r a n k == 0 ) c o u t << ” T e s t f o r a l l t o A l l G a t h e r \n” ; f o r ( i n t i = 0 ; i < K ; i++) localPart [ i ] = rank ;

120 121 122 123 124

allToAllGather ( localPart , K , allParts ) ;

125

s i z e o f ( d o u b l e ) , M P I _ D O U B L E , ←֓

126

sleep ( rank ) ; c o u t << r a n k << ” : ” ; f o r ( i n t i = 0 ; i < K * N ; i++) c o u t << a l l P a r t s [ i ] << ” ” ; c o u t << e n d l ;

127 128 129 130 131 132 133

MPI_Finalize () ; delete [ ] localPart ; delete [ ] allParts ; return 0;

134 135 136 137 138

}

(b) The number of steps/rounds required is N − 1 which is obviously much higher than the lgN of the butterfly scheme. However, the data transmitted in every step by the allToAllGather function is constant, while in the butterfly algorithm they grow exponentially in size by a factor of 2. (c) Assuming that the blocks destined for each process have a size of K bytes, and that the communications during a round take place simultaneously, then the total communication costs are: (ring) tcomm =

N −2 X i=0

ly) t(butterf = comm

lgN −1 X i=0

(butterf ly)

l · K = (N − 1) · l · K

(5.5)

l · K · 2i = l · K(2lgN − 1) = (N − 1) · l · K (5.6)

(ring)

The tcomm = tcomm result may seem surprising, but there is an important factor that has been intentionally ignored in this setting : communication latency. The start-up overhead can be significant.

The butterfly pattern can offer substantially lower costs if this is factored-in as well.

15. The case study on Diffusion Limited Aggregation in Section 5.20, decomposes the problem’s data on a particle-wise basis. Explore the alternative of partitioning the 2D grid and assigning it to the processes involved.

82

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING What are the benefits and drawbacks of this approach? Is the communication pattern involved different in this case? Does this approach produce similar results to the sequential program and alternative parallel programs?

Answer The benefit of a row-wise geometric decomposition is a more regular communication pattern that is also localized, instead of global. Changes to the crystal formation do not have to be broadcasted to all the nodes. The drawbacks are (a) a skewed load distribution, as the nodes handling the crystal region have in general fewer particles assigned to them, and (b) a potentially much higher communication cost that grows linearly with the number of columns used. Results should be identical to the sequential program, as long as the sequential program also grows the crystal at the end of each time step.

In the following listing (which is part of a project including the dla core.cpp file shown later), the grid is initialized in node 0 (line 74) and scattered in a row-wise fashion to the remaining nodes (line 79).

The grid local to each node is made up of two additional columns (to simplify boundary checking) and two additional rows that contain data handled by the neighboring processes. The two top and bottom rows of each local grid are exchanged with the previous and next process respectively (the first and last processes are obviously an exception). The particles that end-up in the top/bottom rows “migrate” to the previous/next process after the message exchanges taking place in line 92-93 and 99-100.

The loops of lines 113-115 and 120-122 collect those migrating particles.

The bounceUp and bounceDown variables control whether this migration is possible : It is obviously impossible for the very first and very last rows in the overall grid, because that would mean that the particles are leaving completely.

The final state of the grid is gathered by process 0 (line 134) and output to the console in the form of a PBM image (lines 138-149). A schematic of the problem decomposition is shown in Figure 5.2.

1 2 3 4 5 6 7 8

#i n c l u d e <s t d i o . h> #i n c l u d e < s t d l i b . h> #i n c l u d e < s t r i n g . h> #i n c l u d e < a s s e r t . h> #i n c l u d e <math . h> #i n c l u d e <i o s t r e a m > #i n c l u d e ” d l a c o r e . h” #i n c l u d e <mpi . h>

9 10

u s i n g namespace s t d ;

11 12 13 14

#d e f i n e PIC SIZE 100 #d e f i n e PARTICLES 500 #d e f i n e MAX ITER 10000

15 16 17 18 19 20 21 22 23 24

/ *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−* / i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t cols , rows , iter , particles , x , y ; i n t * pic , * p i c 2 , * t m p ; int * nextNodeRows , * prevNodeRows ; i n t r a n k , num , i ; M P I _ I n i t (& a r g c , &a r g v ) ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ;

83

Figure 5.2: Geometric problem decomposition illustration for the solution of Exercise 15.

25 26

M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &n u m ) ; MPI_Request req ;

27 28

29

if

( argc < 2) // u s e d e f a u l t v a l u e s does not s p e c i f y anything cols = PIC_SIZE ; rows = PIC_SIZE ; iter = MAX_ITER ; particles = PARTICLES ;

31 32 33

35 36

} else { cols = atoi rows = atoi particles = iter = atoi

37 38 39 40 41

u s e r ←֓

{

30

34

if

( argv [ 1 ] ) ; ( argv [ 2 ] ) ; atoi ( argv [ 3 ] ) ; ( argv [ 4 ] ) ;

}

42 43

44 45 46

// t o make c o d e s i m p l e r and a v o i d s p e c i a l c a s e s , rows i s t o be a m u l t i p l e o f // t h e number o f p r o c e s s e s rows = ( i n t ) ceil ( rows * 1.0 / num ) * num ; i n t rowsPerNode = rows / num ;

f o r c e d ←֓

47 48 49

// i n i t i a l i z e t h e random number g e n e r a t o r srand ( time (0) ) ;

50 51

52

53 54 55 56 57 58 59 60

// g r i d h as two e x t r a rows , one above and one below t h e a c t u a l ←֓ rows // The e x t r a rows ” b e l o n g ” t o o t h e r nodes , s o a t t h e end o f ←֓ each time s t e p // t h e r e s h o u l d be a p a i r −w i s e r e d u c t i o n o f t h e s e rows i f ( r a n k == 0 ) { p i c = new i n t [ ( c o l s + 2 ) * ( r o w s + 2 ) ] ; p i c 2 = new i n t [ ( c o l s + 2 ) * ( r o w s + 2 ) ] ; } else {

84

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING p i c = new i n t [ ( c o l s + 2 ) * ( r o w s P e r N o d e + 2 ) ] ; p i c 2 = new i n t [ ( c o l s + 2 ) * ( r o w s P e r N o d e + 2 ) ] ;

61 62 63 64 65 66 67 68

} bool bounceUp = f a l s e , bounceDown = f a l s e ; i f ( r a n k == 0 ) bounceUp = true ; e l s e i f ( r a n k == n u m − 1 ) bounceDown = true ;

69 70 71

n e x t N o d e R o w s = new i n t [ 2 * ( c o l s + 2 ) ] ; p r e v N o d e R o w s = new i n t [ 2 * ( c o l s + 2 ) ] ;

72 73 74 75 76

if

( r a n k == 0 ) d l a _ i n i t ( pic2 , rows + 2 , cols + 2 , particles , 1) ; // c l e a n p i c b u f f e r m e m s e t ( pic , 0 , ( c o l s + 2 ) * ( r o w s P e r N o d e + 2 ) * s i z e o f ( i n t ) ) ;

77 78 79

// boundary rows do n o t need t o be communicated now .

M P I _ S c a t t e r ( p i c 2 + c o l s + 2 , ( c o l s + 2 ) * r o w s P e r N o d e , M P I _ I N T ←֓ , p i c + c o l s + 2 , ( c o l s + 2 ) * r o w s P e r N o d e , M P I _ I N T , 0 , ←֓ MPI_COMM_WORLD ) ;

80 81 82 83 84

85 86 87 88 89 90 91 92

93

94

i n t nextID = rank + 1 , prevID = rank − 1; w h i l e (−−i t e r >= 0 ) { d l a _ e v o l v e ( pic , p i c 2 , r o w s P e r N o d e + 2 , c o l s + 2 , b o u n c e U p , ←֓ bounceDown ) ; tmp = pic2 ; pic2 = pic ; pic = tmp ; // e x c h a n g e i n f o r m a t i o n w i t h n e i g h b o r i n g n o d e s // e x c h a n g e w i t h ” p r e v i o u s ” i f ( r a n k != 0 ) { M P I _ I s e n d ( pic , 2 * ( c o l s + 2 ) , M P I _ I N T , p r e v I D , i t e r , ←֓ M P I _ C O M M _ W O R L D , &r e q ) ; M P I _ R e c v ( p r e v N o d e R o w s , 2 * ( c o l s + 2 ) , M P I _ I N T , p r e v I D ←֓ , iter , M P I _ C O M M _ W O R L D , M P I _ S T A T U S _ I G N O R E ) ; }

95 96 97 98 99

100

101

// e x c h a n g e w i t h ” n e x t ” i f ( r a n k != n u m − 1 ) { M P I _ I s e n d ( p i c + r o w s P e r N o d e * ( c o l s + 2 ) , 2 * ( c o l s + ←֓ 2 ) , M P I _ I N T , n e x t I D , i t e r , M P I _ C O M M _ W O R L D , &r e q ) ; M P I _ R e c v ( n e x t N o d e R o w s , 2 * ( c o l s + 2 ) , M P I _ I N T , n e x t I D ←֓ , iter , M P I _ C O M M _ W O R L D , M P I _ S T A T U S _ I G N O R E ) ; }

102 103

104

105 106 107 108 109 110 111

// r e d u c t i o n i s d u p l i c a t e d i n t h e n o d e s s o t h a t t h e y dont ←֓ have t o w a i t // what i s i m p o r t a n t i n t h e n e i g h b o r s ’ rows , i s t h e n e g a t i v e ←֓ cells i f ( r a n k != 0 ) { f o r ( i n t i = 0 ; i < c o l s + 2 ; i++) i f ( prevNodeRows [ i ] < 0) pic [ i ] = prevNodeRows [ i ] ; else pic [ i ] = 0 ; // r e s e t any p a r t i c l e s now on t h e ←֓ o t h e r node

112

f o r ( i n t i = 0 ; i < c o l s + 2 ; i++) i f ( prevNodeRows [ cols + 2 + i ] > 0) p i c [ c o l s + 2 + i ] += p r e v N o d e R o w s [ c o l s + 2 + i ] ; // add p a r t i c l e s coming from t h e o t h e r node

113 114 115

←֓

}

116 117 118 119 120 121 122

if

( r a n k != n u m − 1 ) { f o r ( i n t i = 0 ; i < c o l s + 2 ; i++) i f ( nextNodeRows [ i ] > 0) p i c [ r o w s P e r N o d e * ( c o l s + 2 ) + i ] += n e x t N o d e R o w s [ ←֓ i ]; // add p a r t i c l e s coming from t h e o t h e r ←֓

85 node 123

f o r ( i n t i = 0 ; i < c o l s + 2 ; i++) i f ( nextNodeRows [ cols + 2 + i ] < 0) p i c [ ( r o w s P e r N o d e + 1 ) * ( c o l s + 2 ) + i ] = ←֓ nextNodeRows [ cols + 2 + i ] ; else pic [ ( rowsPerNode + 1) * ( cols + 2) + i ] = 0 ; // r e s e t any p a r t i c l e s now on t h e o t h e r node

124 125 126

127 128

←֓

}

129 130

}

131 132 133

M P I _ G a t h e r ( p i c + c o l s + 2 , r o w s P e r N o d e * ( c o l s + 2 ) , M P I _ I N T , ←֓ p i c + c o l s + 2 , r o w s P e r N o d e * ( c o l s + 2 ) , M P I _ I N T , 0 , ←֓ MPI_COMM_WORLD ) ; / * P r i n t t o s t d o u t a PBM p i c t u r e o f t h e s i m u l a t i o n s p a c e * / i f ( r a n k == 0 ) { p r i n t f ( ”P1\n%i %i \n” , c o l s , r o w s ) ;

134

135 136 137 138 139

f o r ( y = 1 ; y <= r o w s ; y++) { f o r ( x = 1 ; x <= c o l s ; x++) { i f ( pic [ y * ( cols + 2) + x ] < 0) printf ( ”1 ” ) ; else printf ( ”0 ” ) ; } p r i n t f ( ” \n” ) ; }

140 141 142 143 144 145 146 147 148 149 150

}

151 152

MPI_Finalize () ; d e l e t e [ ] pic ; d e l e t e [ ] pic2 ; delete [ ] nextNodeRows ; delete [ ] prevNodeRows ; return 0;

153 154 155 156 157 158 159

}

The dla core.cpp file that supplements the code above is: 160 161

#i n c l u d e < s t d l i b . h> #i n c l u d e ” d l a c o r e . h”

162 163 164 165 166 167 168 169 170 171 172 173 174 175 176

177 178 179 180

/ *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− * Checks t h e p r e s e n c e o f a s t r u c t u r e i n n e i g h b o r i n g c e l l s * p i c p o i n t s t o 2D a r r a y h o l d i n g data , a r r a n g e d i n c o l s * columns . B e c a u s e t h e a r r a y i s two columns and two rows * w i d e r than n e c e s s a r y , t h e r e i s no need t o c h e c k f o r * boundary v a l u e s o f x and y . */ i n t c h e c k _ p r o x i m ( i n t * pic , i n t c o l s , i n t x , i n t y ) { i n t * row0 , * row1 , * row2 ; row0 = pic + ( y − 1) * cols + x − 1 ; row1 = row0 + cols ; row2 = row1 + cols ; i f ( * r o w 0 < 0 | | * ( r o w 0 + 1 ) < 0 | | * ( r o w 0 + 2 ) < 0 | | * r o w 1 < ←֓ 0 | | * ( r o w 1 + 1 ) < 0 | | * ( r o w 1 + 2 ) < 0 | | * r o w 2 < 0 | | * ( ←֓ row2 + 1) < 0 | | *( row2 + 2) < 0) r e t u r n ( −1) ; else return (1) ; }

181 182 183 184 185 186

/ *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−* / / * R e t u r n s −1 ,0 and 1 w i t h e q u a l p r o b a b i l i t y * / i n l i n e int three_way () { return ( random ( ) % 3) − 1 ;

86 187

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING }

188 189 190 191

192 193 194 195 196

/ *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−* / / * I n i t i a l i z e s t h e 2D a r r a y f o r t h e s i m u l a t i o n * / v o i d d l a _ i n i t ( i n t * pic , i n t r o w s , i n t c o l s , i n t p a r t i c l e s , init_seed ) { int i , j , x , y ; f o r ( i = 0 ; i < r o w s ; i++) f o r ( j = 0 ; j < c o l s ; j++) pic [ i * cols + j ] = 0 ;

i n t ←֓

197 198

f o r ( i = 0 ; i < p a r t i c l e s ; i++) / * g e n e r a t e i n i t i a l ←֓ p a r t i c l e p l a c e m e n t */ { x = random ( ) % ( cols − 2) + 1 ; // c o u n t i n g s t a r t s from 1 y = random ( ) % ( rows − 2) + 1 ; i f ( ( y == r o w s / 2 + 1 ) && ( x == c o l s / 2 + 1 ) ) // r e p e a t ←֓ i f true i−−; else p i c [ y * c o l s + x ]++; }

199

200 201 202 203

204 205 206 207 208

if

209 210

( init_seed ) p i c [ ( r o w s / 2 ) * c o l s + ( c o l s / 2 ) ] = −1; s e e d */

/* p l a c e

i n i t i a l ←֓

211 212

}

213 214 215 216 217 218 219 220 221 222

223

/ *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− * S i n g l e step e v o l u t i o n of the simulation .

* * The c e l l v a l u e s r e p r e s e n t : * 0 : empty s p a c e * >0 : m u l t i p l e p a r t i c l e s * <0 : c r y s t a l * * R e t u r n s t h e a d d r e s s o f t h e s t r u c t u r e h o l d i n g t h e l a s t update ←֓ */ i n t * d l a _ e v o l v e ( i n t * pic , i n t * p i c 2 , i n t r o w s , i n t c o l s , b o o l ←֓ bounceUp , bool bounceDown )

224 225

{ int x , y , k ;

226 227 228 229 230

// p r e p a r e a r r a y t o h o l d new s t a t e f o r ( y = 1 ; y < r o w s − 1 ; y++) f o r ( x = 1 ; x < c o l s − 1 ; x++) p i c 2 [ y * c o l s + x ] = p i c [ y * c o l s + x ] > 0 ? 0 : p i c [ y * ←֓ cols + x ] ;

231 232 233 234 235 236 237 238 239 240

f o r ( y = 1 ; y < r o w s − 1 ; y++) f o r ( x = 1 ; x < c o l s − 1 ; x++) f o r ( k = 0 ; k < p i c [ y * c o l s + x ] ; k++) { int new_x = x + three_way () ; i f ( n e w _ x == 0 && b o u n c e U p ) new_x = 1; e l s e i f ( n e w _ x == c o l s − 1 && b o u n c e D o w n ) new_x = cols − 2;

241 242 243 244 245 246

int new_y = y + three_way () ; i f ( n e w _ y == 0 ) new_y = 1; e l s e i f ( n e w _ y == r o w s − 1 ) new_y = rows − 2;

247 248

249 250

251

( pic2 [ new_y * cols + new_x ] > 0) // s t e p s i n t o ←֓ empty s p a c e p i c 2 [ n e w _ y * c o l s + n e w _ x ]++; e l s e i f ( p i c 2 [ n e w _ y * c o l s + n e w _ x ] == 0 ) // s t e p s ←֓ i n t o u n c h e c ke d s p a c e { if

87 p i c 2 [ n e w _ y * c o l s + n e w _ x ] = c h e c k _ p r o x i m ( p i c 2 , ←֓ cols , new_x , new_y ) ;

252

}

253

}

254

return pic2 ;

255 256

}

16. Implement a 3D-space Diffusion Limited Aggregation simulation, by extending the solutions provided in Section 5.20.

Answer The project reported in Section 5.20 consists of two source code files: dla core.c and dla mpi.c. The transition to 3D requires significant changes only to the functions in dla core.c, which handle the particles as represented by the PartStr structure (which in turn gains a z coordinate).

The complete code (with a minor conversion to C++) for dla core.cpp is shown below, the major differences being limited to: (a) Lines 24-33 of the check proxim function, that now has to check 27 possible neighboring positions, scattered over 3 planes along the z-axis. (b) The calculation of the indices into the grid array (as seen in lines 69, 109 and 142), that need to take into consideration the third dimension.

1 2 3 4

/ * Core l i b r a r y f o r s i m u l a t i n g 3D DLA phenomena * / #i n c l u d e < s t d l i b . h> #i n c l u d e < s t r i n g . h> #i n c l u d e ” d l a c o r e . h”

5 6 7 8 9 10 11 12 13

14 15 16 17 18

/ *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− * Checks t h e p r e s e n c e o f a s t r u c t u r e i n n e i g h b o r i n g c e l l s * p i c p o i n t s t o 3D a r r a y h o l d i n g data , a r r a n g e d i n depth number * o f c o l s X c o l u m n s p l a n e s . B e c a u s e t h e a r r a y i s ” s u r r o u n d e d ” by * an empty cube o f c e l l s , t h e r e i s no need t o c h e c k f o r * boundary v a l u e s o f x , y and z . */ i n t c h e c k _ p r o x i m ( i n t * pic , i n t r o w s , i n t c o l s , i n t x , i n t y , i n t ←֓ z) { i n t * plane0 , * plane1 , * plane2 ; plane0 = pic + ( z − 1) * rows * cols ; plane1 = pic + z * rows * cols ; plane2 = pic + ( z + 1) * rows * cols ;

19 20 21 22 23 24

25

26

27

28

29

30

31

i n t row0 , row1 , row2 ; row0 = ( y − 1) * cols + x − 1 ; row1 = row0 + cols ; row2 = row1 + cols ; i f ( plane0 [ row0 ] < 0 | | plane0 [ row0 2] < 0 | | plane0 [ row1 ] < 0 | | plane0 [ row1 2] < 0 | | plane0 [ row2 ] < 0 | | plane0 [ row2 2] < 0 | | plane1 [ row0 ] < 0 | | plane1 [ row0 2] < 0 | | plane1 [ row1 ] < 0 | | plane1 [ row1 2] < 0 | | plane1 [ row2 ] < 0 | | plane1 [ row2 2] < 0 | | plane2 [ row0 ] < 0 | | plane2 [ row0 2] < 0 | | plane2 [ row1 ] < 0 | | plane2 [ row1 2] < 0 | |

+ 1] < 0 | |

p l a n e 0 [ r o w 0 + ←֓

+ 1] < 0 | |

p l a n e 0 [ r o w 1 + ←֓

+ 1] < 0 | |

p l a n e 0 [ r o w 2 + ←֓

+ 1] < 0 | |

p l a n e 1 [ r o w 0 + ←֓

+ 1] < 0 | |

p l a n e 1 [ r o w 1 + ←֓

+ 1] < 0 | |

p l a n e 1 [ r o w 2 + ←֓

+ 1] < 0 | |

p l a n e 2 [ r o w 0 + ←֓

+ 1] < 0 | |

p l a n e 2 [ r o w 1 + ←֓

88

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING plane2 [ row2 ] < 0 | | 2 ] < 0) r e t u r n ( −1) ; else return (1) ;

32

33 34 35 36

plane2 [ row2 + 1] < 0 | |

p l a n e 2 [ r o w 2 + ←֓

}

37 38 39 40 41 42 43

/ *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−* / / * R e t u r n s −1 ,0 and 1 w i t h e q u a l p r o b a b i l i t y * / i n l i n e int three_way () { return ( random ( ) % 3) − 1 ; }

44 45 46

47

48 49 50

/ *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−* / / * I n i t i a l i z e s t h e 3D a r r a y and a r r a y o f p a r t i c l e s f o r t h e ←֓ s i m u l a t i o n */ v o i d d l a _ i n i t _ p l i s t ( i n t * pic , i n t r o w s , i n t c o l s , i n t d e p t h , ←֓ PartStr * p , i n t particles , i n t init_seed ) { i n t i , x , y , z , idx ; m e m s e t ( pic , 0 , r o w s * c o l s * d e p t h * s i z e o f ( i n t ) ) ;

51 52

f o r ( i = 0 ; i < p a r t i c l e s ; i++) / * g e n e r a t e i n i t i a l ←֓ p a r t i c l e p l a c e m e n t */ { x = random ( ) % ( cols − 2) + 1 ; // c o u n t i n g s t a r t s from 1 y = random ( ) % ( rows − 2) + 1 ; z = random ( ) % ( rows − 2) + 1 ; i f ( ( y == r o w s / 2 + 1 ) && ( x == c o l s / 2 + 1 ) && ( z == ←֓ depth / 2 + 1) ) // r e p e a t i f t r u e i−−; else { p[i ]. x = x; p[i ]. y = y; p[i ]. z = z; } } i f ( init_seed ) { i d x = ( d e p t h / 2 ) * r o w s * c o l s + ( r o w s / 2 ) * c o l s + ( c o l s ←֓ / 2) ; p i c [ i d x ] = −1; /* p l a c e i n i t i a l s e e d */ }

53

54 55 56 57 58

59 60 61 62 63 64 65 66 67 68 69

70 71 72 73

}

74 75 76 77 78 79 80 81 82 83 84

/ *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− * S i n g l e step e v o l u t i o n of the simulation .

* * The c e l l v a l u e s r e p r e s e n t : * 0 : empty s p a c e * <0 : c r y s t a l −− f o r c o n s i s t e n c y w i t h a l t e r n a t i v e f o r m u l a t i o n * * P a r t i c l e s are held in a separate array . */ v o i d d l a _ e v o l v e _ p l i s t ( i n t * pic , i n t r o w s , i n t c o l s , i n t d e p t h , ←֓ P a r t S t r * p , i n t * p a r t i c l e s , P a r t S t r * c h a n g e s , i n t * ←֓ numChanges )

85 86 87

{ int i ;

* numChanges = 0;

88 89 90 91 92 93 94 95

f o r ( i = 0 ; i < * p a r t i c l e s ; i++) { int new_x = p [ i ] . x + three_way () ; i f ( n e w _ x == 0 ) // bounce o f f new_x = 1; e l s e i f ( n e w _ x == c o l s − 1 ) new_x = cols − 2;

96 97

int new_y = p [ i ] . y + three_way () ;

boundaries

89 if

( n e w _ y == 0 ) // bounce o f f new_y = 1; e l s e i f ( n e w _ y == r o w s − 1 ) new_y = rows − 2;

98 99 100 101

boundaries

102

int new_z = p [ i ] . z + three_way () ; i f ( n e w _ z == 0 ) // bounce o f f new_z = 1; e l s e i f ( n e w _ z == d e p t h − 1 ) new_z = depth − 2;

103 104 105 106 107

boundaries

108

i n t idx = new_z * rows * cols + new_y * cols + new_x ; i f ( p i c [ i d x ] == 0 ) // s t e p s i n t o empty s p a c e { i n t t u r n C r y s t a l = c h e c k _ p r o x i m ( pic , r o w s , c o l s , n e w _ x , ←֓ new_y , new_z ) ; i f ( turnCrystal < 0) { // r e c o r d c r y s t a l change changes [* numChanges ] . x = new_x ; changes [* numChanges ] . y = new_y ; changes [* numChanges ] . z = new_z ; ( * n u m C h a n g e s ) ++;

109 110 111 112

113 114 115 116 117 118 119 120 121 122 123 124

} else {

125 126 127

// change p o s i t i o n t o p a r t i c l e p [ i ] . x = new_x ; p [ i ] . y = new_y ; p [ i ] . z = new_z ;

128 129 130

}

131

}

132

}

133 134

// e r a s e p a r t i c l e from l i s t p [ i ] = p [(* particles ) − 1 ] ; i−−; ( * p a r t i c l e s )−−;

}

135 136 137

138 139 140 141 142

143 144 145

/ *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−* / v o i d a p p l y _ c h a n g e s ( i n t * pic , i n t r o w s , i n t c o l s , P a r t S t r * ←֓ changes , i n t numChanges ) { int i ; f o r ( i = 0 ; i < n u m C h a n g e s ; i++) { i n t i d x = c h a n g e s [ i ] . z * r o w s * c o l s + c h a n g e s [ i ] . y * c o l s ←֓ + changes [ i ] . x ; p i c [ i d x ] = −1; } }

On the other hand, the contents of dla mpi.cpp file shown below, require only minor changes. These are limited to: (a) Lines 206-220, where the derived MPI datatype for PartStr is defined. (b) Lines 247-258 that output the resulting structure as a CSV file. This CSV file can be imported in ParaView (http://www.paraview.org/ Wiki/ParaView/Data_formats) for viewing purposes.

146 147 148 149

/ * DLA 3D MPI program * P a r t i c l e s a r e e v e n l y d i s t r i b u t e d among n o d e s / p r o c e s s e s * G. B a r l a s , 2014 */

150 151 152 153 154

#i n c l u d e <s t d i o . h> #i n c l u d e < s t d l i b . h> #i n c l u d e < a s s e r t . h> #i n c l u d e ” d l a c o r e . h”

90 155

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING #i n c l u d e <mpi . h>

156 157 158 159

#d e f i n e PIC SIZE 100 #d e f i n e PARTICLES 1000 #d e f i n e MAX ITER 10000

160 161 162 163 164 165 166 167 168 169 170 171

/ *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−* / i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t cols , rows , depth , iter , particles , x , y ; i n t * pic ; PartStr *p , * changes , * t o t a l C h a n g e s ; i n t r a n k , num , i , n u m C h a n g e s , n u m T o t a l C h a n g e s ; int * changesPerNode , * buffDispl ; M P I _ I n i t (& a r g c , &a r g v ) ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &n u m ) ;

172 173

174

if

( argc < 2) // u s e d e f a u l t v a l u e s does not s p e c i f y anything cols = PIC_SIZE + 2; rows = PIC_SIZE + 2; depth = PIC_SIZE + 2; iter = MAX_ITER ; particles = PARTICLES ;

176 177 178 179

181 182

} else { cols = atoi ( argv [ 1 ] ) + 2; rows = atoi ( argv [ 2 ] ) + 2; depth = atoi ( argv [ 3 ] ) + 2; particles = atoi ( argv [ 4 ] ) ; iter = atoi ( argv [ 5 ] ) ;

183 184 185 186 187 188

u s e r ←֓

{

175

180

if

}

189 190 191

// i n i t i a l i z e t h e random number g e n e r a t o r srand ( time (0) ) ;

192 193 194 195

196 197 198 199

200

201 202 203

i n t particlesPerNode = particles / num ; i f ( r a n k == n u m − 1 ) p a r t i c l e s P e r N o d e = p a r t i c l e s − p a r t i c l e s P e r N o d e * ( n u m − 1 ) ; ←֓ // i n c a s e p a r t i c l e s c a n n o t be s p l i t e v e n l y // p r i n t f (”% i h as %i \n ” , rank , p a r t i c l e s P e r N o d e ) ; pic = ( i n t *) malloc ( s i z e o f ( i n t ) * cols * rows * depth ) ; p = ( PartStr *) malloc ( s i z e o f ( PartStr ) * particlesPerNode ) ; c h a n g e s = ( P a r t S t r * ) m a l l o c ( s i z e o f ( P a r t S t r ) * ←֓ particlesPerNode ) ; t o t a l C h a n g e s = ( P a r t S t r * ) m a l l o c ( s i z e o f ( P a r t S t r ) * ←֓ particlesPerNode ) ; changesPerNode = ( i n t *) malloc ( s i z e o f ( i n t ) * num ) ; buffDispl = ( i n t *) malloc ( s i z e o f ( i n t ) * num ) ; a s s e r t ( p i c != 0 && p != 0 && c h a n g e s != 0 && t o t a l C h a n g e s != 0←֓ && c h a n g e s P e r N o d e != 0 ) ;

204 205 206 207 208 209 210

// MPI u s e r t y p e d e c l a r a t i o n int lengths [ 3 ] = { 1 , 1 , 1 }; MPI_Datatype types [ 3 ] = { MPI_INT , MPI_INT , MPI_INT }; M P I _ A i n t add1 , add2 , add3 ; MPI_Aint displ [ 3 ] ; MPI_Datatype Point ;

211 212 213 214 215 216 217

M P I _ A d d r e s s ( p , &a d d 1 ) ; M P I _ A d d r e s s (&( p [ 0 ] . y ) , &a d d 2 ) ; M P I _ A d d r e s s (&( p [ 0 ] . z ) , &a d d 3 ) ; displ [ 0 ] = 0; displ [ 1 ] = add2 − add1 ; displ [ 2 ] = add3 − add1 ;

218 219 220

M P I _ T y p e _ s t r u c t ( 3 , l e n g t h s , d i s p l , t y p e s , &P o i n t ) ; M P I _ T y p e _ c o m m i t (& P o i n t ) ;

221 222

d l a _ i n i t _ p l i s t ( pic , r o w s , c o l s , d e p t h , p , p a r t i c l e s P e r N o d e , 1 ) ←֓ ;

91 w h i l e (−−i t e r ) { d l a _ e v o l v e _ p l i s t ( pic , r o w s , c o l s , d e p t h , p , &←֓ p a r t i c l e s P e r N o d e , c h a n g e s , &n u m C h a n g e s ) ;

223 224 225

226

// e x c h a n g e i n f o r m a t i o n w i t h o t h e r n o d e s M P I _ A l l g a t h e r (& n u m C h a n g e s , 1 , M P I _ I N T , c h a n g e s P e r N o d e , 1 , ←֓ MPI_INT , MPI_COMM_WORLD ) ; // c a l c u l a t e o f f s e t s numTotalChanges = 0; f o r ( i = 0 ; i < n u m ; i++) { buffDispl [ i ] = numTotalChanges ; n u m T o t a l C h a n g e s += c h a n g e s P e r N o d e [ i ] ; }

227 228

229 230 231 232 233 234 235 236

if {

237 238

( numTotalChanges > 0) M P I _ A l l g a t h e r v ( c h a n g e s , n u m C h a n g e s , P o i n t , ←֓ t o t a l C h a n g e s , c h a n g e s P e r N o d e , b u f f D i s p l , P o i n t , ←֓ MPI_COMM_WORLD ) ; a p p l y _ c h a n g e s ( pic , r o w s , c o l s , t o t a l C h a n g e s , ←֓ numTotalChanges ) ;

239

240

}

241

}

242 243

if

244

( r a n k == 0 ) {

245

// s a v e d a t a p o i n t s a s a CSV f i l e t h a t can be i m p o r t e d i n ←֓ Paraview F I L E * f = f o p e n ( ” d l a . c s v ” , ”w+t ” ) ; f p r i n t f ( f , ”x c oord , y c oord , z c oord , s c a l a r \n” ) ; i n t idx = 0 ; f o r ( i n t z = 0 ; z < d e p t h ; z++) f o r ( i n t y = 0 ; y < r o w s ; y++) f o r ( i n t x = 0 ; x < c o l s ; x++) { i f ( p i c [ i d x ] == −1) f p r i n t f ( f , ”%i , %i , %i , 1\n” , x , y , z ) ; i d x ++; } fclose ( f ) ;

246

247 248 249 250 251 252 253 254 255 256 257 258

}

259 260

M P I _ R e d u c e (& p a r t i c l e s P e r N o d e , &p a r t i c l e s , 1 , M P I _ I N T , M P I _ S U M , ←֓ 0 , MPI_COMM_WORLD ) ; i f ( r a n k == 0 ) f p r i n t f ( s t d e r r , ” Remaining p a r t i c l e s %i \n” , p a r t i c l e s ) ;

261

262 263 264

free ( pic ) ; free ( p ) ; free ( changes ) ; free ( changesPerNode ) ; free ( buffDispl ) ; MPI_Finalize () ; return 0;

265 266 267 268 269 270 271 272

}

17. It is not unusual in NoWs setups, to have machines with different capabilities. Create an MPI program that would have each process read from a file a number of integers that is proportional to its CPU speed as indicated by the operating frequency (its “clock”). Use appropriate filetypes and views to perform the data distribution in a cyclic-block manner.

Answer Assuming that the input file is broken into BLOCKSIZE pieces 1 , the solution 1 Actually, it is an implicit requirement that the number of etype elements in the file is a multiple of the filetype which is in our case equal to BLOCKSIZE in length. Otherwise, the “excess” part of the file will not be read by any process.

92

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING is assign to each node, a part of a block proportional to its clock speed.

The mathematical formulation is identical to the one used in Exercise 6 and Equations 5.1 and 5.2. Each node i creates a filetype that corresponds to a part of block equal to localSizei = ⌊parti · BLOCKSIZE⌋

(5.7)

in size, that starts at offset localOf fi =

i−1 X j=0

⌊partj · BLOCKSIZE⌋

(5.8)

from the beginning of a block.

In order to make sure that the sum of all parts equals BLOCKSIZE, the last of N nodes is assigned a part equal to

localSizeN −1 = BLOCKSIZE −

N −2 X j=0

⌊partj · BLOCKSIZE⌋

(5.9)

The resulting source code is shown below. Lines 24-31 deal with the local retrieval and global all-to-all distribution of the clock frequencies. The all-to-all scattering operation is required so that the computation of the individual parts can be duplicated in all nodes, without the need to have them communicated by the master node.

Lines 33-57 implements Equations 5.1, 5.2, 5.7 and 5.8. Lines 55, 56, executed solely on the last node of the communicator, deal with the truncations issues depicted by Equation 5.9.

Lines 65 and 66 declare a different local filetype based on the characteristics of each node. This in turn is used to read the data, one filetype at a time, in lines 77-90.

Commented-out lines 67, 81 and 87 represent alternative ways for accomplishing the desired results.

1

2 3 4 5 6 7

// Works o n l y i f t h e actualBlock #i n c l u d e <mpi . h> #i n c l u d e <s t d i o . h> #i n c l u d e <s t d l i b . h> #i n c l u d e <s t r i n g . h> #i n c l u d e <u n i s t d . h> #i n c l u d e <v e c t o r >

filesize

i s a m u l t i p l e o f t h e c a l c u l a t e d ←֓

8 9

u s i n g namespace s t d ;

10 11

#d e f i n e BLOCKSIZE 1000

12 13 14 15 16 17 18 19 20

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t r a n k , num , i ; M P I _ I n i t (& a r g c , &a r g v ) ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &n u m ) ; MPI_Status status ;

93 21 22

f l o a t clockMHz ; f l o a t * allClock = NULL ;

23 24

25 26

FILE * fp = popen ( ”/ bin / cat / proc / c p u i n f o u s r / b i n / head −n 1 | / u s r / b i n /gawk −F r”) ; f s c a n f ( fp , ”%f ” , &c l o c k M H z ) ; pclose ( fp ) ;

| / b i n / g r e p MHz | /←֓ ’ : ’ ’ { p r i n t $ 2 } ’ ” , ”←֓

27 28

a l l C l o c k = new f l o a t [ n u m ] ;

29 30

31

// a l l g a t h e r s o c a l c u l a t i o n s can be r e p l i c a t e d a c r o s s a l l ←֓ nodes M P I _ A l l g a t h e r (& c l o c k M H z , 1 , M P I _ F L O A T , a l l C l o c k , 1 , M P I _ F L O A T , ←֓ MPI_COMM_WORLD ) ;

32 33 34 35

f l o a t sumClocks = allClock [ 0 ] ; f o r ( i = 1 ; i < n u m ; i++) s u m C l o c k s += a l l C l o c k [ i ] ;

36 37 38 39

f l o a t part [ num ] ; f o r ( i = 0 ; i < n u m ; i++) part [ i ] = allClock [ i ] / sumClocks ;

40 41 42 43 44 45 46 47 48 49 50 51 52

int int int for {

localSize ; localOff = 0; actualBlock = 0; ( i = 0 ; i < r a n k ; i++) l o c a l O f f += p a r t [ i ] * B L O C K S I Z E ; a c t u a l B l o c k += p a r t [ i ] * B L O C K S I Z E ;

} localSize = part [ rank ] * BLOCKSIZE ; a c t u a l B l o c k += l o c a l S i z e ; f o r ( i = r a n k + 1 ; i < n u m ; i++) a c t u a l B l o c k += p a r t [ i ] * B L O C K S I Z E ;

53 54

55 56 57

// l a s t node p i c k i n g up t h e s l a c k , s o t h a t a c t u a l B l o c k == ←֓ BLOCKSIZE i f ( r a n k == num −1) l o c a l S i z e += ( B L O C K S I Z E − a c t u a l B l o c k ) ; actualBlock = BLOCKSIZE ;

58 59 60

MPI_File f ; MPI_File_open ( MPI_COMM_WORLD , argv [ 1 ] , M P I _ I N F O _ N U L L , &f ) ;

M P I _ M O D E _ R D O N L Y , ←֓

61 62 63

MPI_Datatype filetype ; int starts = 0;

64 65

66 67

68

M P I _ T y p e _ c r e a t e _ s u b a r r a y ( 1 , &a c t u a l B l o c k , &l o c a l S i z e , &s t a r t s , ←֓ M P I _ O R D E R _ C , M P I _ I N T , &f i l e t y p e ) ; M P I _ T y p e _ c o m m i t (& f i l e t y p e ) ; // M P I F i l e s e t v i e w ( f , 0 , MPI INT , f i l e t y p e , ( c h a r * ) ” n a t i v e ←֓ ” , MPI INFO NULL ) ; M P I _ F i l e _ s e t _ v i e w ( f , l o c a l O f f * s i z e o f ( i n t ) , M P I _ I N T , ←֓ filetype , ( char *) ” native ” , MPI_INFO_NULL ) ;

69 70 71

v e c t o r < i n t >d a t a ; i n t temp [ BLOCKSIZE ] ;

72 73 74 75

76

77 78 79 80 81 82 83

MPI_Offset filesize ; M P I _ F i l e _ g e t _ s i z e ( f , &f i l e s i z e ) ; // g e t s i z e i n b y t e s f i l e s i z e /= s i z e o f ( i n t ) ; // c o n v e r t s i z e i n number o f ←֓ items long long i n t pos = 0 ; // l o c a l O f f ; // i n i t i a l f i l e ←֓ p o s i t i o n per p r o c e s s while ( pos < filesize ) { M P I _ F i l e _ r e a d ( f , t e m p , 1 , f i l e t y p e , &s t a t u s ) ; i n t cnt ; // MPI Get count (& s t a t u s , f i l e t y p e , &c n t ) ; // g e t t h e number o f d a t a r e a d i n e t y p e u n i t s M P I _ G e t _ c o u n t (& s t a t u s , M P I _ I N T , &c n t ) ;

94

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING

84 85

p o s += a c t u a l B l o c k ; for ( int i = 0; i < localSize ; f o r ( i n t i = 0 ; i < c n t ; i++) data . push_back ( temp [ i ] ) ;

86 87

//

88 89

i ++)

}

90 91

sleep ( rank ) ; c o u t << r a n k << ” r e a d ” << d a t a . s i z e ( ) << ” numbers . ” << e n d l ←֓ ; f o r ( i n t i = 0 ; i < 3 0 ; i++) c o u t << d a t a [ i ] << ” ” ; c o u t << ” . . . . L a s t one i s : ” << d a t a [ d a t a . s i z e ( ) − 1 ] ; c o u t << e n d l ;

92 93

94 95 96 97 98

MPI_Finalize () ; return 0;

99 100 101

}

18. The details of the trapezoidal rule for computing a function integral are discussed in Section 3.5.2. Implement an MPI-based version of the trapezoidal rule using: (a) dynamic partitioning and (b) static partitioning.

For part (a) you can base your solution on master-worker implementation of Section 5.22.1.

Answer (a) The following listing is a derivation of the master-worker implementation of Section 5.22.1. Given the simple nature of the problem and the associated I/O, the resulting code is much simpler than the original. The only essential customizations are: the adaptation of the derived datatype used to communicate the description of a “job”, and the elimination of the “job assignment tracking” feature (supported by the assignedPart array in Listing 5.38), as in this case the results are scalar, floating point numbers that are accumulated by the master node (line 121).

1 2 3 4 5 6

#i n c l u d e <s t d i o . h> #i n c l u d e < s t d l i b . h> #i n c l u d e <i o s t r e a m > #i n c l u d e <math . h> #i n c l u d e <mpi . h> u s i n g namespace s t d ;

7 8 9 10 11 12 13

// ************************************************************ // Communication t a g s #d e f i n e NULLRESULTTAG 0 #d e f i n e RESULTTAG 1 #d e f i n e WORKITEMTAG 2 #d e f i n e ENDTAG 3

14 15 16 17 18 19 20 21

// ************************************************************ struct WorkItem { double start ; double end ; int divisions ; };

22 23 24 25 26 27

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− double func ( double x ) { return fabs ( sin ( x ) ) ; }

28 29 30 31

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− d o u b l e i n t e g r C a l c ( d o u b l e st , d o u b l e en , i n t d i v ) {

95 // c a l c u l a t e a r e a double localRes = 0 ; double step = ( en − st ) / div ; double x ; x = st ; l o c a l R e s = func ( st ) + func ( en ) ; l o c a l R e s /= 2 ; f o r ( i n t i = 1 ; i < d i v ; i++) { x += s t e p ; l o c a l R e s += f u n c ( x ) ; } l o c a l R e s *= s t e p ; return localRes ;

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46

}

47 48 49 50 51

// ************************************************************ void registerWorkItem ( MPI_Datatype * workItemType ) { struct WorkItem sample ;

52

int blklen [ 3 ] = { 1 , 1 , 1 }; M P I _ A i n t d i s p l [ 3 ] , off , b a s e ; MPI_Datatype types [ 3 ] = { MPI_DOUBLE , MPI_DOUBLE , MPI_INT };

53 54 55 56

displ [ 0 ] = 0; M P I _ G e t _ a d d r e s s (&( s a m p l e . s t a r t ) , &b a s e ) ; M P I _ G e t _ a d d r e s s (&( s a m p l e . e n d ) , &o f f ) ; displ [ 1 ] = off − base ; M P I _ G e t _ a d d r e s s (&( s a m p l e . d i v i s i o n s ) , &o f f ) ; displ [ 2 ] = off − base ;

57 58 59 60 61 62 63

M P I _ T y p e _ c r e a t e _ s t r u c t ( 3 , blklen , displ , types , MPI_Type_commit ( workItemType ) ;

64 65 66

workItemType ) ;

}

67 68 69 70 71 72 73 74

// ************************************************************ i n t main ( i n t argc , char * argv [ ] ) { i n t N , rank ; double start_time , end_time ; MPI_Status status ; MPI_Request request ;

75 76 77 78

M P I _ I n i t (& a r g c , &a r g v ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &N ) ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ;

79 80 81

MPI_Datatype workItemType ; r e g i s t e r W o r k I t e m (& w o r k I t e m T y p e ) ;

82 83 84 85 86 87 88

if

( r a n k == 0 )

// m a s t e r c o d e

{ double LOWERLIMIT = 0 ; double UPPERLIMIT = 10; i n t NDIV = 1000000; int NJOBS = 10;

89 90 91

if

( argc > 4) { LOWERLIMIT = atof ( argv [ 1 ] ) ; UPPERLIMIT = atof ( argv [ 2 ] ) ; NDIV = atoi ( argv [ 3 ] ) ; NJOBS = atoi ( argv [ 4 ] ) ;

92 93 94 95 96

}

97 98 99 100 101 102 103 104 105

double total = 0 , partialRes ; i n t divPerJob = NDIV / NJOBS ; W o r k I t e m * w = new W o r k I t e m [ N J O B S ] ; double jobStep = ( UPPERLIMIT − LOWERLIMIT ) / NJOBS ; double jobSt = LOWERLIMIT ; double jobEnd = jobSt + jobStep ; f o r ( i n t i = 0 ; i < N J O B S ; i++) {

96

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING w [ i ] . start = jobSt ; w [ i ] . end = jobEnd ; w [ i ] . divisions = divPerJob ; jobSt = jobEnd ; j o b E n d += j o b S t e p ;

106 107 108 109 110

}

111 112

// now d i s t r i b u t e t h e work i t e m t o t h e w o r k e r n o d e s f o r ( i n t i = 0 ; i < N J O B S ; i++) { M P I _ R e c v (& p a r t i a l R e s , 1 , M P I _ D O U B L E , M P I _ A N Y _ S O U R C E , ←֓ M P I _ A N Y _ T A G , M P I _ C O M M _ W O R L D , &s t a t u s ) ; int workerID = status . MPI_SOURCE ; i n t tag = status . MPI_TAG ; M P I _ I s e n d (&( w [ i ] ) , 1 , w o r k I t e m T y p e , w o r k e r I D , ←֓ W O R K I T E M T A G , M P I _ C O M M _ W O R L D , &r e q u e s t ) ; i f ( t a g == R E S U L T T A G ) t o t a l += p a r t i a l R e s ; }

113 114 115 116

117 118 119

120 121 122 123

// now s e n d t e r m i n a t i o n m e s s a g e s f o r ( i n t i = 1 ; i < N ; i++) { M P I _ R e c v (& p a r t i a l R e s , 1 , M P I _ D O U B L E , M P I _ A N Y _ S O U R C E , ←֓ M P I _ A N Y _ T A G , M P I _ C O M M _ W O R L D , &s t a t u s ) ; int workerID = status . MPI_SOURCE ; i n t tag = status . MPI_TAG ;

124 125 126 127

128 129 130

if

( t a g == R E S U L T T A G ) t o t a l += p a r t i a l R e s ; M P I _ I s e n d ( N U L L , 0 , w o r k I t e m T y p e , w o r k e r I D , E N D T A G , ←֓ M P I _ C O M M _ W O R L D , &r e q u e s t ) ;

131 132 133

}

134 135

c o u t << ” R e s u l t delete [ ] w ;

136 137

} else {

138 139 140

i s ” << t o t a l << e n d l ; // w o r k e r c o d e

M P I _ S e n d ( N U L L , 0 , M P I _ D O U B L E , 0 , N U L L R E S U L T T A G , ←֓ M P I _ C O M M _ W O R L D ) ; // e s t a b l i s h communication w i t h ←֓ master while (1) { WorkItem w ; M P I _ R e c v (&w , 1 , w o r k I t e m T y p e , 0 , M P I _ A N Y _ T A G , ←֓ M P I _ C O M M _ W O R L D , &s t a t u s ) ; // g e t a new work ←֓ item i n t tag = status . MPI_TAG ; i f ( t a g == E N D T A G ) break ;

141

142 143 144 145

146 147 148 149

d o u b l e p a r t R e s = i n t e g r C a l c ( w . s t a r t , w . end , w . ←֓ divisions ) ; M P I _ S e n d (& p a r t R e s , 1 , M P I _ D O U B L E , 0 , R E S U L T T A G , ←֓ MPI_COMM_WORLD ) ; // r e t u r n t h e r e s u l t s

150

151

}

152

}

153 154

MPI_Finalize () ; return 0;

155 156 157

}

(b) The static partitioning is a much simpler proposition. Each node can query the command-line parameters and determine (without receiving any information from the “master”) the part of the integration range that is dedicate to it (lines 211, 212). Following the partial integration of line 213, with a reduction (line 216) completes the calculation.

158 159 160

#i n c l u d e <s t d i o . h> #i n c l u d e < s t d l i b . h> #i n c l u d e <i o s t r e a m >

97 161 162 163

#i n c l u d e <math . h> #i n c l u d e <mpi . h> u s i n g namespace s t d ;

164 165 166 167 168 169

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− double func ( double x ) { return fabs ( sin ( x ) ) ; }

170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− d o u b l e i n t e g r C a l c ( d o u b l e st , d o u b l e en , i n t d i v ) { // c a l c u l a t e a r e a double localRes = 0 ; double step = ( en − st ) / div ; double x ; x = st ; l o c a l R e s = func ( st ) + func ( en ) ; l o c a l R e s /= 2 ; f o r ( i n t i = 1 ; i < d i v ; i++) { x += s t e p ; l o c a l R e s += f u n c ( x ) ; } l o c a l R e s *= s t e p ; return localRes ; }

189 190 191 192 193 194 195 196

// ************************************************************ i n t main ( i n t argc , char * argv [ ] ) { i n t N , rank ; double LOWERLIMIT = 0 ; double UPPERLIMIT = 10; i n t NDIV = 1000000;

197

M P I _ I n i t (& a r g c , &a r g v ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &N ) ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ;

198 199 200 201

if

202

( argc > 3) {

203

LOWERLIMIT = atof ( argv [ 1 ] ) ; UPPERLIMIT = atof ( argv [ 2 ] ) ; NDIV = atoi ( argv [ 3 ] ) ;

204 205 206

}

207 208

i n t divPerNode = NDIV / N ; double nodeStep = ( UPPERLIMIT − LOWERLIMIT ) / N ; double localSt = LOWERLIMIT + rank * nodeStep ; double localEnd = localSt + nodeStep ; double localRes = integrCalc ( localSt , localEnd , double total ;

209 210 211 212 213 214

divPerNode ) ;

215

M P I _ R e d u c e (& l o c a l R e s , &t o t a l , 1 , M P I _ D O U B L E , M P I _ S U M , 0 , ←֓ MPI_COMM_WORLD ) ; i f ( r a n k == 0 ) c o u t << ” R e s u l t i s ” << t o t a l << e n d l ;

216

217 218 219

MPI_Finalize () ; return 0;

220 221 222

}

19. Use the master-worker code provided in Section 5.22, as a basis for the creation of a hierarchical master-worker configuration, where nodes are organized in a 3-level (or higher) tree instead of the 2-level tree of the simple setup. The secondary master nodes of the middle tree layer(s) should be responsible for managing the load distribution in their subtrees, while the primary master at the root of the tree should be responsible for

98

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING the overall workload distribution.

Answer The calculation of the Mandelbrot fractal is by no means a suitable target application for this kind of setup. So this solution is more of a proof-ofconcept, than a practical way to speed-up the particular calculation.

In the listing that follows, nodes form a 3-level hierarchy, with SECONDARYMASTER nodes serving in the intermediate layer. The worker nodes calculate the secondary master they are reporting to, with the simple calculation of line 347. The major points can be summarized as follows:

The master node initially distributes groups of BATCHSIZE number of work items (lines 234-242). Subsequently, it sends to each secondary master a single work item for every partial result received (lines 245255). When all the work items have been distributed, the master sends termination messages for each partial result received (lines 257266).

The secondary master nodes maintain a local pool of work items, that is initialized by the bulk work items sent by the primary master (lines 287-289). The work items are sent one-by-one to the workers that communicate with the secondary masters. The image parts/results that are sent back, are forwarded to the primary master (lines 296 and 316).

The secondary master nodes -expect to- receive a new work item for each result they send back. The new work item is appended to the local pool (line 320), so it is the next one to be sent to a worker.

The worker assignment is delayed, until the work item in question is received from the master node (lines 301-307).

Obviously, a proper, circular, FIFO queue setup would be more beneficial for the handling of each local work item pool. To avoid making an already lengthy code even longer, this is not pursued.

The worker node code is identical to the one reported in Section 5.22, with the exception of line 347.

1 2 3 4 5 6 7 8

#i n c l u d e <s t d i o . h> #i n c l u d e < s t d l i b . h> #i n c l u d e <i o s t r e a m > #i n c l u d e <math . h> #i n c l u d e ”mpi . h” #i n c l u d e <QImage> #i n c l u d e <QRgb> u s i n g namespace s t d ;

9 10 11 12 13 14 15

// ************************************************************ // Communication t a g s #d e f i n e NULLRESULTTAG 0 #d e f i n e RESULTTAG 1 #d e f i n e WORKITEMTAG 2 #d e f i n e ENDTAG 3

16 17

18

19 20

#d e f i n e BATCHSIZE 10 / * Job s a r e i n i t i a l l y s e n d t o ←֓ s e c o n d a r y m a s t e r n o d e s i n g r o u p s o f BATCHSIZE * B A T C H S I Z E h a s t o b e g r e a t e r ←֓ t h a n 1 , o t h e r w i s e t h e ←֓ secondary master nodes * terminate prematurely */

99 21

22

#d e f i n e SECONDARYMASTER 2 / * Number o f s e c o n d a r y m a s t e r ←֓ n o d e s */ // ************************************************************

23 24 25 26 27 28

typedef struct WorkItem { double upperX , upperY , lowerX , lowerY ; i n t pixelsX , pixelsY , imageX , imageY , imgPartID ; } WorkItem ;

29 30 31 32 33 34 35 36 37

// ************************************************************ // C l a s s f o r computing a f r a c t a l s e t p a r t c l a s s MandelCompute { private : double upperX , upperY , lowerX , lowerY ; i n t pixelsX , pixelsY , imgID ; i n t * img ;

38 39 40

s t a t i c int MAXITER ; i n t d i v e r g e ( d o u b l e cx , d o u b l e c y ) ;

41 42 43 44 45 46 47 48

public : MandelCompute () ; void init ( W o r k I t e m * wi ) ; ˜ MandelCompute () ; int * compute () ; }; int MandelCompute : : MAXITER = 255;

49 50

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

i n t M a n d e l C o m p u t e : : d i v e r g e ( d o u b l e cx , d o u b l e c y ) { i n t iter = 0; d o u b l e v x = cx , v y = cy , tx , t y ; w h i l e ( i t e r < M A X I T E R && ( v x * v x + v y * v y ) < 4 ) { tx = vx * vx − vy * vy + cx ; ty = 2 * vx * vy + cy ; vx = tx ; vy = ty ; i t e r ++; } return iter ; }

66 67 68 69 70 71 72

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− MandelCompute : : ˜ MandelCompute () { i f ( i m g != N U L L ) d e l e t e [ ] img ; }

73 74 75 76 77 78

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− MandelCompute : : MandelCompute () { img = NULL ; }

79 80

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

81 82 83 84 85 86 87

void M a n d e l C o m p u t e : : init ( W o r k I t e m * wi ) { u p p e r X = wi−>u p p e r X ; u p p e r Y = wi−>u p p e r Y ; l o w e r X = wi−>l o w e r X ; l o w e r Y = wi−>l o w e r Y ;

88 89

90 91 92

if

( i m g == N U L L pixelsY )

||

p i x e l s X != wi−>p i x e l s X

{ if

( i m g != N U L L ) d e l e t e [ ] img ;

||

p i x e l s Y != wi−>←֓

100

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING // img h as an e x t r a e l e m e n t f o r s t o r i n g t h e ID o f t h e ←֓ p i c t u r e p a r t computed i m g = new i n t [ ( wi−>p i x e l s X ) * ( wi−>p i x e l s Y ) + 1 ] ;

93

94

} p i x e l s X = wi−>p i x e l s X ; p i x e l s Y = wi−>p i x e l s Y ; i m g I D = wi−>i m g P a r t I D ;

95 96 97 98 99

}

100 101

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

102 103 104 105 106

int * MandelCompute : : compute () { double stepx = ( lowerX − upperX ) / pixelsX ; double stepy = ( upperY − lowerY ) / pixelsY ;

107

f o r ( i n t i = 0 ; i < p i x e l s X ; i++) f o r ( i n t j = 0 ; j < p i x e l s Y ; j++) { double tempx , tempy ; tempx = upperX + i * stepx ; tempy = upperY − j * stepy ; img [ j * pixelsX + i ] = diverge ( tempx , tempy ) ; } img [ pixelsX * pixelsY ] = imgID ; return img ;

108 109 110 111 112 113 114 115 116 117 118

}

119 120 121 122 123

// ************************************************************ void registerWorkItem ( MPI_Datatype * workItemType ) { struct WorkItem sample ;

124

int blklen [ 2 ] ; M P I _ A i n t d i s p l [ 2 ] , off , b a s e ; MPI_Datatype types [ 2 ] ;

125 126 127 128

blklen [ 0 ] = 4; blklen [ 1 ] = 5; communicated r e g a r d l e s s

129 130

// t h e p a r t ’ s l o c a t i o n

i s ←֓

131

types [ 0 ] = MPI_DOUBLE ; types [ 1 ] = MPI_INT ;

132 133 134

displ [ 0 ] = 0; M P I _ G e t _ a d d r e s s (&( s a m p l e . u p p e r X ) , &b a s e ) ; M P I _ G e t _ a d d r e s s (&( s a m p l e . p i x e l s X ) , &o f f ) ; displ [ 1 ] = off − base ;

135 136 137 138 139

M P I _ T y p e _ c r e a t e _ s t r u c t ( 2 , blklen , displ , types , MPI_Type_commit ( workItemType ) ;

140 141 142

workItemType ) ;

}

143 144 145 146

147 148 149 150 151 152

153 154

// ************************************************************ // Uses t h e d i v e r g e n c e i t e r a t i o n s t o p s e u d o c o l o r t h e f r a c t a l s e t v o i d s a v e P i x e l s ( Q I m a g e * img , i n t * i m g P a r t , i n t i m a g e X , i n t ←֓ imageY , i n t height , i n t width ) { f o r ( i n t i = 0 ; i < w i d t h ; i++) f o r ( i n t j = 0 ; j < h e i g h t ; j++) { int color = imgPart [ j * width + i ] ; img−>s e t P i x e l ( i m a g e X + i , i m a g e Y + j , q R g b ( 2 5 6 − c o l o r , ←֓ 256 − c o l o r , 256 − c o l o r ) ) ; } }

155 156 157 158 159 160 161 162

// ************************************************************ i n t main ( i n t argc , char * argv [ ] ) { i n t N , rank ; double start_time , end_time ; MPI_Status status ; MPI_Request request ;

101 163 164

start_time = MPI_Wtime () ;

165

// visit Library Genesis for more solution manuals: libgen.is, libgen.st, libgen.rs, forum.mhut.org

166

M P I _ I n i t (& a r g c , &a r g v ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &N ) ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ;

167 168 169 170 171

if {

172

173 174

( N < SECONDARYMASTER * 2 + 1) c e r r << a r g v [ 0 ] << ” n e e d s a t l e a s t ” << S E C O N D A R Y M A S T E R * ←֓ 2 + 1 << ” n o d e s t o run \n” ; MPI_Abort ( MPI_COMM_WORLD , 2) ;

}

175 176 177

MPI_Datatype workItemType ; r e g i s t e r W o r k I t e m (& w o r k I t e m T y p e ) ;

178 179 180 181 182 183 184

185 186

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i f ( r a n k == 0 ) // p r i m a r y m a s t e r c o d e { i f ( argc < 6) { c e r r << a r g v [ 0 ] << ” upperCornerX upperCornerY ←֓ lowerCornerX lowerCornerY w o r k I t e m P i x e l s P e r S i d e \n”←֓ ; MPI_Abort ( MPI_COMM_WORLD , 1) ; }

187 188 189 190 191 192 193

double upperCornerX , upperCornerY ; double lowerCornerX , lowerCornerY ; double partXSpan , partYSpan ; int workItemPixelsPerSide ; i n t Xparts , Yparts ; i n t imgX = 1024 , imgY = 768;

194 195 196 197 198 199

upperCornerX = atof ( argv [ 1 ] ) ; upperCornerY = atof ( argv [ 2 ] ) ; lowerCornerX = atof ( argv [ 3 ] ) ; lowerCornerY = atof ( argv [ 4 ] ) ; workItemPixelsPerSide = atoi ( argv [ 5 ] ) ;

200 201

202 203 204 205

// make s u r e t h a t t h e image s i z e i s e v e n l y d i v i d e d i n work ←֓ items Xparts = ( i n t ) ceil ( imgX * 1.0 / workItemPixelsPerSide ) ; Yparts = ( i n t ) ceil ( imgY * 1.0 / workItemPixelsPerSide ) ; imgX = Xparts * workItemPixelsPerSide ; imgY = Yparts * workItemPixelsPerSide ;

206 207 208 209

partXSpan = ( lowerCornerX − upperCornerX ) / Xparts ; partYSpan = ( upperCornerY − lowerCornerY ) / Yparts ; Q I m a g e * i m g = new Q I m a g e ( i m g X , i m g Y , Q I m a g e : : F o r m a t _ R G B 3 2 ) ←֓ ;

210 211 212 213 214 215 216

// p r e p a r e t h e work i t e m s i n i n d i v i d u a l s t r u c t u r e s W o r k I t e m * w = new W o r k I t e m [ X p a r t s * Y p a r t s ] ; f o r ( i n t i = 0 ; i < X p a r t s ; i++) f o r ( i n t j = 0 ; j < Y p a r t s ; j++) { i n t idx = j * Xparts + i ;

217

w [ idx ] . upperX w [ idx ] . upperY w [ idx ] . lowerX w [ idx ] . lowerY

218 219 220 221

= = = =

upperCornerX + i * partXSpan ; upperCornerY − j * partYSpan ; upperCornerX + ( i + 1) * partXSpan ; upperCornerY − ( j + 1) * partYSpan ;

222

w [ idx ] . imageX = i * workItemPixelsPerSide ; w [ idx ] . imageY = j * workItemPixelsPerSide ; w [ idx ] . pixelsX = workItemPixelsPerSide ; w [ idx ] . pixelsY = workItemPixelsPerSide ; w [ idx ] . imgPartID = idx ;

223 224 225 226 227 228

}

229 230 231

// now d i s t r i b u t e t h e work i t e m t o t h e w o r k e r n o d e s i n t * a s s i g n e d P a r t = new i n t [ S E C O N D A R Y M A S T E R ] ; // keep ←֓

102

232

233 234 235 236

237 238

239 240

241 242

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING t r a c k o f how many j o b s e a c h s e c o n d a r y m a s t e r i s ←֓ assigned i n t * i m g P a r t = new i n t [ w o r k I t e m P i x e l s P e r S i d e * ←֓ workItemPixelsPerSide + 1 ] ; // f o r c o l l e c t i n g ←֓ results int assignedCounter = 0; f o r ( i n t i = 1 ; i <= S E C O N D A R Y M A S T E R ; i++) { M P I _ R e c v ( N U L L , 0 , M P I _ I N T , M P I _ A N Y _ S O U R C E , ←֓ N U L L R E S U L T T A G , M P I _ C O M M _ W O R L D , &s t a t u s ) ; int secMasterID = status . MPI_SOURCE ; i n t t o B e A s s i g n e d = m i n ( B A T C H S I Z E , X p a r t s * Y p a r t s − ←֓ assignedCounter ) ; assignedPart [ secMasterID ] = toBeAssigned ; M P I _ I s e n d ( w + a s s i g n e d C o u n t e r , t o B e A s s i g n e d , ←֓ w o r k I t e m T y p e , s e c M a s t e r I D , W O R K I T E M T A G , ←֓ M P I _ C O M M _ W O R L D , &r e q u e s t ) ; a s s i g n e d C o u n t e r += t o B e A s s i g n e d ; }

243 244 245 246 247

248 249

250

int remainPartToCollect = Xparts * Yparts ; while ( assignedCounter < Xparts * Yparts ) { M P I _ R e c v ( i m g P a r t , w o r k I t e m P i x e l s P e r S i d e * ←֓ w o r k I t e m P i x e l s P e r S i d e + 1 , M P I _ I N T , M P I _ A N Y _ S O U R C E ←֓ , R E S U L T T A G , M P I _ C O M M _ W O R L D , &s t a t u s ) ; int secMasterID = status . MPI_SOURCE ; i n t w i d x = i m g P a r t [ w o r k I t e m P i x e l s P e r S i d e * ←֓ workItemPixelsPerSide ] ; // a s s i g n e d P a r t [ workerID←֓ ]; M P I _ I s e n d (&( w [ a s s i g n e d C o u n t e r ] ) , 1 , w o r k I t e m T y p e , ←֓ s e c M a s t e r I D , W O R K I T E M T A G , M P I _ C O M M _ W O R L D , &r e q u e s t ←֓ );

251

s a v e P i x e l s ( img , i m g P a r t , w [ w i d x ] . i m a g e X , w [ w i d x ] . ←֓ i m a g e Y , w o r k I t e m P i x e l s P e r S i d e , ←֓ workItemPixelsPerSide ) ; a s s i g n e d C o u n t e r ++; r e m a i n P a r t T o C o l l e c t −−;

252

253 254 255

}

256 257 258 259

260 261

262

while ( remainPartToCollect > 0) { M P I _ R e c v ( i m g P a r t , w o r k I t e m P i x e l s P e r S i d e * ←֓ w o r k I t e m P i x e l s P e r S i d e + 1 , M P I _ I N T , M P I _ A N Y _ S O U R C E ←֓ , R E S U L T T A G , M P I _ C O M M _ W O R L D , &s t a t u s ) ; int secMasterID = status . MPI_SOURCE ; i n t w i d x = i m g P a r t [ w o r k I t e m P i x e l s P e r S i d e * ←֓ workItemPixelsPerSide ] ; M P I _ I s e n d ( N U L L , 0 , w o r k I t e m T y p e , s e c M a s t e r I D , E N D T A G , ←֓ M P I _ C O M M _ W O R L D , &r e q u e s t ) ;

263

s a v e P i x e l s ( img , i m g P a r t , w [ w i d x ] . i m a g e X , w [ w i d x ] . ←֓ i m a g e Y , w o r k I t e m P i x e l s P e r S i d e , ←֓ workItemPixelsPerSide ) ; r e m a i n P a r t T o C o l l e c t −−;

264

265 266

}

267 268 269

img−>s a v e ( ” mandel . png ” , ”PNG” , 0 ) ; r e s u l t i n g image

// s a v e t h e ←֓

270 271 272 273

delete [ ] w ; delete [ ] assignedPart ; delete [ ] imgPart ;

274 275 276 277 278 279 280 281 282

end_time = MPI_Wtime () ; c o u t << ” T o t a l t i m e : ” << e n d _ t i m e − s t a r t _ t i m e << e n d l ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− e l s e i f ( r a n k <= S E C O N D A R Y M A S T E R ) // s e c o n d a r y m a s t e r c o d e { i n t workItemPixelsPerSide = atoi ( argv [ 5 ] ) ; W o r k I t e m * l o c a l P o o l = new W o r k I t e m [ B A T C H S I Z E ] ;

103 283

284

i n t * i m g P a r t = new i n t [ w o r k I t e m P i x e l s P e r S i d e * ←֓ workItemPixelsPerSide + 1 ] ; // f o r c o l l e c t i n g ←֓ results int poolCount = 0;

285 286 287

288

289

// g e t b a t c h o f l o a d from p r i m a r y m a s t e r M P I _ S e n d ( N U L L , 0 , M P I _ I N T , 0 , N U L L R E S U L T T A G , ←֓ MPI_COMM_WORLD ) ; // e s t a b l i s h communication w i t h ←֓ primary master M P I _ R e c v ( l o c a l P o o l , B A T C H S I Z E , w o r k I t e m T y p e , 0 , ←֓ W O R K I T E M T A G , M P I _ C O M M _ W O R L D , &s t a t u s ) ; M P I _ G e t _ c o u n t (& s t a t u s , w o r k I t e m T y p e , &p o o l C o u n t ) ; ←֓ // g e t t h e s i z e o f t h e a s s i g n m e n t

290 291 292 293 294 295 296

297 298

MPI_Request jobReq ; bool waitFlag = f a l s e ; int pendingResults = 0; while ( poolCount > 0) { M P I _ R e c v ( i m g P a r t , w o r k I t e m P i x e l s P e r S i d e * ←֓ w o r k I t e m P i x e l s P e r S i d e + 1 , M P I _ I N T , M P I _ A N Y _ S O U R C E ←֓ , M P I _ A N Y _ T A G , M P I _ C O M M _ W O R L D , &s t a t u s ) ; int workerID = status . MPI_SOURCE ; i n t tag = status . MPI_TAG ;

299

// w a i t f o r j o b ( s ) t o be r e c e i v e d from t h e M P I I r e c v ←֓ c a l l f o l l o w i n g below i f ( waitFlag ) { MPI_Status st2 ; M P I _ W a i t (& j o b R e q , &s t 2 ) ; i f ( s t 2 . M P I _ T A G == W O R K I T E M T A G ) p o o l C o u n t ++; } p o o l C o u n t −−; p e n d i n g R e s u l t s ++; M P I _ I s e n d ( l o c a l P o o l + p o o l C o u n t , 1 , w o r k I t e m T y p e , ←֓ w o r k e r I D , W O R K I T E M T A G , M P I _ C O M M _ W O R L D , &r e q u e s t ) ; i f ( t a g == R E S U L T T A G ) { p e n d i n g R e s u l t s −−;

300

301 302 303 304 305 306 307 308 309 310

311 312 313 314

// f o r w a r d r e s u l t s t o p r i m a r y m a s t e r M P I _ S e n d ( i m g P a r t , w o r k I t e m P i x e l s P e r S i d e * ←֓ w o r k I t e m P i x e l s P e r S i d e + 1 , M P I _ I N T , 0 , ←֓ RESULTTAG , MPI_COMM_WORLD ) ;

315 316

317

// g e t new work i t e m a f t e r t h e c o r r e s p o n d i n g memory←֓ i s f r e e t o a c c e p t new d a t a M P I _ W a i t (& r e q u e s t , &s t a t u s ) ; M P I _ I r e c v ( l o c a l P o o l + p o o l C o u n t , 1 , w o r k I t e m T y p e , ←֓ 0 , M P I _ A N Y _ T A G , M P I _ C O M M _ W O R L D , &j o b R e q ) ; waitFlag = true ;

318

319 320

321

}

322 323

}

324 325 326 327 328

329 330

// now s e n d t e r m i n a t i o n m e s s a g e s while ( pendingResults > 0) { M P I _ R e c v ( i m g P a r t , w o r k I t e m P i x e l s P e r S i d e * ←֓ w o r k I t e m P i x e l s P e r S i d e + 1 , M P I _ I N T , M P I _ A N Y _ S O U R C E ←֓ , M P I _ A N Y _ T A G , M P I _ C O M M _ W O R L D , &s t a t u s ) ; int workerID = status . MPI_SOURCE ; i n t tag = status . MPI_TAG ;

331 332

333 334 335

M P I _ I s e n d ( N U L L , 0 , w o r k I t e m T y p e , w o r k e r I D , E N D T A G , ←֓ M P I _ C O M M _ W O R L D , &r e q u e s t ) ; i f ( t a g == R E S U L T T A G ) { p e n d i n g R e s u l t s −−;

336 337 338

// f o r w a r d r e s u l t s t o p r i m a r y m a s t e r M P I _ S e n d ( i m g P a r t , w o r k I t e m P i x e l s P e r S i d e * ←֓ w o r k I t e m P i x e l s P e r S i d e + 1 , M P I _ I N T , 0 , ←֓

104

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING RESULTTAG , } delete [ ] imgPart ; delete [ ] localPool ;

340 341 342 343 344 345 346 347

348 349

350 351 352 353

354 355 356

MPI_COMM_WORLD ) ;

}

339

} //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− else // w o r k e r c o d e { i n t myMasterID = ( rank % SECONDARYMASTER ) + 1; // which ←֓ master to r e p o r t to MandelCompute c ; M P I _ S e n d ( N U L L , 0 , M P I _ I N T , m y M a s t e r I D , N U L L R E S U L T T A G , ←֓ MPI_COMM_WORLD ) ; // e s t a b l i s h communication w i t h ←֓ master while (1) { WorkItem w ; M P I _ R e c v (&w , 1 , w o r k I t e m T y p e , m y M a s t e r I D , M P I _ A N Y _ T A G , ←֓ M P I _ C O M M _ W O R L D , &s t a t u s ) ; // g e t a new work ←֓ item i n t tag = status . MPI_TAG ; i f ( t a g == E N D T A G ) break ;

357

c . i n i t (&w ) ; i n t * res = c . compute ( ) ; M P I _ S e n d ( res , w . p i x e l s X * w . p i x e l s Y + 1 , M P I _ I N T , ←֓ myMasterID , RESULTTAG , MPI_COMM_WORLD ) ; // ←֓ return the r e s u l t s

358 359 360

}

361

}

362 363

MPI_Finalize () ; return 0;

364 365 366

}

20. Modify the multi-threaded master-worker code of Section 5.22.2 so that there are two separate threads in each worker process for communication : one for receiving work items and one for sending back results. What are the benefits and drawbacks of this arrangement?

Answer The benefits are two-fold: (a) The decoupled “sender” and “receiver” codes are simpler to write and maintain. The less complicated logic is evident, as a simple comparison to Listing 5.40 can attest. (b) The two communication operations can proceed at different speeds, potentially improving the utilization of the computing resources.

The modifications required are limited to the part of the main function that deal with worker communications (lines 391-436 of Listing 5.40). The additional ResultSender class shown below, is used to spawn a thread that handles communication to the master node exclusively, leaving the main thread to handle communication from the master node.

In terms of data items, the only change affects the assigned counter, which now needs to be changed from two threads. Instead of using a mutex to control access to it, its type is changed to QAtomicInt. The following listing contains only the changes/additions to Listing 5.40, in order to reduce clutter.

105 1 2 3 4 5 6 7 8 9

10 11

... c l a s s ResultSender : public QThread { private : Q u e u e M o n i t o r < M a n d e l R e s u l t * >* o u t q u e ; int numWorkerThreads ; QAtomicInt * assigned ; public : R e s u l t S e n d e r ( Q u e u e M o n i t o r < M a n d e l R e s u l t * >* oq , i n t nw , ←֓ Q A t o m i c I n t * a s ) : o u t q u e ( o q ) , n u m W o r k e r T h r e a d s ( n w ) , ←֓ a s s i g n e d ( as ) {} void run ( ) ; };

12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

// ************************************************************ void ResultSender : : run ( ) { while ( numWorkerThreads ) { MandelResult * res ; i f ( o u t q u e −>a v a i l I t e m s ( ) > 0 ) { r e s = o u t q u e −>d e q u e ( ) ; i f ( r e s == N U L L ) { n u m W o r k e r T h r e a d s −−; } else { MPI_Request r ; MPI_Status s ;

30

M P I _ I s e n d ( res−>g e t R e s u l t A d d r e s s ( ) , res−>←֓ g e t R e s u l t S i z e ( ) , M a n d e l R e s u l t : : t y p e , 0 , ←֓ R E S U L T T A G , M P I _ C O M M _ W O R L D , &r ) ; // ←֓ return the r e s u l t s M P I _ W a i t (&r , &s ) ;

31

32 33

o u t q u e −>r e l e a s e ( r e s ) ; a s s i g n e d −>f e t c h A n d A d d O r d e r e d ( −1) ;

34 35

}

36

}

37

}

38 39

}

40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55

// ************************************************************ i n t main ( i n t argc , char * argv [ ] ) { ... } else // w o r k e r c o d e { ... // one l o o p f o r s e n d i n g and r e c v m e s s a g e s bool endOfWork = f a l s e ; int numWorkerThreads = numCores ; QAtomicInt assigned (0) ; // spawn t h e t h r e a d f o r s e n d i n g r e s u l t s back R e s u l t S e n d e r r s ( o u t q u e , n u m W o r k e r T h r e a d s , &a s s i g n e d ) ; rs . start ( ) ;

56 57 58 59 60 61

62 63 64 65 66 67

// j o b r e c e i v i n g p a r t while ( ! endOfWork ) { M a n d e l W o r k I t e m * w = i n q u e −>r e s e r v e ( ) ; M P I _ R e c v ( w , 1 , M a n d e l W o r k I t e m : : t y p e , 0 , M P I _ A N Y _ T A G , ←֓ M P I _ C O M M _ W O R L D , &s t a t u s ) ; // g e t a new work ←֓ item i n t tag = status . MPI_TAG ; i f ( t a g == E N D T A G ) { f o r ( i n t i = 0 ; i < n u m C o r e s ; i++) i n q u e −>e n q u e ( N U L L ) ; endOfWork = true ;

106

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING } else {

68 69 70

i n q u e −>e n q u e ( w ) ; assigned . fetchAndAddOrdered (1) ;

71 72

}

73

}

74 75

f o r ( i n t i = 0 ; i < n u m C o r e s ; i++) t h r [ i]−> w a i t ( ) ; rs . wait ( ) ;

76 77 78 79

...

21. Conway’s Game of Life is played on a rectangular grid of cells that may or may not contain an organism. The state of the cells is updated at each time step by applying the following set of rules:

Every organism with 2-3 neighbors survives.

Every organism with 4 or more neighbors dies from overpopulation.

Every organism with 0 - 1 neighbors dies from isolation.

Every empty cell adjacent to 3 organisms gives birth to a new one.

Create an MPI program that evolves a board of arbitrary size (dimensions could be specified at the command-line) over several iterations. The board could be randomly generated or read from a file.

Try applying the geometric decomposition pattern to partition the work among your processes. One example could be to evenly split the board row-wise. It is clear that each process can update its part of the board only by knowing the state of the bottom board row resident in the previous process, and the state of the top board row resident in the next process (the boundary processes being an exception).

Answer visit Library Genesis for more solution manuals: libgen.is, libgen.st, libgen.rs, forum.mhut.org The following solution uses geometric decomposition to distribute the board evenly among the MPI processes, in a row-wise fashion. At the end of each time step, the processes communicate their top and bottom rows with their neighbors, to allow proper calculation of neighbor cell counts. The arrangement is similar to the one used in Exercise 15, for the DLA problem. The difference with the depiction shown in Figure 5.2 is that only one instead of two rows is communicated per message.

1 2 3 4 5 6 7

// Game o f L i f e // G. B a r l a s , Dec . 2014 #i n c l u d e <s t d i o . h> #i n c l u d e < s t d l i b . h> #i n c l u d e < s t r i n g . h> #i n c l u d e <math . h> #i n c l u d e <mpi . h>

8 9

u s i n g namespace s t d ;

10 11 12 13

#d e f i n e PIC SIZE 100 #d e f i n e PARTICLES 500 #d e f i n e MAX ITER 10

14 15 16 17 18 19

#d e f i n e DUMPIMAGES / *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−* / i n t c o u n t N e i g h b o r s ( u n s i g n e d c h a r * pic , i n t c o l s , i n t x , { i n t count = − pic [ cols * y + x ] ;

int y)

107 f o r ( i n t i = −1; i < 2 ; i++) f o r ( i n t j = −1; j < 2 ; j++) c o u n t += p i c [ c o l s * ( y + j ) + ( x + i ) ] ;

20 21 22 23

return count ;

24 25

}

26 27 28 29

30 31 32 33 34

/ *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−* / / * I n i t i a l i z e s t h e 2D a r r a y f o r t h e s i m u l a t i o n * / v o i d b o a r d _ i n i t ( u n s i g n e d c h a r * pic , i n t r o w s , i n t c o l s , particles ) { int i , j , x , y ; f o r ( i = 0 ; i < r o w s ; i++) f o r ( j = 0 ; j < c o l s ; j++) pic [ i * cols + j ] = f a l s e ;

i n t ←֓

35 36

f o r ( i = 0 ; i < p a r t i c l e s ; i++) p a r t i c l e p l a c e m e n t */ { x = random ( ) % ( cols − 2) + 1 ; y = random ( ) % ( rows − 2) + 1 ; pic [ y * cols + x ] = true ; }

37

38 39 40 41 42 43

/* g e n e r a t e

i n i t i a l ←֓

// c o u n t i n g s t a r t s from 1

}

44 45 46 47 48

49 50 51

/ *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− * S i n g l e step e v o l u t i o n of the simulation . */ v o i d b o a r d _ e v o l v e ( u n s i g n e d c h a r * pic , u n s i g n e d c h a r * p i c 2 , rows , i n t cols ) { // p r e p a r e a r r a y t o h o l d new s t a t e memset ( pic2 , 0 , s i z e o f ( unsigned char ) * rows * cols ) ;

i n t ←֓

52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79

f o r ( i n t y = 1 ; y < r o w s − 1 ; y++) f o r ( i n t x = 1 ; x < c o l s − 1 ; x++) { i n t idx = y * cols + x ; i n t c = c o u n t N e i g h b o r s ( pic , c o l s , x , y ) ; i f ( pic [ idx ] ) // c e l l i n −p l a c e { switch ( c ) { // C a s e s a r e commented o u t b e c a u s e p i c 2 i s r e s e t a l r e a d y // case 0: // case 1: // d e a t h − l o n i l e s s // p i c 2 [ i d x ]= f a l s e ; // break ; case 2: case 3: // s u r v i v a l pic2 [ idx ] = true ; break ; default : // d e a t h − o v e r c r o w d e d // p i c 2 [ i d x ]= f a l s e ; break ; } } e l s e i f ( c == 3 ) // b i r t h i n empty c e l l pic2 [ idx ] = true ; } }

80 81 82

83 84 85

/ *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−* / v o i d s a v e I m a g e ( u n s i g n e d c h a r * pic , i n t r o w s , i n t c o l s , c h a r * ←֓ fname ) { F I L E * f = f o p e n ( f n a m e , ”w+t ” ) ; f p r i n t f ( f , ”P1\n%i %i \n” , c o l s , r o w s ) ;

86 87 88 89

f o r ( i n t y = 1 ; y <= r o w s ; y++) { f o r ( i n t x = 1 ; x <= c o l s ; x++)

108

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING {

90

92 93 94

} f p r i n t f ( f , ” \n” ) ;

95 96

} fclose ( f ) ;

97 98 99

( pic [ y * ( cols + 2) + x ] ) fprintf ( f , ”1 ” ) ; else fprintf ( f , ”0 ” ) ; if

91

}

100 101 102 103 104 105 106 107 108 109 110

/ *−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−* / i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t cols , rows , iter , particles , x , y ; u n s i g n e d c h a r * pic , * p i c 2 , * t m p ; i n t r a n k , num , i ; M P I _ I n i t (& a r g c , &a r g v ) ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &n u m ) ; M P I _ R e q u e s t req , r e q 2 , r e q 3 ;

111 112

if

( argc < 2) // u s e d e f a u l t v a l u e s s p e c i f y anything cols = PIC_SIZE ; rows = PIC_SIZE ; iter = MAX_ITER ; particles = PARTICLES ;

114 115 116 117

119 120

u s e r d o e s n o t ←֓

{

113

118

if

} else { cols = atoi rows = atoi particles = iter = atoi

121 122 123 124

( argv [ 1 ] ) ; ( argv [ 2 ] ) ; atoi ( argv [ 3 ] ) ; ( argv [ 4 ] ) ;

}

125 126 127

128 129 130

// t o make c o d e s i m p l e r and a v o i d s p e c i a l c a s e s , rows i s t o be a m u l t i p l e o f // t h e number o f p r o c e s s e s rows = ( i n t ) ceil ( rows * 1.0 / num ) * num ; i n t rowsPerNode = rows / num ;

f o r c e d ←֓

131 132

s r a n d ( t i m e ( 0 ) ) ; // i n i t i a l i z e

t h e random number g e n e r a t o r

133 134

135

136 137 138 139 140 141 142 143 144 145 146

// g r i d h as two e x t r a rows , one above and one below t h e a c t u a l ←֓ rows // The e x t r a rows ” b e l o n g ” t o o t h e r nodes , s o a t t h e end o f ←֓ each time s t e p // t h e y s h o u l d be e xc h an ge d i f ( r a n k == 0 ) { p i c = new u n s i g n e d c h a r [ ( c o l s + 2 ) * ( r o w s + 2 ) ] ; p i c 2 = new u n s i g n e d c h a r [ ( c o l s + 2 ) * ( r o w s + 2 ) ] ; } else { p i c = new u n s i g n e d c h a r [ ( c o l s + 2 ) * ( r o w s P e r N o d e + 2 ) ] ; p i c 2 = new u n s i g n e d c h a r [ ( c o l s + 2 ) * ( r o w s P e r N o d e + 2 ) ] ; }

147 148 149 150 151

if

( r a n k == 0 ) b o a r d _ i n i t ( pic2 , rows + 2 , cols + 2 , p a r t i c l e s ) ; // c l e a n p i c b u f f e r m e m s e t ( pic , 0 , ( c o l s + 2 ) * ( r o w s P e r N o d e + 2 ) * s i z e o f ( ←֓ unsigned char ) ) ;

152 153 154

// boundary rows do n o t need t o be communicated now .

M P I _ S c a t t e r ( p i c 2 + c o l s + 2 , ( c o l s + 2 ) * r o w s P e r N o d e , ←֓ M P I _ U N S I G N E D _ C H A R , p i c + c o l s + 2 , ( c o l s + 2 ) * ←֓ rowsPerNode , MPI_UNSIGNED_CHAR , 0 , MPI_COMM_WORLD ) ;

155 156

i n t nextID = rank + 1 , prevID = rank − 1;

109 157 158 159 160 161

162 163 164 165 166 167 168

i n t step = 0; w h i l e (++s t e p < i t e r ) { #i f d e f DUMPIMAGES M P I _ G a t h e r ( p i c + c o l s + 2 , r o w s P e r N o d e * ( c o l s + 2 ) , ←֓ M P I _ U N S I G N E D _ C H A R , p i c + c o l s + 2 , r o w s P e r N o d e * ( c o l s ←֓ + 2) , MPI_UNSIGNED_CHAR , 0 , MPI_COMM_WORLD ) ; i f ( r a n k == 0 ) { char fname [ 2 0 ] ; s p r i n t f ( f n a m e , ”%05 i . pbm” , s t e p ) ; s a v e I m a g e ( pic , r o w s , c o l s , f n a m e ) ; } #e n d i f

169

b o a r d _ e v o l v e ( pic , p i c 2 , r o w s P e r N o d e + 2 , c o l s + 2 ) ; tmp = pic2 ; pic2 = pic ; pic = tmp ; // e x c h a n g e i n f o r m a t i o n w i t h ” p r e v i o u s ” node i f ( r a n k != 0 ) { M P I _ I s e n d ( p i c + c o l s + 2 , c o l s + 2 , M P I _ U N S I G N E D _ C H A R , ←֓ p r e v I D , s t e p , M P I _ C O M M _ W O R L D , &r e q ) ; M P I _ I r e c v ( pic , c o l s + 2 , M P I _ U N S I G N E D _ C H A R , p r e v I D , ←֓ s t e p , M P I _ C O M M _ W O R L D , &r e q 2 ) ; }

170 171 172 173 174 175 176 177

178

179 180

// e x c h a n g e w i t h ” n e x t ” i f ( r a n k != n u m − 1 ) { M P I _ I s e n d ( p i c + r o w s P e r N o d e * ( c o l s + 2 ) , c o l s + 2 , ←֓ M P I _ U N S I G N E D _ C H A R , n e x t I D , s t e p , M P I _ C O M M _ W O R L D , &←֓ req ) ; M P I _ I r e c v ( p i c + ( r o w s P e r N o d e + 1 ) * ( c o l s + 2 ) , c o l s +←֓ 2 , M P I _ U N S I G N E D _ C H A R , n e x t I D , s t e p , ←֓ M P I _ C O M M _ W O R L D , &r e q 3 ) ; }

181 182 183 184

185

186 187

// b l o c k u n t i l m e s s a g e s a r e r e c e i v e d i f ( r a n k != 0 ) M P I _ W a i t (& r e q 2 , M P I _ S T A T U S _ I G N O R E ) ; i f ( r a n k != n u m − 1 ) M P I _ W a i t (& r e q 3 , M P I _ S T A T U S _ I G N O R E ) ;

188 189 190 191 192

}

193 194

M P I _ G a t h e r ( p i c + c o l s + 2 , r o w s P e r N o d e * ( c o l s + 2 ) , ←֓ M P I _ U N S I G N E D _ C H A R , p i c + c o l s + 2 , r o w s P e r N o d e * ( c o l s + ←֓ 2) , MPI_UNSIGNED_CHAR , 0 , MPI_COMM_WORLD ) ; i f ( r a n k == 0 ) s a v e I m a g e ( pic , r o w s , c o l s , ” o u t . pbm” ) ;

195

196 197 198

MPI_Finalize () ; d e l e t e [ ] pic ; d e l e t e [ ] pic2 ; return 0;

199 200 201 202 203

}

22. Radix sort is a linear complexity non-comparison-based sorting algorithm that is suitable for concurrent execution. Radix sort sorts data by separating them into groups based on their digits (for integers) or characters (for strings). The data must be of fixed range, i.e. the number of bits or characters used must be known a priori.

Radix sort comes in two forms: Least-Significant Digit radix sort (LSD) or Most-Significant Digit radix sort (MSD). The latter is suitable for parallel execution, as data which are partitioned in groups can be operated independently in subsequent phases of the algorithm. The MSD algorithm,

110

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING which is very close to bucket-sort 2 , can be implemented recursively as shown in the following pseudocode, for binary data. An extension for data with non-binary digits (strings) is straightforward. The use of the auxiliary array B allows the sorting to be stable.

1 2 3 4 5 6 7

// I n p u t : a r r a y A, w i t h N e l e m e n t s , e a c h D b i t s l o n g . // Output : A h o l d s s o r t e d d a t a radix_sort (A , N ) a l l o c a t e memory B equal in size to A t m p <− r a d i x _ a u x ( A , N , B , D −1) i f t m p <> A // i f s o r t e d d a t a end up i n B a r r a y copy B to A

8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

// A u x i l i a r y r e c u r s i v e f u n c t i o n // Use o f te mp orary a r r a y B f o r a s t a b l e s o r t // R e t u r n s l o c a t i o n o f s o r t e d d a t a radix_aux ( A , N , B , k ) i f k = −1 O r N<2 // b a s e c a s e f o r t e r m i n a t i o n return A else l e t r b e t h e n u m b e r o f i t e m s i n A , w i t h k−t h b i t s e t t o 0 r e s e t I d x <− 0 s e t I d x <− r f o r i <− 0 t o N−1 // s e p a r a t e d a t a i n two b i n s i f k−t h b i t o f A [ i ] i s s e t store A [ i ] in B [ setIdx ] s e t I d x <− s e t I d x + 1 else store A [ i ] in B [ r e s e t I d x ] r e s e t I d x <− r e s e t I d x + 1

26 27

28

29 30 31 32 33 34 35 36 37

t m p 1 <− r a d i x _ a u x ( B , r , A , k −1) // s o r t b i n w i t h a 0 ←֓ k−th b i t , u s i n g t h e ( k−1)−th b i t t m p 2 <− r a d i x _ a u x ( B+r , N−r , A+r , k −1) // s o r t b i n w i t h a 1 ←֓ k−th b i t , u s i n g t h e ( k−1)−th b i t i f t m p 1 + r <> t m p 2 // p o i n t e r c o m p a r i s o n if r > N − r // make t h e s m a l l e s t copy c o p y N−r e l e m e n t s f r o m t m p 2 t o t m p 1+r return tmp1 else c o p y r e l e m e n t s f r o m t m p 1 t o t m p 2−r r e t u r n t m p 2−r else return tmp1

Use divide & conquer decomposition as your decomposition pattern (see Section 2.3.2) to design and implement an MPI radix sort.

Answer The algorithm design is influenced by the initial distribution of data, in the same way that bucketsort does. For example, it could be possible that all the nodes get a portion of the initial data, sort it according to one of the bits and exchange the resulting two parts. In this solution we consider the alternative where the data are originally resident at the 0ranked node, which also performs the first sorting phase according to the most significant bit of the data values.

The following listing implements the divide & conquer decomposition pattern, whereas the data after each pass/phase of the algorithm that splits them into two groups, are split between two nodes. The overall communication pattern resembles a scattering operation. Node 0 first splits the 2 The major difference between radix sort and bucket sort is in how the keys are examined: piece-wise in radix sort and as a whole in bucket sort. In the latter, an arbitrary number of buckets can be prescribed.

111 input into two pieces according to the value of the most significant bit, keeps one part for itself and sends the other part to node 1. During the second pass/phase, node 0 splits the remaining data and sends one of the parts to node 2, while node 1 performs the same operation for its part of the data and sends one of the parts to node 3. The process is repeated as many times as the number of bits used by the data values, but the migration of data to other nodes is limited by the number of MPI processes (check of line 120).

The key points of the following listing are:

The data are initialized in node 0, and are sorted by having all the nodes call the radixsort function in line 195. The radixsort function sets-up the auxiliary memory needed for a stable sort (line 171) and calls the recursive function radixSort aux which performs the actual sorting.

In radixSort aux each node determines the phase at which it will join the sorting operation (line 84), as well as the node from which it will receive its assignment (line 108), and the nodes to which it will subsequently assign parts of its workload (line 118).

As with the quicksort algorithm, the split of parts is not even. In the event that a node does not have data to operate on (base case M<=0 of line 88), it terminates, but not before informing all its sub-ordinate nodes that they should terminate as well (loop of lines 94 to 99).

The radixSort aux behaves differently during its first invocation, which is flagged by setting the phase parameter to -1. Namely, data are received from another node (line 109), and sent back to it following the completion of the “local” sort operation (line 153).

The auxiliary array pointed to by aux is used to hold the data during the split operation carried out by function radixPhaseK. Data are shifted between the two memory locations (data and aux) during the different phases, with radixSort aux maintaining a pointer (tmp1) to the actual location. The data should eventually return to their original location. If for any reason (e.g. an odd number of bits is used in the sorting) this is not done, the check in line 174 restores them.

The radixSort aux function also makes sure that the two data portions produced by the application of radixPhaseK in line 117, end up in the same location after they are independently sorted (lines 120-128 for the “left” part and line 131 for the “right” part). The necessary move is minimized by the logic of lines 134-148.

1 2 3 4 5 6 7

#i n c l u d e <mpi . h> #i n c l u d e <s t r i n g . h> #i n c l u d e <s t d i o . h> #i n c l u d e <s t d l i b . h> #i n c l u d e <i o s t r e a m > #i n c l u d e <math . h> #i n c l u d e <u n i s t d . h>

8 9

u s i n g namespace s t d ;

10 11

#d e f i n e ASSIGNTAG 0

112 12

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING #d e f i n e SORTEDRESTAG 1

13 14 15

const i n t MINV = 0; const i n t MAXV = 10000;

16 17 18 19 20 21 22 23 24 25 26 27 28

// ***************************************** // f i r s t b i t i s a t p o s i t i o n 1 i n t msb ( i n t i ) { i n t res = 0 ; while ( i ) { i >>= 1 ; r e s ++; } return res ; }

29 30 31 32 33 34 35 36 37 38 39 40 41 42

43 44 45 46 47 48 49 50 51

// ***************************************** // A u x i l i a r y f u n c t i o n t h a t p e r f o r m s a p h a s e // o f a r a d i x s o r t a l g o r i t h m , b a s e d on t h e // k−th b i t . Temporary a r r a y B i s u s e d f o r // a s t a b l e s o r t . // R e t u r n s l o c a t i o n t h e l o c a t i o n where t h e // s o r t e d d a t a ” s p l i t a p a r t ” . So B [ r ] i s t h e // f i r s t e l e m e n t w i t h a 1− b i t a t p o s i t i o n k i n t r a d i x P h a s e K ( i n t *A , i n t N , i n t *B , i n t k ) { i f ( ( k == −1) | | ( N < 2 ) ) // b a s e c a s e f o r t e r m i n a t i o n { m e m c p y ( B , A , N * s i z e o f ( i n t ) ) ; // d a t a a r e e x p e c t e d t o ←֓ move t o B return 0; } else { int r = 0; i n t b i t M a s k = ( 1 << k ) ; f o r ( i n t i = 0 ; i < N ; i++) i f ( ( A [ i ] & b i t M a s k ) == 0 ) r++;

52

int resetIdx = 0 , setIdx = r ; f o r ( i n t i = 0 ; i < N ; i++) { i f ( ( A [ i ] & bitMask ) > 0) { B [ setIdx ] = A [ i ] ; s e t I d x ++; } else { B [ resetIdx ] = A [ i ] ; r e s e t I d x ++; } } return r ;

53 54 55 56 57 58 59 60 61 62 63 64 65 66 67

}

68 69

}

70 71 72 73 74 75 76 77

78 79 80 81 82

// ***************************************** // R e c u r s i v e r a d i s o r t f u n c t i o n . Based on t h e // phase , d a t a maybe s e n t t o o t h e r n o d e s f o r // t h e s u b s e q u e n t p h a s e s . // t o t a l B i t s c o u n t s t h e b i t s l e f t f o r c o m p a r i s o n // p h a s e==−1 f l a g s t h e f i r s t c a l l t o t h i s f u n c t i o n i n t * r a d i x S o r t _ a u x ( i n t * d a t a , i n t M , i n t * aux , i n t t o t a l B i t s , ←֓ i n t r a n k , i n t num , i n t p h a s e = −1) { int newPhase ; int sourceID ; i n t * tmp1 , * tmp2 ; i f ( p h a s e == −1) // i f f i r s t c a l l t o r e c u r s i v e ←֓ function

113 {

83

newPhase = msb ( rank ) ; t o t a l B i t s −= n e w P h a s e ; // c o r r e s p o n d s t o when t h e node ←֓ w i l l j o i n the s o r t i n g

84 85

}

86 87 88

if

( t o t a l B i t s <= 0 | | M <= 0 ) // b a s e c a s e s , no more s o r t i n g t o ←֓ be done {

89

// have t o t e r m i n a t e any n o d e s w a i t i n g f o r d a t a i f ( p h a s e != −1) { i n t p a r t n e r I D = r a n k ˆ ( 1 << p h a s e ) ; while ( partnerID < num ) { M P I _ S e n d ( N U L L , 0 , M P I _ I N T , p a r t n e r I D , A S S I G N T A G , ←֓ MPI_COMM_WORLD ) ; p h a s e ++; p a r t n e r I D = r a n k ˆ ( 1 << p h a s e ) ; } } return data ;

90 91 92 93 94 95 96

97 98 99 100 101

}

102 103 104 105 106 107 108 109

110

111

// f i r s t g e t d a t a a s s i g n m e n t M P I _ S t a t u s st ; i f ( p h a s e == −1 && r a n k != 0 ) { s o u r c e I D = r a n k ˆ ( 1 << ( n e w P h a s e − 1 ) ) ; M P I _ R e c v ( d a t a , M , M P I _ I N T , s o u r c e I D , A S S I G N T A G , ←֓ M P I _ C O M M _ W O R L D , &s t ) ; M P I _ G e t _ c o u n t (&st , M P I _ I N T , &M ) ; // g e t how many where ←֓ actually received }

112 113 114

if

( p h a s e != −1) newPhase = phase ;

115 116 117 118 119 120 121 122

123 124 125 126 127

128

// now s p l i t t h e d a t a i n two g r o u p s b a s e d on a b i t v a l u e i n t s p l i t = r a d i x P h a s e K ( d a t a , M , aux , t o t a l B i t s − 1 ) ; i n t p a r t n e r I D = r a n k ˆ ( 1 << n e w P h a s e ) ; // e i t h e r s e n d one p a r t t o a n o t h e r node i f ( partnerID < num ) { M P I _ S e n d ( aux , s p l i t , M P I _ I N T , p a r t n e r I D , A S S I G N T A G , ←֓ MPI_COMM_WORLD ) ; } // o r s o r t i t l o c a l l y else { t m p 1 = r a d i x S o r t _ a u x ( aux , s p l i t , d a t a , t o t a l B i t s − 1 , r a n k ←֓ , num , n e w P h a s e + 1 ) ; }

129 130 131

t m p 2 = r a d i x S o r t _ a u x ( a u x + s p l i t , M − s p l i t , d a t a + s p l i t , ←֓ t o t a l B i t s − 1 , r a n k , num , n e w P h a s e + 1 ) ;

132 133 134

135 136 137 138 139 140 141

142 143 144 145 146

// make s u r e a l l d a t a end−up a t t h e same p l a c e , p o i n t e d by tmp1 i f ( partnerID < num ) // no need t o move d a t a i f t h e o t h e r ←֓ p a r t h as been s e n t o f f tmp1 = tmp2 − split ; e l s e i f ( t m p 2 != t m p 1 + s p l i t ) { // m i n i m i z e movement by moving t h e s m a l l e r p a r t i f ( split > M / 2) { m e m c p y ( t m p 1 + s p l i t , t m p 2 , ( M − s p l i t ) * s i z e o f ( i n t ) ) ←֓ ; } else { memcpy ( tmp2 − split , tmp1 , split * s i z e o f ( i n t ) ) ; tmp1 = tmp2 ;

114

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING }

147

}

148 149

// c o l l e c t i f d a t a were s e n t away i f ( partnerID < num ) { M P I _ R e c v ( t m p 1 , s p l i t , M P I _ I N T , p a r t n e r I D , S O R T E D R E S T A G , ←֓ M P I _ C O M M _ W O R L D , &s t ) ; }

150 151 152 153

154 155

// b e f o r e r e c u r s i o n ends , s e n d back t o s o u r c e i f ( p h a s e == −1 && r a n k != 0 ) M P I _ S e n d ( t m p 1 , M , M P I _ I N T , s o u r c e I D , S O R T E D R E S T A G , ←֓ MPI_COMM_WORLD ) ;

156 157 158

159

return tmp1 ;

160 161

}

162 163 164 165 166 167 168

// ***************************************** void r a d i x S o r t ( i n t * data , i n t M , i n t t o t a l B i t s ) { i n t rank , num ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &n u m ) ;

169

i n t myPhase = msb ( rank ) ; i n t * a u x = new i n t [ M ] ;

170 171 172

i n t * t m p = r a d i x S o r t _ a u x ( d a t a , M , aux , t o t a l B i t s , r a n k , n u m ) ; i f ( t m p != d a t a ) m e m c p y ( d a t a , tmp , M * s i z e o f ( i n t ) ) ; d e l e t e [ ] aux ;

173 174 175 176 177

}

178 179 180

// *****************************************

181 182 183 184

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { M P I _ I n i t (& a r g c , &a r g v ) ;

185 186 187 188 189 190 191

i n t r a n k , num , t o t a l B i t s ; M P I _ C o m m _ r a n k ( M P I _ C O M M _ W O R L D , &r a n k ) ; M P I _ C o m m _ s i z e ( M P I _ C O M M _ W O R L D , &n u m ) ; MPI_Status status ; i n t M = atoi ( argv [ 1 ] ) ; // g e t s i z e o f d a t a t o s o r t i n t * data ; // d a t a b u f f e r

192 193

d a t a = new i n t [ M ] ;

194 195

totalBits = ceil ( log2 ( MAXV ) ) ; for radix sort

// t o t a l number o f p h a s e s ←֓

196 197 198 199 200 201 202 203 204 205

// i n i t i a l i z e d a t a a r r a y i n rank 0 i f ( r a n k == 0 ) { srand ( time (0) ) ; f o r ( i n t i = 0 ; i < M ; i++) { data [ i ] = ( rand () % ( MAXV − MINV ) ) + MINV ; } }

206 207

r a d i x S o r t ( data , M , t o t a l B i t s ) ;

208 209 210 211 212 213 214 215

// c o r r e c t n e s s c h e c k i f ( r a n k == 0 ) { f o r ( i n t i = 1 ; i < M ; i++) i f ( data [ i ] < data [ i − 1 ] ) c o u t << ”ERROR\n” ; }

216 217

MPI_Finalize () ;

115 d e l e t e [ ] data ; return 0;

218 219 220

}

116

CHAPTER 5. DISTRIBUTED MEMORY PROGRAMMING

Chapter 6

GPU Programming Exercises 1. An array of type float elements is to be processed in a one-elementper-thread fashion by a GPU. Suggest an execution configuration for the following scenarios: (a) The array is 1-D and of size N . The target GPU has 8 SMs, each with 16 SPs. (b) The array is 2-D and of size N xN . The target GPU has 5 SMs, each with 48 SPs. For each of the above scenario, calculate what is the minimum size that N should satisfy to make the GPU computation a desirable alternative to CPU computation.

Answer The number of data items should be high enough to provide work for all SPs. (a) A total of 8·16 = 128 SPs require at least N ≥ 128. Ideally, N should be a multiple of 128. An execution configuration can be obtained with the following code: i n t b l o c k = 16 * 4 ; // 64 t h r e a d s , a m u l t i p l e o f 16 i n t grid = ( ( N − 1 ) / block ) + 1; foo<<< g r i d , b l o c k > > >(...) ;

(b) A total of 5 · 48 = 240 SPs require at least N 2 ≥ 240 ⇒ N ≥ 15.5.

Ideally, N 2 should be a multiple of 240. An execution configuration can be obtained with the following code: d i m 3 b l o c k ( 1 6 , 1 2 ) ; // 192 t h r e a d s , a m u l t i p l e o f 48 d i m 3 g r i d ( ( N − 1 ) / 16 + 1 , ( N − 1 ) / 12 + 1 ) ; foo<<< g r i d , b l o c k > > >(...) ;

2. A reduction is an operation frequently encountered in a many algorithms: summing-up the elements of an array, finding the minimum, maximum, etc. One possible solution to a CUDA kernel that calculates the sum of an array would be: 117

118

CHAPTER 6. GPU PROGRAMMING _ _ g l o b a l _ _ v o i d s u m ( f l o a t * in , f l o a t * o u t ) { __shared__ f l o a t localStore [ ] ; // t o s p e e d u p d a t a a c c e s s int globalID = threadIdx . x + blockIdx . x * blockDim . x ; int localID = threadIdx . x ; l o c a l S t o r e [ l o c a l I D ] = in [ g l o b a l I D ] ; // copy t o s h a r e d memory f o r ( i n t i =1; i< b l o c k D i m . x ; i * =2) { i f ( l o c a l I D % ( 2 * i ) == 0 ) l o c a l S t o r e [ l o c a l I D ] += l o c a l S t o r e [ l o c a l I D + i ] ; __syncthreads () ; } i f ( l o c a l I D == 0 ) out [ blockIdx . x ] = localStore [ 0 ] ; }

The above needs to be called multiple times, each time reducing the size of the data by the number of threads in a block. Data at each kernel launch should be multiples of the block size.

Analyze the above code in relation to the following criteria and how they reflect on the execution speed: thread divergence, memory coalescing, use of SPs within a warp. Suggest modifications that would improve the performance.

Answer The shown kernel scores low in all the mentioned areas, i.e. thread divergence, memory coalescing, and use of SPs within a warp. Analytically, the issues can be identified as follows:

Thread divergence : The if condition inside the for-loop causes threads to diverge. During the first iteration only the threads with an ID which is a multiple of 2 can execute the if code body. During the second iteration only the ones with an ID which is a multiple of 4 can do this, and so on.

Memory coalescing : There is no regard to memory coalescing inside the for loop. In fact, as the loop progresses, the number of bank conflicts increases.

SP utilization : As soon as the for loop starts executing, an exponentially growing number of SPs with each iteration, become inactive, as they do not satisfy the if condition.

The following program can produce the same result with only one kernel invocation, with maximum SP utilization and minimum thread divergence: 1 2 3

#i n c l u d e <s t d i o . h> #i n c l u d e < s t d l i b . h> #i n c l u d e <cuda . h>

4 5

#d e f i n e MAXVALUE 10000

6 7 8 9 10 11 12 13

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− v o i d n u m b e r G e n ( i n t N , i n t max , i n t * s t o r e ) { int i ; srand ( time (0) ) ; f o r ( i = 0 ; i < N ; i++) rand ( ) % max ;

119 14

}

15 16 17 18 19

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− __global__ void sum ( i n t *d , i n t N , i n t * odds ) { extern __shared__ int count [ ] ;

20

i n t myID = blockIdx . x * blockDim . x + threadIdx . x ; int numThr = blockDim . x * gridDim . x ; int localID = threadIdx . x ; count [ localID ] = 0; i f ( myID < N ) { count [ localID ] = d [ myID ] ; i n t idx = myID + numThr ; // p a r t i a l sum while ( idx < N ) { c o u n t [ l o c a l I D ] += d [ i d x ] ; i d x += n u m T h r ; }

21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

__syncthreads () ; // b l o c k −wide p a r t i a l sum c a l c u l a t i o n i n t step = 1; w h i l e ( ( ( l o c a l I D | s t e p ) < b l o c k D i m . x ) && ( ( l o c a l I D & s t e p ) ←֓ == 0 ) ) { c o u n t [ l o c a l I D ] += c o u n t [ l o c a l I D | s t e p ] ; s t e p *= 2 ; __syncthreads () ; }

36 37 38 39

40 41 42 43 44 45

// now t h e b l o c k −wide p a r t i a l sum i s // add t o g l o b a l c o u n t e r i f ( l o c a l I D == 0 ) a t o m i c A d d ( odds , count [ 0 ] ) ;

46 47 48 49

}

50 51

i n count [ 0 ]

}

52 53 54 55 56

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t N = atoi ( argv [ 1 ] ) ;

57 58

i n t * ha , * h r e s , * da , * d r e s ; pointers

// h o s t ( h * ) and d e v i c e ( d * ) ←֓

59 60 61

h a = new i n t [ N ] ; h r e s = new i n t [ 1 ] ;

62 63 64

c u d a M a l l o c ( ( v o i d * * ) &da , s i z e o f ( i n t ) * N ) ; c u d a M a l l o c ( ( v o i d * * ) &d r e s , s i z e o f ( i n t ) * 1 ) ;

65 66

n u m b e r G e n ( N , MAXVALUE , ha ) ;

67 68 69

c u d a M e m c p y ( da , ha , s i z e o f ( i n t ) * N , c u d a M e m s e t ( dres , 0 , s i z e o f ( i n t ) ) ;

cudaMemcpyHostToDevice ) ;

70 71 72 73 74

i n t blockSize , gridSize ; blockSize = 256; gridSize = 16; s u m <<< g r i d S i z e , b l o c k S i z e , N , dres ) ;

b l o c k S i z e * s i z e o f ( i n t ) >>> ( da , ←֓

75 76

c u d a M e m c p y ( hres , dres ,

s i z e o f ( int ) , cudaMemcpyDeviceToHost ) ;

77 78

p r i n t f ( ”%i \n” , * h r e s ) ;

79 80 81 82 83 84

c u d a F r e e ( ( void *) da ) ; cudaFree ( ( void *) dres ) ; d e l e t e [ ] ha ; d e l e t e [ ] hres ; cudaDeviceReset () ;

120

CHAPTER 6. GPU PROGRAMMING

85

return 0;

86 87

}

The kernel of lines 17-51 allocates shared memory in the form of one integer per block thread. This memory is dynamically allocated during the kernel invocation (line 74). Each thread initially spends its time calculating a partial sum by going over the input data with a stride equal to the total number of threads (while loop of lines 30-34). Subsequently, a block-wide partial sum is calculated by reducing the shared memory counters with the loop of lines 39-44. The resulting sum in count[0] is added to the global memory counter with an atomic operation.

3. The reduction operation discussed in the previous exercise, is a special case of the “scan” or prefix-sum operation that can be applied to the elements of a vector or list. In general, the operator applied can be any of summation, subtraction, minimum, maximum, etc.. Implement a CUDA kernel capable of performing a prefix-sum operation. The “Prefix-Sums and Their Applications” paper by Guy Blelloch, available at http:// www.cs.cmu.edu/~guyb/papers/Ble93.pdf, is a wonderful resource for learning more on the topic.

Answer The solution is an extension of the one given in the previous exercise.

The only differences lie in how the initial partial result for each thread is initialized (lines 34-45), in the operator used to perform the reduction (lines 56-67 and 78-89), and in the atomic operation used to consolidate the block-wide partial results with the global memory result (lines 98-109).

This program can be easily extended to handle any prefix-sum operation by introducing another symbolic constant representing it (following the ones defined in lines 8-10), and appropriately augmenting the switch statements.

1 2 3 4

#i n c l u d e <s t d i o . h> #i n c l u d e < s t d l i b . h> #i n c l u d e < l i m i t s . h> #i n c l u d e <cuda . h>

5 6

#d e f i n e MAXVALUE 10000

7 8 9 10

#d e f i n e SUM OP 0 #d e f i n e MIN OP 1 #d e f i n e MAX OP 2

11 12 13 14 15 16 17 18 19 20 21

#d e f i n e min (A, B) ( ( A>B) ? B : A) #d e f i n e max (A, B) ( ( A<B) ? B : A) //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− v o i d n u m b e r G e n ( i n t N , i n t max , i n t * s t o r e ) { int i ; srand ( time (0) ) ; f o r ( i = 0 ; i < N ; i++) rand ( ) % max ; }

22 23

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

24 25 26 27 28

__global__ void reduce ( i n t *d , i n t N , { extern __shared__ int partRes [ ] ;

i n t * res ,

i n t OP )

121 29 30 31

i n t myID = blockIdx . x * blockDim . x + threadIdx . x ; int numThr = blockDim . x * gridDim . x ; int localID = threadIdx . x ;

32 33 34 35 36 37 38 39 40 41 42 43 44 45

// i n i t i a l i z e a l l t h e t h r e a d s ’ s t o r a g e switch ( OP ) { case SUM_OP : partRes [ localID ] = 0; break ; case MIN_OP : partRes [ localID ] = INT_MAX ; break ; case MAX_OP : partRes [ localID ] = INT_MIN ; break ; };

regardless

46 47 48 49 50

if

( myID < N ) { partRes [ localID ] = d [ myID ] ; i n t idx = myID + numThr ;

51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69

// p a r t i a l r e s u l t while ( idx < N ) { i n t tmp = d [ idx ] ; switch ( OP ) { case SUM_OP : p a r t R e s [ l o c a l I D ] += t m p ; break ; case MIN_OP : partRes [ localID ] = min ( partRes [ localID ] , tmp ) ; break ; case MAX_OP : partRes [ localID ] = max ( partRes [ localID ] , tmp ) ; break ; }; i d x += n u m T h r ; }

70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93

__syncthreads () ; // b l o c k −wide p a r t i a l r e s u l t c a l c u l a t i o n i n t step = 1; i n t otherIdx = localID | step ; w h i l e ( ( o t h e r I d x < b l o c k D i m . x ) && ( ( l o c a l I D & s t e p ) == 0 ) ) { i n t tmp = partRes [ otherIdx ] ; switch ( OP ) { case SUM_OP : p a r t R e s [ l o c a l I D ] += t m p ; break ; case MIN_OP : partRes [ localID ] = min ( partRes [ localID ] , tmp ) ; break ; case MAX_OP : partRes [ localID ] = max ( partRes [ localID ] , tmp ) ; break ; }; s t e p <<= 1 ; otherIdx = localID | step ; __syncthreads () ; }

94 95 96 97 98 99 100 101 102

// now t h e b l o c k −wide p a r t i a l sum i s // add t o g l o b a l c o u n t e r i f ( l o c a l I D == 0 ) switch ( OP ) { case SUM_OP : a t o m i c A d d ( res , p a r t R e s [ 0 ] ) ; break ;

in partRes [ 0 ]

122

CHAPTER 6. GPU PROGRAMMING case MIN_OP : a t o m i c M i n ( res , p a r t R e s [ 0 ] ) ; break ; case MAX_OP : a t o m i c M a x ( res , p a r t R e s [ 0 ] ) ; break ; };

103 104 105 106 107 108 109

}

110 111

}

112 113 114 115 116 117

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− int sharedSize ( int b ) { return b * s i z e o f ( int ) ; }

118 119

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

120 121 122 123

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t N = atoi ( argv [ 1 ] ) ;

124

i n t * ha , * h r e s , * da , * d r e s ; pointers

125

// h o s t ( h * ) and d e v i c e ( d * ) ←֓

126

h a = new i n t [ N ] ; h r e s = new i n t [ 1 ] ;

127 128 129

c u d a M a l l o c ( ( v o i d * * ) &da , s i z e o f ( i n t ) * N ) ; c u d a M a l l o c ( ( v o i d * * ) &d r e s , s i z e o f ( i n t ) * 1 ) ;

130 131 132

n u m b e r G e n ( N , MAXVALUE , ha ) ;

133 134

c u d a M e m c p y ( da , ha , s i z e o f ( i n t ) * N , c u d a M e m s e t ( dres , 0 , s i z e o f ( i n t ) ) ;

135 136

cudaMemcpyHostToDevice ) ;

137

i n t blockSize , gridSize ; blockSize = 256; gridSize = 16; r e d u c e <<< g r i d S i z e , b l o c k S i z e , da , N , d r e s , M A X _ O P ) ;

138 139 140 141

b l o c k S i z e * s i z e o f ( i n t ) >>> ( ←֓

142

c u d a M e m c p y ( hres , dres ,

143

s i z e o f ( int ) , cudaMemcpyDeviceToHost ) ;

144

p r i n t f ( ”%i \n” , * h r e s ) ;

145 146

c u d a F r e e ( ( void *) da ) ; cudaFree ( ( void *) dres ) ; d e l e t e [ ] ha ; d e l e t e [ ] hres ; cudaDeviceReset () ;

147 148 149 150 151 152

return 0;

153 154

}

The code shown here is not nearly as optimum as it could be; it is a tradeoff between readability and performance. A more thorough derivation of an optimum CUDA reduction kernel is covered in http://developer. download.nvidia.com/assets/cuda/files/reduction.pdf, although the material in question targets a Compute Capability 1.0 device, which exhibits many more quirks than contemporary architectures.

4. Create CUDA implementations of the gsl stats mean() and gsl stats variance() functions offered by the GNU Scientific Library, that produce the mean and variance statistics of an array of type double data. Their signatures are: d o u b l e g s l _ s t a t s _ m e a n ( c o n s t d o u b l e D A T A [ ] , // P o i n t e r t o i n p u t ←֓ data

123 size_t STRIDE , // S t e p u s e d t o r e a d ←֓ t h e i n p u t . Normally t h i s s h o u l d be ←֓ s e t to 1 . size_t N ) ; // S i z e o f DATA a r r a y d o u b l e g s l _ s t a t s _ v a r i a n c e ( c o n s t d o u b l e D A T A [ ] , // Same a s above . size_t STRIDE , size_t N ) ;

Assuming that the STRIDE is 1, create a memory access pattern that utilizes coalescing. Suggest ways to deal with the problem if the STRIDE is not 1.

Answer The gsl stats mean and gsl stats variance functions are implemented as host front-ends, to allow for the hiding of the implementation details (such as memory management and grid/block design) from the application that calls them.

The solution is based on the reduction approach that is used in the two previous exercises. Two kernels are employed for calculating the sum of values and the sum of the square of the values respectively, in order for the variance to be calculated from the formula: PN −1 2 PN −1 !2 i=0 vi i=0 vi var = − (6.1) N N where N is the number of values.

In order to deal with a STRIDE parameter which is different from 1, the gsl stats mean and gsl stats variance front-end functions, copy the data to be actually processed in another array (lines 127-130 and 169172). The benefits are two-fold:

A considerably smaller volume of data cross the PCIe bus, significantly reducing the communication cost.

The data items reside in contiguous memory locations allowing for memory coalescing to take place, at least to the extent allowed by the double data type.

The last major point of the following listing is the atomicAdd device function of lines 22-34, that is an operation for double-type operands that is missing from the arsenal of CUDA primitives.

1 2 3

#i n c l u d e <s t d i o . h> #i n c l u d e < s t d l i b . h> #i n c l u d e <cuda . h>

4 5 6 7 8 9 10 11 12 13 14

#d e f i n e MAXVALUE 10000 #d e f i n e min (A, B) ( ( A>B) ? B : A) //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− v o i d n u m b e r G e n ( i n t N , i n t max , d o u b l e * s t o r e ) { int i ; srand ( time (0) ) ; f o r ( i = 0 ; i < N ; i++) store [ i ] = i ; // rand ( ) % max ; }

15 16 17

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− int sharedSize ( int b )

124 18 19 20 21 22 23 24 25

26 27 28 29 30 31 32 33 34 35

CHAPTER 6. GPU PROGRAMMING {

return b * s i z e o f ( double ) ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− __device__ double atomicAdd ( double * address , double val ) { unsigned long long i n t * address_as_ull = ( u n s i g n e d l o n g l o n g i n t ←֓ *) address ; unsigned long long i n t old = * address_as_ull , assumed ; do { assumed = old ; old = atomicCAS ( address_as_ull , assumed , __double_as_longlong ( val + __longlong_as_double ( assumed ) ) ) ; } w h i l e ( a s s u m e d != o l d ) ; return __longlong_as_double ( old ) ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

36 37 38 39

__global__ void sum ( double *d , i n t N , double * totalSum ) { extern __shared__ double partialSum [ ] ;

40

i n t myID = blockIdx . x * blockDim . x + threadIdx . x ; int numThr = blockDim . x * gridDim . x ; int localID = threadIdx . x ; partialSum [ localID ] = 0; i f ( myID < N ) { partialSum [ localID ] = d [ myID ] ; i n t idx = myID + numThr ; // p a r t i a l sum while ( idx < N ) { p a r t i a l S u m [ l o c a l I D ] += d [ i d x ] ; i d x += n u m T h r ; }

41 42 43 44 45 46 47 48 49 50 51 52 53 54 55

__syncthreads () ;

56 57

// r e d u c t i o n i n t step = 1; i n t otherIdx = localID | step ; w h i l e ( ( o t h e r I d x < b l o c k D i m . x ) && ( ( l o c a l I D & s t e p ) == 0 ) ) { p a r t i a l S u m [ l o c a l I D ] += p a r t i a l S u m [ o t h e r I d x ] ; s t e p <<= 1 ; otherIdx = localID | step ; __syncthreads () ; }

58 59 60 61 62 63 64 65 66 67 68

// now t h e b l o c k −wide p a r t i a l sum i s i n p a r t i a l S u m [ 0 ] // add t o g l o b a l c o u n t e r i f ( l o c a l I D == 0 ) atomicAdd ( totalSum , partialSum [ 0 ] ) ;

69 70 71 72

}

73 74

}

75 76

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

77 78 79 80

__global__ void sum2 ( double *d , i n t N , double * totalSum ) { extern __shared__ double partialSum [ ] ;

81 82 83 84 85 86 87 88 89 90

i n t myID = blockIdx . x * blockDim . x + threadIdx . x ; int numThr = blockDim . x * gridDim . x ; int localID = threadIdx . x ; partialSum [ localID ] = 0; i f ( myID < N ) { partialSum [ localID ] = d [ myID ]* d [ myID ] ; i n t idx = myID + numThr ; // p a r t i a l sum

125 while ( idx < N ) { p a r t i a l S u m [ l o c a l I D ] += d [ i d x ] * d [ i d x ] ; i d x += n u m T h r ; }

91 92 93 94 95 96

__syncthreads () ;

97 98

// r e d u c t i o n i n t step = 1; i n t otherIdx = localID | step ; w h i l e ( ( o t h e r I d x < b l o c k D i m . x ) && ( ( l o c a l I D & s t e p ) == 0 ) ) { p a r t i a l S u m [ l o c a l I D ] += p a r t i a l S u m [ o t h e r I d x ] ; s t e p <<= 1 ; otherIdx = localID | step ; __syncthreads () ; }

99 100 101 102 103 104 105 106 107 108 109

// now t h e b l o c k −wide p a r t i a l sum i s i n p a r t i a l S u m [ 0 ] // add t o g l o b a l c o u n t e r i f ( l o c a l I D == 0 ) atomicAdd ( totalSum , partialSum [ 0 ] ) ;

110 111 112 113

}

114 115

}

116 117

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

118 119

120 121 122

d o u b l e g s l _ s t a t s _ m e a n ( c o n s t d o u b l e D A T A [ ] , s i z e _ t S T R I D E , s i z e _ t ←֓ N) { d o u b l e * ha , * da , * d r e s ; // h o s t ( h * ) and d e v i c e ( d * ) p o i n t e r s double hres ;

123 124 125 126 127 128 129 130 131 132 133 134 135 136

int Neffective ; i f ( STRIDE !=1) { N e f f e c t i v e = ( N −1)/ S T R I D E + 1 ; h a = new d o u b l e [ N e f f e c t i v e ] ; f o r ( i n t i =0 , j =0; i<N ; i+=S T R I D E , j++) ha [ j ] = DATA [ i ] ; } else { ha = ( double *) DATA ; Neffective = N ; }

137 138 139

c u d a M a l l o c ( ( v o i d * * ) &da , s i z e o f ( d o u b l e ) * N e f f e c t i v e ) ; c u d a M a l l o c ( ( v o i d * * ) &d r e s , s i z e o f ( d o u b l e ) * 1 ) ;

140 141

142

c u d a M e m c p y ( da , ha , s i z e o f ( d o u b l e ) * N e f f e c t i v e , ←֓ cudaMemcpyHostToDevice ) ; c u d a M e m s e t ( dres , 0 , s i z e o f ( double ) ) ;

143 144 145

146

i n t blockSize , gridSize ; c u d a O c c u p a n c y M a x P o t e n t i a l B l o c k S i z e V a r i a b l e S M e m (& g r i d S i z e , &←֓ b l o c k S i z e , ( v o i d * ) sum , s h a r e d S i z e , N e f f e c t i v e ) ; gridSize = min ( gridSize , Neffective / blockSize ) ;

147 148

s u m <<< g r i d S i z e , b l o c k S i z e , da , N e f f e c t i v e , d r e s ) ;

b l o c k S i z e * s i z e o f ( d o u b l e ) >>> ( ←֓

149 150

c u d a M e m c p y (& h r e s , d r e s , s i z e o f ( d o u b l e ) , ←֓ cudaMemcpyDeviceToHost ) ;

151 152 153

i f ( STRIDE !=1) d e l e t e [ ] ha ;

154 155 156 157 158 159

c u d a F r e e ( ( void *) da ) ; cudaFree ( ( void *) dres ) ; return hres / Neffective ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

126

CHAPTER 6. GPU PROGRAMMING

160 161

162 163 164

d o u b l e g s l _ s t a t s _ v a r i a n c e ( c o n s t d o u b l e D A T A [ ] , s i z e _ t S T R I D E , ←֓ size_t N ) { d o u b l e * ha , * da , * d r e s ; // h o s t ( h * ) and d e v i c e ( d * ) p o i n t e r s double hsum , hsum2 ;

165

int Neffective ; i f ( STRIDE !=1) { N e f f e c t i v e = ( N −1)/ S T R I D E + 1 ; h a = new d o u b l e [ N e f f e c t i v e ] ; f o r ( i n t i =0 , j =0; i<N ; i+=S T R I D E , j++) ha [ j ] = DATA [ i ] ; } else { ha = ( double *) DATA ; Neffective = N ; }

166 167 168 169 170 171 172 173 174 175 176 177 178 179

c u d a M a l l o c ( ( v o i d * * ) &da , s i z e o f ( d o u b l e ) * N e f f e c t i v e ) ; c u d a M a l l o c ( ( v o i d * * ) &d r e s , s i z e o f ( d o u b l e ) * 1 ) ;

180 181 182

c u d a M e m c p y ( da , ha , s i z e o f ( d o u b l e ) * N e f f e c t i v e , ←֓ cudaMemcpyHostToDevice ) ; c u d a M e m s e t ( dres , 0 , s i z e o f ( double ) ) ;

183

184 185

// f i r s t c a l c u l a t e t h e sum o f v a l u e s i n t blockSize , gridSize ; c u d a O c c u p a n c y M a x P o t e n t i a l B l o c k S i z e V a r i a b l e S M e m (& g r i d S i z e , &←֓ b l o c k S i z e , ( v o i d * ) sum , s h a r e d S i z e , N e f f e c t i v e ) ;

186 187 188

189

gridSize = min ( gridSize , Neffective / blockSize ) ; s u m <<< g r i d S i z e , b l o c k S i z e , b l o c k S i z e * s i z e o f ( d o u b l e ) >>> ( ←֓ da , N e f f e c t i v e , d r e s ) ;

190 191

192

c u d a M e m c p y (& h s u m , d r e s , s i z e o f ( d o u b l e ) , ←֓ cudaMemcpyDeviceToHost ) ;

193

194

// t h e n c a l c u l a t e t h e sum o f s q u a r e s c u d a O c c u p a n c y M a x P o t e n t i a l B l o c k S i z e V a r i a b l e S M e m (& g r i d S i z e , &←֓ blockSize , ( void *) sum2 , sharedSize , N e f f e c t i v e ) ; gridSize = min ( gridSize , Neffective / blockSize ) ;

195 196

197 198

// r e s e t t h e r e s u l t v a r i a b l e c u d a M e m s e t ( dres , 0 , s i z e o f ( double ) ) ; s u m 2 <<< g r i d S i z e , b l o c k S i z e , b l o c k S i z e * s i z e o f ( d o u b l e ) >>> ( ←֓ da , N e f f e c t i v e , d r e s ) ;

199 200 201

202

c u d a M e m c p y (& h s u m 2 , d r e s , s i z e o f ( d o u b l e ) , ←֓ cudaMemcpyDeviceToHost ) ;

203

204

i f ( STRIDE !=1) d e l e t e [ ] ha ;

205 206 207 208 209 210 211 212

}

c u d a F r e e ( ( void *) da ) ; cudaFree ( ( void *) dres ) ; double avg = hsum / Neffective ; return hsum2 / Neffective − avg * avg ;

213 214

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

215 216 217 218

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t N = atoi ( argv [ 1 ] ) ;

219 220

double * data ;

221 222

d a t a = new d o u b l e [ N ] ;

223 224 225

numberGen ( N , MAXVALUE , data ) ;

127 p r i n t f ( ”Avg : % l f \n” , g s l _ s t a t s _ m e a n ( d a t a , 1 , N ) ) ; p r i n t f ( ” Var : % l f \n” , g s l _ s t a t s _ v a r i a n c e ( d a t a , 1 , N ) ) ;

226 227 228

d e l e t e [ ] data ; cudaDeviceReset () ;

229 230 231

return 0;

232 233

}

5. Design and implement a CUDA program for calculating the histogram of a 24-bit color image. In this case, three separate histograms will be produced, one for each color component of the image.

Answer The histogram-calculating kernels of Section 6.7.3 can be used as a basis for the solution. The only thing that needs to be explicitly addressed, is the handling of the three color components. Sending the raw RGB data to the GPU could result in having to launch just one kernel only.

However, the memory layout in that case would be equivalent to an array of structures, which as discussed in Section 6.7.4 should be avoided.

The alternative, is to rearrange the pixel data in color planes, as shown in lines 84-110, packing four pixel values per integer (line 106). This arrangement also permits the use of the histogram kernel presented in Listing 6.22, without any modifications.

1 2 3 4 5 6 7

#i n c l u d e <s t d i o . h> #i n c l u d e < s t d l i b . h> #i n c l u d e <u n i s t d . h> #i n c l u d e <math . h> #i n c l u d e <cuda . h> #i n c l u d e <QImage> #i n c l u d e <QRgb>

8 9 10

const i n t BINS = 256; const i n t BINS4ALL = BINS * 32;

11 12

13 14 15 16 17 18 19 20

//←֓

***************************************************************** ←֓

_ _ g l o b a l _ _ v o i d G P U _ h i s t o g r a m _ a t o m i c ( i n t * in , i n t N , { int gloID = blockIdx . x * blockDim . x + threadIdx . x ; int locID = threadIdx . x ; int GRIDSIZE = gridDim . x * blockDim . x ; __shared__ int localH [ BINS4ALL ] ; int bankID = locID % warpSize ; int i ;

int *h )

21 22 23 24

// i n i t i a l i z e t h e l o c a l , s h a r e d −memory b i n s f o r ( i = l o c I D ; i < B I N S 4 A L L ; i += b l o c k D i m . x ) localH [ i ] = 0;

25 26 27

// w a i t f o r a l l warps t o c o m p l e t e t h e p r e v i o u s s t e p __syncthreads () ;

28 29 30 31

32 33

// s t a r t p r o c e s s i n g t h e image d a t a int * mySharedBank = localH + bankID ; i f ( b l o c k D i m . x > w a r p S i z e ) // i f t h e b l o c k s i z e e x c e e d s t h e ←֓ w a r p S i z e , i t i s p o s s i b l e m u l t i p l e warps run a t t h e same ←֓ time f o r ( i = g l o I D ; i < N ; i += G R I D S I Z E ) {

34 35 36 37

i n t temp = in [ i ] ; i n t v = temp & 0 xFF ; i n t v 2 = ( t e m p >> 8 ) & 0 x F F ;

128

CHAPTER 6. GPU PROGRAMMING i n t v 3 = ( t e m p >> 1 6 ) & 0 x F F ; i n t v 4 = ( t e m p >> 2 4 ) & 0 x F F ; a t o m i c A d d ( m y S h a r e d B a n k + ( v << 5 ) , 1 ) ; a t o m i c A d d ( m y S h a r e d B a n k + ( v 2 << 5 ) , 1 ) ; a t o m i c A d d ( m y S h a r e d B a n k + ( v 3 << 5 ) , 1 ) ; a t o m i c A d d ( m y S h a r e d B a n k + ( v 4 << 5 ) , 1 ) ;

38 39 40 41 42 43

} else f o r ( i = g l o I D ; i < N ; i += G R I D S I Z E ) {

44 45 46 47 48

i n t temp = in [ i ] ; i n t v = temp & 0 xFF ; i n t v 2 = ( t e m p >> 8 ) & 0 x F F ; i n t v 3 = ( t e m p >> 1 6 ) & 0 x F F ; i n t v 4 = ( t e m p >> 2 4 ) & 0 x F F ; m y S h a r e d B a n k [ v << 5]++; // Optimized v e r s i o n o f l o c a l H [ ←֓ bankID + v * w a r p S i z e ]++ m y S h a r e d B a n k [ v 2 << 5]++; m y S h a r e d B a n k [ v 3 << 5]++; m y S h a r e d B a n k [ v 4 << 5]++;

49 50 51 52 53 54

55 56 57

}

58 59

// w a i t f o r a l l warps t o c o m p l e t e t h e l o c a l b e f o r e updating the g l o b a l counts __syncthreads () ;

60

61

c a l c u l a t i o n s , ←֓

62

// u s e a t o m i c o p e r a t i o n s t o add t h e l o c a l f i n d i n g s t o t h e ←֓ g l o b a l memory b i n s f o r ( i = l o c I D ; i < B I N S 4 A L L ; i += b l o c k D i m . x ) a t o m i c A d d ( h + ( i >> 5 ) , l o c a l H [ i ] ) ; // Optimized v e r s i o n o f ←֓ atomicAdd ( h + ( i / w a r p S i z e ) , l o c a l H [ i ] ) ;

63

64 65

66

}

67 68

69 70 71 72 73

//←֓

***************************************************************** ←֓

v o i d h i s t o g r a m R G B _ F E ( Q I m a g e &img , { unsigned i n t * d_in , * h_in ; int * d_hist ; i n t N , Nx , N y ;

i n t * hist )

74 75 76 77

Nx = img . width ( ) ; Ny = img . height ( ) ; N = ceil ( ( Nx * Ny ) / 4 . 0 ) ;

78 79 80 81

h _ i n = new u n s i g n e d i n t [ N ] ; c u d a M a l l o c ( ( v o i d * * ) &d _ i n , s i z e o f ( i n t ) * N ) ; c u d a M a l l o c ( ( v o i d * * ) &d _ h i s t , s i z e o f ( i n t ) * B I N S ) ;

82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105

i n t g r i d S i z e =16 , b l o c k S i z e =256; f o r ( i n t c o l o r =0; c o l o r <3; c o l o r++) { i n t i d x =0; f o r ( i n t y =0; y<N y ; y++) f o r ( i n t x =0; x<N x ; x+=4) { u n s i g n e d i n t f o u r P i x e l s =0; f o r ( i n t i =0; i <4; i++) { unsigned i n t tmp ; switch ( color ) { case 0: t m p = q R e d ( i m g . p i x e l ( x+i , y ) ) ; break ; case 1: t m p = q G r e e n ( i m g . p i x e l ( x+i , y ) ) ; break ; default : t m p = q B l u e ( i m g . p i x e l ( x+i , y ) ) ; break ; }

129 f o u r P i x e l s = ( f o u r P i x e l s <<8) | t m p ; } h_in [ idx ] = fourPixels ; i d x ++;

106 107 108 109

}

110 111

c u d a M e m c p y ( d _ i n , h _ i n , s i z e o f ( i n t ) * N , ←֓ cudaMemcpyHostToDevice ) ; cudaMemset ( d_hist , 0 , BINS * s i z e o f ( i n t ) ) ;

112

113 114

G P U _ h i s t o g r a m _ a t o m i c <<< g r i d S i z e , d_in , N , d_hist ) ;

115

b l o c k S i z e >>> ( ( i n t * ) ←֓

116

c u d a M e m c p y ( h i s t+c o l o r * B I N S , d _ h i s t , cudaMemcpyDeviceToHost ) ;

117

s i z e o f ( i n t ) * B I N S , ←֓

}

118 119

cudaFree ( ( void *) d_in ) ; cudaFree ( ( void *) d_hist ) ; d e l e t e [ ] h_in ;

120 121 122 123

}

124 125

126 127

//←֓

***************************************************************** ←֓

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) {

128

QImage pic ; pic . load ( argv [ 1 ] ) ;

129 130 131

i n t * h i s t = new i n t [ 3 * B I N S ] ; h i s t o g r a m R G B _ F E ( pic , h i s t ) ;

132 133 134

f o r ( i n t i =0; i <3; i++) { f o r ( i n t j =0; j <10; j++) p r i n t f ( ”%i ” , h i s t [ B I N S * i+j ] ) ; p r i n t f ( ” . . . \ n” ) ; } cudaDeviceReset () ; d e l e t e [ ] hist ; return 0;

135 136 137 138 139 140 141 142 143 144

}

The above program, which relies on Qt for image I/O, can be compiled with: $

n v c c −X c o m p i l e r −f P I C −m 6 4 −O 2 −D _ R E E N T R A N T \ −D Q T _ N O _ D E B U G −D Q T _ G U I _ L I B −D Q T _ C O R E _ L I B −I / u s r / l i b / x 8 6 _ 6 4 −l i n u x −g n u / q t 5 / m k s p e c s / l i n u x −g++−64 −I . −I / u s r / i n c l u d e / q t 5 −I / u s r / i n c l u d e / q t 5 / Q t G u i −I / u s r / i n c l u d e / q t 5 / Q t C o r e −l Q t 5 C o r e −l Q t 5 G u i −a r c h=s m _ 2 1 −o h i s t o g r a m _ a t o m i c h i s t o g r a m _ a t o m i c . c u

6. Create a variation of the program in Listing 6.24 to discover and plot the memory copy speed for host-to-device and device-to-host operations and for all the available types of host memory allocations : pageable, pinned and mapped. In order to test the last type of allocation, you will have to call a kernel that will try to access the memory, triggering the transfer. Compare your results with the ones returned by the $CUDA/samples/1 Utilities/bandwidthTest sample program.

Answer

130

CHAPTER 6. GPU PROGRAMMING This is an exercise that reveals a great deal about the underpinnings of the zero-copy memory mechanism. The program shown below transfers increasing amounts of data from the host to the device, using pageable, pinned and mapped memory allocation. In the last case, a dummy kernel is invoked to trigger the transfer of memory. The program parameters are: (a) The number of iterations for calculating averages. (b) The increment (in bytes) to use in testing increasing data chunks. (c) The maximum data size (in bytes) to test for.

The maximum device memory is allocated in one go, as dictated by the third command-line parameter, in lines 59-66. The for loop of lines 68 to 72, calls the timeCopyH2D function once for every type of device memory and for increasing data sizes.

The timeCopyH2D function uses a stream and events in the same fashion as Listing 6.24, to calculate the average time taken to transfer data from the host to the device. When a zero-copy memory arrangement is used, the dummyKernel function is also invoked (line 33) in order to trigger the memory transfer instead of the explicit call of line 36.

1 2

#i n c l u d e <s t d i o . h> #i n c l u d e <cuda . h>

3 4 5 6 7 8 9 10 11 12

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // a dummy k e r n e l j u s t f o r i n i t i a t i n g memory t r a n s f e r s when // mapped memory i s u s e d __global__ void dummyKernel ( unsigned char *b , i n t N ) { i n t idx = blockIdx . x * blockDim . x + threadIdx . x ; i f ( idx < N ) b [ idx ] = idx % 256; }

13 14 15

16 17 18 19 20

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− f l o a t t i m e C o p y H 2 D ( u n s i g n e d c h a r * h , u n s i g n e d c h a r * d , i n t N , i n t ←֓ iter , bool Z E R O C O P Y = f a l s e ) { cudaEvent_t startT , endT ; cudaStream_t str ; f l o a t duration ; int i ;

21 22 23 24

c u d a E v e n t C r e a t e (& s t a r t T ) ; c u d a E v e n t C r e a t e (& e n d T ) ; c u d a S t r e a m C r e a t e (& s t r ) ;

25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

cudaEventRecord ( startT , str ) ; f o r ( i = 0 ; i < i t e r ; i++) { i f ( ZEROCOPY ) { int block = 1024; i n t grid = ( N − 1) / block + 1 ; d u m m y K e r n e l <<< 1 , 1 , 0 , s t r >>> ( h , N ) ; } else cudaMemcpyAsync ( d , h , N , cudaMemcpyHostToDevice , str ) ; } c u d a E v e n t R e c o r d ( endT , str ) ; cudaEventSynchronize ( endT ) ; c u d a E v e n t E l a p s e d T i m e (& d u r a t i o n , s t a r t T , e n d T ) ;

41 42 43 44

cudaStreamDestroy ( str ) ; cudaEventDestroy ( startT ) ; cudaEventDestroy ( endT ) ;

131 return ( duration / iter ) /1000;

45 46

// c o n v e r t t o s e c

}

47 48 49 50 51 52 53 54 55 56

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t iter = atoi ( argv [ 1 ] ) ; i n t step = atoi ( argv [ 2 ] ) ; i n t MAXDATASIZE = atoi ( argv [ 3 ] ) ; unsigned char * h_data , * d_data ; unsigned char * h_pinned ; unsigned char * h_zerocopy ;

57

// p a g e a b l e d a t a a l l o c a t i o n h_data = ( unsigned char *) malloc ( MAXDATASIZE ) ; c u d a M a l l o c ( ( v o i d * * ) &d _ d a t a , M A X D A T A S I Z E ) ;

58 59 60 61

// p i n n e d h o s t d a t a a l l o c a t i o n c u d a M a l l o c H o s t ( ( v o i d * * ) &h _ p i n n e d ,

62 63

MAXDATASIZE ) ;

64

// z e r o −copy d a t a a l l o c a t i o n c u d a H o s t A l l o c ( ( v o i d * * ) &h _ z e r o c o p y , M A X D A T A S I Z E , ←֓ cudaHostAllocMapped ) ; p r i n t f ( ” SIZE PAGEABLE PINNED ZERO−COPY\n” ) ; f o r ( i n t d a t a S i z e = 0 ; d a t a S i z e <= M A X D A T A S I Z E ; d a t a S i z e += ←֓ step ) p r i n t f ( ”%i %f %f %f \n” , d a t a S i z e , d a t a S i z e / t i m e C o p y H 2 D ( h _ d a t a , d _ d a t a , d a t a S i z e , ←֓ i t e r ) / (1<<20) , d a t a S i z e / t i m e C o p y H 2 D ( h _ p i n n e d , d _ d a t a , d a t a S i z e ←֓ , i t e r ) / (1<<20) , d a t a S i z e / t i m e C o p y H 2 D ( h _ z e r o c o p y , d _ d a t a , ←֓ d a t a S i z e , i t e r , t r u e ) / (1<<20) ) ;

65 66

67 68

69 70

71

72

73

cudaFreeHost ( h_pinned ) ; cudaFreeHost ( h_zerocopy ) ; free ( h_data ) ; cudaFree ( d_data ) ; cudaDeviceReset () ; return 1;

74 75 76 77 78 79 80

}

A sample of the results (in MB/sec) as tested on a GTX 870M GPU using the CUDA 6.5.14 SDK, are shown below: . / m e m c p y T e s t 100 1000000 1 0 0 0 0 0 0 0 S I Z E P A G E A B L E P I N N E D Z E R O−C O P Y 1000000 8 7 2 1 . 1 9 3 3 5 9 1 1 4 0 7 . 1 5 0 3 9 1 2 5 1 6 4 5 . 0 4 6 8 7 5 2000000 9 8 7 0 . 7 8 7 1 0 9 1 1 4 3 1 . 9 3 9 4 5 3 4 7 7 7 1 6 . 1 5 6 2 5 0 3000000 8 8 1 6 . 8 4 6 6 8 0 1 1 3 3 5 . 1 4 3 5 5 5 7 2 9 3 7 6 . 5 0 0 0 0 0 4000000 8 4 2 8 . 8 9 0 6 2 5 1 0 8 7 2 . 2 7 5 3 9 1 9 5 4 5 9 0 . 6 8 7 5 0 0 5000000 8 9 9 6 . 8 9 5 5 0 8 1 0 9 1 0 . 6 5 4 2 9 7 1 2 7 3 6 0 3 . 5 0 0 0 0 0 6000000 9 0 6 4 . 4 3 0 6 6 4 1 1 6 8 8 . 8 1 2 5 0 0 1 5 8 6 9 1 8 . 1 2 5 0 0 0 7000000 9 6 3 6 . 1 4 9 4 1 4 1 1 6 4 8 . 3 8 2 8 1 2 1 9 1 6 3 7 2 . 1 2 5 0 0 0 8000000 9 6 1 3 . 3 1 1 5 2 3 1 1 7 1 0 . 1 6 4 0 6 2 2 1 4 8 3 0 2 . 2 5 0 0 0 0 9000000 9 6 5 6 . 3 5 1 5 6 2 1 1 6 7 8 . 5 9 5 7 0 3 2 4 9 7 8 6 6 . 5 0 0 0 0 0 10000000 9543.507812 11627.116211 2602141.250000

$

The results for pageable and pinned host memory are consistent with the speeds reported from the $CUDA/samples/1 Utilities/bandwidthTest utility. However, the results for the zero-copy memory are clearly bogus!

It seems that the issue is that the single thread launched in line 33 does not trigger the transfer of the entire designated data block. Forcing the invocation of the kernel for all the tested memory arrangements, results in the following timeCopyH2D function: 1

float

2

{

t i m e C o p y H 2 D ( unsigned char *h , unsigned char *d , iter , bool Z E R O C O P Y = f a l s e )

int N ,

i n t ←֓

132

CHAPTER 6. GPU PROGRAMMING cudaEvent_t startT , endT ; cudaStream_t str ; f l o a t duration ; int i ;

3 4 5 6 7

c u d a E v e n t C r e a t e (& s t a r t T ) ; c u d a E v e n t C r e a t e (& e n d T ) ; c u d a S t r e a m C r e a t e (& s t r ) ;

8 9 10 11

cudaEventRecord ( startT , str ) ; f o r ( i = 0 ; i < i t e r ; i++) { int block = 1024; i n t grid = ( N − 1) / block + 1 ; i f ( ! ZEROCOPY ) { cudaMemcpyAsync ( d , h , N , cudaMemcpyHostToDevice , str ) ; d u m m y K e r n e l <<< g r i d , b l o c k , 0 , s t r >>> ( d , N ) ; } else d u m m y K e r n e l <<< g r i d , b l o c k , 0 , s t r >>> ( h , N ) ; } c u d a E v e n t R e c o r d ( endT , str ) ; cudaEventSynchronize ( endT ) ; c u d a E v e n t E l a p s e d T i m e (& d u r a t i o n , s t a r t T , e n d T ) ;

12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

cudaStreamDestroy ( str ) ; cudaEventDestroy ( startT ) ; cudaEventDestroy ( endT ) ; return ( duration / iter ) /1000;

29 30 31 32 33

}

The results of the modified program are shown below: . / m e m c p y T e s t _ w i t h K e r n e l 100 1000000 1 0 0 0 0 0 0 0 S I Z E P A G E A B L E P I N N E D Z E R O−C O P Y 1000000 7 1 3 9 . 2 9 0 0 3 9 7 2 3 6 . 4 8 8 7 7 0 4 7 2 4 . 3 9 3 0 6 6 2000000 7 6 2 2 . 3 2 1 7 7 7 7 6 9 3 . 1 8 5 5 4 7 5 1 8 9 . 9 1 7 4 8 0 3000000 7 6 2 7 . 7 0 2 1 4 8 8 0 5 7 . 0 8 3 9 8 4 5 1 4 1 . 4 1 0 1 5 6 4000000 6 6 3 0 . 3 0 8 1 0 5 7 8 9 8 . 5 5 7 1 2 9 4 4 2 0 . 7 8 8 0 8 6 5000000 7 4 8 2 . 4 6 3 3 7 9 8 0 9 1 . 9 2 1 3 8 7 4 9 5 4 . 0 7 9 1 0 2 6000000 7 5 2 6 . 9 6 3 3 7 9 8 1 6 8 . 9 1 4 0 6 2 5 0 3 4 . 6 2 2 0 7 0 7000000 7 3 0 4 . 7 8 0 2 7 3 8 1 9 1 . 1 8 3 1 0 5 4 9 7 9 . 7 2 3 6 3 3 8000000 7 4 2 9 . 4 3 4 5 7 0 8 2 3 5 . 0 9 3 7 5 0 4 9 8 1 . 7 1 4 8 4 4 9000000 7 4 1 3 . 1 4 4 5 3 1 8 2 0 3 . 0 4 8 8 2 8 4 8 5 6 . 5 9 1 7 9 7 10000000 7374.308594 8240.675781 4991.039062

$

The above clearly illustrate that zero-copy memory is a recommended solution only if just part of the host data are required for the computation.

Otherwise, even pageable memory is preferable in terms of performance.

The delays introduced by the suspension of the threads that access the memory accumulate to a substantial performance hit.

7. The Mandelbrot set calculators of Section 6.12.1 are limited to a maximum of 1024 iterations per pixel. However, the beauty of the Mandelbrot set is revealed for thousands or millions of iterations. Modify one or more of the solution of Section 6.12.1 so that up to 216 − 1 iterations can be performed for each pixel. Profile your program and analyze its performance. What is the grid/block arrangement that yields the best performance?

Answer The modifications required are minimum as they are concentrated on the memory management part of the code. Using the #1 version of the solution described in Section 6.12.1.1 as the basis of the answer, we need to modify only the kernel.cu file of Listing 6.29, so that the pixel values are

133 stored as short integers (instead of unsigned characters). The changes to Listing 6.29 are documented below: // L i n e 5 , c h a n g i n g t h e upper i t e r a t i o n s t a t i c const int MAXITER = 0 xFFFF ;

limit

// L i n e 2 6 , c h a n g i n g t h e s i g n a t u r e o f t h e k e r n e l t o accommodate ←֓ the d i f f e r e n t type o f p i x e l s t o r a g e _ _ g l o b a l _ _ v o i d m a n d e l K e r n e l ( u n s i g n e d s h o r t i n t * d _ r e s , d o u b l e ←֓ u p p e r X , d o u b l e u p p e r Y , d o u b l e s t e p X , d o u b l e s t e p Y , i n t r e s X , ←֓ i n t resY , i n t pitch ) // L i n e s 5 2 , 53 unsigned short * h_res ; unsigned short * d_res ; // L i n e s 5 8 , 59 C U D A _ C H E C K _ R E T U R N ( c u d a M a l l o c ( ( v o i d * * ) &d _ r e s , r e s X * r e s Y * ←֓ s i z e o f ( unsigned short i n t ) ) ) ; h _ r e s = ( u n s i g n e d s h o r t i n t * ) m a l l o c ( r e s Y * p i t c h * s i z e o f ( ←֓ unsigned short i n t ) ) ; // L i n e 7 1 , g e t t i n g t h e d i f f e r e n t −t y p e r e s u l t s C U D A _ C H E C K _ R E T U R N ( c u d a M e m c p y ( h _ r e s , d _ r e s , r e s Y * p i t c h * ←֓ s i z e o f ( unsigned short i n t ) , cudaMemcpyDeviceToHost ) ) ;

8. Examine the following modifications for improving the stand-alone CUDA AES implementations of Section ??: (a) Move the tables from constant to shared memory : the constant memory cache is faster than global memory, but it is still slower than shared memory. Will the speed improvement offset the need to copy the tables to constant memory for every block of threads? (b) Process multiple 16-byte blocks per thread : turning rijndaelGPUEncrypt() into a device function and introducing another global function as a front-end to it, to be called by rijndaelEncryptFE(), should require the smallest possible effort.

Modify the source code of the Version #2 program in order to introduce your chosen changes, and measure the performance improvement obtained. . Which of offers the biggest -if any- performance improvement?

Answer (a) Moving the tables from constant to shared memory The following prefix code is required in the rijndaelGPUEncrypt function contained in the rijndael device.cu file, to copy the five encoding tables residing in constant memory to shared memory: _ _ g l o b a l _ _ v o i d r i j n d a e l G P U E n c r y p t ( i n t n r o u n d s , u 3 2 * d a t a , ←֓ int N) { __shared__ u32 sTe0 [ 2 5 6 ] ; __shared__ u32 sTe1 [ 2 5 6 ] ; __shared__ u32 sTe2 [ 2 5 6 ] ; __shared__ u32 sTe3 [ 2 5 6 ] ; __shared__ u32 sTe4 [ 2 5 6 ] ; f o r ( i n t i=t h r e a d I d x . x ; i< 2 5 6 ; i+= b l o c k D i m . x ) { sTe0 [ i ] = Te0 [ i ] ; sTe1 [ i ] = Te1 [ i ] ; sTe2 [ i ] = Te2 [ i ] ; sTe3 [ i ] = Te3 [ i ] ;

134

CHAPTER 6. GPU PROGRAMMING sTe4 [ i ] = Te4 [ i ] ; } __syncthreads () ; ...

(b) Processing multiple 16-byte blocks per thread This modification can be achieved with fewer changes, by reducing the size of the grid and introducing a loop in the rijndaelGPUEncrypt function. The resulting code, combining the changes of the previous step, is shown below: // c h a n g e s t o f i l e r i j n d a e l h o s t s t r e a m s . cu c o n s t i n t B L O C K S P E R T H R E A D =4; v o i d r i j n d a e l E n c r y p t F E ( c o n s t u 3 2 * rk , i n t k e y b i t s , ←֓ u n s i g n e d c h a r * p l a i n t e x t , u n s i g n e d c h a r * c i p h e r t e x t , i n t ←֓ N , i n t thrPerBlock = 256) { . . . // g r i d c a l c u l a t i o n i n t g r i d = c e i l ( ( n u m D a t a B l o c k s * 1 . 0 ) / ( t h r P e r B l o c k * ←֓ BLOCKSPERTHREAD ) ) ; r i j n d a e l G P U E n c r y p t <<< g r i d , t h r P e r B l o c k , 0 , s t r [ ←֓ w h i c h S t r e a m ] >>> ( n r o u n d s , d _ b u f f e r [ w h i c h S t r e a m ] , ←֓ toSend ) ; .

.

.

// c h a n g e s t o f i l e r i j n d a e l d e v i c e . cu _ _ g l o b a l _ _ v o i d r i j n d a e l G P U E n c r y p t ( i n t n r o u n d s , u 3 2 * d a t a , ←֓ int N) { __shared__ u32 sTe0 [ 2 5 6 ] ; __shared__ u32 sTe1 [ 2 5 6 ] ; __shared__ u32 sTe2 [ 2 5 6 ] ; __shared__ u32 sTe3 [ 2 5 6 ] ; __shared__ u32 sTe4 [ 2 5 6 ] ; f o r ( i n t i = t h r e a d I d x . x ; i < 2 5 6 ; i += b l o c k D i m . x ) { sTe0 [ i ] = Te0 [ i ] ; sTe1 [ i ] = Te1 [ i ] ; sTe2 [ i ] = Te2 [ i ] ; sTe3 [ i ] = Te3 [ i ] ; sTe4 [ i ] = Te4 [ i ] ; } __syncthreads () ; i n t myID = blockIdx . x * blockDim . x + threadIdx . x ; int totalThr = gridDim . x * blockDim . x ; f o r ( ; m y I D < ( N >> 4 ) ; m y I D += t o t a l T h r ) { u 3 2 s0 , s1 , s2 , s3 , t0 , t1 , t2 , t 3 ; c o n s t u32 * rk = d_rk ; // t o a v o i d c h a n g i n g d r k u32 aux ; i n t m y D a t a I d x = m y I D << 2 ; // * 4 t o

o f f s e t 16 b y t e s

// c o d e below t h i s l i n e i s s h a r e d w i t h t h e o r i g i n a l ←֓ rijndaelGPUEncrypt f u n c t i o n #i f n d e f FULL UNROLL int r ; . . . data [ m y D a t a I d x + 3 ] = R E S H U F F L E ( s3 ) ; } }

In terms of performance, there is virtually no change as the encryption/decryption of a single block already constitutes a sufficiently

135 heavy computational load.

9. The MPI cluster AES implementations of Section 6.12.2 does not provide overlapping of communication and computation. This issue could be addressed if a new “work item” was downloaded by the worker nodes while the GPU was processing an already downloaded part. Modify the MPI solution to provide this functionality. Do you expect any problems with the load-balancing of the modified solution?

Answer A solution to this problem could be that each worker sets-up buffer space to hold three work items, and starts processing one while using a nonblocking send and a non-blocking receive operations to communicate the completed and next pending work items respectively.

The only modifications required, relate to the worker code as shown below for the case of a GPU worker. An identical arrangement is possible for a CPU worker. The communication/computation overlap is achieved by using immediate communication primitives to exchange data with the master (lines 31, 36) while computation is taking place (line 38). There is no need to use immediate functions in lines 29 and 34, as scalar primitive types can be buffered by MPI : 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

// c h a n g e s t o f i l e AES MPI/ main . cpp i f ( r a n k == 0 ) . . . else // GPU w o r k e r { i n t workItemSize = atoi ( argv [ 3 ] ) ; i n t thrPerBlock = atoi ( argv [ 4 ] ) ; i n t p o s [ 3 ] = { −1, −1, −1 } ; int totalWork = 0; unsigned char * workBuff [ 3 ] ; w o r k B u f f [ 0 ] = new u n s i g n e d c h a r [ w o r k I t e m S i z e ] ; w o r k B u f f [ 1 ] = new u n s i g n e d c h a r [ w o r k I t e m S i z e ] ; w o r k B u f f [ 2 ] = new u n s i g n e d c h a r [ w o r k I t e m S i z e ] ; int actualSize [ 3 ] ; MPI_Request getRq , sendRq ;

16 17

18 19

int activeBuff = 0; management int completedBuff = 2; int nextBuff = 1;

// i n d i c e s

f o r work i t e m b u f f e r ←֓

20 21

22

23

24

25 26 27 28 29

30 31

M P I _ S e n d (&( p o s [ c o m p l e t e d B u f f ] ) , 1 , M P I _ I N T , 0 , T A G _ R E S , ←֓ MPI_COMM_WORLD ) ; M P I _ R e c v (&( p o s [ a c t i v e B u f f ] ) , 1 , M P I _ I N T , 0 , T A G _ W O R K , ←֓ M P I _ C O M M _ W O R L D , &s t a t ) ; M P I _ R e c v ( w o r k B u f f [ a c t i v e B u f f ] , w o r k I t e m S i z e , ←֓ M P I _ U N S I G N E D _ C H A R , 0 , T A G _ D A T A , M P I _ C O M M _ W O R L D , &s t a t ) ←֓ ; M P I _ G e t _ c o u n t (& s t a t , M P I _ U N S I G N E D _ C H A R , &( a c t u a l S i z e [ ←֓ activeBuff ] ) ) ; t o t a l W o r k += a c t u a l S i z e [ a c t i v e B u f f ] ; w h i l e ( p o s [ a c t i v e B u f f ] >= 0 ) { // s e n d c o m p l e t e d i t e m w i t h immediate s e n d M P I _ S e n d (&( p o s [ c o m p l e t e d B u f f ] ) , 1 , M P I _ I N T , 0 , T A G _ R E S ←֓ , MPI_COMM_WORLD ) ; i f ( pos [ completedBuff ] > 0) M P I _ I s e n d ( w o r k B u f f [ c o m p l e t e d B u f f ] , a c t u a l S i z e [ ←֓ c o m p l e t e d B u f f ] , M P I _ U N S I G N E D _ C H A R , 0 , T A G _ D A T A , ←֓ M P I _ C O M M _ W O R L D , &s e n d R q ) ;

32 33

// g e t n e x t i t e m w h i l e p r o c e s s i n g

136

CHAPTER 6. GPU PROGRAMMING M P I _ R e c v (&( p o s [ n e x t B u f f ] ) , 1 , M P I _ I N T , 0 , T A G _ W O R K , ←֓ M P I _ C O M M _ W O R L D , &s t a t ) ; i f ( pos [ nextBuff ] > 0) M P I _ I r e c v ( w o r k B u f f [ n e x t B u f f ] , w o r k I t e m S i z e , ←֓ M P I _ U N S I G N E D _ C H A R , 0 , T A G _ D A T A , M P I _ C O M M _ W O R L D , ←֓ &g e t R q ) ;

34

35 36

37

r i j n d a e l E n c r y p t F E ( rk , k e y b i t s , w o r k B u f f [ a c t i v e B u f f ] , ←֓ w o r k B u f f [ a c t i v e B u f f ] , a c t u a l S i z e [ a c t i v e B u f f ] , ←֓ thrPerBlock ) ;

38

39

// make s u r e c o m m u n i c a t i o n s a r e c o m p l e t e b e f o r e u p d a t i n g ←֓ t h e i n d i c e s and c o m n t i n u i n g i f ( pos [ completedBuff ] > 0) M P I _ W a i t (& s e n d R q , M P I _ S T A T U S _ I G N O R E ) ; i f ( pos [ nextBuff ] > 0) { M P I _ W a i t (& g e t R q , &s t a t ) ; M P I _ G e t _ c o u n t (& s t a t , M P I _ U N S I G N E D _ C H A R , &(←֓ actualSize [ nextBuff ] ) ) ; t o t a l W o r k += a c t u a l S i z e [ n e x t B u f f ] ; } // update i n d i c e s completedBuff = activeBuff ; activeBuff = nextBuff ; nextBuff = ( nextBuff + 1) % 3 ; } rijndaelShutdown () ; c o u t << ” Worker ” << r a n k << ” p r o c e s s e d ” << t o t a l W o r k << ←֓ endl ; delete [ ] workBuff [ 0 ] ; delete [ ] workBuff [ 1 ] ; delete [ ] workBuff [ 2 ] ;

40

41 42 43 44 45 46

47 48 49 50 51 52 53 54 55

56 57 58 59

}

The three work item buffers are addressed individually by the completedBuff, activeBuff and nextBuff indices. Each buffer is accompanied by a dedicated position (pos array) and length (actualSize array) variables. While one buffer is being processed (pointed to by activeBuff), another holding previous results (pointed to by completedBuff) is being send to the master node (line 31) and the third one (pointed to by nextBuff) is being filled with new input data (line 36).

In all the communication transactions, the pos array members guard against sending dummy data to the master (line 30) and against waiting for new data when the input is exhausted (lines 26 and 35).

A problem may arise with the load balancing of this solution if a very slow worker is participating in the proceedings. It is possible that it can delay the termination of the overall execution by receiving a pending work item that could be processed faster at another node.

10. Modify the MPI cluster AES implementations of Section 6.12.2 so that only two types of messages are needed for data exchange, instead of the current three. How can this be combined with the modification of the previous exercise?

Answer The messages tagged by the TAG WORK label can be eliminated from the communication protocol if the master node maintains a record of which part of the data is assigned to each worker node respectively. When a single work item is assigned to each worker, a simple integer array with

137 as many element as the number of workers, is sufficient for handling this requirement. int assignedWorkItem [ comm_size ] ;

If more than a single work item is assigned at a time to a worker, then the assignedWorkItem array has to become an array of containers, such as a vector<int>: v e c t o r <i n t > a s s i g n e d W o r k I t e m [ c o m m _ s i z e ] ;

The termination of the workers can be achieved by sending a work item of length zero.

There is also another alternative. The whole point of this exercise is to reduce the number of messages sent, while at the same time being able to identify the origin/destination in the input/output data of the data parts being exchanged. Towards this end, we could utilize the tag of the messages to carry this information (it is of type int). The only drawback of collapsing the two messages into one is that there is a need for an extra data copy from the generic message buffer used to collect the incoming results (line 33) to the output data repository (line 38): 1 2 3 4 5 6 7 8 9

if

( r a n k == 0 ) {

unsigned char * iobuf ; unsigned char * recvbuf ; i f ( ( f = f o p e n ( a r g v [ 1 ] , ” r ” ) ) == N U L L ) { f p r i n t f ( s t d e r r , ”Can ’ t open %s \n” , a r g v [ 1 ] ) ; exit ( EXIT_FAILURE ) ; }

10 11

i n t workItemSize = atoi ( argv [ 3 ] ) ;

12 13 14 15

fseek ( f , 0 , SEEK_END ) ; lSize = ftell ( f ) ; rewind ( f ) ;

16 17 18 19 20 21 22

i o b u f = new u n s i g n e d c h a r [ l S i z e ] ; r e c v b u f = new u n s i g n e d c h a r [ w o r k I t e m S i z e ] ; a s s e r t ( i o b u f != N U L L ) ; a s s e r t ( r e c v b u f != N U L L ) ; fread ( iobuf , 1 , lSize , f ) ; fclose ( f ) ;

23 24 25 26

timeval tim ; g e t t i m e o f d a y (& tim , N U L L ) ; double tm2 = tim . tv_sec + ( tim . tv_usec / 1000000.0) ;

27 28 29 30 31 32 33

34 35 36 37 38 39

// m a s t e r main l o o p i n t pos = 0 ; while ( pos < lSize ) { i n t retPos , recvSize ; M P I _ R e c v ( r e c v b u f , w o r k I t e m S i z e , M P I _ U N S I G N E D _ C H A R , ←֓ M P I _ A N Y _ S O U R C E , M P I _ A N Y _ T A G , M P I _ C O M M _ W O R L D , &s t a t ←֓ ); M P I _ G e t _ c o u n t (& s t a t , M P I _ U N S I G N E D _ C H A R , &r e c v S i z e ) ; i f ( recvSize > 0) { retPos = stat . MPI_TAG ; memcpy ( iobuf + retPos , recvbuf , recvSize ) ; }

40 41 42

// a s s i g n n e x t work i t e m i n t a c t u a l S i z e = ( w o r k I t e m S i z e < l S i z e − p o s ) ? ←֓ workItemSize : ( lSize − pos ) ;

138

CHAPTER 6. GPU PROGRAMMING M P I _ S e n d ( i o b u f + pos , a c t u a l S i z e , M P I _ U N S I G N E D _ C H A R , ←֓ s t a t . M P I _ S O U R C E , pos , M P I _ C O M M _ W O R L D ) ; p o s += a c t u a l S i z e ;

43

44

}

45 46

// w a i t f o r l a s t r e s u l t s f o r ( i n t i = 1 ; i < c o m m _ s i z e ; i++) { i n t retPos , recvSize ; M P I _ R e c v ( r e c v b u f , w o r k I t e m S i z e , M P I _ U N S I G N E D _ C H A R , ←֓ M P I _ A N Y _ S O U R C E , M P I _ A N Y _ T A G , M P I _ C O M M _ W O R L D , &s t a t ←֓ ); M P I _ G e t _ c o u n t (& s t a t , M P I _ U N S I G N E D _ C H A R , &r e c v S i z e ) ; i f ( recvSize > 0) { retPos = stat . MPI_TAG ; memcpy ( iobuf + retPos , recvbuf , recvSize ) ; }

47 48 49 50 51

52 53 54 55 56 57 58

// s e n d t e r m i n a t i o n s i g n a l M P I _ S e n d ( r e c v b u f , 0 , M P I _ U N S I G N E D _ C H A R , s t a t . ←֓ MPI_SOURCE , 0 , MPI_COMM_WORLD ) ;

59 60

} g e t t i m e o f d a y (& tim , N U L L ) ; double tm3 = tim . tv_sec + ( tim . tv_usec / 1000000.0) ;

61 62 63 64

FILE * fout ; i f ( ( f o u t = f o p e n ( a r g v [ 2 ] , ”w” ) ) == N U L L ) { f p r i n t f ( s t d e r r , ”Can ’ t open %s \n” , a r g v [ 2 ] ) ; exit ( EXIT_FAILURE ) ; } fwrite ( iobuf , 1 , lSize , fout ) ; fclose ( fout ) ; d e l e t e [ ] iobuf ; delete [ ] recvbuf ;

65 66 67 68 69 70 71 72 73 74 75

g e t t i m e o f d a y (& t i m e M a i n , N U L L ) ; d o u b l e t m 4 = t i m e M a i n . t v _ s e c + ( t i m e M a i n . t v _ u s e c / ←֓ 1000000.0) ;

76 77

78

// p r i n t −o u t some t i m i n g i n f o r m a t i o n p r i n t f ( ” %.9 l f \ t %.9 l f \n” , t m 4 − tm1 , t m 3 − t m 2 ) ;

79 80 81 82 83 84 85 86 87 88 89 90 91 92 93

} else {

// GPU w o r k e r i n t workItemSize = atoi ( argv [ 3 ] ) ; i n t thrPerBlock = atoi ( argv [ 4 ] ) ; i n t pos [ 3 ] = { 0 , 0 , 0 } ; int totalWork = 0; unsigned char * workBuff [ 3 ] ; w o r k B u f f [ 0 ] = new u n s i g n e d c h a r [ w o r k I t e m S i z e ] ; w o r k B u f f [ 1 ] = new u n s i g n e d c h a r [ w o r k I t e m S i z e ] ; w o r k B u f f [ 2 ] = new u n s i g n e d c h a r [ w o r k I t e m S i z e ] ; int actualSize [ 3 ] = { 0 , 0 , 0 }; MPI_Request getRq , sendRq ;

94 95

96 97

int activeBuff = 0; management int completedBuff = 2; int nextBuff = 1;

// i n d i c e s

f o r work i t e m b u f f e r ←֓

98 99

100

101 102

103 104 105 106

M P I _ S e n d ( w o r k B u f f [ c o m p l e t e d B u f f ] , 0 , M P I _ U N S I G N E D _ C H A R , 0 , ←֓ pos [ completedBuff ] , MPI_COMM_WORLD ) ; M P I _ R e c v ( w o r k B u f f [ a c t i v e B u f f ] , w o r k I t e m S i z e , ←֓ M P I _ U N S I G N E D _ C H A R , 0 , M P I _ A N Y _ T A G , M P I _ C O M M _ W O R L D , &←֓ stat ) ; pos [ activeBuff ] = stat . MPI_TAG ; M P I _ G e t _ c o u n t (& s t a t , M P I _ U N S I G N E D _ C H A R , &( a c t u a l S i z e [ ←֓ activeBuff ] ) ) ; t o t a l W o r k += a c t u a l S i z e [ a c t i v e B u f f ] ; while ( actualSize [ activeBuff ] > 0) { // s e n d c o m p l e t e d i t e m w i t h immediate s e n d

139 M P I _ I s e n d ( w o r k B u f f [ c o m p l e t e d B u f f ] , a c t u a l S i z e [ ←֓ c o m p l e t e d B u f f ] , M P I _ U N S I G N E D _ C H A R , 0 , p o s [ ←֓ c o m p l e t e d B u f f ] , M P I _ C O M M _ W O R L D , &s e n d R q ) ;

107

108

// g e t n e x t i t e m w h i l e p r o c e s s i n g M P I _ I r e c v ( w o r k B u f f [ n e x t B u f f ] , w o r k I t e m S i z e , ←֓ M P I _ U N S I G N E D _ C H A R , 0 , M P I _ A N Y _ T A G , M P I _ C O M M _ W O R L D , ←֓ &g e t R q ) ;

109 110

111

r i j n d a e l E n c r y p t F E ( rk , k e y b i t s , w o r k B u f f [ a c t i v e B u f f ] , ←֓ w o r k B u f f [ a c t i v e B u f f ] , a c t u a l S i z e [ a c t i v e B u f f ] , ←֓ thrPerBlock ) ;

112

113

// make s u r e c o m m u n i c a t i o n s a r e c o m p l e t e b e f o r e ←֓ u p d a t i n g t h e i n d i c e s and c o m n t i n u i n g M P I _ W a i t (& s e n d R q , M P I _ S T A T U S _ I G N O R E ) ; M P I _ W a i t (& g e t R q , &s t a t ) ; M P I _ G e t _ c o u n t (& s t a t , M P I _ U N S I G N E D _ C H A R , &( a c t u a l S i z e [ ←֓ nextBuff ] ) ) ; pos [ nextBuff ] = stat . MPI_TAG ; t o t a l W o r k += a c t u a l S i z e [ n e x t B u f f ] ;

114

115 116 117

118 119 120

completedBuff = activeBuff ; activeBuff = nextBuff ; nextBuff = ( nextBuff + 1) % 3 ;

121 122 123

} rijndaelShutdown () ; c o u t << ” Worker ” << r a n k << ” p r o c e s s e d ” << t o t a l W o r k << ←֓ endl ; delete [ ] workBuff [ 0 ] ; delete [ ] workBuff [ 1 ] ; delete [ ] workBuff [ 2 ] ;

124 125 126

127 128 129 130

}

In the above code the explicit communication of the buffer positions is eliminated. Termination and first job request conditions are detected by checking the length of the input (line 35) and output data (line 104) communicated respectively.

11. The whole point of the multi-core “adventure”, is to accelerate our programs. This should be our sole focus, beyond any mis- or pre-conceptions.

The evaluation of the different AES parallel implementations conducted in Section 6.12.2.4 considered only the encryption process, disregarding any I/O costs incurred. Perform your own experiment where the overall execution time is considered and not just the encryption time. Make sure that the file cache provided by the operating system is not utilized by:

Either, calling the following from the command-line (root permissions are required): $

sync ; echo 3 > / proc / sys / vm / d r o p _ c a c h e s

Or, calling the posix fadvice() function from within your program, prior to any I/O: #i n c l u d e <u n i s t d . h> #i n c l u d e < f c n t l . h> int main ( int argc , char * argv [ ] ) { int fd ; f d = o p e n ( a r g v [ 1 ] , O _ R D O N L Y ) ; // O p e n t h e f i l e h o l d i n g t h e ←֓ input data f d a t a s y n c ( fd ) ; p o s i x _ f a d v i s e ( fd , 0 , 0 , P O S I X _ F A D V _ D O N T N E E D ) ; // c l e a r c a c h e close ( fd ) ; . . . }

140

CHAPTER 6. GPU PROGRAMMING Analyze your findings.

Answer This is an activity that is to be carried out by the students. The following command line can be used for generating a random 32MB input file to be used in testing: $

dd

i f =/d e v / u r a n d o m o f=i n 3 2 M B y t e s b s =32M c o u n t=1

12. Modify the program shown in Listing ??, so that the reduction phases are shortened to one, using the shuffling-down operation multiple times.

Answer 1 2 3

__global__ void countOdds ( i n t *d , { extern __shared__ int count [ ] ;

int N ,

i n t * odds )

4

i n t myID = blockIdx . x * blockDim . x + threadIdx . x ; int totalThr = blockDim . x * gridDim . x ; int localID = threadIdx . x ; i n t t o t a l W a r p s = b l o c k D i m . x / 32 + ( b l o c k D i m . x % 32 != 0 ) ; int warpID = localID / 32; count [ localID ] = 0; f o r ( i n t i=m y I D ; i < N ; i+= t o t a l T h r ) c o u n t [ l o c a l I D ] += ( d [ i ] % 2 ) ; __syncthreads () ;

5 6 7 8 9 10 11 12 13 14

// r e d u c t i o n p h a s e : sum up t h e warp r e s u l t s while ( totalWarps > 0) { unsigned mask = 0 xffffffff ; i n t val = count [ localID ] ; f o r ( i n t o f f s e t = 1 6 ; o f f s e t > 0 ; o f f s e t /= 2 ) { v a l += _ _ s h f l _ d o w n _ s y n c ( m a s k , val , o f f s e t ) ; } count [ localID ] = 0; t o t a l W a r p s /= 3 2 ; __syncthreads () ; i f ( l o c a l I D % 32 == 0 ) // l a n e 0 i n a l l warps count [ warpID ] = val ; __syncthreads () ; i f ( w a r p I D ==0) p r i n t f ( ”## %i %i %i \n” , w a r p I D , l o c a l I D , c o u n t [ l o c a l I D ] ) ; }

15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

i f ( w a r p I D ==0) { unsigned mask = 0 xffffffff ; i n t val = count [ localID ] ; f o r ( i n t o f f s e t = 1 6 ; o f f s e t > 0 ; o f f s e t /= 2 ) { v a l += _ _ s h f l _ d o w n _ s y n c ( m a s k , val , o f f s e t ) ; } count [ 0 ] = val ; } // add t o g l o b a l c o u n t e r i f ( l o c a l I D == 0 ) a t o m i c A d d ( odds , count [ 0 ] ) ;

34 35 36 37 38 39 40 41 42 43 44 45 46 47

}

13. Listing ?? presents a kernel that uses grid synchronization for filtering an array of numbers. The problem is that we can only launch one block per SM, and given that each thread processes one data element, this limits the maximum problem size we can handle. Write a modified version of the kernel in Listing ?? that processes multiple elements per thread. Are there any shortcomings that you will have to deal with?

141 Answer The following kernel processes multiple elements per thread. The shortcoming is that the filtered data are reordered in the destination array relative to the source array. By sacrificing coalesced memory access, this can be overcome.

1 2

3 4 5 6 7

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− _ _ g l o b a l _ _ v o i d f i l t e r O d d s ( i n t * src , i n t * d e s t , i n t N , oddBlockCount ) { i n t myID = blockIdx . x * blockDim . x + threadIdx . x ; int totalThr = blockDim . x * gridDim . x ; int myCount = 0; // t h r e a d s p e c i f i c int myDestOffset ; // t h r e a d s p e c i f i c

i n t * ←֓

8 9 10

11 12

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // S t e p 1 : p r o c e s s t h e p a r t o f t h e i n p u t a s s i g n e d t o t h e ←֓ thread f o r ( i n t i = m y I D ; i < N ; i += t o t a l T h r ) m y C o u n t += s r c [ i ] % 2 ;

13 14 15 16 17 18 19 20 21 22 23 24 25 26

27

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // S t e p 2 : f i n d t h e number o f odds a s s i g n e d t o e a c h warp c o a l e s c e d _ g r o u p cg = c o a l e s c e d _ t h r e a d s ( ) ; i n t val = myCount ; f o r ( i n t o f f s e t = 1 ; o f f s e t < 3 2 ; o f f s e t *= 2 ) { i n t t m p = c g . s h f l _ u p ( val , o f f s e t ) ; i f ( c g . t h r e a d _ r a n k ( ) >= o f f s e t ) // s o u r c e e x i s t s ? v a l += t m p ; } myDestOffset = val ; int groupOffset = 0; i f ( c g . t h r e a d _ r a n k ( ) == c g . s i z e ( ) − 1 ) // l a s t t h r e a d ←֓ h as t h e t o t a l c o u n t f o r t h e group groupOffset = atomicAdd ( oddBlockCount + blockIdx . x , val ) ;

28 29 30

// p r o p a g a t e t o t h e whole group / warp g r o u p O f f s e t = cg . shfl ( groupOffset , cg . size ( ) − 1) ;

31 32 33 34 35 36 37

// ” S h i f t ” warp o f f s e t s up by one p o s i t i o n val = cg . s h f l _ u p ( myDestOffset , 1) ; i f ( c g . t h r e a d _ r a n k ( ) == 0 ) myDestOffset = 0; else myDestOffset = val ;

38 39 40 41

// a d j u s t o f f s e t s f o r t h e b l o c k i f ( cg . t h r e a d _ r a n k ( ) > 0) m y D e s t O f f s e t += g r o u p O f f s e t ;

42 43 44 45

// w a i t f o r a l l b l o c k s t o r e a c h t h i s p o i n t grid_group g = this_grid () ; g . sync () ;

46 47 48 49

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // S t e p 3 : f i n d t h e b l o c k − s p e c i f i c o f f s e t s int * accumm = oddBlockCount + gridDim . x ;

50 51 52 53 54 55 56 57

if

( myID < gridDim . x ) { i n t tmp = atomicAdd ( accumm , oddBlockCount [ myID ] = tmp ;

} // w a i t f o r g . sync () ;

all

the block

offsets

oddBlockCount [ myID ] ) ; t o be updated

58 59 60

// a d j u s t t h e t h r e a d o f f s e t s f o r t h e g r i d m y D e s t O f f s e t += o d d B l o c k C o u n t [ b l o c k I d x . x ] ;

61 62 63 64 65

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // S t e p 4 : move t h e d a t a f o r ( i n t i = m y I D ; i < N ; i += t o t a l T h r ) {

142

CHAPTER 6. GPU PROGRAMMING i n t tmp = src [ i ] ; i f ( tmp % 2) d e s t [ m y D e s t O f f s e t ++] = t m p ;

66 67 68

}

69 70

}

Chapter 7

GPU and Accelerator Programming : OpenCL Exercises 1. Write a device query program that will report all the OpenCL devices present in your computer, along with the version of OpenCL they support, the number of compute units, the size of the global memory and the size of the local memory available in each one.

Answer 1

#i n c l u d e <i o s t r e a m >

2 3 4 5 6 7

#i f d e f APPLE #i n c l u d e <OpenCL/ c l . h> #e l s e #i n c l u d e <CL/ c l . h> #e n d i f

8 9

u s i n g namespace s t d ;

10 11 12 13 14 15 16 17 18

const int MAXNUMDEV = 10; i n t main () { cl_int errNum ; cl_uint numPlatforms ; cl_platform_id platforms [ 2 ] ; cl_device_id devID [ MAXNUMDEV ] ; cl_uint numDev ;

19 20 21 22 23 24 25 26

// Get a r e f e r e n c e t o an o b j e c t r e p r e s e n t i n g a p l a t f o r m e r r N u m = c l G e t P l a t f o r m I D s ( 2 , p l a t f o r m s , &n u m P l a t f o r m s ) ; i f ( e r r N u m != C L _ S U C C E S S | | n u m P l a t f o r m s <= 0 ) { c e r r << ” F a i l e d t o f i n d any OpenCL p l a t f o r m s . ” << e n d l ; return 1; }

27 28 29 30 31 32

33

char devName [ 1 0 0 ] ; size_t nameLen ; f o r ( i n t p I D = 0 ; p I D < n u m P l a t f o r m s ; p I D++) { e r r N u m = c l G e t P l a t f o r m I n f o ( p l a t f o r m s [ p I D ] , ←֓ C L _ P L A T F O R M _ V E R S I O N , 1 0 0 , ( v o i d * ) d e v N a m e , &n a m e L e n ) ; p r i n t f ( ” P l a t f o r m : %s \n” , d e v N a m e ) ;

34 35

// Get t h e d e v i c e IDs matching t h e CL DEVICE TYPE p arame te r ←֓ , up t o t h e MAXNUMDEV l i m i t

143

144CHAPTER 7. GPU AND ACCELERATOR PROGRAMMING : OPENCL e r r N u m = c l G e t D e v i c e I D s ( p l a t f o r m s [ p I D ] , C L _ D E V I C E _ T Y P E _ A L L ←֓ , M A X N U M D E V , d e v I D , &n u m D e v ) ; i f ( e r r N u m != C L _ S U C C E S S | | n u m D e v <= 0 ) { c e r r << ” F a i l e d t o f i n d any OpenCL d e v i c e s . ” << e n d l ; return 2; }

36

37 38 39 40 41 42

f o r ( i n t i = 0 ; i < n u m D e v ; i++) { e r r N u m = c l G e t D e v i c e I n f o ( d e v I D [ i ] , C L _ D E V I C E _ N A M E , ←֓ 1 0 0 , ( v o i d * ) d e v N a m e , &n a m e L e n ) ; i f ( e r r N u m == C L _ S U C C E S S ) c o u t << ” \ t D e v i c e ” << i << ” i s ” << d e v N a m e << e n d l ←֓ ;

43 44 45

46 47

48

c l _ u i n t cu ; e r r N u m = c l G e t D e v i c e I n f o ( d e v I D [ i ] , ←֓ C L _ D E V I C E _ M A X _ C O M P U T E _ U N I T S , s i z e o f ( c l _ u i n t ) , ( ←֓ v o i d * ) &cu , N U L L ) ; i f ( e r r N u m == C L _ S U C C E S S ) c o u t << ” \ t D e v i c e ” << i << ” h as ” << c u << ” ←֓ compute u n i t s \n” ;

49 50

51 52

53

cl_ulong globalMem ; e r r N u m = c l G e t D e v i c e I n f o ( d e v I D [ i ] , ←֓ C L _ D E V I C E _ G L O B A L _ M E M _ S I Z E , s i z e o f ( c l _ u l o n g ) , ( ←֓ v o i d * ) &g l o b a l M e m , N U L L ) ; i f ( e r r N u m == C L _ S U C C E S S ) c o u t << ” \ t D e v i c e ” << i << ” h as ” << g l o b a l M e m << ”←֓ b y t e s g l o b a l memory\n” ;

54 55

56 57

58

cl_ulong locallMem ; e r r N u m = c l G e t D e v i c e I n f o ( d e v I D [ i ] , ←֓ C L _ D E V I C E _ L O C A L _ M E M _ S I Z E , s i z e o f ( c l _ u l o n g ) , ( v o i d ←֓ * ) &l o c a l l M e m , N U L L ) ; i f ( e r r N u m == C L _ S U C C E S S ) c o u t << ” \ t D e v i c e ” << i << ” h as ” << l o c a l l M e m << ”←֓ b y t e s l o c a l memory\n” ;

59 60

61 62

}

63

}

64 65

return 0;

66 67

}

2. Implement the trapezoid rule integration technique in OpenMP for an arbitrary function f (). As explained in Section ??, the definite integral between [a, b] of f () can be approximated by the formula

h

n−1 f (a) + f (b) X f (xi ) + 2 i=1

!

where n is the number of divisions and h = b−a n . Explore the following possibilities : (a) have the device calculate the target function’s f (xi ) values and the host accumulate the sum. (b) Have the device perform all the calculations, including the reduction of the final result. For the latter approach employ work group functions.

Answer Host program: 1 2 3 4 5

#i n c l u d e < s t d l i b . h> #i n c l u d e <i o s t r e a m > #i n c l u d e <s t d i o . h> #i n c l u d e <atomic>

145 6

#i n c l u d e ” . . / c l u t i l i t y . h”

7 8

u s i n g namespace s t d ;

9 10 11 12 13 14 15 16

s t a t i c cl_context cont = 0; // i n i t i a l i z e f o r c l e a n u p c h e c k s t a t i c cl_command_queue q = 0; s t a t i c c l _ p r o g r a m pr = 0 ; s t a t i c cl_kernel kern = 0; s t a t i c bool OCL2support = true ; s t a t i c f l o a t * data = 0; // r e s u l t d a t a s t a t i c cl_mem d_data = 0; // r e s u l t d a t a i f OpenCL 2 . 0 i s n o t ←֓ supported

17 18 19 20 21 22

// ************************************************************ void cleanUp ( ) { i f ( k e r n != 0 ) clReleaseKernel ( kern ) ;

23 24

if

( p r != 0 ) c l R e l e a s e P r o g r a m ( pr ) ;

if

( q != 0 ) clReleaseCommandQueue ( q ) ;

25 26 27 28 29

if

30

( c o n t != 0 ) {

31

if

( OCL2support ) c l S V M F r e e ( cont , data ) ; else clReleaseMemObject ( d_data ) ; clReleaseContext ( cont ) ;

32 33 34 35 36

}

37 38

}

39 40 41

42 43 44 45 46

// ************************************************************ // Host f r o n t −end f u n c t i o n t h a t a l l o c a t e s t h e memory and l a u n c h e s ←֓ t h e GPU k e r n e l f l o a t h o s t F E ( f l o a t st , f l o a t en , i n t d i v ) { cl_int errNum ; cl_device_id devID ; cl_event kernComplete ;

47 48 49

// c o n t e x t and queue s e t u p s e t u p D e v i c e ( cont , q , NULL , devID ,

CL_DEVICE_TYPE_ALL ) ;

50 51 52 53

54 55

// d i s c o v e r t h e number o f CUs c l _ u i n t cu ; e r r N u m = c l G e t D e v i c e I n f o ( d e v I D , C L _ D E V I C E _ M A X _ C O M P U T E _ U N I T S , ←֓ s i z e o f ( c l _ u i n t ) , ( v o i d * ) &cu , N U L L ) ; i f ( e r r N u m != C L _ S U C C E S S ) c u =10;

56 57 58 59

60 61 62 63 64 65

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // r e a d program s o u r c e and b u i l d i t i f ( ! s e t u p P r o g r a m A n d K e r n e l ( d e v I D , c o n t , ” t r a p e z o i d . c l ” , pr , ”←֓ i n t e g r a t e ” , kern ) ) { i f ( p r != 0 ) p r i n t f ( ” E r r o r : %s \n” , g e t C o m p i l a t i o n E r r o r ( pr , d e v I D ) ) ; cleanUp () ; return 1; }

66 67 68 69 70

71

72 73

// c h e c k i f SVM i s s u p p o r t e d char attr [ 2 0 ] ; size_t attrLen ; c l G e t D e v i c e I n f o ( d e v I D , C L _ D E V I C E _ V E R S I O N , 2 0 , ( v o i d * ) a t t r , &←֓ attrLen ) ; i f ( a t t r [ 7 ] == ’ 1 ’ ) // C h a r a c t e r a t p o s i t i o n 7 i s t h e ←֓ OpenCL major v e r s i o n OCL2support = f a l s e ;

146CHAPTER 7. GPU AND ACCELERATOR PROGRAMMING : OPENCL if {

74 75 76

77

( OCL2support ) data = ( f l o a t *) c l S V M A l l o c ( cont , C L _ M E M _ R E A D _ W R I T E , ( f l o a t ) , 0) ; * d a t a =0;

s i z e o f ←֓

} else {

78 79 80 81

d_data = c l C r e a t e B u f f e r ( cont , C L _ M E M _ R E A D _ W R I T E , f l o a t ) , N U L L , &e r r N u m ) ; f l o a t p a t t e r n =0; c l E n q u e u e F i l l B u f f e r ( q , d _ d a t a , ( v o i d * )&p a t t e r n , f l o a t ) , 0 , s i z e o f ( f l o a t ) , 0 , NULL , NULL ) ; d a t a = new f l o a t [ 1 ] ;

82

83 84

85

s i z e o f ( ←֓ s i z e o f ( ←֓

}

86 87

if

( OCL2support ) errNum = c l S e t K e r n e l A r g S V M P o i n t e r ( kern , 0 , data ) ; else e r r N u m = c l S e t K e r n e l A r g ( k e r n , 0 , s i z e o f ( c l _ m e m ) , ( v o i d * ) &←֓ d_data ) ;

88 89 90 91

92

e r r N u m |= c l S e t K e r n e l A r g ( k e r n , 1 , s i z e o f ( f l o a t ) , ( v o i d * ) &s t ←֓ ); e r r N u m |= c l S e t K e r n e l A r g ( k e r n , 2 , s i z e o f ( f l o a t ) , ( v o i d * ) &e n ←֓ ); e r r N u m |= c l S e t K e r n e l A r g ( k e r n , 3 , s i z e o f ( i n t ) , ( v o i d * ) &d i v ) ←֓ ;

93

94

95

96

if

97

( isError ( errNum , ” Failed to s e t k e r n e l parameters ” ) ) {

98

cleanUp () ; return 1;

99 100

}

101 102

s i z e _ t i d x S p a c e [ ] = { c u * 256 } ; s i z e _ t l o c a l W o r k S i z e [ ] = { 256 } ;

103 104 105

// l a u n c h GPU k e r n e l e r r N u m = c l E n q u e u e N D R a n g e K e r n e l ( q , k e r n , 1 , N U L L , i d x S p a c e , ←֓ l o c a l W o r k S i z e , 0 , N U L L , &k e r n C o m p l e t e ) ; i f ( isError ( errNum , ” Failed to launch k e r n e l ” ) ) { cleanUp () ; return 1; }

106 107

108 109 110 111 112 113

if

( ! OCL2support ) e r r N u m = c l E n q u e u e R e a d B u f f e r ( q , d _ d a t a , C L _ T R U E , 0 , s i z e o f ( ←֓ f l o a t ) , data , 0 , NULL , NULL ) ; else { //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // w a i t f o r t h e k e r n e l t o f i n i s h cl_event evlist [ ] = { kernComplete }; errNum = clWaitForEvents (1 , evlist ) ; i f ( isError ( errNum , ” Failed to wait f o r event ” ) ) { cleanUp () ; return 1; } }

114 115

116 117 118 119 120 121 122 123 124 125 126 127 128

// c l e a n −up a l l o c a t e d memory f l o a t r e s=* d a t a ; cleanUp () ; return res ;

129 130 131 132 133

}

134 135

// ************************************************************

136 137 138

i n t main ( i n t argc , char * argv [ ] ) {

147 f l o a t st , e n ; int division ;

139 140 141

st = atof ( argv [ 1 ] ) ; en = atof ( argv [ 2 ] ) ; division = atoi ( argv [ 3 ] ) ;

142 143 144 145

c o u t << ” R e s u l t

146

i s : ” << h o s t F E ( st , en , d i v i s i o n ) << e n d l ;

147

return 0;

148 149

}

Device code: 1 2 3 4

float f( float x) { return x * x + 2 * sin ( x ) ; }

5 6 7 8 9 10 11 12

v o i d a t o m i c _ f l o a t _ a d d i t i o n ( v o l a t i l e a t o m i c _ f l o a t * ptr , f l o a t a ) { f l o a t o l d v=a t o m i c _ l o a d ( p t r ) ; f l o a t n e w v=o l d v+a ; w h i l e ( ! a t o m i c _ c o m p a r e _ e x c h a n g e _ s t r o n g ( ptr , &o l d v , n e w v ) ) n e w v=o l d v+a ; }

13 14

15 16 17 18

k e r n e l v o i d i n t e g r a t e ( g l o b a l a t o m i c _ f l o a t * res , f l o a t st , en , i n t d i v ) { size_t ID = g e t _ g l o b a l _ i d ( 0 ) ; size_t localID = get_local_id (0) ; i n t t o t a l W I = g e t _ g l o b a l _ s i z e ( 0 ) ; // t o t a l work i t e m s

f l o a t ←֓

19

f l o a t localRes = 0; f l o a t step = ( en − st ) / div ; f l o a t stride = totalWI * step ; f l o a t x = s t + ( I D +1) * s t e p ; // s k i p f ( s t ) by a d d i n g 1 t o ID

20 21 22 23 24

f o r ( i n t i = I D ; i < div −1; i+=t o t a l W I ) { l o c a l R e s += f ( x ) ; x += s t r i d e ; } // f i r s t work i t e m i n t h e whole i n d e x r a n g e i f ( I D==0) l o c a l R e s += ( f ( s t ) + f ( e n ) ) / 2 ;

25 26 27 28 29 30 31 32 33

l o c a l R e s *= s t e p ;

34 35

// r e d u c e t h e work group l o c a l r e s u l t f l o a t workGroupSum = work_group_reduce_add ( localRes ) ;

36 37 38

i f ( l o c a l I D ==0) p r i n t f ( ”WG % l i

39 40

: %f \n” , g e t _ g r o u p _ i d ( 0 ) , w o r k G r o u p S u m ) ;

41

// f i r s t work i t e m i n grpup add t h e l o c a l sum i f ( l o c a l I D ==0) a t o m i c _ f l o a t _ a d d i t i o n ( res , w o r k G r o u p S u m ) ;

42 43 44 45

}

3. Use the R 1solution of the previous exercise to calculate π. It is known that π = 4 0 x21+1 dx. Test your program with single and double floating point precision arithmetic. How does this change influence the accuracy and performance? Answer To accommodate the double-precision calculation, one needs to change the float type variables to double. OpenCL requires an extension for handling 64-bit atomics, which can be enabled by entering in the device code the following lines: 1

#pragma OPENCL EXTENSION c l k h r i n t 6 4 b a s e a t o m i c s

: enable

148CHAPTER 7. GPU AND ACCELERATOR PROGRAMMING : OPENCL 2

#pragma OPENCL EXTENSION c l k h r i n t 6 4 e x t e n d e d a t o m i c s

: enable

4. Write an OpenCL program to calculate the outer product of two vectors.

If we have two vectors U = (u1 , u2 , . . . , un ) and V = (v1 , v2 , . . . , vm ), their outer product is a matrix nxm:   u1 v 1 u1 v 2 . . . u1 v m  u2 v 1 u2 v 2 . . . u2 v m    (7.1) U ⊗V = . .. ..  ..  .. . . .  un v 1

u1 v 2

...

un v m

Make your program accept as command-line parameters the n and m sizes, so that you can experiment with different workloads.

Answer Host code: 1 2 3 4

#i n c l u d e < s t d l i b . h> #i n c l u d e <i o s t r e a m > #i n c l u d e <s t d i o . h> #i n c l u d e <memory>

5 6

#i n c l u d e ” . . / c l u t i l i t y . h”

7 8

u s i n g namespace s t d ;

9 10 11 12 13 14 15 16

static static static static static static static

cl_context cont = 0; cl_command_queue q = 0; c l _ p r o g r a m pr = 0 ; cl_kernel kern = 0; cl_mem d_u = 0 ; cl_mem d_v = 0 ; cl_mem d_outer = 0;

// i n i t i a l i z e

f o r cleanup check

17 18 19 20 21 22

// ************************************************************ void cleanUp ( ) { i f ( k e r n != 0 ) clReleaseKernel ( kern ) ;

23 24

if

( p r != 0 ) c l R e l e a s e P r o g r a m ( pr ) ;

if

( q != 0 ) clReleaseCommandQueue ( q ) ;

25 26 27 28 29

if

30

( c o n t != 0 ) {

31

clReleaseMemObject ( d_u ) ; clReleaseMemObject ( d_v ) ; clReleaseMemObject ( d_outer ) ; clReleaseContext ( cont ) ;

32 33 34 35

}

36 37

}

38 39 40

41

42 43 44 45 46

// ************************************************************ // Host f r o n t −end f u n c t i o n t h a t a l l o c a t e s t h e memory and l a u n c h e s ←֓ t h e GPU k e r n e l // v o i d hostFE ( u n i q u e p t r < f l o a t [ ] > u , i n t n , u n i q u e p t r < ←֓ f l o a t [ ] > v , i n t m, u n i q u e p t r < f l o a t [ ] > o u t e r ) void hostFE ( f l o a t * u , i n t n , f l o a t * v , i n t m , f l o a t * outer ) { cl_int errNum ; cl_device_id devID ; cl_event kernComplete ;

47 48

// c o n t e x t and queue s e t u p

149 s e t u p D e v i c e ( cont , q , NULL , devID ,

49

CL_DEVICE_TYPE_ALL ) ;

50

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // r e a d program s o u r c e and b u i l d i t i f ( ! s e t u p P r o g r a m A n d K e r n e l ( d e v I D , c o n t , ” o u t e r . c l ” , pr , ”←֓ outerProduct ” , kern ) ) { i f ( p r != 0 ) p r i n t f ( ” E r r o r : %s \n” , g e t C o m p i l a t i o n E r r o r ( pr , d e v I D ) ) ; cleanUp () ; return ; }

51 52 53

54 55 56 57 58 59 60 61

d _ u = c l C r e a t e B u f f e r ( c o n t , C L _ M E M _ R E A D _ O N L Y , n * s i z e o f ( f l o a t ←֓ ) , N U L L , &e r r N u m ) ; i f ( i s E r r o r ( e r r N u m , ” F a i l e d t o a l l o c a t e memory” ) ) { cleanUp () ; return ; } d _ v = c l C r e a t e B u f f e r ( c o n t , C L _ M E M _ R E A D _ O N L Y , m * s i z e o f ( f l o a t ←֓ ) , N U L L , &e r r N u m ) ; i f ( i s E r r o r ( e r r N u m , ” F a i l e d t o a l l o c a t e memory” ) ) { cleanUp () ; return ; } d _ o u t e r = c l C r e a t e B u f f e r ( c o n t , C L _ M E M _ R E A D _ W R I T E , n * m * ←֓ s i z e o f ( f l o a t ) , N U L L , &e r r N u m ) ; i f ( i s E r r o r ( e r r N u m , ” F a i l e d t o a l l o c a t e memory” ) ) { cleanUp () ; return ; }

62

63 64 65 66 67 68

69 70 71 72 73 74

75 76 77 78 79 80

e r r N u m = c l E n q u e u e W r i t e B u f f e r ( q , d_u , C L _ T R U E , 0 , n * s i z e o f ( ←֓ f l o a t ) , u , 0 , NULL , NULL ) ; e r r N u m = c l E n q u e u e W r i t e B u f f e r ( q , d_v , C L _ T R U E , 0 , m * s i z e o f ( ←֓ f l o a t ) , v , 0 , NULL , NULL ) ;

81

82

83

e r r N u m |= c l S e t K e r n e l A r g d_u ) ; e r r N u m |= c l S e t K e r n e l A r g e r r N u m |= c l S e t K e r n e l A r g d_v ) ; e r r N u m |= c l S e t K e r n e l A r g e r r N u m |= c l S e t K e r n e l A r g d_outer ) ;

84

85 86

87 88

( k e r n , 0 , s i z e o f ( c l _ m e m ) , ( v o i d * ) &←֓ ( k e r n , 1 , s i z e o f ( i n t ) , ( v o i d * ) &n ) ; ( k e r n , 2 , s i z e o f ( c l _ m e m ) , ( v o i d * ) &←֓ ( k e r n , 3 , s i z e o f ( i n t ) , ( v o i d * ) &m ) ; ( k e r n , 4 , s i z e o f ( c l _ m e m ) , ( v o i d * ) &←֓

89

if

90

( isError ( errNum , ” Failed to s e t k e r n e l parameters ” ) ) {

91

cleanUp () ; return ;

92 93

}

94 95

s i z e _ t d i m X = ( n + 1 5 ) / 16 * 1 6 ; s i z e _ t d i m Y = ( m + 1 5 ) / 16 * 1 6 ; c o u t << d i m X << ” ” << d i m Y << e n d l ; size_t i d x S p a c e [ ] = { dimX , dimY } ; s i z e _ t l o c a l W o r k S i z e [ ] = { 1 6 , 16 } ;

96 97 98 99 100 101

// l a u n c h GPU k e r n e l e r r N u m = c l E n q u e u e N D R a n g e K e r n e l ( q , k e r n , 2 , N U L L , i d x S p a c e , ←֓ l o c a l W o r k S i z e , 0 , N U L L , &k e r n C o m p l e t e ) ; i f ( isError ( errNum , ” Failed to launch k e r n e l ” ) ) { cleanUp () ; return ; }

102 103

104 105 106 107 108 109

e r r N u m = c l E n q u e u e R e a d B u f f e r ( q , d _ o u t e r , C L _ T R U E , 0 , n * m * ←֓ s i z e o f ( f l o a t ) , outer , 0 , NULL , NULL ) ;

110

111

}

150CHAPTER 7. GPU AND ACCELERATOR PROGRAMMING : OPENCL 112 113

// ************************************************************

114 115 116 117

i n t main ( i n t argc , char * argv [ ] ) { int n , m ;

118

n = atof ( argv [ 1 ] ) ; m = atof ( argv [ 2 ] ) ;

119 120 121

unique_ptr < f l o a t [ ] > u = make_unique < f l o a t [ ] > ( n ) ; unique_ptr < f l o a t [ ] > v = make_unique < f l o a t [ ] > ( m ) ; unique_ptr < f l o a t [ ] > outer = make_unique < f l o a t [ ] > ( n * m ) ;

122 123 124 125

f o r ( i n t i = 0 ; i < n ; i++) u[i] = i; f o r ( i n t i = 0 ; i < m ; i++) v[i] = i;

126 127 128 129 130

hostFE ( u . get ( ) , n , v . get ( ) , m , outer . get ( ) ) ;

131 132

// p r i n t −o u t t h e r e s u l t f o r ( i n t j = 0 ; j < m ; j++) { f o r ( i n t i = 0 ; i < n ; i++) c o u t << o u t e r [ j * n + i ] << ” ” ; c o u t << e n d l ; }

133 134 135 136 137 138 139 140

cleanUp () ; return 0;

141 142 143

}

Device code: 1

2 3 4

kernel void o u t e r P r o d u c t ( global f l o a t *u , int m , global f l o a t * outer ) { size_t xID = get_global_id (0) ; size_t yID = get_global_id (1) ;

int n , global

f l o a t * v , ←֓

5

i f ( x I D < n && y I D < m ) { outer [ yID * n + xID ] = u [ xID ] * v [ yID ] ; }

6 7 8 9 10

}

5. Write an OpenCL program to calculate the inner product of two vectors.

If we have two vectors U = (u1 , u2 , . . . , un ) and V = (v1 , v2 , . .P . , vn ), their n inner or dot product is a scalar defined as the sum: U · V = i=1 ui · vi .

Experiment with two kernels: one that assigns one work item per product ui · vi , and one that assigns multiple products per work item. Which of the two kernels performs best?

Also try running your program with different work group sizes, e.g. 32, 64, 128, 256, 512. How is the execution time affected?

Answer Host code: 1 2 3 4

#i n c l u d e < s t d l i b . h> #i n c l u d e <i o s t r e a m > #i n c l u d e <s t d i o . h> #i n c l u d e <memory>

5 6

#i n c l u d e ” . . / c l u t i l i t y . h”

7 8 9

u s i n g namespace s t d ;

151 10 11 12 13 14 15 16

static static static static static static static

cl_context cont = 0; cl_command_queue q = 0; c l _ p r o g r a m pr = 0 ; cl_kernel kern = 0; cl_mem d_u = 0 ; cl_mem d_v = 0 ; cl_mem d_inner = 0;

// i n i t i a l i z e

f o r cleanup check

17 18 19 20 21 22

// ************************************************************ void cleanUp ( ) { i f ( k e r n != 0 ) clReleaseKernel ( kern ) ;

23 24

if

( p r != 0 ) c l R e l e a s e P r o g r a m ( pr ) ;

if

( q != 0 ) clReleaseCommandQueue ( q ) ;

25 26 27 28 29

if

30

( c o n t != 0 ) {

31

clReleaseMemObject ( d_u ) ; clReleaseMemObject ( d_v ) ; clReleaseMemObject ( d_inner ) ; clReleaseContext ( cont ) ;

32 33 34 35

}

36 37

}

38 39 40

41

42 43 44 45

// ************************************************************ // Host f r o n t −end f u n c t i o n t h a t a l l o c a t e s t h e memory and l a u n c h e s ←֓ t h e GPU k e r n e l v o i d h o s t F E ( f l o a t * u , i n t n , f l o a t * v , f l o a t &i n n e r , i n t ←֓ groupSize ) { cl_int errNum ; cl_device_id devID ; cl_event kernComplete ;

46 47 48

// c o n t e x t and queue s e t u p s e t u p D e v i c e ( cont , q , NULL , devID ,

CL_DEVICE_TYPE_ALL ) ;

49 50 51 52

53 54 55 56 57 58

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // r e a d program s o u r c e and b u i l d i t i f ( ! s e t u p P r o g r a m A n d K e r n e l ( d e v I D , c o n t , ” i n n e r . c l ” , pr , ”←֓ innerProduct ” , kern ) ) { i f ( p r != 0 ) p r i n t f ( ” E r r o r : %s \n” , g e t C o m p i l a t i o n E r r o r ( pr , d e v I D ) ) ; cleanUp () ; return ; }

59 60

61 62 63 64 65 66

67 68 69 70 71 72

73 74 75 76 77

d _ u = c l C r e a t e B u f f e r ( c o n t , C L _ M E M _ R E A D _ O N L Y , n * s i z e o f ( f l o a t ←֓ ) , N U L L , &e r r N u m ) ; i f ( i s E r r o r ( e r r N u m , ” F a i l e d t o a l l o c a t e memory” ) ) { cleanUp () ; return ; } d _ v = c l C r e a t e B u f f e r ( c o n t , C L _ M E M _ R E A D _ O N L Y , n * s i z e o f ( f l o a t ←֓ ) , N U L L , &e r r N u m ) ; i f ( i s E r r o r ( e r r N u m , ” F a i l e d t o a l l o c a t e memory” ) ) { cleanUp () ; return ; } d _ i n n e r = c l C r e a t e B u f f e r ( c o n t , C L _ M E M _ R E A D _ W R I T E , s i z e o f ( ←֓ f l o a t ) , N U L L , &e r r N u m ) ; i f ( i s E r r o r ( e r r N u m , ” F a i l e d t o a l l o c a t e memory” ) ) { cleanUp () ; return ; }

152CHAPTER 7. GPU AND ACCELERATOR PROGRAMMING : OPENCL e r r N u m = c l E n q u e u e W r i t e B u f f e r ( q , d_u , C L _ T R U E , 0 , n * s i z e o f ( ←֓ f l o a t ) , u , 0 , NULL , NULL ) ; e r r N u m = c l E n q u e u e W r i t e B u f f e r ( q , d_v , C L _ T R U E , 0 , n * s i z e o f ( ←֓ f l o a t ) , v , 0 , NULL , NULL ) ; f l o a t p a t t e r n =0; c l E n q u e u e F i l l B u f f e r ( q , d _ i n n e r , ( v o i d * )&p a t t e r n , s i z e o f ( f l o a t ←֓ ) , 0 , s i z e o f ( f l o a t ) , 0 , NULL , NULL ) ;

78

79

80 81

82

e r r N u m |= c l S e t K e r n e l A r g d_u ) ; e r r N u m |= c l S e t K e r n e l A r g e r r N u m |= c l S e t K e r n e l A r g d_v ) ; e r r N u m |= c l S e t K e r n e l A r g d_inner ) ;

83

84 85

86

( k e r n , 0 , s i z e o f ( c l _ m e m ) , ( v o i d * ) &←֓ ( k e r n , 1 , s i z e o f ( i n t ) , ( v o i d * ) &n ) ; ( k e r n , 2 , s i z e o f ( c l _ m e m ) , ( v o i d * ) &←֓ ( k e r n , 3 , s i z e o f ( c l _ m e m ) , ( v o i d * ) &←֓

87

if

88

( isError ( errNum , ” Failed to s e t k e r n e l parameters ” ) ) {

89

cleanUp () ; return ;

90 91

}

92 93

s i z e _ t d i m X = ( n + g r o u p S i z e −1) / g r o u p S i z e * g r o u p S i z e ; size_t idxSpace [ ] = { dimX }; size_t localWorkSize [ ] = { ( size_t ) groupSize };

94 95 96 97

// l a u n c h GPU k e r n e l e r r N u m = c l E n q u e u e N D R a n g e K e r n e l ( q , k e r n , 1 , N U L L , i d x S p a c e , ←֓ l o c a l W o r k S i z e , 0 , N U L L , &k e r n C o m p l e t e ) ; i f ( isError ( errNum , ” Failed to launch k e r n e l ” ) ) { cleanUp () ; return ; }

98 99

100 101 102 103 104 105

e r r N u m = c l E n q u e u e R e a d B u f f e r ( q , d _ i n n e r , C L _ T R U E , 0 , s i z e o f ( ←֓ f l o a t ) , &i n n e r , 0 , N U L L , N U L L ) ;

106

107

}

108 109

// ************************************************************

110 111 112 113

i n t main ( i n t argc , char * argv [ ] ) { int n , groupSize ;

114

n = atof ( argv [ 1 ] ) ; groupSize = atof ( argv [ 2 ] ) ;

115 116 117

unique_ptr < f l o a t [ ] > u = make_unique < f l o a t [ ] > ( n ) ; unique_ptr < f l o a t [ ] > v = make_unique < f l o a t [ ] > ( n ) ; f l o a t inner ;

118 119 120 121

f o r ( i n t i = 0 ; i < n ; i++) u [ i ] = 1; f o r ( i n t i = 0 ; i < n ; i++) v [ i ] = 1;

122 123 124 125 126

hostFE ( u . get ( ) , n , v . get ( ) , inner ,

127

groupSize ) ;

128

// p r i n t −o u t t h e r e s u l t c o u t << i n n e r << e n d l ;

129 130 131

cleanUp () ; return 0;

132 133 134

}

Device code: 1 2 3 4 5 6

v o i d a t o m i c _ f l o a t _ a d d i t i o n ( v o l a t i l e a t o m i c _ f l o a t * ptr , f l o a t a ) { f l o a t o l d v=a t o m i c _ l o a d ( p t r ) ; f l o a t n e w v=o l d v+a ; w h i l e ( ! a t o m i c _ c o m p a r e _ e x c h a n g e _ s t r o n g ( ptr , &o l d v , n e w v ) ) n e w v=o l d v+a ;

153 7

}

8 9

10 11 12

kernel void i n n e r P r o d u c t ( global f l o a t *u , global v o l a t i l e f l o a t * outer ) { size_t xID = get_global_id (0) ; size_t localID = get_local_id (0) ;

int n , global

f l o a t * v , ←֓

13

float

14

l o c a l R e s =0;

15

// p e r f o r m c a l c u l a t i o n o n l y i f i f ( xID < n ) localRes = u [ xID ] * v [ xID ] ;

16 17 18

c o r r e s p o n d i n g data e x i s t

19

// r e d u c e t h e work group l o c a l r e s u l t f l o a t workGroupSum = work_group_reduce_add ( localRes ) ;

20 21 22

// f i r s t work i t e m i n group add t h e l o c a l sum i f ( l o c a l I D ==0) a t o m i c _ f l o a t _ a d d i t i o n ( outer , workGroupSum ) ;

23 24 25 26

}

6. Write an OpenCL program for calculating the histogram of a grayscale image. Use an OpenCL image object to represent the input data. Use one work item per pixel.

Answer Host code: 1 2 3 4

#i n c l u d e < s t d l i b . h> #i n c l u d e <i o s t r e a m > #i n c l u d e <s t d i o . h> #i n c l u d e <memory>

5 6 7

#i n c l u d e ” . . / c l u t i l i t y . h” #i n c l u d e ” . . / common/pgm . cpp ”

8 9

u s i n g namespace s t d ;

10 11 12 13 14 15 16 17 18 19 20 21

s t a t i c cl_context cont = 0; // i n i t i a l i z e f o r c l e a n u p c h e c k s t a t i c cl_command_queue q = 0; s t a t i c c l _ p r o g r a m pr = 0 ; s t a t i c cl_kernel kern = 0; s t a t i c cl_mem d_hist = 0; s t a t i c cl_mem d_inImg = 0; // ************************************************************ void cleanUp ( ) { i f ( k e r n != 0 ) clReleaseKernel ( kern ) ;

22 23

if

( p r != 0 ) c l R e l e a s e P r o g r a m ( pr ) ;

if

( q != 0 ) clReleaseCommandQueue ( q ) ;

if

( d _ i n I m g != 0 ) clReleaseMemObject ( d_inImg ) ;

if

( d _ h i s t != 0 ) clReleaseMemObject ( d_hist ) ;

if

( c o n t != 0 ) clReleaseContext ( cont ) ;

24 25 26 27 28 29 30 31 32 33 34 35 36 37

}

38 39 40

41

// ************************************************************ // Host f r o n t −end f u n c t i o n t h a t a l l o c a t e s t h e memory and l a u n c h e s ←֓ t h e GPU k e r n e l v o i d h o s t F E ( P G M I m a g e &img , i n t * h i s t )

154CHAPTER 7. GPU AND ACCELERATOR PROGRAMMING : OPENCL 42 43 44 45

{ cl_int errNum ; cl_device_id devID ; cl_event kernComplete ;

46 47

i n t n C o l = i m g . n u m _ c o l o r s +1;

48 49 50

// c o n t e x t and queue s e t u p s e t u p D e v i c e ( cont , q , NULL , devID ,

CL_DEVICE_TYPE_ALL ) ;

51 52 53 54

55 56 57 58 59 60

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // r e a d program s o u r c e and b u i l d i t i f ( ! s e t u p P r o g r a m A n d K e r n e l ( d e v I D , c o n t , ” h i s t o g r a m . c l ” , pr , ”←֓ histogram ” , kern ) ) { i f ( p r != 0 ) p r i n t f ( ” E r r o r : %s \n” , g e t C o m p i l a t i o n E r r o r ( pr , d e v I D ) ) ; cleanUp () ; return ; }

61 62 63 64 65 66 67 68 69 70

// image s p e c i f i c a t i o n cl_image_desc imgDescr ; imgDescr . image_type = CL_MEM_OBJECT_IMAGE2D ; imgDescr . image_width = img . x_dim ; imgDescr . image_height = img . y_dim ; imgDescr . image_row_pitch = 0; imgDescr . num_mip_levels = 0; imgDescr . num_samples = 0; imgDescr . mem_object = NULL ;

71 72 73 74

cl_image_format imgFmr ; imgFmr . image_channel_order = CL_R ; imgFmr . image_channel_data_type = CL_UNSIGNED_INT8 ;

75 76 77

78 79 80 81 82

// I n p u t image d _ i n I m g = c l C r e a t e I m a g e ( c o n t , C L _ M E M _ R E A D _ O N L Y | ←֓ C L _ M E M _ C O P Y _ H O S T _ P T R , &i m g F m r , &i m g D e s c r , i m g . p i x e l s , &←֓ errNum ) ; i f ( i s E r r o r ( e r r N u m , ” F a i l e d t o c r e a t e d e v i c e i n p u t image ” ) ) { cleanUp () ; return ; }

83 84 85

86

87 88 89 90 91

// Image s a m p l e r c l _ s a m p l e r _ p r o p e r t i e s p r o p [ ] = { C L _ S A M P L E R _ N O R M A L I Z E D _ C O O R D S , ←֓ CL_FALSE , 0 }; c l _ s a m p l e r s a m p l e r = c l C r e a t e S a m p l e r W i t h P r o p e r t i e s ( c o n t , p r o p , ←֓ &e r r N u m ) ; i f ( isError ( errNum , ” Failed to c r e a t e sampler ” ) ) { cleanUp () ; return ; }

92 93

94 95 96 97 98 99 100

d _ h i s t = c l C r e a t e B u f f e r ( c o n t , C L _ M E M _ R E A D _ O N L Y , n C o l * s i z e o f ←֓ ( i n t ) , N U L L , &e r r N u m ) ; i f ( i s E r r o r ( e r r N u m , ” F a i l e d t o a l l o c a t e memory” ) ) { cleanUp () ; return ; } i n t p a t t e r n =0; c l E n q u e u e F i l l B u f f e r ( q , d _ h i s t , ( v o i d * )&p a t t e r n , s i z e o f ( i n t ) , ←֓ 0 , nCol * s i z e o f ( i n t ) , 0 , NULL , NULL ) ;

101 102

103

104

105

e r r N u m |= c l S e t K e r n e l A r g d_inImg ) ; e r r N u m |= c l S e t K e r n e l A r g x_dim ) ; e r r N u m |= c l S e t K e r n e l A r g y_dim ) ; e r r N u m |= c l S e t K e r n e l A r g

( k e r n , 0 , s i z e o f ( c l _ m e m ) , ( v o i d * ) &←֓ ( k e r n , 1 , s i z e o f ( i n t ) , ( v o i d * ) &i m g . ←֓ ( k e r n , 2 , s i z e o f ( i n t ) , ( v o i d * ) &i m g . ←֓ ( k e r n , 3 , s i z e o f ( c l _ m e m ) , ( v o i d * ) &←֓

155 d_hist ) ; e r r N u m |= c l S e t K e r n e l A r g ( k e r n , 4 , s i z e o f ( i n t ) , ( v o i d * ) &n C o l ←֓ ); e r r N u m |= c l S e t K e r n e l A r g ( k e r n , 5 , s i z e o f ( c l _ s a m p l e r ) , ( v o i d ←֓ * ) &s a m p l e r ) ;

106

107

108

if

109

( isError ( errNum , ” Failed to s e t k e r n e l parameters ” ) ) {

110

cleanUp () ; return ;

111 112

}

113 114 115

size_t size_t size_t size_t

116 117 118 119

d i m X = ( i m g . x _ d i m + 1 5 ) / 16 * 1 6 ; d i m Y = ( i m g . y _ d i m + 1 5 ) / 16 * 1 6 ; i d x S p a c e [ ] = { dimX , dimY } ; l o c a l W o r k S i z e [ ] = { 1 6 , 16 } ;

120

// l a u n c h GPU k e r n e l e r r N u m = c l E n q u e u e N D R a n g e K e r n e l ( q , k e r n , 2 , N U L L , i d x S p a c e , ←֓ l o c a l W o r k S i z e , 0 , N U L L , &k e r n C o m p l e t e ) ; i f ( isError ( errNum , ” Failed to launch k e r n e l ” ) ) { cleanUp () ; return ; }

121 122

123 124 125 126 127 128

e r r N u m = c l E n q u e u e R e a d B u f f e r ( q , d _ h i s t , C L _ T R U E , 0 , n C o l * ←֓ s i z e o f ( i n t ) , hist , 0 , NULL , NULL ) ;

129

130

}

131 132

// ************************************************************

133 134 135 136

i n t main ( i n t argc , char * argv [ ] ) { PGMImage inImg ( argv [ 1 ] ) ;

137

i n t n C o l = i n I m g . n u m _ c o l o r s +1; unique_ptr < i n t [ ] > hist = make_unique < i n t [ ] > ( nCol ) ;

138 139 140

hostFE ( inImg , hist . get ( ) ) ;

141 142

// p r i n t −o u t t h e r e s u l t f o r ( i n t i =0; i<n C o l ; i++) c o u t << i << ” ” << h i s t [ i ] << e n d l ;

143 144 145 146

cleanUp () ; return 0;

147 148 149

}

Device code: 1

2 3 4 5 6

k e r n e l v o i d h i s t o g r a m ( r e a d _ o n l y i m a g e 2 d _ t in , global i n t * hist , i n t nCol , s a m p l e r _ t s ) { size_t xID = get_global_id (0) ; size_t yID = get_global_id (1) ; size_t xThreads = get_global_size (0) ; size_t yThreads = get_global_size (1) ;

int w ,

i n t h , ←֓

7

i f ( x I D < w && y I D < h ) { i n t 2 c o o r d = { xID , y I D } ; i n t 4 c o l o r = ( i n t 4 ) r e a d _ i m a g e i ( in , s , c o o r d ) ; i n t gray = color . x ; a t o m i c _ a d d ( h i s t+g r a y , 1 ) ; }

8 9 10 11 12 13 14 15

}

7. Modify the program of the previous exercise so that you process multiple pixels per work item.

Answer

156CHAPTER 7. GPU AND ACCELERATOR PROGRAMMING : OPENCL The only changes in the host code, affect the index space generation. The code below basically partitions the image across the y-axis amongst the CUs. 1 2 3 4 5 6

7 8 9 10 11

. . . v o i d h o s t F E ( P G M I m a g e &img , i n t * h i s t ) { // d i s c o v e r t h e number o f CUs c l _ u i n t cu ; e r r N u m = c l G e t D e v i c e I n f o ( d e v I D , C L _ D E V I C E _ M A X _ C O M P U T E _ U N I T S , ←֓ s i z e o f ( c l _ u i n t ) , ( v o i d * ) &cu , N U L L ) ; i f ( e r r N u m != C L _ S U C C E S S ) { cleanUp () ; return ; }

12

size_t size_t size_t size_t

13 14 15 16

dimX = 16; dimY = cu * 1 6 ; i d x S p a c e [ ] = { dimX , dimY } ; l o c a l W o r k S i z e [ ] = { 1 6 , 16 } ;

17 18 19

20

// l a u n c h GPU k e r n e l e r r N u m = c l E n q u e u e N D R a n g e K e r n e l ( q , k e r n , 2 , N U L L , i d x S p a c e , ←֓ l o c a l W o r k S i z e , 0 , N U L L , &k e r n C o m p l e t e ) ; . . .

In the device code, loops allow the iteration over the whole image: 1

2 3 4 5 6 7

k e r n e l v o i d h i s t o g r a m ( r e a d _ o n l y i m a g e 2 d _ t in , global i n t * hist , i n t nCol , s a m p l e r _ t s ) { size_t xID = get_global_id (0) ; size_t yID = get_global_id (1) ; size_t xThreads = get_global_size (0) ; size_t yThreads = get_global_size (1) ; size_t blockID = get_local_linear_id () ;

int w ,

i n t h , ←֓

8

f o r ( i n t i=x I D ; i<w ; i+= x T h r e a d s ) f o r ( i n t j=y I D ; j<h ; j+= y T h r e a d s ) { int2 coord = { i , j }; i n t 4 c o l o r = ( i n t 4 ) r e a d _ i m a g e i ( in , s , c o o r d ) ; i n t gray = color . x ; a t o m i c _ a d d ( h i s t+g r a y , 1 ) ; }

9 10 11 12 13 14 15 16 17

}

8. Modify the histogram calculation of the previous exercise so that your kernel uses local memory to speed up the process. How does your new version compare with the previous one in terms of execution time?

Answer The only changes in the host code, affect the addition of one more kernel parameter, for the dynamically allocated local memory: 1 2 3 4 5 6

. . . v o i d h o s t F E ( P G M I m a g e &img , i n t * h i s t ) { . . . e r r N u m |= c l S e t K e r n e l A r g ( k e r n , 6 , s i z e o f ( i n t ) * n C o l , N U L L ) ; . . .

In the device code, the local memory has to be initialized before the image can be processed. Fences separate the blocks of code that need to complete before the next one:

157 1

2 3 4 5 6 7 8

k e r n e l v o i d h i s t o g r a m ( r e a d _ o n l y i m a g e 2 d _ t in , i n t w , i n t h , ←֓ global i n t * hist , i n t nCol , s a m p l e r _ t s , local i n t * l o c H i s t ) { size_t xID = get_global_id (0) ; size_t yID = get_global_id (1) ; size_t xThreads = get_global_size (0) ; size_t yThreads = get_global_size (1) ; size_t localID = get_local_linear_id () ; size_t groupSize = get_local_size (0) * get_local_size (1) ;

9

// l o c a l memory i n i t f o r ( i n t i=l o c a l I D ; i<n C o l ; i+=g r o u p S i z e ) locHist [ i ]=0; work_group_barrier ( CLK_LOCAL_MEM_FENCE ) ;

10 11 12 13 14

// image p r o c e s s . f o r ( i n t i=x I D ; i<w ; i+= x T h r e a d s ) f o r ( i n t j=y I D ; j<h ; j+= y T h r e a d s ) { int2 coord = { i , j }; i n t 4 c o l o r = ( i n t 4 ) r e a d _ i m a g e i ( in , s , c o o r d ) ; i n t gray = color . x ; a t o m i c _ a d d ( l o c H i s t+g r a y , 1 ) ; }

15 16 17 18 19 20 21 22 23 24

work_group_barrier ( CLK_LOCAL_MEM_FENCE ) ;

25 26

// a d d i n g t h e l o c a l h i s t o g r a m t o t h e g l o b a l one f o r ( i n t i=l o c a l I D ; i<n C o l ; i+=g r o u p S i z e ) a t o m i c _ a d d ( h i s t+i , l o c H i s t [ i ] ) ;

27 28 29 30 31

}

9. Create OpenCL implementations of the gsl stats mean() and gsl stats variance() functions offered by the GNU Scientific Library, that produce the mean and variance statistics of a supplied array. To avoid issues with 64-bit extension support, you can settle for the following signatures: f l o a t g s l _ s t a t s _ m e a n ( c o n s t f l o a t D A T A [ ] , // P o i n t e r t o i n p u t ←֓ data size_t STRIDE , // S t e p u s e d t o r e a d t h e ←֓ i n p u t . Normally t h i s s h o u l d be s e t t o ←֓ 1. size_t N ) ; // S i z e o f DATA a r r a y f l o a t g s l _ s t a t s _ v a r i a n c e ( c o n s t f l o a t D A T A [ ] , // Same a s above . size_t STRIDE , size_t N ) ;

Answer The host code filters out the data that need to be send to the device, if STRIDE is different than 1, by copying the data to be used in a new array. This reduces the amount of data that need to be transferred. The host code is factorized to reduce the number of lines required. This is a substandard solution for execution efficiency, as the data are transferred twice: 1 2 3 4 5

#i n c l u d e < s t d l i b . h> #i n c l u d e <i o s t r e a m > #i n c l u d e <s t d i o . h> #i n c l u d e <memory> #i n c l u d e <l i m i t s >

6 7

#i n c l u d e ” . . / c l u t i l i t y . h”

8 9

u s i n g namespace s t d ;

10 11 12

static static

cl_context cont = 0; cl_command_queue q = 0;

// i n i t i a l i z e

f o r cleanup check

158CHAPTER 7. GPU AND ACCELERATOR PROGRAMMING : OPENCL 13 14 15 16 17 18 19 20 21 22

s t a t i c c l _ p r o g r a m pr = 0 ; s t a t i c cl_kernel kern = 0; s t a t i c cl_mem d_data = 0; s t a t i c cl_mem d_res = 0; c o n s t i n t G R O U P S I Z E =256; // ************************************************************ void cleanUp ( ) { i f ( k e r n != 0 ) clReleaseKernel ( kern ) ;

23 24

if

( p r != 0 ) c l R e l e a s e P r o g r a m ( pr ) ;

if

( q != 0 ) clReleaseCommandQueue ( q ) ;

if

( d _ d a t a != 0 ) clReleaseMemObject ( d_data ) ;

if

( d _ r e s != 0 ) clReleaseMemObject ( d_res ) ;

if

( c o n t != 0 ) clReleaseContext ( cont ) ;

25 26 27 28 29 30 31 32 33 34 35 36 37 38

}

39 40 41 42

43 44 45 46

// ************************************************************ f l o a t h o s t F E ( c o n s t f l o a t D A T A [ ] , s i z e _ t S T R I D E , s i z e _ t N , c h a r * ←֓ kernName ) { cl_int errNum ; cl_device_id devID ; cl_event kernComplete ;

47 48 49 50

// p r e p r o c e s s h o s t d a t a i n c a s e STRIDE i s n o t 1 f l o a t * ha ; // h o s t ( h * ) and d e v i c e ( d * ) p o i n t e r s f l o a t hres ;

51 52

u n i q u e _ p t r <f l o a t [ ] > t m p D a t a=n u l l p t r ;

53 54 55 56 57 58 59 60 61 62 63 64 65 66 67

int Neffective ; i f ( STRIDE !=1) { N e f f e c t i v e = ( N −1)/ S T R I D E + 1 ; t m p D a t a = m a k e _ u n i q u e <f l o a t [ ] > ( N e f f e c t i v e ) ; f o r ( i n t i =0 , j =0; i<N ; i+=S T R I D E , j++) tmpData [ j ] = DATA [ i ] ; ha = t m p D a t a . get ( ) ; } else { ha = ( f l o a t *) DATA ; Neffective = N ; }

68 69 70 71

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // c o n t e x t and queue s e t u p s e t u p D e v i c e ( cont , q , NULL , devID , C L _ D E V I C E _ T Y P E _ A L L ) ;

72 73 74 75

76 77 78 79 80 81

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // r e a d program s o u r c e and b u i l d i t i f ( ! s e t u p P r o g r a m A n d K e r n e l ( d e v I D , c o n t , ” s t a t i s t i c s . c l ” , pr , ←֓ kernName , kern ) ) { i f ( p r != 0 ) p r i n t f ( ” E r r o r : %s \n” , g e t C o m p i l a t i o n E r r o r ( pr , d e v I D ) ) ; cleanUp () ; return 0; }

82 83

d_data = c l C r e a t e B u f f e r ( cont , C L _ M E M _ R E A D _ O N L Y , s i z e o f ( f l o a t ) , N U L L , &e r r N u m ) ;

N e f f e c t i v e * ←֓

159 e r r N u m |= c l E n q u e u e W r i t e B u f f e r ( q , d _ d a t a , C L _ T R U E , 0 , ←֓ N e f f e c t i v e * s i z e o f ( f l o a t ) , ha , 0 , N U L L , N U L L ) ; i f ( i s E r r o r ( e r r N u m , ” F a i l e d t o a l l o c a t e and o r w r i t e d a t a ” ) ) { cleanUp () ; return 0; }

84

85 86 87 88 89 90

d_res = c l C r e a t e B u f f e r ( cont , C L _ M E M _ R E A D _ W R I T E , , N U L L , &e r r N u m ) ; f l o a t p a t t e r n =0; c l E n q u e u e F i l l B u f f e r ( q , d _ r e s , ( v o i d * )&p a t t e r n , 0 , s i z e o f ( f l o a t ) , 0 , NULL , NULL ) ;

91

92 93

s i z e o f ( f l o a t ) ←֓ s i z e o f ( f l o a t ) , ←֓

94

e r r N u m |= c l S e t K e r n e l A r g d_data ) ; e r r N u m |= c l S e t K e r n e l A r g Neffective ) ; e r r N u m |= c l S e t K e r n e l A r g d_res ) ; e r r N u m |= c l S e t K e r n e l A r g NULL ) ;

95

96

97

98

( k e r n , 0 , s i z e o f ( c l _ m e m ) , ( v o i d * ) &←֓ ( k e r n , 1 , s i z e o f ( i n t ) , ( v o i d * ) &←֓ ( k e r n , 2 , s i z e o f ( c l _ m e m ) , ( v o i d * ) &←֓ ( k e r n , 3 , s i z e o f ( f l o a t ) * G R O U P S I Z E , ←֓

99

if

100

( isError ( errNum , ” Failed to s e t k e r n e l parameters ” ) ) {

101

cleanUp () ; return 0;

102 103

}

104 105 106

// d i s c o v e r t h e number o f CUs c l _ u i n t cu ; e r r N u m = c l G e t D e v i c e I n f o ( d e v I D , C L _ D E V I C E _ M A X _ C O M P U T E _ U N I T S , ←֓ s i z e o f ( c l _ u i n t ) , ( v o i d * ) &cu , N U L L ) ; i f ( e r r N u m != C L _ S U C C E S S ) { cleanUp () ; return 0; }

107 108 109

110 111 112 113 114 115

// l i m i t t h e number o f work g r o u p s i f t h e y w i l l n o t have any ←֓ work i f ( cu * G R O U P S I Z E > N ) c u = ( N+G R O U P S I Z E −1)/ G R O U P S I Z E ; size_t i d x S p a c e [ ] = { G R O U P S I Z E * cu } ; size_t localWorkSize [ ] = { GROUPSIZE };

116

117 118 119 120 121

// l a u n c h GPU k e r n e l e r r N u m = c l E n q u e u e N D R a n g e K e r n e l ( q , k e r n , 1 , N U L L , i d x S p a c e , ←֓ l o c a l W o r k S i z e , 0 , N U L L , &k e r n C o m p l e t e ) ; i f ( isError ( errNum , ” Failed to launch k e r n e l ” ) ) { cleanUp () ; return 0; }

122 123

124 125 126 127 128 129

e r r N u m = c l E n q u e u e R e a d B u f f e r ( q , d _ r e s , C L _ T R U E , 0 , s i z e o f ( ←֓ f l o a t ) , &h r e s , 0 , N U L L , N U L L ) ; cleanUp () ; return hres / Neffective ;

130

131 132 133

}

134 135 136

137 138 139 140 141

142 143 144

// ************************************************************ f l o a t g s l _ s t a t s _ m e a n ( c o n s t f l o a t D A T A [ ] , s i z e _ t S T R I D E , s i z e _ t N←֓ ) { r e t u r n h o s t F E ( D A T A , S T R I D E , N , ”sum” ) ; } // ************************************************************ f l o a t g s l _ s t a t s _ v a r i a n c e ( c o n s t f l o a t D A T A [ ] , s i z e _ t S T R I D E , ←֓ size_t N ) { f l o a t s , s2 ; s = h o s t F E ( D A T A , S T R I D E , N , ”sum” ) ;

160CHAPTER 7. GPU AND ACCELERATOR PROGRAMMING : OPENCL 145 146 147 148 149 150 151 152

}

s 2 = h o s t F E ( D A T A , S T R I D E , N , ”sum2” ) ; int Neffective ; i f ( STRIDE !=1) N e f f e c t i v e = ( N −1)/ S T R I D E + 1 ; else Neffective = N ; r e t u r n ( s2 − s *s ) / N e f f e c t i v e ;

153 154

// ************************************************************

155 156 157 158 159

i n t main ( i n t argc , char * argv [ ] ) { f l o a t st , e n ; int N ;

160

st = atof ( argv [ 1 ] ) ; en = atof ( argv [ 2 ] ) ; N = atoi ( argv [ 3 ] ) ;

161 162 163 164

u n i q u e _ p t r <f l o a t [ ] > d a t a=m a k e _ u n i q u e <f l o a t [ ] > ( N ) ; f l o a t s c a l e= n u m e r i c _ l i m i t s <i n t > : : m a x ( ) / ( en−s t ) ; f o r ( i n t i =0; i<N ; i++) d a t a [ i ] = r a n d ( ) / s c a l e+ s t ;

165 166 167 168 169

p r i n t f ( ”Avg : % l f \n” , g s l _ s t a t s _ m e a n ( d a t a . g e t ( ) , 1 , N ) ) ; p r i n t f ( ” Var : % l f \n” , g s l _ s t a t s _ v a r i a n c e ( d a t a . g e t ( ) , 1 , N ) ) ;

170 171 172

return 0;

173 174

}

Device code: 1 2 3 4 5 6 7

v o i d a t o m i c _ f l o a t _ a d d i t i o n ( v o l a t i l e a t o m i c _ f l o a t * ptr , f l o a t a ) { f l o a t o l d v=a t o m i c _ l o a d ( p t r ) ; f l o a t n e w v=o l d v+a ; w h i l e ( ! a t o m i c _ c o m p a r e _ e x c h a n g e _ s t r o n g ( ptr , &o l d v , n e w v ) ) n e w v=o l d v+a ; }

8 9 10

11 12 13 14

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− kernel void sum ( global f l o a t *d , i n t N , global local f l o a t * localSum ) { size_t ID = g e t _ g l o b a l _ i d ( 0 ) ; size_t numThreads = get_global_size (0) ; size_t localID = get_local_linear_id () ;

f l o a t * t o t a l S u m , ←֓

15

localSum [ localID ]=0;

16 17

f o r ( i n t i=I D ; i<N ; i+=n u m T h r e a d s ) l o c a l S u m [ l o c a l I D ] += d [ i ] ;

18 19 20

float

21

g r o u p T o t a l=w o r k _ g r o u p _ r e d u c e _ a d d ( l o c a l S u m [ l o c a l I D ] ) ;

22

if

23 24 25

( l o c a l I D == 0 ) atomic_float_addition ( totalSum ,

groupTotal ) ;

}

26 27 28

29 30 31 32

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− kernel void sum2 ( global f l o a t *d , i n t N , global local f l o a t * localSum ) { size_t ID = g e t _ g l o b a l _ i d ( 0 ) ; size_t numThreads = get_global_size (0) ; size_t localID = get_local_linear_id () ;

33 34

localSum [ localID ]=0;

35 36 37 38 39

f o r ( i n t i=I D ; i<N ; i+=n u m T h r e a d s ) { f l o a t t m p=d [ i ] ; l o c a l S u m [ l o c a l I D ] += t m p * t m p ;

f l o a t * t o t a l S u m , ←֓

161 }

40 41

float

42

g r o u p T o t a l=w o r k _ g r o u p _ r e d u c e _ a d d ( l o c a l S u m [ l o c a l I D ] ) ;

43

if

44

( l o c a l I D == 0 ) atomic_float_addition ( totalSum ,

45 46

groupTotal ) ;

}

10. In the filtering example of Section ??, two kernels are employed in order to get the filtered data in order in the output array. Solve the problem using one kernel, by relaxing the keeping-the-order constraint.

Answer The only thing that needs to be modified, is that the kernel should atomically increment a global counter to get the next free position in the filtered data array: Host code: 1 2 3

#i n c l u d e <i o s t r e a m > #i n c l u d e <s t r i n g > #i n c l u d e <t i m e . h>

4 5

#i n c l u d e ” . . / c l u t i l i t y . h”

6 7

u s i n g namespace s t d ;

8 9 10 11 12 13 14 15

static static static static static static static

cl_context cont = 0; cl_command_queue q = 0; c l _ p r o g r a m pr = 0 ; cl_kernel kern = 0; i n t * data = 0; int * filteredData = 0; int * groupCounts = 0;

// i n i t i a l i z e

f o r cleanup check

16 17 18 19 20 21

//============================================ void cleanUp ( ) { i f ( k e r n != 0 ) clReleaseKernel ( kern ) ;

22 23

if

( p r != 0 ) c l R e l e a s e P r o g r a m ( pr ) ;

if

( q != 0 ) clReleaseCommandQueue ( q ) ;

24 25 26 27 28

if

29

( c o n t != 0 ) {

30 31

if

( data ) c l S V M F r e e ( cont , data ) ;

if

( filteredData ) c l S V M F r e e ( cont ,

filteredData ) ;

( groupCounts ) c l S V M F r e e ( cont ,

groupCounts ) ;

32 33 34 35 36

if

37 38 39

clReleaseContext ( cont ) ;

40

}

41 42

}

43 44 45 46 47 48 49

//============================================ i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { cl_int errNum ; cl_device_id devID ; cl_event countKernComplete , moveKernComplete ;

50 51 52 53

if

( argc < 2) { p r i n t f ( ”%s #num data \n” , a r g v [ 0 ] ) ;

162CHAPTER 7. GPU AND ACCELERATOR PROGRAMMING : OPENCL 54 55 56

return 0; } i n t N = atoi ( argv [ 1 ] ) ;

57 58 59

// c o n t e x t and queue s e t u p s e t u p D e v i c e ( cont , q , NULL , devID ) ;

60 61 62 63

64 65 66 67 68 69

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // r e a d program s o u r c e and b u i l d i t i f ( ! s e t u p P r o g r a m A n d K e r n e l ( d e v I D , c o n t , ” f i l t e r . c l ” , pr , ”←֓ f i l t e r O d d s ” , kern ) ) { i f ( p r != 0 ) p r i n t f ( ” E r r o r : %s \n” , g e t C o m p i l a t i o n E r r o r ( pr , d e v I D ) ) ; cleanUp () ; return 1; }

70 71 72 73

74 75 76 77 78 79 80

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // SVM memory a l l o c a t i o n i n t * d a t a = ( i n t * ) c l S V M A l l o c ( c o n t , C L _ M E M _ R E A D _ W R I T E , N * ←֓ s i z e o f ( i n t ) , 0) ; i f ( d a t a == N U L L ) { cleanUp () ; return 1; } f o r ( i n t i = 0 ; i < N ; i++) data [ i ] = i ;

81 82

83 84 85 86 87

i n t * f i l t e r e d D a t a = ( i n t * ) c l S V M A l l o c ( c o n t , C L _ M E M _ R E A D _ W R I T E ←֓ , N * s i z e o f ( i n t ) , 0) ; i f ( f i l t e r e d D a t a == N U L L ) { cleanUp () ; return 1; }

88 89 90

91 92 93 94 95 96

i n t numWorkGroups = ( N + 255) / 256; i n t * f i l t C o u n t = ( i n t * ) c l S V M A l l o c ( c o n t , C L _ M E M _ R E A D _ W R I T E , ←֓ s i z e o f ( i n t ) , 0) ; i f ( f i l t C o u n t == N U L L ) { cleanUp () ; return 1; } * f i l t C o u n t =0;

97 98 99

100 101 102

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // s p e c i f y t h e p a r a m e t e r s t h a t w i l l be u s e d f o r t h e k e r n e l ←֓ execution // s e t u p t h e work i t e m i n d e x s p a c e and group s i z e size_t idxSpace [ ] = { N }; s i z e _ t l o c a l W o r k S i z e [ ] = { 256 } ;

103 104 105 106 107 108 109 110 111 112 113

// enqueue k e r n e l e x e c u t i o n r e q u e s t errNum = c l S e t K e r n e l A r g S V M P o i n t e r ( kern , 0 , data ) ; e r r N u m |= c l S e t K e r n e l A r g ( k e r n , 1 , s i z e o f ( i n t ) , ( v o i d * ) &N ) ; e r r N u m |= c l S e t K e r n e l A r g S V M P o i n t e r ( k e r n , 2 , f i l t e r e d D a t a ) ; e r r N u m |= c l S e t K e r n e l A r g S V M P o i n t e r ( k e r n , 3 , f i l t C o u n t ) ; i f ( isError ( errNum , ” Failed to s e t k e r n e l 1 parameters ” ) ) { cleanUp () ; return 1; }

114 115

116 117 118 119 120 121

e r r N u m = c l E n q u e u e N D R a n g e K e r n e l ( q , k e r n , 1 , N U L L , i d x S p a c e , ←֓ l o c a l W o r k S i z e , 0 , N U L L , &c o u n t K e r n C o m p l e t e ) ; i f ( isError ( errNum , ” Failed to launch k e r n e l 1 ” ) ) { cleanUp () ; return 1; }

163 //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // w a i t f o r t h e k e r n e l t o f i n i s h cl_event evlist [ ] = { countKernComplete }; errNum = clWaitForEvents (1 , evlist ) ; i f ( isError ( errNum , ” Failed to wait f o r event ” ) ) { cleanUp () ; return 1; }

122 123 124 125 126 127 128 129 130 131

c o u t << ” F i l t e r e d d a t a : ” << * f i l t C o u n t << e n d l ;

132 133

f o r ( i n t i = 0 ; i < * f i l t C o u n t ; i++) c o u t << f i l t e r e d D a t a [ i ] << ” ” ; c o u t << e n d l ;

134 135 136 137

// r e l e a s e r e s o u r c e s cleanUp () ; return 0;

138 139 140 141

}

Device code: 1 2 3 4

i n l i n e bool predicate ( int i ) { return i % 2; }

5 6

7 8 9 10 11 12 13 14 15

k e r n e l v o i d f i l t e r O d d s ( g l o b a l i n t * src , i n t N , g l o b a l i n t * d e s t , ←֓ global v o l a t i l e int * filtCount ) { i n t ID = g e t _ g l o b a l _ i d ( 0 ) ; i f ( ID < N ) { i n t h a v e O d d = p r e d i c a t e ( src [ ID ] ) ; i f ( haveOdd ) { i n t offset = atomic_fetch_add ( filtCount , 1) ; dest [ offset ] = src [ ID ] ;

16

}

17

}

18 19

}

11. Conduct an experiment to find out how changing the image resolution and/or the values of the symbolic constants DEVICEALLOC and CPUALLOC influences the distribution of the load in the hybrid Mandelbrot set calculation program of Section ??.

Answer N/A 12. The hybrid Mandelbrot calculator of Section ?? partitions the target image row-wise. Modify the code so that the partitioning is done column-wise.

As there are more columns than rows in an image, do you think this will allow for better load balancing and faster execution?

Answer The device code remains identical. The changes affect mainly the LoadMonitor class and the way the computed data are moved to the QImage object: 1 2 3 4 5 6

. . . // ************************************************************ struct BlockDescr { double upperX , upperY , stepX , stepY ; i n t width , height ;

164CHAPTER 7. GPU AND ACCELERATOR PROGRAMMING : OPENCL 7 8 9 10

int startCol ; }; // ************************************************************ const int DEVICEALLOC = 64;

11 12 13 14 15 16 17 18 19 20 21

22 23 24 25 26 27 28 29

30 31 32

c l a s s LoadMonitor { private : mutex l ; i n t width , height ; double upperX , upperY ; int currentCol = 0; double xStep , yStep ; public : L o a d M o n i t o r ( d o u b l e uX , d o u b l e uY , d o u b l e lX , d o u b l e lY , i n t w , ←֓ i n t h ) : upperX ( uX ) , upperY ( uY ) , width ( w ) , height ( h ) { xStep = ( lX − upperX ) / width ; yStep = ( upperY − lY ) / height ; } bool getNextBlock ( BlockDescr & d , bool isDevice = f a l s e ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // r e t u r n s t r u e i f t h e d s t r u c t u r e i s p o p u l a t e d . F a l s e i f work i s ←֓ done bool LoadMonitor : : getNextBlock ( BlockDescr & d , bool isDevice ) { l o c k _ g u a r d < mutex > lg ( l ) ;

33

if

34 35

( c u r r e n t C o l == w i d t h ) return f a l s e ;

36

d . upperY = upperY ; d . upperX = upperX + xStep * currentCol ; d . stepX = xStep ; d . stepY = yStep ; d . height = height ; d . startCol = currentCol ; i n t t o A l l o c = ( w i d t h − c u r r e n t C o l >= D E V I C E A L L O C ) ? D E V I C E A L L O C ←֓ : width − currentCol ; c u r r e n t C o l+=t o A l l o c ; d . width = toAlloc ; return true ;

37 38 39 40 41 42 43

44 45 46 47

}

48 49 50 51 52 53 54 55 56 57 58 59

60 61 62

// ************************************************************ c l a s s HostFE { private : cl_context cont = 0; // i n i t i a l i z e f o r c l e a n u p c h e c k cl_command_queue q = 0; c l _ p r o g r a m pr = 0 ; cl_kernel kern = 0; bool OCL2support = true ; u n s i g n e d c h a r * d a t a = 0 ; // image d a t a cl_mem d_data = 0; // image d a t a i f OpenCL 2 . 0 i s n o t ←֓ supported shared_ptr < QImage > img ; shared_ptr < LoadMonitor > mon ; cl_device_type t ;

63 64 65 66

67 68 69 70 71 72 73 74 75

public : i n t c o l s P r o c e s s e d =0; H o s t F E ( s h a r e d _ p t r < Q I m a g e > img , mon , c l _ d e v i c e _ t y p e t ) { t h i s −>i m g=i m g ; t h i s −>m o n=m o n ; t h i s −>t=t ; } void operator ( ) ( ) ; void cleanUp ( ) ; };

s h a r e d _ p t r < L o a d M o n i t o r > ←֓

165 76 77 78 79 80

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void HostFE : : cleanUp ( ) { i f ( k e r n != 0 ) clReleaseKernel ( kern ) ;

81 82

if

( p r != 0 ) c l R e l e a s e P r o g r a m ( pr ) ;

if

( q != 0 ) clReleaseCommandQueue ( q ) ;

83 84 85 86 87

if

88

( c o n t != 0 ) {

89

if

( OCL2support ) c l S V M F r e e ( cont , data ) ; else clReleaseMemObject ( d_data ) ; clReleaseContext ( cont ) ;

90 91 92 93 94

}

95 96

}

97 98 99

100 101 102 103 104 105

// ************************************************************ // Host f r o n t −end f u n c t i o n t h a t a l l o c a t e s t h e memory and l a u n c h e s ←֓ t h e GPU k e r n e l void HostFE : : operator ( ) ( ) { BlockDescr work ; cl_int errNum ; cl_device_id devID ; cl_event kernComplete ;

106 107 108

// c o n t e x t and queue s e t u p s e t u p D e v i c e ( cont , q , NULL , devID , t ) ;

109 110 111 112

113 114 115 116 117 118

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // r e a d program s o u r c e and b u i l d i t i f ( ! s e t u p P r o g r a m A n d K e r n e l ( d e v I D , c o n t , ” k e r n e l . c l ” , pr , ”←֓ mandelKernel ” , kern ) ) { i f ( p r != 0 ) p r i n t f ( ” E r r o r : %s \n” , g e t C o m p i l a t i o n E r r o r ( pr , d e v I D ) ) ; cleanUp () ; return ; }

119 120 121 122 123

124

125

// c h e c k i f SVM i s s u p p o r t e d char attr [ 2 0 ] ; size_t attrLen ; c l G e t D e v i c e I n f o ( d e v I D , C L _ D E V I C E _ V E R S I O N , 2 0 , ( v o i d * ) a t t r , &←֓ attrLen ) ; i f ( a t t r [ 7 ] == ’ 1 ’ ) // C h a r a c t e r a t p o s i t i o n 7 i s t h e ←֓ OpenCL major v e r s i o n OCL2support = f a l s e ;

126 127 128

i n t threadX , threadY ;

129 130 131 132 133 134 135

136 137 138 139

140 141 142

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // main t h r e a d l o o p w h i l e ( mon−>g e t N e x t B l o c k ( w o r k , t r u e ) ) { c o l s P r o c e s s e d+=w o r k . w i d t h ; // memory a l l o c a t i o n and some i n i t i a l i z a t i o n s done o n l y ←֓ during the f i r s t i t e r a t i o n i f ( d a t a == 0 ) { i f ( OCL2support ) d a t a = ( u n s i g n e d c h a r * ) c l S V M A l l o c ( c o n t , ←֓ C L _ M E M _ R E A D _ W R I T E , w o r k . w i d t h * w o r k . h e i g h t * ←֓ s i z e o f ( unsigned char ) , 0) ; else { d _ d a t a = c l C r e a t e B u f f e r ( c o n t , C L _ M E M _ R E A D _ W R I T E , ←֓

166CHAPTER 7. GPU AND ACCELERATOR PROGRAMMING : OPENCL w o r k . w i d t h * w o r k . h e i g h t * s i z e o f ( u n s i g n e d ←֓ c h a r ) , N U L L , &e r r N u m ) ; d a t a = new u n s i g n e d c h a r [ w o r k . w i d t h * w o r k . h e i g h t ] ;

143

} threadX = ( i n t ) ceil ( work . width * 1.0 / THR_BLK_X ) ; threadY = ( i n t ) ceil ( work . height * 1.0 / THR_BLK_Y ) ; t h r e a d X = ( ( t h r e a d X + B L O C K _ S I D E − 1 ) / B L O C K _ S I D E ) * ←֓ BLOCK_SIDE ; // make s u r e t h a t a work i t e m ←֓ d i v i d e s the idxSpace evenly t h r e a d Y = ( ( t h r e a d Y + B L O C K _ S I D E − 1 ) / B L O C K _ S I D E ) * ←֓ BLOCK_SIDE ;

144 145 146 147

148

}

149 150 151 152 153 154 155

if

( OCL2support ) errNum = c l S e t K e r n e l A r g S V M P o i n t e r ( kern , 0 , data ) ; else e r r N u m = c l S e t K e r n e l A r g ( k e r n , 0 , s i z e o f ( c l _ m e m ) , ( v o i d ←֓ * ) &d _ d a t a ) ;

156 157

158

159

160

161

162

163 164 165 166 167

e r r N u m |= c l S e t K e r n e l A r g ( k e r n , * ) &w o r k . u p p e r X ) ; e r r N u m |= c l S e t K e r n e l A r g ( k e r n , * ) &w o r k . u p p e r Y ) ; e r r N u m |= c l S e t K e r n e l A r g ( k e r n , * ) &w o r k . s t e p X ) ; e r r N u m |= c l S e t K e r n e l A r g ( k e r n , * ) &w o r k . s t e p Y ) ; e r r N u m |= c l S e t K e r n e l A r g ( k e r n , work . width ) ; e r r N u m |= c l S e t K e r n e l A r g ( k e r n , work . height ) ; i f ( isError ( errNum , ” Failed to { cleanUp () ; return ; }

1 , s i z e o f ( d o u b l e ) , ( v o i d ←֓ 2 , s i z e o f ( d o u b l e ) , ( v o i d ←֓ 3 , s i z e o f ( d o u b l e ) , ( v o i d ←֓ 4 , s i z e o f ( d o u b l e ) , ( v o i d ←֓ 5 , s i z e o f ( i n t ) , ( v o i d * ) &←֓ 6 , s i z e o f ( i n t ) , ( v o i d * ) &←֓ s e t ke rne l parameters ” ) )

168 169 170

size_t idxSpace [ ] = { threadX , threadY }; size_t localWorkSize [ ] = { BLOCK_SIDE , BLOCK_SIDE };

171 172 173

174 175 176 177 178

// l a u n c h GPU k e r n e l e r r N u m = c l E n q u e u e N D R a n g e K e r n e l ( q , k e r n , 2 , N U L L , i d x S p a c e ←֓ , l o c a l W o r k S i z e , 0 , N U L L , &k e r n C o m p l e t e ) ; i f ( isError ( errNum , ” Failed to launch k e r n e l ” ) ) { cleanUp () ; return ; }

179 180 181

182 183 184 185 186 187 188 189 190 191 192 193

if

( ! OCL2support ) e r r N u m = c l E n q u e u e R e a d B u f f e r ( q , d _ d a t a , C L _ T R U E , 0 , w o r k ←֓ . w i d t h * w o r k . h e i g h t * s i z e o f ( u n s i g n e d c h a r ) , d a t a , ←֓ 0 , NULL , NULL ) ; else { //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // w a i t f o r t h e k e r n e l t o f i n i s h cl_event evlist [ ] = { kernComplete }; errNum = clWaitForEvents (1 , evlist ) ; i f ( isError ( errNum , ” Failed to wait f o r event ” ) ) { cleanUp () ; return ; } }

194 195 196 197 198 199 200 201

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // copy r e s u l t s i n t o QImage o b j e c t f o r ( i n t j = 0 ; j < w o r k . h e i g h t ; j++) f o r ( i n t i = 0 ; i < w o r k . w i d t h ; i++) { i n t color = data [ j * work . width + i ] ; img−>s e t P i x e l ( i + w o r k . s t a r t C o l , j , q R g b ( 2 5 6 − ←֓

167 c o l o r , 256 − c o l o r , 256 − c o l o r ) ) ; }

202

} // c l e a n −up a l l o c a t e d memory c e r r << t << ” P r o c e s s e d ” << c o l s P r o c e s s e d << ” rows \n” ; cleanUp () ;

203 204 205 206 207

}

168CHAPTER 7. GPU AND ACCELERATOR PROGRAMMING : OPENCL

Chapter 8

Shared-memory programming : OpenMP Exercises 1. Modify the program in Listing 4.1, so that printf is used instead of cout.

Do you see any difference in the output compared to the one reported in Section 4.2? Can you explain it?

Answer 1 2 3

#i n c l u d e < s t d l i b . h> #i n c l u d e <s t d i o . h> #i n c l u d e <omp . h>

4 5

u s i n g namespace s t d ;

6 7 8 9 10 11

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t numThr = atoi ( argv [ 1 ] ) ; #pragma omp p a r a l l e l n u m t h r e a d s ( numThr ) p r i n t f ( ” H e l l o from t h r e a d %i \n” , o m p _ g e t _ t h r e a d _ n u m ( ) ) ;

12

return 0;

13 14

}

The output does not appear mangled, as the printf strings are build and output as a unit, while the cout object goes through several method invocations.

2. In the matrix multiplication example of Section 4.4.2 we could get three perfectly nested loops, if we initialize the C matrix outside: double A [ K ] [ L ] ; double B [ L ] [ M ] ; double C [ K ] [ M ] ; . . . #pragma omp p a r a l l e l f o r c o l l a p s e ( 3 ) f o r ( i n t i = 0 ; i < K ; i++) f o r ( i n t j = 0 ; j < M ; j++) f o r ( i n t k = 0 ; k < L ; k++) C [ i ] [ j ] += A [ i ] [ k ] * B [ k ] [ j ] ;

Is the above code correct? If not, what kind of modification is required for fixing it?

169

170

CHAPTER 8. SHARED-MEMORY PROGRAMMING : OPENMP

j:0,M−1 0

1

2

3

4

5

i:1,K−2

1

2

3

4

Figure 8.1: Dependency graph for Exercise 3.

Answer The code is not correct, because for the inner loop C[i][j] is effectively a reduction variable. To eliminate the race condition, a critical section is needed: #pragma omp p a r a l l e l f o r c o l l a p s e ( 3 ) f o r ( i n t i = 0 ; i < K ; i++) f o r ( i n t j = 0 ; j < M ; j++) f o r ( i n t k = 0 ; k < L ; k++) #pragma omp c r i t i c a l C [ i ] [ j ] += A [ i ] [ k ] * B [ k ] [ j ] ;

In terms of performance, this will effectively make all threads execute sequentially, because unnamed critical constructs use the same mutex.

A better alternative is to make sure that each execution of the inner loop is assigned to a single thread, by using a schedule clause, which is equivalent to converting the collapse(3) clause to collapse(2): #pragma omp p a r a l l e l f o r c o l l a p s e ( 3 ) s c h e d u l e ( s t a t i c , L ) f o r ( i n t i = 0 ; i < K ; i++) f o r ( i n t j = 0 ; j < M ; j++) f o r ( i n t k = 0 ; k < L ; k++) C [ i ] [ j ] += A [ i ] [ k ] * B [ k ] [ j ] ;

3. Draw the iteration space dependency graph for the following program: f o r ( i n t i = 1 ; i < K −1; i++) f o r ( i n t j = 0 ; j < M ; j++) a [ i ] [ j ] = a [ i −1][ j ] + a [ i +1][ j ] ;

What kind of dependencies exist? How can you eliminate them?

Answer The ISDG is shown in Figure 8.1.

We have a flow dependence from a[i-1][j] and an anti-dependence for a[i+1][j]. The anti-dependence can be eliminated by creating a copy a2 of the original matrix a. The flow dependence can be honored (not removed) by parallelizing only the inner loop, resulting in the following code:

171 f o r ( i n t i = 0 ; i < K −1; i++) f o r ( i n t j = 0 ; j < M ; j++) a2 [ i ] [ j ] = a [ i +1][ j ] ; f o r ( i n t i = 1 ; i < K −1; i++) #pragma omp p a r a l l e l f o r f o r ( i n t j = 0 ; j < M ; j++) a [ i ] [ j ] = a [ i −1][ j ] + a2 [ i ] [ j ] ;

4. Create a C++ program for visualizing the thread iteration assignment performed by a parallel for directive, for different schedule schemes.

Your program should have a per-thread vector that accumulates the loop control variable values assigned to each thread. Use this program, and appropriate schedule settings, to experiment with multiple schemes, without recompiling your program. This is a sample of the output you should be able to get for a loop of 100 iterations: e x p o r t O M P _ S C H E D U L E=” s t a t i c , 5 ” ./ solution T h r e a d 0 : 0 1 2 3 4 40 41 42 43 44 80 81 82 83 84 T h r e a d 1 : 5 6 7 8 9 45 46 47 48 49 85 86 87 88 89 T h r e a d 2 : 10 11 12 13 14 50 51 52 53 54 90 91 92 93 94 T h r e a d 3 : 15 16 17 18 19 55 56 57 58 59 95 96 97 98 99 T h r e a d 4 : 20 21 22 23 24 60 61 62 63 64 T h r e a d 5 : 25 26 27 28 29 65 66 67 68 69 T h r e a d 6 : 30 31 32 33 34 70 71 72 73 74 T h r e a d 7 : 35 36 37 38 39 75 76 77 78 79

$

$

Answer The solution is based on having a vector per thread (line 11), to store the values of the loop control variable (line 18): 1 2 3 4

#i n c l u d e <i o s t r e a m > #i n c l u d e < s t d l i b . h> #i n c l u d e <v e c t o r > #i n c l u d e <omp . h>

5 6

u s i n g namespace s t d ;

7 8 9 10 11

i n t main () { int N = omp_get_max_threads () ; v e c t o r < i n t >* s t o r e = new v e c t o r < i n t >[ N ] ;

12 13 14 15 16 17 18 19

#pragma omp p a r a l l e l f o r s c h e d u l e ( r u n t i m e ) f o r ( i n t k = 0 ; k < 1 0 0 ; k++) { i n t idx = omp_get_thread_num ( ) ; store [ idx ] . push_back ( k ) ; }

20

// o u t p u t f o r ( i n t i = 0 ; i < N ; i++) { c o u t << ” Thread ” << i << ” : ” ; f o r ( i n t j = 0 ; j < s t o r e [ i ] . s i z e ( ) ; j++) c o u t << s t o r e [ i ] [ j ] << ” ” ; c o u t << e n d l ; }

21 22 23 24 25 26 27 28 29

return 0;

30 31

}

5. In Listing 4.19 we examined the issue of processing a linked-list concurrently. Is there a way to improve this program for a doubly-linked list?

Write the corresponding program.

172

CHAPTER 8. SHARED-MEMORY PROGRAMMING : OPENMP Answer In the following code, the list is traversed in two opposite directions, allowing the generation of two tasks per iteration of the while loop of lines 61-75.

1 2 3 4 5

#i n c l u d e <i o s t r e a m > #i n c l u d e <math . h> #i n c l u d e < s t d l i b . h> #i n c l u d e <omp . h> #i n c l u d e <u n i s t d . h>

6 7

u s i n g namespace s t d ;

8 9 10 11 12 13 14 15 16 17

18

19 20 21 22 23

// t e m p l a t e s t r u c t u r e f o r a l i s t ’ s node template < c l a s s T > s t r u c t Node { T info ; Node * next ; Node * prev ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // Appends a v a l u e a t t h e end o f a l i s t p o i n t e d by t h e head * h ←֓ and t a i l * t t e m p l a t e < c l a s s T > v o i d a p p e n d ( i n t v , N o d e < T > ** h , N o d e < T←֓ > ** t ) { N o d e < T > * t m p = new N o d e < T > ( ) ; tmp−>i n f o = v ; tmp−>n e x t = N U L L ; tmp−>p r e v = * t ;

24

if

25

{

26

( * t == N U L L )

27 28

} else {

29 30 31 32 33

}

34 35

*h = tmp ; *t = tmp ;

( * t )−>n e x t = t m p ; *t = tmp ;

}

36 37 38 39 40 41 42

43

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // f u n c t i o n s t u b f o r p r o c e s s i n g a node ’ s d a t a template < c l a s s T > void process ( Node < T > *p ) { #pragma omp c r i t i c a l c o u t << p−>i n f o << ” by t h r e a d ” << o m p _ g e t _ t h r e a d _ n u m ( ) << ←֓ endl ; }

44 45 46 47 48 49 50 51 52 53

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t main ( i n t argc , char * argv [ ] ) { // b u i l d a sample l i s t i n t N = atoi ( argv [ 1 ] ) ; N o d e < i n t >* h e a d = N U L L ; N o d e < i n t >* t a i l = N U L L ; f o r ( i n t i = 0 ; i < N ; i++) a p p e n d ( i , &h e a d , &t a i l ) ;

54 55 56 57 58 59 60 61 62 63 64 65

#pragma omp p a r a l l e l { #pragma omp s i n g l e { N o d e < i n t >* t m p 1 = h e a d ; N o d e < i n t >* t m p 2 = t a i l ; w h i l e ( t m p 1 != N U L L && t m p 1 != t m p 2 ) { #pragma omp t a s k process ( tmp1 ) ; #pragma omp t a s k

173 process ( tmp2 ) ;

66 67

t m p 1 = t m p 1−>n e x t ; i f ( t m p 1 == t m p 2 ) check { tmp1 = NULL ; break ; } t m p 2 = t m p 2−>p r e v ;

68 69

70 71 72 73 74

} // p r o c e s s t h e l a s t node , list i f ( t m p 1 != N U L L ) process ( tmp1 ) ;

75 76

77 78

// even−s i z e d

list

t e r m i n a t i o n ←֓

i n t h e m i d d l e o f an odd−s i z e d ←֓

}

79

}

80 81

return 0;

82 83

}

6. Write a program for traversing and processing the elements of a binary tree in parallel, using a pre-order, in-order or post-order traversal. Use the task construct to that effect.

Answer In-order traversal cannot be parallelized as in enforces a total order between the tree nodes.

1 2 3 4 5 6 7 8 9 10 11 12

//−−−−−−−−−−−−−−−−−−−−−−−− t e m p l a t e < typename T > s t r u c t T r e e N o d e { T value ; TreeNode < T > * left ; TreeNode < T > * right ; }; //−−−−−−−−−−−−−−−−−−−−−−−− // empty p l a c e h o l d e r f o r a p r o c e s s i n g f u n c t i o n t e m p l a t e < typename T > v o i d p r o c e s s S t u b ( T n ) { }

13 14

//−−−−−−−−−−−−−−−−−−−−−−−−

15 16

17 18 19

t e m p l a t e < typename T > v o i d p r e O r d e r ( T r e e N o d e < T > * n , v o i d * ←֓ process ( T n ) ) { i f ( n == N U L L ) return ;

20 21 22 23 24 25 26 27

p r o c e s s ( n−>v a l u e ) ; #pragma omp p a r a l l e l { #pragma omp s i n g l e { #pragma omp t a s k p r e O r d e r ( n−>l e f t , p r o c e s s ) ;

28 29 30 31 32 33 34

// keep u s i n g t h i s t h r e a d f o r t h e o t h e r c h i l d p r e O r d e r ( n−>r i g h t , p r o c e s s ) ; #pragma omp t a s k w a i t } } }

35 36

//−−−−−−−−−−−−−−−−−−−−−−−−

37 38

39 40

t e m p l a t e < typename T > v o i d p o s t O r d e r ( T r e e N o d e < T > * n , v o i d * ←֓ process ( T n ) ) { i f ( n == N U L L )

174

CHAPTER 8. SHARED-MEMORY PROGRAMMING : OPENMP return ;

41 42 43 44 45 46 47 48

#pragma omp p a r a l l e l { #pragma omp s i n g l e { #pragma omp t a s k p o s t O r d e r ( n−>l e f t , p r o c e s s ) ;

49 50 51 52 53 54 55 56

// keep u s i n g t h i s t h r e a d f o r t h e o t h e r c h i l d p o s t O r d e r ( n−>r i g h t , p r o c e s s ) ; #pragma omp t a s k w a i t } } p r o c e s s ( n−>v a l u e ) ; }

57 58

//−−−−−−−−−−−−−−−−−−−−−−−−

59 60

61 62

63 64 65

// i n O r d e r c a n n o t be p a r a l l e l i z e d a s i n e n f o r c e s a t o t a l o r d e r ←֓ between t h e t r e e n o d e s . // p o s t O r d e r and p r e O r d e r o n l y have a p a r t i a l o r d e r i n g t e m p l a t e < typename T > v o i d i n O r d e r ( T r e e N o d e < T > * n , v o i d * ←֓ process ( T n ) ) { i f ( n == N U L L ) return ;

66

i n O r d e r ( n−>l e f t , p r o c e s s ) ; p r o c e s s ( n−>v a l u e ) ; i n O r d e r ( n−>r i g h t , p r o c e s s ) ;

67 68 69 70 71

}

7. Modify the program of the previous exercise, so that undeferred tasks are generated after the traversal has moved beyond the fifth level of the tree (assume that the root sits at level 0).

Answer The required modifications relative to the answer of the previous question, are shown below. The solution is based on the introduction of a function parameter for counting the depth of the recursion/depth of the tree nodes examined. The default value of “0” means that the caller does not need to be aware of this parameter.

1

2 3 4

t e m p l a t e < typename T > v o i d p r e O r d e r ( T r e e N o d e < T > * n , v o i d * ←֓ p r o c e s s ( T n ) , i n t l e v e l =0) { i f ( n == N U L L ) return ;

5 6 7 8 9 10 11 12

p r o c e s s ( n−>v a l u e ) ; #pragma omp p a r a l l e l { #pragma omp s i n g l e { #pragma omp t a s k i f ( l e v e l >=5) p r e O r d e r ( n−>l e f t , p r o c e s s , l e v e l +1) ;

13 14 15 16 17 18 19

// keep u s i n g t h i s t h r e a d f o r t h e o t h e r c h i l d p r e O r d e r ( n−>r i g h t , p r o c e s s , l e v e l +1) ; #pragma omp t a s k w a i t } } }

20 21

//−−−−−−−−−−−−−−−−−−−−−−−−

22 23

t e m p l a t e < typename T > v o i d p o s t O r d e r ( T r e e N o d e < T > * n , v o i d * ←֓ p r o c e s s ( T n ) , i n t l e v e l =0)

175 24 25 26

{ if

( n == N U L L ) return ;

27 28 29 30 31 32 33

#pragma omp p a r a l l e l { #pragma omp s i n g l e { #pragma omp t a s k i f ( l e v e l >=5) p o s t O r d e r ( n−>l e f t , p r o c e s s , l e v e l +1) ;

34 35 36 37 38 39 40 41

// keep u s i n g t h i s t h r e a d f o r t h e o t h e r c h i l d p o s t O r d e r ( n−>r i g h t , p r o c e s s , l e v e l +1) ; #pragma omp t a s k w a i t } } p r o c e s s ( n−>v a l u e ) ; }

8. Modify the Fibonacci sequence-calculating program of Listing 4.20, so that it counts the number of child tasks generated.

Answer Use of an atomic pragma is required for properly incrementing the shared counter. Only changes relative to Listing 4.20 are shown.

1

int numTasks = 0;

2 3 4 5 6 7 8 9 10 11 12 13 14

i n t fib ( i n t i ) { i n t t1 , t 2 ; i f ( i == 0 | | i == 1 ) return 1; else { #pragma omp t a s k s h a r e d ( t 1 ) { #pragma omp a t o m i c n u m T a s k s ++; t1 = fib ( i − 1) ;

i f ( i >25) m e r g e a b l e

15 16 17 18 19 20 21 22

} // keep t h r e a d f o r c a l l i n g t2 = fib ( i − 2) ; #pragma omp t a s k w a i t r e t u r n t1 + t2 ; } }

function again

9. Use the task directive to create a solution to the single producer, multiple consumers problem, with a variable number of consumers, as specified in the command-line.

Answer The following code solves the integration example that is presented in Section 4.5.1.1 and solved with a fixed number of consumers in Listing 4.18. To minimize clutter, we show only the main function that generates the consumer threads (via a for loop in lines 21-25) and executes the producer thread code (lines 28-52).

1 2 3 4 5 6

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i f ( a r g c == 1 ) { c e r r << ” Usage ” << a r g v [ 0 ] << ” #t h r e a d s #j o b s \n” ; exit (1) ;

176

CHAPTER 8. SHARED-MEMORY PROGRAMMING : OPENMP } i n t N = atoi ( argv [ 1 ] ) ; i n t J = atoi ( argv [ 2 ] ) ;

7 8 9 10 11 12 13 14 15 16

17 18 19 20 21 22 23 24

25

S l i c e * b u f f e r = new S l i c e [ B U F F S I Z E ] ; i n t in = 0 , out = 0 ; QSemaphore avail , buffSlots ( BUFFSIZE ) ; QMutex l , integLock ; double integral = 0 ; #pragma omp p a r a l l e l d e f a u l t ( none ) s h a r e d ( b u f f e r , i n , out , a v a i l , ←֓ b u f f S l o t s , l , i n t e g L o c k , i n t e g r a l , J , N, c o u t ) n u m t h r e a d s ( ←֓ N+1) { #pragma omp s i n g l e { // c o n s u m e r s ’ p a r t f o r ( i n t i = 0 ; i < N ; i++) { #pragma omp t a s k i n t e g r C a l c ( b u f f e r , b u f f S l o t s , a v a i l , l , out , i n t e g L o c k ←֓ , integral ) ; }

26 27 28 29 30 31 32 33 34 35

// p r o d u c e r t h r e a d , r e s p o n s i b l e f o r h an d i n g o u t ’ j o b s ’ double divLen = ( UPPERLIMIT − LOWERLIMIT ) / J ; d o u b l e st , e n d = L O W E R L I M I T ; f o r ( i n t i = 0 ; i < J ; i++) { st = end ; e n d += d i v L e n ; i f ( i == J − 1 ) end = UPPERLIMIT ;

36

buffSlots . acquire () ; buffer [ in ] . start = st ; buffer [ in ] . end = end ; buffer [ in ] . d i v i s i o n s = 1 0 0 0 ; in = ( in + 1) % B U F F S I Z E ; avail . release () ;

37 38 39 40 41 42

}

43 44

// put t e r m i n a t i o n s e n t i n e l s i n b u f f e r f o r ( i n t i = 0 ; i < N ; i++) { buffSlots . acquire () ; buffer [ in ] . d i v i s i o n s = 0 ; in = ( in + 1) % B U F F S I Z E ; avail . release () ; }

45 46 47 48 49 50 51 52 53 54 55 56 57

// w a i t f o r a l l t h r e a d s t o f i n i s h #pragma omp t a s k w a i t } }

58

c o u t << ” R e s u l t delete [ ] buffer ;

59 60

is

: ” << i n t e g r a l << e n d l ;

61

return 0;

62 63

}

10. Use the task directive and its depend clause, to model the dependency graph of Figure 4.12. You can use stub functions to represent each of the tasks in the figure.

Answer 1 2 3 4 5

#i n c l u d e <i o s t r e a m > #i n c l u d e < s t d l i b . h> #i n c l u d e <omp . h> #i n c l u d e <u n i s t d . h>

177

T1

T3

T2

T4

T6

T5

Figure 8.2: A dependency graph consisting of six tasks.

6

u s i n g namespace s t d ;

7 8 9 10

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { int x , y , z ;

11 12 13 14 15 16 17 18 19

#pragma omp p a r a l l e l { #pragma omp s i n g l e { #pragma omp t a s k s h a r e d ( x ) depend ( o u t : x ) { // T1 c o u t << ”T1\n” ; }

20 21 22 23 24

#pragma omp t a s k s h a r e d ( x ) depend ( o u t : y , z ) { // T3 c o u t << ”T3\n” ; }

25 26 27 28 29 30

#pragma omp t a s k s h a r e d ( x ) depend ( i n o u t { // T2 sleep (1) ; c o u t << ”T2\n” ; }

: x , y)

31 32 33 34 35

#pragma omp t a s k s h a r e d ( x ) depend ( i n { // T4 c o u t << ”T4\n” ; }

: y)

#pragma omp t a s k s h a r e d ( x ) depend ( i n { // T5 sleep (1) ; c o u t << ”T5\n” ; }

: y)

#pragma omp t a s k s h a r e d ( x ) depend ( i n { // T6 c o u t << ”T6\n” ; }

: z)

36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

#pragma omp t a s k w a i t } }

51 52 53

return 0; }

11. Finding the odd integers in a array can be accomplished by the following

178

CHAPTER 8. SHARED-MEMORY PROGRAMMING : OPENMP OpenMP code: i n t data [ N ] ; i n t o d d C o u n t =0; #pragma omp p a r a l l e l f o r f o r ( i n t i = 0 ; i < N ; i++) i f ( data [ i ] % 2 ) #pragma omp a t o m i c o d d C o u n t ++;

Modify the above code, so that there is no need for a critical or atomic directive, by introducing a counter array, with one element per thread.

Compare the performance of your version with and without cache false sharing. If we were to use a reduction variable, what false-sharing elimination technique, does this correspond to?

Answer The following listing contains four different versions of the code snippet in question, encapsulated in appropriately named functions: 1 2 3 4 5 6 7 8 9 10 11

// O r i g i n a l c o d e wih a t o m i c i n t c o u n t _ a t o m i c ( i n t * data , i n t N ) { int oddCount = 0; #pragma omp p a r a l l e l f o r f o r ( i n t i = 0 ; i < N ; i++) i f ( data [ i ] % 2) #pragma omp a t o m i c o d d C o u n t ++; return oddCount ; }

12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // Array o f c o u n t e r s , but t h e r e i s f a l s e s h a r i n g i n t c o u n t _ f a l s e _ s h a r i n g ( i n t * data , i n t N ) { int numThr = omp_get_max_threads () ; int oddCount [ numThr ] ; memset ( oddCount , 0 , numThr * s i z e o f ( i n t ) ) ; int totalCount = 0; #pragma omp p a r a l l e l f o r f o r ( i n t i = 0 ; i < N ; i++) { i f ( data [ i ] % 2) o d d C o u n t [ o m p _ g e t _ t h r e a d _ n u m ( ) ]++; }

27

f o r ( i n t i = 0 ; i < n u m T h r ; i++) t o t a l C o u n t += o d d C o u n t [ i ] ; return totalCount ;

28 29 30 31

}

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // E l i m i n a t e s f a l s e s h a r i n g w i t h p ad d i n g i n t c o u n t _ w i t h o u t _ f a l s e _ s h a r i n g ( i n t * data , i n t N ) { int numThr = omp_get_max_threads () ; int oddCount [ numThr * 8 ] ; memset ( oddCount , 0 , numThr * 8 * s i z e o f ( i n t ) ) ; int totalCount = 0; #pragma omp p a r a l l e l f o r f o r ( i n t i = 0 ; i < N ; i++) { i f ( data [ i ] % 2) o d d C o u n t [ o m p _ g e t _ t h r e a d _ n u m ( ) * 8]++; }

47

f o r ( i n t i = 0 ; i < n u m T h r ; i++) t o t a l C o u n t += o d d C o u n t [ i * 8 ] ; return totalCount ;

48 49 50 51

}

179 52 53 54 55 56 57 58 59 60 61 62 63 64

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t c o u n t _ w i t h _ r e d u c t i o n ( i n t * data , i n t N ) { int oddCount = 0; #pragma omp p a r a l l e l f o r r e d u c t i o n ( + : oddCount ) f o r ( i n t i = 0 ; i < N ; i++) { i f ( data [ i ] % 2) o d d C o u n t ++; } return oddCount ; }

The use of a reduction variable is equivalent to using private variables for eliminating false sharing.

12. Quicksort rightfully holds the place of the top-performer amongst generic sorting algorithms. It’s design can be considered a reflection of mergesort, although both algorithms employ a divide-and-conquer design. Write a sequential implementation of quicksort and proceed to parallelize it using OpenMP constructs. Measure the speedup that can be achieved, and compare it to the performance that can be obtained from the bottom-up mergesort of Section 4.8.1.

Answer The solution involves the use of task constructs in lines and . The code incorporates several quicksort optimizations such as median-of-three pivot element selection (lines 54-62) and a switch to insertion sort once the size of the data fall below a fixed threshold (lines 92-96). To reduce the generation of tasks, a similar heuristic to the one described in Section 4.8.2 is used ( thresh variable in line 121).

In terms of performance, the program more closely matches the top-down mergesort, failing to fully exploit a multicore CPU because of its recursive nature. The bottom-up mergesort was about twice as fast as the quicksort on an i7 3770K CPU. 1 2 3 4 5 6

#i n c l u d e < s t d l i b . h> #i n c l u d e <s t d i o . h> #i n c l u d e <u n i s t d . h> #i n c l u d e <omp . h> #i n c l u d e <QTime> #i n c l u d e <i o s t r e a m >

7 8

u s i n g namespace s t d ;

9 10 11 12

c o n s t i n t I N S E R T I O N _ T H R E S H O L D =50; c o n s t i n t m a x T a s k s =256; i n t _ t h r e s h _ ; // u s e d f o r t h e i f c l a u s e s

13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

// ************************************************ void i n s e r t i o n S o r t ( i n t * data , i n t N ) { f o r ( i n t i = 1 ; i < N ; i++) { i n t tmp = data [ i ] ; i n t l o c=i −1; w h i l e ( l o c >=0 && d a t a [ l o c ] > t m p ) { d a t a [ l o c +1] = d a t a [ l o c ] ; loc −−; } d a t a [ l o c +1] = t m p ; }

180 28

CHAPTER 8. SHARED-MEMORY PROGRAMMING : OPENMP }

29 30 31 32 33 34 35 36 37

// ************************************************ v o i d n u m b e r G e n ( i n t N , i n t max , i n t * s t o r e ) { int i ; srand ( time (0) ) ; f o r ( i = 0 ; i < N ; i++) store [ i ] = rand ( ) % max ; }

38 39 40 41 42 43 44 45

// ************************************************ void swap ( i n t * data , i n t x , i n t y ) { i n t temp = data [ x ] ; data [ x ] = data [ y ] ; data [ y ] = temp ; }

46 47 48 49 50

// ************************************************ i n t p a r t i t i o n ( i n t * data , i n t N ) { int i = 0 , j = N ;

51

// median o f 3 i n t a=d a t a [ 0 ] , b=d a t a [ N >> 1 ] , c=d a t a [ N − 1 ] ; i f ( ( a > b && b >c ) | | ( c > b && b >a ) ) { s w a p ( d a t a , 0 , N>>1) ; } e l s e i f ( ( a > c && c >b ) | | ( b > c && c>a ) ) { s w a p ( d a t a , 0 , N −1) ; }

52 53 54 55 56 57 58 59 60 61 62

i n t pivot = data [ 0 ] ;

63 64

do {

65 66

do {

67 68

i++; } w h i l e ( p i v o t > d a t a [ i ] && i < N ) ;

69 70 71 72

do {

73 74

j−−; } w h i l e ( p i v o t < d a t a [ j ] && j > 0 ) ; swap ( data , i , j ) ;

75 76 77 78

} while ( i < j ) ;

79 80 81

swap ( data , i , j ) ;

82 83

swap ( data , 0 , j ) ; return j ;

84 85 86

}

87 88 89 90 91 92 93 94 95 96 97

// ************************************************ void QSort ( i n t * data , i n t N ) { i f ( N <= 1 ) return ; e l s e i f ( N<I N S E R T I O N _ T H R E S H O L D ) { i n s e r t i o n S o r t ( data , N ) ; return ; }

98 99 100 101

i n t p i v o t P o s = p a r t i t i o n ( data , N ) ; #pragma omp p a r a l l e l {

181 102 103 104 105 106 107 108 109 110 111

#pragma omp s i n g l e { #pragma omp t a s k i f (N > t h r e s h ) mergeable QSort ( data , p i v o t P o s ) ; #pragma omp t a s k i f (N > t h r e s h ) mergeable Q S o r t (&( d a t a [ p i v o t P o s + 1 ] ) , N − p i v o t P o s − 1 ) ; #pragma omp t a s k w a i t } } }

112 113 114 115 116 117 118 119 120 121 122

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t main ( i n t argc , char * argv [ ] ) { i f ( a r g c == 1 ) { f p r i n t f ( s t d e r r , ”%s N\n” , a r g v [ 0 ] ) ; exit (0) ; } i n t N = atoi ( argv [ 1 ] ) ; _thresh_ = 2.0 * N / ( maxTasks + 1) ;

123

i n t * d a t a = new i n t [ N ] ; numberGen ( N , 1000 , data ) ; QTime t ; t . start () ;

124 125 126 127 128

QSort ( data , N ) ;

129 130

c o u t << ” S o r t e d i n ” << t . e l a p s e d ( ) / 1 0 0 0 . 0 << ” s e c \n” ;

131 132

d e l e t e [ ] data ; return 0;

133 134 135

} #pragma omp t a r g e t teams #pragma omp p a r a l l e l f o r c o l l a p s e ( 2 ) f o r ( i n t i =0; i<N ; i++) f o r ( i n t j =0; j<N ; j++) f o r ( i n t k =0; k<N ; k++) c [ i ] [ j]+= a [ i ] [ k ] * b [ k ] [ j ] ;

What would be the outcome of this change?

Answer This change cannot be done, as it will force the execution of the parallel for by all the teams. If effect, the computation will be duplicated as many times as the number of teams.

13. Assuming we have a dual socket platform with 32 cores per CPU, each supporting two hardware threads, write the appropriate environment variable so that the following places are defined: (a) Each socket is a place. (b) Each set of four cores constitutes a place. (c) Each set of 16 hardware threads is a place.

Answer (a) OMP PLACES=“sockets” or OMP PLACES=“{0:32}:2:32” (b) OMP PLACES=“{0:4}:16:4” (c) OMP PLACES=“{0:16}:2:16”

182

CHAPTER 8. SHARED-MEMORY PROGRAMMING : OPENMP

14. The AMD Epyc 7742 CPU comes with 64 cores, each supporting two hardware threads. The cores are organized in the form of CCD chiplets, each hosting eight cores and their dedicated 32MiB L3 cache. The L3 cache is split between two Core Complexes (CCXs). Explain what will happen for each of the following settings in terms of thread placement: (a) OMP PLACES=“ll caches”, OMP PROC BIND=“close”, for 64 threads and the master thread at p0 (b) OMP PLACES=“{0:4}:16:4”, OMP PROC BIND=“master”, for 256 threads and the master thread at p1 (c) OMP PLACES=“{0:8}:8:8”, OMP PROC BIND=“spread”, for 16 threads and the master thread at p2 (d) OMP PLACES=“{0:8}:8:8”, OMP PROC BIND=“spread”, for 256 threads and the master thread at p3 (e) OMP PLACES=“cores”, OMP PROC BIND=“spread”, for 64 threads and the master thread at p4 (f) OMP PLACES=“cores”, OMP PROC BIND=“spread”, for 16 threads and the master thread at p5 Answer (a) There will be 16 places, one for each CCX. The threads will be placed as follows: 0-3 at p0 , 4-7 at p1 , etc. (b) There will be 16 places defined, each made of 4 hardware threads. All the threads will be placed at p1 . (c) There will be 8 places. The threads will be placed as follows: 0-1 at p2 , 2-3 at p3 , etc. (d) There will be 8 places. The threads will be placed as follows: 0-31 at p3 , 32-63 at p4 , etc. (e) There will be 64 places. The threads will be placed as follows: 0 at p4 , 1 at p5 , etc. (f) There will be 64 places. The places will be grouped into 16 new groups p′0 = {p0 , p1 , . . . , p3 }, etc. The threads will be placed as follows: 0 at {p4 , p5 , p6 , p7 }, 1 at {p8 , p9 , p10 , p11 }, etc.

Chapter 9

The Thrust Template Library Exercises 1. Develop a Thrust program for calculating the inner product of two vectors by using the thrust::transform algorithm and an appropriate functor.

Answer The thrust::transform algorithm can be used to multiple the elements of two vectors, but it has to be accompanied by the thrust::reduce algorithm in order to sum-up the partial products. In the following code, two randomly-initialized vectors, of user-supplied size (as specified by a single command-line parameter), are multiplied accordingly. The result is computed on both the CPU and GPU in order to compare the results: #i n c l u d e <i o s t r e a m > #i n c l u d e < s t d l i b . h> #i n c l u d e <t h r u s t / d e v i c e v e c t o r . h> #i n c l u d e <t h r u s t / h o s t v e c t o r . h> #i n c l u d e <t h r u s t / t r a n s f o r m . h> #i n c l u d e <t h r u s t / r e d u c e . h> u s i n g namespace s t d ; struct innerProdFunct { __host__ __device__ { return x * y ; } };

float

o p e r a t o r ( ) ( f l o a t &x ,

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t N = atoi ( argv [ 1 ] ) ; t h r u s t : : h o s t _ v e c t o r < f l o a t >h _ x ( N ) ; t h r u s t : : h o s t _ v e c t o r < f l o a t >h _ y ( N ) ; t h r u s t : : h o s t _ v e c t o r < f l o a t >h _ p r o d ( N ) ; t h r u s t : : d e v i c e _ v e c t o r < f l o a t >d _ x ; t h r u s t : : d e v i c e _ v e c t o r < f l o a t >d _ y ; t h r u s t : : d e v i c e _ v e c t o r < f l o a t >d _ p r o d ( N ) ; srand ( time (0) ) ; f o r ( i n t i = 0 ; i < N ; i++) { h_x [ i ] = rand ( ) % 1000;

183

f l o a t &y )

184

CHAPTER 9. THE THRUST TEMPLATE LIBRARY h_y [ i ] = rand ( ) % 1000; } // copy d a t a t o d e v i c e d_x = h_x ; d_y = h_y ; // d e v i c e −b a s e d c a l c u l a t i o n t h r u s t : : t r a n s f o r m ( d _ x . b e g i n ( ) , d _ x . e n d ( ) , d _ y . b e g i n ( ) , ←֓ d_prod . begin () , innerProdFunct () ) ; f l o a t res = thrust : : reduce ( d_prod . begin ( ) , d_prod . end ( ) ) ; c o u t << ” I n n e r p r o d u c t on GPU i s ” << r e s << e n d l ; // h o s t c a l c u l a t i o n t h r u s t : : t r a n s f o r m ( h _ x . b e g i n ( ) , h _ x . e n d ( ) , h _ y . b e g i n ( ) , ←֓ h_prod . begin () , innerProdFunct () ) ; res = thrust : : reduce ( h_prod . begin ( ) , h_prod . end ( ) ) ; c o u t << ” I n n e r p r o d u c t on CPU i s ” << r e s << e n d l ; return 0; }

2. Calculate the inner product of two vectors by using the thrust::inner product algorithm. Compare the performance of this version with the performance of a CUDA-based solution.

Answer The use of the thrust::inner product can replace the sequence of thrust::transform and thrust::reduce of the previous exercise. The resulting program does not require a functor for the calculation to take place: #i n c l u d e <i o s t r e a m > #i n c l u d e < s t d l i b . h> #i n c l u d e <t h r u s t / d e v i c e v e c t o r . h> #i n c l u d e <t h r u s t / h o s t v e c t o r . h> #i n c l u d e <t h r u s t / i n n e r p r o d u c t . h> u s i n g namespace s t d ; i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t N = atoi ( argv [ 1 ] ) ; t h r u s t : : h o s t _ v e c t o r < f l o a t >h _ x ( N ) ; t h r u s t : : h o s t _ v e c t o r < f l o a t >h _ y ( N ) ; t h r u s t : : d e v i c e _ v e c t o r < f l o a t >d _ x ; t h r u s t : : d e v i c e _ v e c t o r < f l o a t >d _ y ; srand ( time (0) ) ; f o r ( i n t i = 0 ; i < N ; i++) { h_x [ i ] = rand ( ) % 1000; h_y [ i ] = rand ( ) % 1000; } d_x = h_x ; d_y = h_y ; f l o a t res = thrust : : inner_product ( d_x . begin ( ) , d_x . end ( ) , d_y . begin ( ) , 0, // i n i t i a l r e d u c t i o n v a l u e t h r u s t : : p l u s < f l o a t >() , // r e d u c t i o n t h r u s t : : m u l t i p l i e s < f l o a t >() ) ; // ←֓ transformation c o u t << ” I n n e r p r o d u c t f u n c t i o n on GPU p r o d u c e s ” << r e s << ←֓ endl ; return 0; }

3. Use Thrust algorithms to find the absolute maximum of an array of values.

185 Answer The solution can be based on the thrust::transform reduce algorithm, which permits kernel fusion. A custom unary functor (lines 8-14) can be used to calculate the absolute values during the transformation phase, with the built-in thrust::maximum functor used in the reduction phase.

1 2 3 4 5

#i n c l u d e <i o s t r e a m > #i n c l u d e < s t d l i b . h> #i n c l u d e <t h r u s t / d e v i c e v e c t o r . h> #i n c l u d e <t h r u s t / h o s t v e c t o r . h> #i n c l u d e <t h r u s t / t r a n s f o r m r e d u c e . h>

6 7 8 9 10 11 12 13 14

u s i n g namespace s t d ; struct absFunct { __host__ __device__ { return fabs ( x ) ; } };

float

o p e r a t o r ( ) ( f l o a t &x )

15 16 17 18 19 20

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t N = atoi ( argv [ 1 ] ) ; t h r u s t : : h o s t _ v e c t o r < f l o a t >h _ x ( N ) ; t h r u s t : : d e v i c e _ v e c t o r < f l o a t >d _ x ;

21

srand ( time (0) ) ; f o r ( i n t i = 0 ; i < N ; i++) { h_x [ i ] = rand ( ) % 1000; }

22 23 24 25 26 27

d_x = h_x ; f l o a t max = thrust : : transform_reduce ( d_x . begin ( ) , d_x . end ( ) , absFunct () , 0, t h r u s t : : m a x i m u m < f l o a t ←֓ >() ) ; c o u t << ”Max by GPU: ” << m a x << e n d l ;

28 29 30 31 32

33 34 35

max = 0 ; f o r ( i n t i = 0 ; i < N ; i++) { f l o a t temp = fabs ( h_x [ i ] ) ; i f ( temp > max ) max = temp ; } c o u t << ”Max by CPU: ” << m a x << e n d l ;

36 37 38 39 40 41 42 43 44

return 0;

45 46

}

4. Write a Thrust program for calculating the sum of two matrices.

Answer The solution can be based on the thrust::transform algorithm, combined with the built-in thrust::plus functor. The matrices can be simply treated as vectors, as their dimensions have no influence on the computation (as they do in multiplication). In the following code the user supplies the dimensions of the matrices to be randomly generated for testing the code.

A thrust::host vector is used for initializing the two matrices (lines 39-40 and 43-44) before passing them to the device (lines 41 and 45) and for retrieving and printing out the result (lines 49-50).

186 1 2 3 4 5 6

CHAPTER 9. THE THRUST TEMPLATE LIBRARY #i n c l u d e <i o s t r e a m > #i n c l u d e <t h r u s t / h o s t v e c t o r . h> #i n c l u d e <t h r u s t / d e v i c e v e c t o r . h> #i n c l u d e <t h r u s t / t r a n s f o r m . h> #i n c l u d e <t h r u s t / random . h> #i n c l u d e <math . h>

7 8

u s i n g namespace s t d ;

9 10 11 12 13 14 15 16 17 18 19 20 21 22

v o i d p r i n t M a t r i x ( t h r u s t : : h o s t _ v e c t o r < f l o a t >&m , { f o r ( i n t i = 0 ; i < N ; i++) { c o u t << ” | ” ; f o r ( i n t j = 0 ; j < M ; j++) { c o u t << m [ i * M + j ] << ” ” ; } c o u t << ” | ” << e n d l ; } c o u t << ”===========================\n” ; }

int N ,

int M)

23 24 25 26 27 28 29

// **************************************************** i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { // i n i t i a l i z e t h e RNG thrust : : default_random_engine rng ( time (0) ) ; t h r u s t : : u n i f o r m _ i n t _ d i s t r i b u t i o n < i n t >u n i D i s t r ( −10000 , ←֓ 10000) ;

30

i n t N = atoi ( argv [ 1 ] ) ; i n t M = atoi ( argv [ 2 ] ) ;

31 32 33

// g e n e r a t e t h e d a t a on t h e h o s t and move them t o t h e d e v i c e t h r u s t : : d e v i c e _ v e c t o r < f l o a t >x ( N * M ) ; t h r u s t : : d e v i c e _ v e c t o r < f l o a t >y ( N * M ) ; t h r u s t : : d e v i c e _ v e c t o r < f l o a t >z ( N * M ) ; t h r u s t : : h o s t _ v e c t o r < f l o a t >a u x ( N * M ) ; f o r ( i n t i = 0 ; i < x . s i z e ( ) ; i++) aux [ i ] = uniDistr ( rng ) ; x = aux ; p r i n t M a t r i x ( aux , N , M ) ; f o r ( i n t i = 0 ; i < x . s i z e ( ) ; i++) aux [ i ] = uniDistr ( rng ) ; y = aux ; p r i n t M a t r i x ( aux , N , M ) ;

34 35 36 37 38 39 40 41 42 43 44 45 46 47

t h r u s t : : t r a n s f o r m ( x . b e g i n ( ) , x . e n d ( ) , y . b e g i n ( ) , z . b e g i n ( ) ←֓ , t h r u s t : : p l u s < f l o a t >() ) ; aux = z ; p r i n t M a t r i x ( aux , N , M ) ;

48

49 50 51

return 0;

52 53

}

5. Write a Thrust program for calculating the definite integral of a function f (x) over a range [a, b], using the trapezoidal rule. Consider a solution that avoids the need to create a vector for all the x-values for which a trapezoid is calculated. The details of the trapezoidal rule can be found in Section 3.5.2.

Answer The answer is to employ a thrust::counting iterator to generate a sequence of indices, and a functor which scales them to generate x values for the calculation of the desired function.

The functor shown in the listing below (lines 13-28) is initialized with the appropriate offset (data member a) and scale (data member h) to convert

187 an index i to an appropriate xi value (line 25), as dictated by the equation that calculates the integral of a function f () in the range [a, b] : h

n−1 f (a) + f (b) X f (xi ) + 2 i=1

!

where x0 = a, xn = b, xi = a + i · h and h = b−a n .

Line 26 calculates the value of the function. The thrust::transform reduce algorithm of lines 39-42, sums-up the produced function values, effectively calculating the summation term of the above equation.

By having the initial reduction value set to ( f(0.0f) + f(N * 1.0f) ) /2, (f(0.0f) evaluates to f(a) and f(N * 1.0f) to f(b))the reduction calculates the complete parenthesis expression. The multiplication by h in line 43 completes the result.

1 2 3 4 5 6

#i n c l u d e <i o s t r e a m > #i n c l u d e < s t d l i b . h> #i n c l u d e <math . h> #i n c l u d e <t h r u s t / i t e r a t o r / c o u n t i n g i t e r a t o r . h> #i n c l u d e <t h r u s t / t r a n s f o r m r e d u c e . h> #i n c l u d e <t h r u s t / f u n c t i o n a l . h>

7 8 9

const const

float float

LOWERLIMIT = 0; // s t a n d s f o r r a n g e l i m i t a U P P E R L I M I T = 1 0 ; // s t a n d s f o r r a n g e l i m i t b

10 11 12 13 14 15

u s i n g namespace s t d ; // ***************************************** struct integrFunct { float h , a ;

16

i n t e g r F u n c t ( f l o a t st ,

17

float

width )

{

18

a = st ; h = width ;

19 20

}

21 22

__host__ __device__ f l o a t { float x = a + h * i ; return fabs ( sin ( x ) ) ; }

23 24 25 26 27 28 29 30 31 32

operator () ( f l o a t i )

}; // ***************************************** i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t N = atoi ( argv [ 1 ] ) ;

33

f l o a t h = ( UPPERLIMIT − LOWERLIMIT ) / N ; ; integrFunct f ( LOWERLIMIT , h ) ;

34 35 36

t h r u s t : : c o u n t i n g _ i t e r a t o r < f l o a t >x ( 1 ) ;

37 38

f l o a t res = thrust : : transform_reduce ( x , x + N − 1 , f, ( f ( 0 . 0 f ) + f ( N * 1 . 0 f ) ) ←֓ /2 , t h r u s t : : p l u s < f l o a t >() ) ; r e s *= h ; c o u t << ” D e f i n i t e i n t e g r a l i s ” << r e s << e n d l ;

39 40 41

42 43 44 45

return 0;

46 47

}

6. Use Thrust to calculate the mean and variance of a data set X of cardinality N . For convenience, the corresponding formulas are:

188

CHAPTER 9. THE THRUST TEMPLATE LIBRARY

E[X] =

PN −1 i=0

xi

N

σ 2 = E[X 2 ] − (E[X])2

(9.1) (9.2)

Answer This is a straight-forward application of the thrust::reduce algorithm for calculating the sum of the data set values, and the thrust::transform reduce algorithm for calculating the sum of the square data set values. The remaining calculations are very simple. #i n c l u d e <i o s t r e a m > #i n c l u d e < s t d l i b . h> #i n c l u d e <math . h> #i n c l u d e <t h r u s t / h o s t v e c t o r . h> #i n c l u d e <t h r u s t / d e v i c e v e c t o r . h> #i n c l u d e <t h r u s t / r e d u c e . h> #i n c l u d e <t h r u s t / random . h> #i n c l u d e <t h r u s t / t r a n s f o r m r e d u c e . h> #i n c l u d e <t h r u s t / f u n c t i o n a l . h> u s i n g namespace s t d ; // ***************************************** struct sqrFunct { __host__ __device__ f l o a t operator () ( f l o a t x ) { return x*x ; } }; // ***************************************** i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t N = atoi ( argv [ 1 ] ) ; // i n i t i a l i z e t h e RNG thrust : : default_random_engine rng ( time (0) ) ; t h r u s t : : u n i f o r m _ i n t _ d i s t r i b u t i o n <i n t > u n i D i s t r ( − 1 0 0 0 0 , 1 0 0 0 0 ) ; // g e n e r a t e t h e d a t a on t h e h o s t and move them t o t h e d e v i c e t h r u s t : : d e v i c e _ v e c t o r < f l o a t >d _ x ( N ) ; t h r u s t : : h o s t _ v e c t o r <f l o a t > h _ x ( N ) ; f o r ( i n t i = 0 ; i < N ; i++) h _ x [ i ] = u n i D i s t r ( r n g ) ; d_x = h_x ; // c a l c u l a t e t h e sum o f v a l u e s f l o a t avg = thrust : : reduce ( d_x . begin ( ) , d_x . end ( ) ) ; a v g /= N ; // c a l c u l a t e t h e sum o f s q u a r e d v a l u e s f l o a t sum2 = thrust : : transform_reduce ( d_x . begin ( ) , d_x . end ( ) , sqrFunct () , 0, t h r u s t : : p l u s < f l o a t >() ) ←֓ ; c o u t << ”Mean : ” << a v g << e n d l ; c o u t << ” V a r i a n c e : ” << s u m 2 / N − a v g * a v g << e n d l ; return 0; }

7. Measure the performance of the thrust::sort algorithm in Thrust by sorting varying volumes of data. Compare the achieved times with the STL version running on the host. For this purpose create a big array (make sure it is not too big to fit in the GPU’s memory) and populate it with random data.

189 Answer The following listing employs the POSIX high-resolution timing functions (see Appendix C.2) for reporting the time it takes to sort a variablesize array of integers using STL on the host and Thrust on the device respectively. The program requires three command-line parameters corresponding to the minimum (Nmin) and maximum array size (Nmax), and the iteration step used in testing (Nstep).

Once the arrays are initialized and placed in their respective memory spaces, the loop of lines 40-52 proceeds to sort an increasing part of the arrays and report the time and processing rate in million integers/sec units.

1 2 3 4 5 6 7

#i n c l u d e <i o s t r e a m > #i n c l u d e <a l g o r i t h m > #i n c l u d e < s t d l i b . h> #i n c l u d e <t i m e . h> #i n c l u d e <t h r u s t / d e v i c e v e c t o r . h> #i n c l u d e <t h r u s t / h o s t v e c t o r . h> #i n c l u d e <t h r u s t / s o r t . h>

8 9

u s i n g namespace s t d ;

10 11

// ***********************************************

12 13 14 15 16 17 18 19

double hrclock_sec ( ) { t i m e s p e c ts ; c l o c k _ g e t t i m e ( C L O C K _ R E A L T I M E , &t s ) ; double aux = ts . tv_sec + ts . t v _ n s e c / 1 0 0 0 0 0 0 0 0 0 . 0 ; return aux ; }

20 21

// ***********************************************

22 23 24 25 26 27 28 29

i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t Nmin = atoi ( argv [ 1 ] ) ; i n t Nmax = atoi ( argv [ 2 ] ) ; i n t Nstep = atoi ( argv [ 3 ] ) ; t h r u s t : : h o s t _ v e c t o r < f l o a t >h _ x ( N m a x ) ; t h r u s t : : d e v i c e _ v e c t o r < f l o a t >d _ x ;

30 31 32 33 34 35

srand ( time (0) ) ; f o r ( i n t i = 0 ; i < N m a x ; i++) { h_x [ i ] = rand ( ) % 1000000; }

36 37

d_x = h_x ;

38 39 40 41 42 43 44 45

d o u b l e c p u _ t i m e , g p u _ t i m e , t0 , t1 , t 2 ; f o r ( i n t i t e m s = N m i n ; i t e m s <= N m a x ; i t e m s += N s t e p ) { t0 = h r c l o c k _ s e c ( ) ; std : : sort ( h_x . begin ( ) , h_x . begin ( ) + items ) ; t1 = h r c l o c k _ s e c ( ) ; c p u _ t i m e = t1 − t0 ;

46

thrust : : sort ( d_x . begin ( ) , d_x . begin ( ) + items ) ; t2 = h r c l o c k _ s e c ( ) ; g p u _ t i m e = t2 − t1 ;

47 48 49 50

c o u t << i t e m s << ” Times CPU,GPU ( s e c ) : ” << c p u _ t i m e << ”←֓ ” << g p u _ t i m e << ” \ t R a t e s ( m i l l i o n i n t e g e r s / s e c ) : ”←֓ << i t e m s / c p u _ t i m e / 1000000 << ” ” << i t e m s / ←֓ g p u _ t i m e / 1000000 << ” \n” ;

51

52

}

53 54

return 0;

190 55

CHAPTER 9. THE THRUST TEMPLATE LIBRARY }

The results of a sample run on a 3.5GHz i7-3770K CPU and a GTX 560Ti GPU are shown below: n v c c −O 2 s o r t T h r u s t . c u − o s o r t T h r u s t . / s o r t T h r u s t 1000000 5000000 500000 1000000 T i m e s CPU , G P U ( s e c ) : 0 . 0 5 4 8 1 8 6 0 . 0 0 2 6 0 7 1 1 million integers / sec ) : 18.242 383.567 1500000 T i m e s CPU , G P U ( s e c ) : 0 . 0 6 4 9 8 6 9 0 . 0 0 3 3 7 3 1 5 million integers / sec ) : 23.0816 444.689 2000000 T i m e s CPU , G P U ( s e c ) : 0 . 0 7 9 1 8 5 5 0 . 0 0 4 1 3 7 0 4 million integers / sec ) : 25.2572 483.438 2500000 T i m e s CPU , G P U ( s e c ) : 0 . 0 9 5 3 4 4 3 0 . 0 0 4 9 1 5 9 5 million integers / sec ) : 26.2208 508.548 3000000 T i m e s CPU , G P U ( s e c ) : 0 . 1 0 3 8 9 6 0 . 0 0 5 8 8 6 7 9 million integers / sec ) : 28.8751 509.615 3500000 T i m e s CPU , G P U ( s e c ) : 0 . 1 2 4 5 2 1 0 . 0 0 6 5 5 7 7 million integers / sec ) : 28.1077 533.723 4000000 T i m e s CPU , G P U ( s e c ) : 0 . 1 2 7 9 6 6 0 . 0 0 7 3 2 5 8 9 million integers / sec ) : 31.2583 546.009 4500000 T i m e s CPU , G P U ( s e c ) : 0 . 1 3 6 4 8 2 0 . 0 0 8 0 1 6 3 5 million integers / sec ) : 32.9713 561.353 5000000 T i m e s CPU , G P U ( s e c ) : 0 . 1 5 1 1 9 4 0 . 0 0 8 7 9 9 0 8 million integers / sec ) : 33.0701 568.241

$

$

R a t e s ( ←֓ R a t e s ( ←֓ R a t e s ( ←֓ R a t e s ( ←֓ R a t e s ( ←֓ R a t e s ( ←֓ R a t e s ( ←֓ R a t e s ( ←֓ R a t e s ( ←֓

8. Measure the performance hit that an Array-of-Structures design approach will have in device-based sorting of 108 randomly generated instances of the following structure: struct TestStr { i n t key ; f l o a t value ; __host__ __device__ b o o l o p e r a t o r <( c o n s t T e s t S t r &o ) c o n s t { return t h i s . key < o . key ; } };

To measure the performance deterioration, you will have to implement and time a Structure-of-Arrays alternative.

Answer The answer is given in the listing below. While the thrust::sort algorithm can used for sorting the array of structures, the thrust::sort by key algorithm needs to be employed for the structure-of-arrays alternative.

The data are randomly generated and are common in both arrangements.

The POSIX high-resolution timing functions (see Appendix C.2) are used for reporting the time. #i n c l u d e <i o s t r e a m > #i n c l u d e <a l g o r i t h m > #i n c l u d e < s t d l i b . h> #i n c l u d e <t i m e . h> #i n c l u d e <t h r u s t / d e v i c e v e c t o r . h> #i n c l u d e <t h r u s t / h o s t v e c t o r . h> #i n c l u d e <t h r u s t / s o r t . h> u s i n g namespace s t d ; // *********************************************** double hrclock_sec ( ) {

191 t i m e s p e c ts ; c l o c k _ g e t t i m e ( C L O C K _ R E A L T I M E , &t s ) ; double aux = ts . tv_sec + ts . t v _ n s e c / 1 0 0 0 0 0 0 0 0 0 . 0 ; return aux ; } // *********************************************** struct TestStr { i n t key ; f l o a t value ; __host__ __device__ bool operator< ( const TestStr & o ) const { r e t u r n t h i s −>k e y < o . k e y ; } }; // *********************************************** i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t N = atoi ( argv [ 1 ] ) ; thrust : : host_vector < TestStr > h_str ( N ) ; thrust : : device_vector < TestStr > d_str ; t h r u s t : : h o s t _ v e c t o r < i n t >h _ k e y s ( N ) ; t h r u s t : : h o s t _ v e c t o r < f l o a t >h _ v a l u e s ( N ) ; t h r u s t : : d e v i c e _ v e c t o r < i n t >d _ k e y s ( N ) ; t h r u s t : : d e v i c e _ v e c t o r < f l o a t >d _ v a l u e s ( N ) ; srand ( time (0) ) ; f o r ( i n t i = 0 ; i < N ; i++) { h_str [ i ] . key = rand ( ) % 1000000; h_str [ i ] . value = rand () % 1000000; h_keys [ i ] = h_str [ i ] . key ; h_values [ i ] = h_str [ i ] . value ; } d_str = h_str ; d_keys = h_keys ; d_values = d_values ; d o u b l e t0 , t1 , t 2 ; t0 = h r c l o c k _ s e c ( ) ; thrust : : sort ( d_str . begin ( ) , d_str . end ( ) ) ; t1 = h r c l o c k _ s e c ( ) ; t h r u s t : : s o r t _ b y _ k e y ( d _ k e y s . b e g i n ( ) , d _ k e y s . e n d ( ) , d _ v a l u e s . ←֓ begin () ) ; t2 = h r c l o c k _ s e c ( ) ; c o u t << N << ” ” << t 1 − t 0 << ” ” << t 2 − t 1 << e n d l ; return 0; }

9. Create a Thrust program for computing the Mandelbrot set. Section 3.8.2 covers the mathematical details of how the set is calculated.

Answer Thrust has the limitation of not providing a 2D counting iterator, that would have been handy in this problem. We can still use the thrust::counting iterator facility as long as we can couple it with a transformation that maps each generated index to a coordinate on the complex plane. In the functor of lines 19-57 in the listing that follows, this is exactly what is taking place.

The functor is initialized with all the information necessary to map what is essentially a pixel counter running from top-to-bottom and left-to-right,

192

CHAPTER 9. THE THRUST TEMPLATE LIBRARY into a pair of complex plane coordinates. The conversion is taking place in lines 40-43. The remaining lines in the operator() method is just a copy of the MandelCompute::diverge method shown in Listing 3.29 in page 127.

The thrust::transform call in line 85 calculates the number of iterations required for each examined complex plane point to diverge. These data are subsequently copied to the host (line 88), prior to using them for painting the pixels of a QImage object (lines 91-96).

1 2

// To c o m p i l e : // nvcc −m64 −O2 −X c o m p i l e r ’− fPIE ’ −D REENTRANT −DQT NO DEBUG −←֓ DQT GUI LIB −DQT CORE LIB −I / u s r / l i b / x 8 6 6 4−l i n u x −gnu / q t 5 /←֓ mkspecs / l i n u x −g++−64 −I . −I / u s r / i n c l u d e / q t 5 −I / u s r / i n c l u d e /←֓ q t 5 /QtGui −I / u s r / i n c l u d e / q t 5 / QtCore −I . −o mandelThrust ←֓ mandelThrust . cu −L / o p t / cuda / l i b 6 4 / −l c u d a r t −r d c=t r u e −a r c h ←֓ =sm 21 −lm −l Q t 5 C o r e −lQt5Gui −L / u s r / l i b / x 8 6 6 4−l i n u x −gnu /

3 4 5 6 7 8 9 10 11 12

#i n c l u d e <i o s t r e a m > #i n c l u d e < s t d l i b . h> #i n c l u d e <math . h> #i n c l u d e <QImage> #i n c l u d e <QRgb> #i n c l u d e <t h r u s t / d e v i c e v e c t o r . h> #i n c l u d e <t h r u s t / h o s t v e c t o r . h> #i n c l u d e <t h r u s t / i t e r a t o r / c o u n t i n g i t e r a t o r . h> #i n c l u d e <t h r u s t / t r a n s f o r m . h>

13 14

u s i n g namespace s t d ;

15 16

const int MAXITER = 255;

17 18 19 20 21 22 23

// ***************************************** struct mandelFunct { double upperX , upperY ; double xStep , yStep ; int imageWidth ;

24 25 26 27 28 29 30 31 32

m a n d e l F u n c t ( d o u b l e uX , d o u b l e uY , d o u b l e xS , d o u b l e yS , { upperX = uX ; upperY = uY ; xStep = xS ; yStep = yS ; imageWidth = w ; }

33 34 35 36 37 38

__host__ __device__ int operator { i n t xIDX , yIDX ; d o u b l e cx , c y ; i n t iter = 0;

() ( int i)

39

xIDX = i % imageWidth ; yIDX = i / imageWidth ; cx = upperX + xIDX * xStep ; cy = upperY − yIDX * yStep ;

40 41 42 43 44

d o u b l e v x = cx , v y = cy , tx , t y ; w h i l e ( i t e r < M A X I T E R && ( v x * v x + v y * v y ) < 4 ) { tx = vx * vx − vy * vy + cx ; ty = 2 * vx * vy + cy ; vx = tx ; vy = ty ; i t e r ++; } return iter ;

45 46 47 48 49 50 51 52 53 54 55 56

}

int w)

193 57

};

58 59 60 61 62 63 64 65

// ***************************************** i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t imgX = 1024 , imgY = 768; double upperCornerX , upperCornerY ; double lowerCornerX , lowerCornerY ; double xStep , yStep ;

66

upperCornerX upperCornerY lowerCornerX lowerCornerY

67 68 69 70

= = = =

atof atof atof atof

( argv [ 1 ] ) ; ( argv [ 2 ] ) ; ( argv [ 3 ] ) ; ( argv [ 4 ] ) ;

71

xStep = ( lowerCornerX − upperCornerX ) / imgX ; yStep = ( upperCornerY − lowerCornerY ) / imgY ;

72 73 74

Q I m a g e * i m g = new Q I m a g e ( i m g X , i m g Y , Q I m a g e : : F o r m a t _ R G B 3 2 ) ;

75 76

mandelFunct f ( upperCornerX , upperCornerY , xStep , yStep , imgX ) ;

77 78

t h r u s t : : h o s t _ v e c t o r < i n t >h _ i t e r ; t h r u s t : : d e v i c e _ v e c t o r < i n t >d _ i t e r ( i m g X * i m g Y ) ;

79 80 81

t h r u s t : : c o u n t i n g _ i t e r a t o r < f l o a t >x ( 0 ) ;

82 83

// c a l c u l a t e t h e p i x e l v a l u e s on t h e d e v i c e thrust : : t r a n s f o r m ( x , x + imgX * imgY , d_iter . begin ( ) , f ) ;

84 85 86

// copy r e s u l t s t o h o s t h_iter = d_iter ;

87 88 89

// draw t h e image p i x e l s a c c o r d i n g l y f o r ( i n t j = 0 ; j < i m g Y ; j++) f o r ( i n t i = 0 ; i < i m g X ; i++) { i n t color = h_iter [ imgX * j + i ] ; img−>s e t P i x e l ( i , j , q R g b ( 2 5 6 − c o l o r , 256 − c o l o r , 256 ←֓ − color ) ) ; } img−>s a v e ( ” mandel . png ” , ”PNG” , 0 ) ; return 0;

90 91 92 93 94 95

96 97 98 99

}

10. A problem related to the furthest-distance point solved in Section 7.5, is the problem of finding the pair of points which are the furthest apart from each other. Create a brute-force solution to this problem, i.e. by examining all pairs of points, using Thrust. Consider the case where the number of points is too big to allow storage of all the calculated distances.

You should avoid duplicate calculations since the distance from point A to B is the same as the distance from B to A. Answer Not being able to store the distances creates both opportunities and difficulties:

The number N of points that can be used as input is not limited by 2 the need to allocate Θ( N 2−N ) storage for the pair distances.

The distances might have to be calculated multiple times.

Each pair of points can be identified by the index of the corresponding cell in the distances table. We can compare the “indices” of two pairs of points, by calculating and comparing the corresponding pair-wise distances. In order to avoid duplicate calculations, we can limit the indices to either of

194

CHAPTER 9. THE THRUST TEMPLATE LIBRARY

Figure 9.1: Illustration of how a pair of points and their distance can be identified by the index of the corresponding cell in the distances matrix. The indices are calculated by numbering the cells in a top-to-bottom and left-to-right fashion, skipping the gray cells. The matrix is not actually allocated. the two table halves, as split by the main diagonal. In the following code we use the lower half. The indices are calculated as illustrated in Figure 9.1, i.e. to identify the pair of point (i, j), with i > j, we use the formula: idx(i,j) =

i−1 X

k=1

k+j =

i(i − 1) +j 2

(9.3)

The opposite mapping, i.e. from an index to a pair of points can be also performed. Given an initial approximation for i :

idx(i,j) =

q i2 − i) + j ⇒ i2 = 2 · idx(i,j) − 2j + i ⇒ i ≈ 2 · idx(i,j) (9.4) 2

we can calculate an exact value for i and j by utilizing the knowledge that i > j: i = ( i n t ) floor ( sqrt ( 2 . 0 * idx ) ) ; w h i l e ( ( j = i d x − i * ( i − 1 ) / 2 ) >= i ) i++;

The solution given below is based on the application of a thrust::reduce algorithm on a thrust::counting iterator that enumerates all the possible cell indices/point pairs. Upon comparing two indices, the supplied functor (lines 20-49) reverses the mapping to convert the two pair indices into two pairs of points, pair A and pair B. The point indices Ai, Aj and Bi, Bj are subsequently used to calculate and compare the square of the −−−→ −−−→ distances kAiAjk2 kBiBjk2 (indices used liberally here to represent the actual points).

The only restriction of the following solution is that the type thrust::counting iterator<int> iterator is appropriate for up to N < 46340. For bigger inputs, a switch to thrust::counting iterator<long> would be necessary, as a 32-bit signed integer is not sufficient for holding the generated indices.

195 1 2 3 4 5 6 7

#i n c l u d e <i o s t r e a m > #i n c l u d e <t h r u s t / h o s t v e c t o r . h> #i n c l u d e <t h r u s t / d e v i c e v e c t o r . h> #i n c l u d e <t h r u s t / i t e r a t o r / c o u n t i n g i t e r a t o r . h> #i n c l u d e <t h r u s t / r e d u c e . h> #i n c l u d e <t h r u s t / random . h> #i n c l u d e <math . h>

8 9

u s i n g namespace s t d ;

10 11 12 13 14 15 16 17

// **************************************************** _ _ h o s t _ _ _ _ d e v i c e _ _ v o i d r e v e r s e M a p ( i n t idx , i n t &i , i n t &j ) { i = ( i n t ) floor ( sqrt ( 2 . 0 * idx ) ) ; w h i l e ( ( j = i d x − i * ( i − 1 ) / 2 ) >= i ) i++; }

18 19 20 21 22

// **************************************************** struct maxFunct { t h r u s t : : d e v i c e _ p t r < i n t >x , y , z ;

23

// comparing d i s t a n c e s o f p a i r s o f p o i n t s A and B a s i n d e x e d by←֓ idxA , idxB __host__ __device__ int operator ( ) ( i n t idxA , i n t idxB ) { i n t Ai , Aj , Bi , B j ; i n t distA , distB ;

24

25 26 27 28 29

r e v e r s e M a p ( i d x A , Ai , A j ) ; r e v e r s e M a p ( i d x B , Bi , B j ) ;

30 31 32

i n t xDiff , yDiff , zDiff ; xDiff = x [ Ai ] − x [ Aj ] ; yDiff = y [ Ai ] − y [ Aj ] ; zDiff = z [ Ai ] − z [ Aj ] ; distA = xDiff * xDiff + yDiff * yDiff + zDiff * zDiff ;

33 34 35 36 37 38

xDiff yDiff zDiff distB

39 40 41 42

= x [ Bi ] − x [ Bj ] ; = y [ Bi ] − y [ Bj ] ; = z [ Bi ] − z [ Bj ] ; = xDiff * xDiff + yDiff * yDiff + zDiff * zDiff ;

43

if

( distA > distB ) return idxA ; else return idxB ;

44 45 46 47

}

48 49

};

50 51 52 53 54 55 56

// **************************************************** i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { // i n i t i a l i z e t h e RNG thrust : : default_random_engine rng ( time (0) ) ; t h r u s t : : u n i f o r m _ i n t _ d i s t r i b u t i o n < i n t >u n i D i s t r ( −10000 , ←֓ 10000) ;

57 58

i n t N = atoi ( argv [ 1 ] ) ;

59 60 61 62 63 64 65 66 67 68 69 70 71 72

// g e n e r a t e t h e d a t a on t h e h o s t and move them t o t h e d e v i c e t h r u s t : : d e v i c e _ v e c t o r < i n t >d _ x ( N ) ; t h r u s t : : d e v i c e _ v e c t o r < i n t >d _ y ( N ) ; t h r u s t : : d e v i c e _ v e c t o r < i n t >d _ z ( N ) ; t h r u s t : : h o s t _ v e c t o r < i n t >h _ x ( N ) ; t h r u s t : : h o s t _ v e c t o r < i n t >h _ y ( N ) ; t h r u s t : : h o s t _ v e c t o r < i n t >h _ z ( N ) ; f o r ( i n t i = 0 ; i < N ; i++) h _ x [ i ] = u n i D i s t r ( r n g ) ; d_x = h_x ; f o r ( i n t i = 0 ; i < N ; i++) h _ y [ i ] = u n i D i s t r ( r n g ) ; d_y = h_y ; f o r ( i n t i = 0 ; i < N ; i++) h _ z [ i ] = u n i D i s t r ( r n g ) ; d_z = h_z ;

196

CHAPTER 9. THE THRUST TEMPLATE LIBRARY

73

// i n i t i a l i z e t h e f u n c t o r t h a t w i l l f i n d t h e maximum d i s t a n c e , ←֓ s o t h a t i t h as a c c e s s t o t h e c o o r d i n a t e a r r a y s maxFunct f ; f . x = d_x . data ( ) ; f . y = d_y . data ( ) ; f . z = d_z . data ( ) ;

74

75 76 77 78 79

// r e d u c e t h e i n d e x o f t h e most d i s t a n t p o i n t i n t f u r t h e s t = t h r u s t : : r e d u c e ( t h r u s t : : c o u n t i n g _ i t e r a t o r < i n t ←֓ >(0) , t h r u s t : : c o u n t i n g _ i t e r a t o r < i n t ←֓ >(( N * N − N ) / 2 ) , 0, f) ;

80 81

82

83 84 85

c o u t << ”The most d i s t a n t p a i r i s ” << f u r t h e s t << e n d l ; i n t Ai , A j ; r e v e r s e M a p ( f u r t h e s t , Ai , A j ) ; c o u t << ” F i r s t p o i n t i s ” << A i << ” ( ” << h _ x [ A i ] << ” ” << h _ y ←֓ [ A i ] << ” ” << h _ z [ A i ] << ” ) ” << e n d l ; c o u t << ” Second p o i n t i s ” << A j << ” ( ” << h _ x [ A j ] << ” ” << ←֓ h _ y [ A j ] << ” ” << h _ z [ A j ] << ” ) ” << e n d l ;

86 87 88 89

90

91

// CPU v e r i f i c a t i o n double m a x _ d i s t = 0 , maxI , maxJ ; f o r ( i n t i = 0 ; i < N − 1 ; i++) f o r ( i n t j = i + 1 ; j < N ; j++) { i n t xDiff , yDiff , zDiff ; xDiff = h_x [ i ] − h_x [ j ] ; yDiff = h_y [ i ] − h_y [ j ] ; zDiff = h_z [ i ] − h_z [ j ] ; i n t dist = xDiff * xDiff + yDiff * yDiff + zDiff * zDiff ; i f ( dist > max_dist ) { max_dist = dist ; maxI = i ; maxJ = j ; } } c o u t << e n d l ; c o u t << ” D i s t a n c e by CPU : ” << m a x _ d i s t << e n d l ; c o u t << ” F i r s t p o i n t i s ” << m a x I << ” ( ” << h _ x [ m a x I ] << ” ” <<←֓ h _ y [ m a x I ] << ” ” << h _ z [ m a x I ] << ” ) ” << e n d l ; c o u t << ” Second p o i n t i s ” << m a x J << ” ( ” << h _ x [ m a x J ] << ” ” ←֓ << h _ y [ m a x J ] << ” ” << h _ z [ m a x J ] << ” ) ” << e n d l ;

92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111

112

113

return 0;

114 115

}

11. The all-pairs-shortest paths graph problem can be solved by the dynamic programming algorithm by Floyd and Warshall. Assuming that the graph is described by a V xV adjacency matrix where V is the number of vertices, then the pseudocode of this algorithm below, shows how the solution is obtain in V stages, where each stage involves the update of the distance matrix via the involvement of an intermediate vertex: // i n i t i a l i z a t i o n p h a s e A l l o c a t e a V x V d i s t a n c e m a t r i x , and s e t a l l e l e m e n t s t o i n f i n i t y f o r each vertex v d i s t a n c e [ v ] [ v ] <− 0 f o r each edge ( u , v ) d i s t a n c e [ u ] [ v ] <− a d j ( u , v ) // t h e w e i g h t o f t h e e d g e ( u , v ) // c o m p u t a t i o n p h a s e f o r k from 1 to V // f o r e a c h k i n t e r m e d i a t e v e r t e x f o r i from 1 to V // r e c o n s i d e r e a c h p a i r o f v e r t i c e s f o r j from 1 to V i f distance [ i ] [ j ] > distance [ i ] [ k ] + distance [ k ] [ j ] d i s t a n c e [ i ] [ j ] <− d i s t a n c e [ i ] [ k ] + d i s t a n c e [ k ] [ j ]

197 end i f return distance

Create a Thrust program that could perform the V individual stages of the algorithm in parallel, i.e. parallelize the two inner loops of the algorithm.

Answer Similarly to the previous exercise for finding the pair of most distant points in a set, we can use a thrust::counting iterator to go over all the elements of the adjacency matrix. Fortunately, as all elements are to be considered1 , the mapping between an index and the adjacency matrix coordinates is straightforward as shown in lines 31,32.

The functor that is used to update the distance of two vertices (lines 2241), is setup so that it has access to the adjacency matrix data and the number of vertices V (lines 64,65). Before each phase of the algorithm executed by the for loop of lines 68-76, the index of the vertex to be used as an intermediate, is also set (line 71).

1 2 3 4 5 6

#i n c l u d e <t h r u s t / h o s t v e c t o r . h> #i n c l u d e <t h r u s t / d e v i c e v e c t o r . h> #i n c l u d e <t h r u s t / i t e r a t o r / c o u n t i n g i t e r a t o r . h> #i n c l u d e <t h r u s t / t r a n s f o r m . h> #i n c l u d e <t h r u s t / random . h> #i n c l u d e <s t d i o . h>

7 8

u s i n g namespace s t d ;

9 10 11 12 13 14 15 16 17 18 19

// **************************************************** v o i d p r i n t A d j ( t h r u s t : : h o s t _ v e c t o r < i n t >&a , i n t V ) { f o r ( i n t i = 0 ; i < V ; i++) { f o r ( i n t j = 0 ; j < V ; j++) p r i n t f ( ”%3 i ” , a [ i * V + j ] ) ; p r i n t f ( ” \n” ) ; } }

20 21 22 23 24 25 26

// **************************************************** struct floydFunct { t h r u s t : : d e v i c e _ p t r < i n t >a d j ; int V ; int k ;

27

// comparing d i s t a n c e s o f p a i r s o f p o i n t s A and B a s i n d e x e d by←֓ i ,j __host__ __device__ int operator ( ) ( i n t idx ) { i n t row = idx / V ; i n t col = idx % V ;

28

29 30 31 32 33

i n t currDist = adj [ idx ] ; i n t otherRouteDist = adj [ row * V + k ] + adj [ k * V + col ] ; i f ( otherRouteDist < currDist ) return otherRouteDist ; else return currDist ;

34 35 36 37 38 39

}

40 41

};

42

1 Actually the k th column and row are unchanged during stage k but we do not exclude them from the computation as this would complicate the code. Not only do we need to change the mapping from the thrust::counting iterator to the table cells, but also the results of each stage’s computation cannot be stored in-place.

198 43 44 45 46 47 48

CHAPTER 9. THE THRUST TEMPLATE LIBRARY // **************************************************** i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { // i n i t i a l i z e t h e RNG thrust : : default_random_engine rng ( time (0) ) ; t h r u s t : : u n i f o r m _ i n t _ d i s t r i b u t i o n < i n t >u n i D i s t r ( 1 , 1 0 0 ) ;

49

i n t V = atoi ( argv [ 1 ] ) ;

50 51

// g e n e r a t e t h e d a t a on t h e h o s t and move them t o t h e d e v i c e t h r u s t : : d e v i c e _ v e c t o r < i n t >d _ a d j ( V * V ) ; t h r u s t : : h o s t _ v e c t o r < i n t >h _ a d j ( V * V ) ; f o r ( i n t i = 0 ; i < V * V ; i++) h_adj [ i ] = uniDistr ( rng ) ; f o r ( i n t i = 0 ; i < V ; i++) h_adj [ i * V + i ] = 0;

52 53 54 55 56 57 58 59

d_adj = h_adj ;

60 61

// i n i t i a l i z e t h e f u n c t o r s o t h a t coordinate arrays floydFunct f ; f.V = V; f . adj = d_adj . data ( ) ;

62

63 64 65

i t h as a c c e s s t o t h e ←֓

66

printAdj ( h_adj , V ) ; f o r ( i n t k = 0 ; k < V ; k++) { // s e t t h e f u n c t o r t o u s e t h e k−th node a s i n t e r m e d i a t e f.k = k; t h r u s t : : t r a n s f o r m ( t h r u s t : : c o u n t i n g _ i t e r a t o r <i n t >(0) , t h r u s t : : c o u n t i n g _ i t e r a t o r <i n t >(V * V ) , d_adj . begin () , f) ; }

67 68 69 70 71 72 73 74 75 76 77

p r i n t f ( ”−−−−−−−−−−−−−−−−\n” ) ; h_adj = d_adj ; printAdj ( h_adj , V ) ;

78 79 80 81

return 0;

82 83

}

Chapter 10

High-level multi-threaded programming with the Qt library Exercises 1. Use the QtConcurrent functionality to implement a prime number checker.

Compare it in terms of speed, efficiency and programming effort to your QThread-based attempt of the previous exercise.

Answer The solution involves a NumberRange class for representing parts of the number range to scan for prime numbers. When an instance of this class is processed by the static NumberRange::process method, the prime numbers in that range are deposited in the designated repository as referenced by the repo pointer.

The main method uses the user-supplied desired length of the NumberRange instances (variable PARTLEN), to populate a vector of them (lines 80-90).

This vector is subsequently processed by using the “mapping” functionality of the QtConcurrent namespace (line 92). The blockingMap function applies the NumberRange::process method on all elements of the input vector, effectively calculating the desired results.

1 2 3 4 5 6 7

#i n c l u d e <QtConcurrent / QtConcurrentMap> #i n c l u d e <QMutex> #i n c l u d e <QMutexLocker> #i n c l u d e <math . h> #i n c l u d e < s t d l i b . h> #i n c l u d e <i o s t r e a m > #i n c l u d e <v e c t o r >

8 9

u s i n g namespace s t d ;

10 11 12 13 14 15 16 17

//=========================================== // u s e d f o r s t o r i n g t h e r e s u l t s i n a t h r e a d −s a f e manner c l a s s Repository { private : v e c t o r <i n t > r e s u l t ; QMutex l ;

199

200CHAPTER 10. HIGH-LEVEL MULTI-THREADED PROGRAMMING WITH THE QT LIBRARY 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71

public : void store ( i n t i ) ; v e c t o r <i n t > * g e t R e s u l t ( ) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void Repository : : store ( i n t i ) { Q M u t e x L o c k e r m l (&l ) ; result . push_back ( i ) ; } //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− v e c t o r <i n t > * R e p o s i t o r y : : g e t R e s u l t ( ) { r e t u r n &r e s u l t ; } //=========================================== c l a s s NumberRange { private : i n t first , last ; Repository * repo ; public : N u m b e r R a n g e ( ) {} N u m b e r R a n g e ( i n t a , i n t b , R e p o s i t o r y * r ) : f i r s t ( a ) , l a s t ( b ) , ←֓ r e p o ( r ) {} v o i d s e t F i r s t ( i n t f ) { f i r s t=f ; } v o i d s e t L a s t ( i n t l ) { l a s t=l ; } v o i d s e t R e p o ( R e p o s i t o r y * r ) { r e p o=r ; } s t a t i c v o i d p r o c e s s ( N u m b e r R a n g e &) ; }; //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− v o i d N u m b e r R a n g e : : p r o c e s s ( N u m b e r R a n g e &n ) { f o r ( i n t i=n . f i r s t ; i<=n . l a s t ; i+=2) { b o o l i s P r i m e=t r u e ; i n t j =3; i n t limit = sqrt ( i ) ; w h i l e ( j<= l i m i t && i s P r i m e ) { i f ( i % j == 0 ) i s P r i m e= f a l s e ; j+=2; } i f ( isPrime ) n . r e p o−>s t o r e ( i ) ; } } //=========================================== i n t main ( i n t argc , char * argv [ ] ) { int a , b ; a=a t o i ( a r g v [ 1 ] ) ; b=a t o i ( a r g v [ 2 ] ) ; i n t P A R T L E N=a t o i ( a r g v [ 3 ] ) ;

72 73 74 75 76

// make s u r e a and b a r e odd a = ( a % 2 == 0 ) ? a+1 : a ; b = ( b % 2 == 0 ) ? b−1 : b ; P A R T L E N = ( P A R T L E N % 2 == 0 ) ? P A R T L E N : P A R T L E N +1; // ←֓ PARTLEN s h o u l d be even

77 78 79 80 81 82 83 84 85 86 87 88 89

Repository rep ; v e c t o r <N u m b e r R a n g e > d a t a ; i n t n u m P a r t s = ( i n t ) c e i l ( ( b−a ) * 1 . 0 / P A R T L E N ) ; data . resize ( numParts ) ; i n t f , l=a −2; f o r ( i n t i =0; i<n u m P a r t s ; i++) { f=l +2; l=f+P A R T L E N ; i f ( l>b ) l=b ; data . at ( i ) . s e t F i r s t ( f ) ; data . at ( i ) . s e t L a s t ( l ) ;

201 d a t a . a t ( i ) . s e t R e p o (& r e p ) ;

90

}

91 92

Q t C o n c u r r e n t : : b l o c k i n g M a p ( data ,

93

NumberRange : : process ) ;

94

c o u t << ” T o t a l p r i m e s f o u n d ” << r e p . g e t R e s u l t ( )−>s i z e ( ) << ←֓ endl ; return 0;

95

96 97

}

To run the above program in order to find all the prime numbers in the range [103 , 108 ], one has to use the following command-line: $

p r i m e C h e c k e r _ V 3 1000 1 0 0 0 0 0 0 0 0 100000

The last parameter concerns the value of PARTLEN. In terms of performance, an identical set of tests as in the previous exercise, results in an average execution time of 0.42 sec, for PARTLEN = 10000, which is on par with the monitor-based solution. In terms of development effort, the QtConcurrent based solution is marginally shorter.

2. Create a big array of randomly generated 2D coordinates (x, y). Each of the coordinates should be a number in the range [-1000, 1000]. Use appropriate QtConcurrent functions to find the points that are in a ring of distances between 100 and 200 from the point of origin. Compare the performance of your solution against a sequential implementation.

Answer In the following program a Point structure is used to hold randomly generated data. A vector<Point> container holds the input data, instead of a vector<Point*>, in order to simplify and speedup memory management.

Upon resizing the vector container in line 57, all the memory required for holding the Point instances is reserved.

Also, in order to avoid the use of the costly sqrt function, the filtering is performed on the squares of the distances.

1 2 3 4 5 6 7 8

#i n c l u d e <QtConcurrent / QtConcurrentMap> #i n c l u d e <QtConcurrent / Q t C o n c u r r e n t F i l t e r > #i n c l u d e <QThreadPool> #i n c l u d e <QTime> #i n c l u d e <b o o s t / b i n d . hpp> #i n c l u d e <v e c t o r > #i n c l u d e <i o s t r e a m > #i n c l u d e < s t d l i b . h>

9 10

u s i n g namespace s t d ;

11 12 13 14 15 16 17 18

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− s t r u c t Point { int x , y ; i n t d2 ; // d i s t a n c e s q u a r e Point () {}; };

19 20 21 22 23 24

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− void d2Calc ( Point & p ) { p . d2 = p . x * p . x + p . y * p . y ; }

25 26 27

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− bool inRing ( Point p , i n t minDist , i n t maxDist )

202CHAPTER 10. HIGH-LEVEL MULTI-THREADED PROGRAMMING WITH THE QT LIBRARY 28

{ i n t d = p . d2 ; i f ( d >= m i n D i s t && d <= m a x D i s t ) return true ; return f a l s e ;

29 30 31 32 33

}

34 35 36 37 38 39 40 41 42

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t main ( i n t argc , char * argv [ ] ) { int numThr ; int numPoints ; vector < Point > data ; d o u b l e tio , t f u l l ; QTime t ;

43

if

44

( argc < 3) {

45

c e r r << a r g v [ 0 ] << ” numThreads numPoints \n” ; exit (1) ;

46 47

} // f o r t i m i n g t . start () ;

48 49 50 51

numThr = atoi ( argv [ 1 ] ) ; numPoints = atoi ( argv [ 2 ] ) ;

52 53 54

// i n i t i a l i z e d a t a a r r a y srand ( time (0) ) ; data . resize ( numPoints ) ; // one memory a l l o c a t i o n o n l y f o r ( i n t i = 0 ; i < n u m P o i n t s ; i++) { d a t a [ i ] . x = r a n d ( ) % 2000 − 1 0 0 0 ; d a t a [ i ] . y = r a n d ( ) % 2000 − 1 0 0 0 ; }

55 56 57 58 59 60 61 62 63

tio = t . elapsed ( ) / 1 0 0 0 . 0 ; // c o n t r o l t h e number o f t h r e a d s Q T h r e a d P o o l : : g l o b a l I n s t a n c e ( )−>s e t M a x T h r e a d C o u n t ( n u m T h r ) ;

64 65 66 67

// c a l c u l a t e d i s t a n c e s f i r s t Q t C o n c u r r e n t : : b l o c k i n g M a p ( data , d2Calc ) ;

68 69 70

// f i l t e r p o i n t e s b a s e d on d i s t a n c e n e x t Q t C o n c u r r e n t : : b l o c k i n g F i l t e r ( d a t a , b o o s t : : b i n d ( i n R i n g , _1 , ←֓ 100 * 1 0 0 , 200 * 2 0 0 ) ) ;

71 72

73

tfull = t . elapsed () / 1000.0; c o u t << ” T o t a l : ” << t f u l l << ” Comp . t i m e : ” << t f u l l − t i o << ←֓ endl ; c o u t << d a t a . s i z e ( ) << e n d l ; return 0;

74 75

76 77 78

}

3. Use the QtConcurrent functionality to implement a parallel bucketsort.

Does the number of buckets play a significant role in your implementation’s performance?.

Answer In the following Listing, the bucketsort template function implements a variation of the algorithm, whereas (i) the input is scanned and distributed to buckets (lines 43-48), (ii) the buckets are sorted separately via STL’s sort function (lines 51-54), and (iii) the resulting partial sorted arrays are copied back to the original data repository (lines 57-62).

The number of buckets matches the number of threads used by QThreadPool, and for this reason the number of buckets is invariably connected with the level of concurrency.

203 To avoid expensive synchronization logic, such as using mutices to control access to the buckets (STL’s vector is not thread safe), each thread is responsible for a dedicated bucket. The bucket array address and bucket index for which a thread is responsible for, are passed as parameters to the filterOp template function that is invoked by QtConcurrent::run in line 45. The minimum value of the input range and the value range per bucket are also passed to enable filterOp to determine the bucket that an item belongs to.

Finally, the sorting is performed via the unary front-end vecSort function that is “mapped” on all the buckets. To this end, a vector of vector<T> pointers is populated in lines 52-53 prior to the application of QtConcurrent::blockingMap. 1 2 3 4 5 6 7 8 9 10

#i n c l u d e <QtConcurrent / QtConcurrentMap> #i n c l u d e <QtConcurrent / QtConcurrentRun> #i n c l u d e <QFuture> #i n c l u d e <QThreadPool> #i n c l u d e <QTime> #i n c l u d e <b o o s t / b i n d . hpp> #i n c l u d e <v e c t o r > #i n c l u d e <a l g o r i t h m > #i n c l u d e <i o s t r e a m > #i n c l u d e < s t d l i b . h>

11 12

u s i n g namespace s t d ;

13 14 15

c o n s t i n t M I N V = −1000; const i n t MAXV = 1000;

16 17 18

19 20 21 22 23 24 25 26 27

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− t e m p l a t e <typename T> i n l i n e v o i d f i l t e r O p ( v e c t o r <T> * v , t a r g e t B u c k e t I d x , v e c t o r <T> * b , T m , T b W i d t h ) { f o r ( i n t i =0; i< v−>s i z e ( ) ; i++) { T t m p = v−>a t ( i ) ; i n t bucketIdx = ( tmp − m ) / bWidth ; i f ( b u c k e t I d x == t a r g e t B u c k e t I d x ) b−>p u s h _ b a c k ( t m p ) ; } }

i n t ←֓

28 29 30 31 32 33

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− t e m p l a t e <typename T> i n l i n e v o i d v e c S o r t ( v e c t o r <T> * v ) { s o r t ( v−>b e g i n ( ) , v−>e n d ( ) ) ; }

34 35 36

37 38

39 40

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− t e m p l a t e <typename T> v o i d b u c k e t S o r t ( v e c t o r <T> &d a t a , c o n s t T &←֓ m i n V , c o n s t T &m a x V ) { i n t n u m B u c k e t s = Q T h r e a d P o o l : : g l o b a l I n s t a n c e ( )−>←֓ m a x T h r e a d C o u n t ( ) ; // match b u c k e t s w i t h t h e number o f ←֓ threads v e c t o r <T> b u c k e t [ n u m B u c k e t s ] ; T bwidth = ( maxV − minV ) / numBuckets ; // r a n g e p e r b u c k e t

41 42 43 44 45

// l a u n c h a s many r e q u e s t s a s t h e number o f b u c k e t s / t h r e a d s Q F u t u r e <v o i d> f [ n u m B u c k e t s ] ; f o r ( i n t i =0; i<n u m B u c k e t s ; i++) f [ i ] = Q t C o n c u r r e n t : : r u n ( b o o s t : : b i n d ( f i l t e r O p <T >, &d a t a , ←֓ i , bucket + i , minV , bwidth ) ) ;

46 47 48

f o r ( i n t i =0; i<n u m B u c k e t s ; i++) f [ i ] . waitForFinished () ;

49 50 51

// s o r t e a c h o f t h e b u c k e t s s e p a r a t e l y v e c t o r < v e c t o r <T> *> v v ;

204CHAPTER 10. HIGH-LEVEL MULTI-THREADED PROGRAMMING WITH THE QT LIBRARY f o r ( i n t i =0; i<n u m B u c k e t s ; i++) v v . p u s h _ b a c k (&( b u c k e t [ i ] ) ) ; Q t C o n c u r r e n t : : b l o c k i n g M a p ( vv , v e c S o r t <T >) ;

52 53 54 55

// copy back t o o r i g i n a l v e c t o r i n t l o c =0; f o r ( i n t i =0; i<n u m B u c k e t s ; i++) { f o r ( i n t j =0; j<b u c k e t [ i ] . s i z e ( ) ; j++) d a t a [ l o c ++] = b u c k e t [ i ] . a t ( j ) ; }

56 57 58 59 60 61 62 63

}

64 65 66 67 68 69 70 71 72

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− i n t main ( i n t argc , char * argv [ ] ) { int numThr ; int numItems ; vector < i n t > data ; d o u b l e tio , t f u l l ; QTime t ;

73

if {

74 75

( argc < 3) c e r r << a r g v [ 0 ] << ” numThreads numItems \n” ; exit (1) ;

76 77

} // f o r t i m i n g t . start () ;

78 79 80 81

numThr = atoi ( argv [ 1 ] ) ; numItems = atoi ( argv [ 2 ] ) ;

82 83 84

// i n i t i a l i z e d a t a a r r a y srand ( time (0) ) ; data . resize ( numItems ) ; // one memory a l l o c a t i o n o n l y f o r ( i n t i = 0 ; i < n u m I t e m s ; i++) { d a t a [ i ] = ( r a n d ( ) % ( M A X V−M I N V ) ) + M I N V ; }

85 86 87 88 89 90 91 92

tio = t . elapsed ( ) / 1 0 0 0 . 0 ; // c o n t r o l t h e number o f t h r e a d s Q T h r e a d P o o l : : g l o b a l I n s t a n c e ( )−>s e t M a x T h r e a d C o u n t ( n u m T h r ) ;

93 94 95 96

b u c k e t S o r t <i n t >( d a t a , M I N V , M A X V ) ;

97 98

tfull = t . elapsed () / 1000.0; c o u t << ” T o t a l : ” << t f u l l << ” Comp . t i m e : ” << t f u l l − t i o ←֓ << e n d l ; return 0;

99 100

101 102

}

4. Create a program that utilizes the QtConccurrent functionality, for calculating the definite integral of a function between a user-supplied range.

Answer

Chapter 11

Load Balancing Exercises 1. Design a brute-force/exhaustive algorithm for determining the optimum subset of N nodes/processors in the case of static load balancing. What is the complexity of your algorithm?

Answer In general, finding the optimum subset of nodes requires enumerating all the possible subsets, i.e. 2N − 1 if we ignore the empty set. The enumeration can be implemented in a variety of ways, the following recursive algorithm being a generic approach that does not impose any restrictions on N .

The following description is based on the assumption that a function to evaluate the execution time on a subset S exists (symbolic name eval() used to represent this function).

The recursive auxiliary function OptimumSetAux, calls itself by including (line 15) or excluding (line 16) the i-th element of the processor set P , until the parameter i becomes equal to N (base case of line 8), which indicates that a subset is ready to be evaluated.

1: procedure OptimumSet(P ) 2: bestS ← ∅ 3: bestT ← inf 4: OptimumSetAux(P ,∅, 0, bestS, bestT ) 5: Return bestS 6: end procedure 7: procedure OptimumSetAux(P , S, i, bestS, bestT ) 8: if i = N then 9: T ← eval(S) 10: if T < bestT then 11: bestS ← S 12: bestT ← T 13: end if 14: else 15: OptimumSetAux(P , S ∪ Pi , i + 1, bestS, bestT ) 205

206

CHAPTER 11. LOAD BALANCING 16:

OptimumSetAux(P , S, i + 1, bestS, bestT )

17: end if 18: end procedure

The complexity of this algorithm depends on the cost of the eval() function. Given that there are 2N subsets to be evaluated, we can conclude that the complexity ∈ Ω(2N ).

2. Design a greedy algorithm for determining the optimum subset of N nodes/processors in the case of static load balancing. Will you always get the same solution as the one provided by an exhaustive algorithm?

Answer A greedy algorithm can be employed that initially sorts the nodes in descending order of their computational power, i.e. ascending of their pi parameter as identified in Table 8.1.

The following algorithm iterates over the sorted nodes, adding one node at a time for as long as the execution time decreases (while loop terminates otherwise via line 14) or until all nodes are examined (line 6). The complex condition of line 9 ensures that sets leading to invalid solutions (i.e. with one or more partk ≤ 0) are not considered.

Actually, the condition in line 9 can be simply “if T < bestT then”, as a negative part to a node enlarges the parts assigned to other nodes, inevitable making the execution time higher.

1: procedure GreedySet(P ) 2: P ′ ← sort(P ) 3: bestS ← P0′ 4: bestT ← eval(bestS) 5: i←1 6: while i < N do 7: S ← bestS ∪ Pi′ 8: T ← eval(S) 9: if T < bestT AND ∀Pk ∈ S, partk > 0 then 10: bestS ← S 11: bestT ← T 12: i←i+1 13: else 14: Break 15: end if 16: end while 17: Return bestS 18: end procedure A heuristic algorithm cannot produce an identical solution to an exhaustive algorithm, unless this is done by chance, or special circumstances/problem settings allow the generation of an optimum solution via a greedy approach.

3. The closed-form solutions in Section 8.3.3.1 for the N-port communication setup, are based on the assumption that the master participates in the processing of the load. Derive the equations that would govern the solution

207 if the master abstained from this task, facilitating only I/O functionality instead.

Answer The answer to this question is already provided in Section 8.3.3.1, albeit in a condensed form, without a length explanation. This exercise serves the purpose of testing student understanding and their ability to follow the proper steps towards the derivation of a solution.

Once we remove the master node from computation duties, Equation 8.9 becomes: (i)

(i)

(j)

(j)

(j) tdistr + t(i) comp + tcoll = tdistr + tcomp + tcoll ⇒

li (a parti L + b) + pi (parti L + ei ) + li (c parti L + d) = lj (a partj L + b) + pj (partj L + ej ) + lj (c partj L + d)

(11.1)

To solve the problem, we need to find one of the assigned parts, e.g. part1 . So from the above equation we can express all parti with i 6= 1 as a function of part1 : parti = part1

p1 e1 − pi ei + (l1 − li )(b + d) l1 (a + c) + p1 + li (a + c) + pi L(li (a + c) + pi )

(11.2)

Then, the normalization equation allows us to get a solution for part1 : N −1 X j=1

partj = 1 ⇒

l1 (a + c) + p1 p1 e1 − p2 e2 + (l1 − l2 )(b + d) + + ... l2 (a + c) + p2 L(l2 (a + c) + p2 ) p1 e1 − pN −1 eN −1 + (l1 − lN −1 )(b + d) l1 (a + c) + p1 + =1⇒ +part1 lN −1 (a + c) + pN −1 L(lN −1 (a + c) + pN −1 )   N −1 N −1 X X l (a + c) + p pj ej − p1 e1 + (lj − l1 )(b + d) 1 1 part1 1 + = 1+ ⇒ l (a + c) + pj L(lj (a + c) + pj ) j=2 j j=2 part1 + part1

part1

N −1 X pj ej − p1 e1 + (lj − l1 )(b + d) l1 (a + c) + p1 =1+ ⇒ l (a + c) + p L(lj (a + c) + pj ) j j j=2 j=1 PN −1 p e −p e1 +(lj −l1 )(b+d) 1 + L−1 j=2 j j l1j (a+c)+p j part1 = (11.3) PN −1 −1 (l1 (a + c) + p1 ) j=1 (lj (a + c) + pj )

N −1 X

4. Write a program that calculates part0 from Equations 8.11 and 8.39 in a linear number of steps.

Answer The following listing contains two functions that calculate part0 and subsequently all parti , for the two problem settings that Equations 8.11 and 8.39 correspond to. The “trick” (especially for Equation 8.39 that involves double summations) is to pre-compute and reuse terms, such as e.g. the

208

CHAPTER 11. LOAD BALANCING Pk

j=1 bj term of Equation. 8.39, via lines 51-53 in OneportBlockSingleInst.

The two functions also return the predicted total execution time.

It is important to note that OneportBlockSingleInst just implements Equations 8.38 and 8.39, without ordering the nodes as mandated by Theorem 8.3.3.

1

#i n c l u d e <i o s t r e a m >

2 3 4 5

6

7 8

9 10 11

u s i n g namespace s t d ; // ********************************************** // S t o r e s t h e computed p a r t i i n a r r a y p a r t and t h e r e t u r n s t h e ←֓ t o t a l e x e c u t i o n time d o u b l e N p o r t B l o c k S i n g l e I n s t ( d o u b l e * p , d o u b l e * e , d o u b l e * l , ←֓ d o u b l e a , d o u b l e b , d o u b l e c , d o u b l e d , i n t N , l o n g L , ←֓ double * part ) { double lacp [ N ] ; // s h o u l d be r e p l a c e d by dynamic a l l o c a t i o n i f ←֓ N i s big double sumTerm [ N ] ; double totalTime ; double nomin , denom ;

12

f o r ( i n t i = 0 ; i < N ; i++) lacp [ i ] = l [ i ] * ( a + c ) + p [ i ] ;

13 14 15

f o r ( i n t i = 1 ; i < N ; i++) sumTerm [ i ] = p [ 0 ] * e [ 0 ] − p [ i ] * e [ i ] − l [ i ] * ( b + d ) ;

16 17 18

nomin = 0; f o r ( i n t i = 1 ; i < N ; i++) n o m i n += s u m T e r m [ i ] / l a c p [ i ] ; nomin = 1 − nomin / L ;

19 20 21 22 23

denom = 1; f o r ( i n t i = 1 ; i < N ; i++) d e n o m += 1 / l a c p [ i ] ; d e n o m *= p [ 0 ] ;

24 25 26 27 28

part [ 0 ] = nomin / denom ;

29 30

f o r ( i n t i = 1 ; i < N ; i++) p a r t [ i ] = p a r t [ 0 ] * p [ 0 ] / l a c p [ i ] + s u m T e r m [ i ] / ( L * l a c p [ i←֓ ]) ;

31 32

33

totalTime = p [ 0 ] * ( part [ 0 ] * L + e [ 0 ] ) ; return totalTime ;

34 35 36

}

37 38 39

40

41 42

43

// ********************************************** // S t o r e s t h e computed p a r t i i n a r r a y p a r t and t h e r e t u r n s t h e ←֓ t o t a l e x e c u t i o n time d o u b l e O n e p o r t B l o c k S i n g l e I n s t ( d o u b l e * p , d o u b l e * e , d o u b l e l , ←֓ double *b , i n t N , long L , double * part ) { d o u b l e p l [ N ] ; // s h o u l d be r e p l a c e d by dynamic a l l o c a t i o n i f N ←֓ i s big double bSum [ N ] ;

44 45 46

double totalTime ; double nomin , denom ;

47 48 49

f o r ( i n t i = 0 ; i < N ; i++) pl [ i ] = p [ i ] + l ;

50 51 52 53

bSum [ 0 ] = 0; f o r ( i n t i = 1 ; i < N ; i++) bSum [ i ] = b [ i ] + bSum [ i − 1 ] ;

54 55 56 57 58

nomin = 0; f o r ( i n t i = 1 ; i < N ; i++) n o m i n += l / p l [ i ] * b S u m [ i ] ; n o m i n /= L ;

209 n o m i n ++; f o r ( i n t i = 1 ; i < N ; i++) n o m i n += l / p l [ i ] ;

59 60 61 62

denom = 0; f o r ( i n t i = 0 ; i < N ; i++) d e n o m += 1 / p l [ i ] ; d e n o m *= p l [ 0 ] ;

63 64 65 66 67

part [ 0 ] = nomin / denom ;

68 69

f o r ( i n t i = 1 ; i < N ; i++) p a r t [ i ] = p a r t [ 0 ] * p l [ 0 ] / p l [ i ] − l / ( L * p l [ i ] ) * b S u m [ i ] ←֓ − l / pl [ i ] ;

70 71

72

totalTime = p [ 0 ] * part [ 0 ] * L ; return totalTime ;

73 74 75

}

76 77 78 79 80 81 82 83 84 85

// ********************************************** i n t main () { // s m a l l t e s t e r d a t a double p [ 4 ] = { 1 , 2 , 3 , 4 } ; double e [ 4 ] = { 1 , 1 , 1 , 1 } ; d o u b l e b [ 4 ] = { 1 0 0 0 , 1 0 0 0 , 1 0 0 0 , 1000 } ; double l [ 4 ] = { 1 , 1 , 1 , 1 } ; double part [ 4 ] ;

86

c o u t << ” t : ” << N p o r t B l o c k S i n g l e I n s t ( p , e , l , 1 , 0 , 1 , 0 , 4 , ←֓ 1 0 0 0 0 0 0 , p a r t ) << e n d l ; f o r ( i n t i = 0 ; i < 4 ; i++) c o u t << p a r t [ i ] << ” ” ; c o u t << e n d l ;

87

88 89 90 91

c o u t << ” t : ” << O n e p o r t B l o c k S i n g l e I n s t ( p , e , 1 , b , 4 , ←֓ 1 0 0 0 0 0 0 , p a r t ) << e n d l ; f o r ( i n t i = 0 ; i < 4 ; i++) c o u t << p a r t [ i ] << ” ” ; c o u t << e n d l ;

92

93 94 95 96

return 0;

97 98

}

5. Solve the example at the end of Section 8.3.3.2, by reversing the computing power of the nodes: p0 = 4·6.6·10−8 , p1 = 3·(6.6·10−8 ), p2 = 2·(6.6·10−8 ) and p3 = (6.6 · 10−8 ).

Answer

Before applying the equations, the nodes (excluding the master) should be sorted according to their bi (pi + l) property. This presents a challenge, as while we can switch the place of a node, the bi overheads are node order specific and not node specific. Hence. changing the order of a node, potentially causes a change in its ei parameter. Fortunately, the closed-form solutions of Section 11.3.3.2 allow the easy calculation of the predicted time for the different permutations of the nodes (surely not advisable for large N , but quite doable for the particular problem).

The results show that load should be distributed in the order P3 , P2 , P1 . The corresponding parts are part0 = 0.156456, part3 = 0.444968, part2 = 0.237034 and part1 = 0.161541, with a total execution time of 1.028sec.

6. An alternative to having the master node distribute the load, is to have all the nodes access a network filesystem and retrieve the data from there.

210

CHAPTER 11. LOAD BALANCING What kind of communication configuration would correspond to such an arrangement? Calculate the partitioning that would be produced in this case, for the same problem setting as the previous question.

Answer Such a design corresponds to a N-port communication configuration. If we assume that all nodes (including P0 which was previously considered as having the load locally) perform the same sequence of getting the load, processing it and sending the results back to the filesystem server, then the minimum execution time is achieved by having all nodes finish at the same time instance, i.e. for every pair of Pi and Pj , we have:

(i)

(i)

(j)

(j)

(j) tdistr + t(i) comp + tcoll = tdistr + tcomp + tcoll ⇒

l(parti L+bi )+pi parti L+l parti L = l(partj L+bj )+pj partj L+l partj L ⇒ parti L(pi + 2 l) = partj L(pj + 2 l) + l(bj − bi ) ⇒ parti = partj

l(bj − bi ) pj + 2 l + pi + 2 l L(pi + 2 l)

(11.4)

Thus, we can express all parti as a function of part0 : parti = part0

p0 + 2 l l(b0 − bi ) + pi + 2 l L(pi + 2 l)

(11.5)

The normalization equation can then yield a closed-form solution for part0 : N −1 X i=0

parti = 1 ⇒ part0

N −1 l X b0 − bi p0 + 2 l + =1⇒ pi + 2 l L i=1 pi + 2 l i=0 PN −1 −b0 1 + Ll i=1 pbii+2 l part0 = PN −1 (p0 + 2 l) i=0 (pi + 2 l)−1

N −1 X

(11.6)

If we assume that the nodes are ordered as specified by the problem statement, then we have b1 = b2 = 2 · 3 · 2160B and b0 = b3 = 3 · 2160B. Using the above equations produces part0 = 0.125605, part1 = 0.165952, part2 = 0.24447 and part1 = 0.463997, with a total execution time of 2.097sec.

7. The partitioning performed using DLT in the kernel convolution example, returns the percent of the input bytes that should be assigned to each node? How can we convert this number to pixels? One may argue that we should instead use image rows. How can we convert the result of our analysis to rows?

Answer As long as there is a linear relationship between the data volume and the desired load unit, there is no need for a conversion. The parti results are

211 percentages and can be used without conversion to split the number of rows as well as the number of pixels.

8. Use the tiobench utility available at http://sourceforge.net/projects/ tiobench/ to measure the performance you can get by concurrent access to a network filesystem (e.g. a NFS volume). Calculate the collective throughput of the server versus the number of threads used.

Answer No model answer is available for this exercise. The students should experiment with the tiobench utility and plot the results. A plot similar to Figure 8.6 should be produced.

9. One way that can be used to improve the distribution cost of input data is to compress them. What kind of compression algorithms could be used in this case? How could we adapt the cost models to reflect this change?

Answer There are two distinct possibilities: (a) The input data are already in compressed form. This imposed restrictions on the compression algorithm and data encoding format, as they should permit random access to the input data. This immediately rules out dynamic compression schemes (e.g. dictionary based) and enforces some sort of partitioning of the compressed data, in the form of packets or blocks. A static Huffman/arithmetic code compression algorithm would be suitable for the task. (b) The data are compressed before the distribution phase. Any compression algorithm could be used in this case, as long as the compressiondecompression overhead does not exceed the gains from the reduced communication time.

Let’s assume that a node decompresses the input upon receiving it, and compresses the results prior to sending them back, spending compri time originalDataSize per load unit. If on average, the compression ratio is cr = compressedDataSize , then, the cost models would have to be modified as follows:

t(i) comp = pi (parti L + ei ) + compri parti L = (pi + compri )(parti L +

(i)

(i)

(11.7)

L a + b) = li ( parti L + b) cr cr

(11.8)

c L + d) = li ( parti L + d) cr cr

(11.9)

tdistr = li (a parti tcoll = li (c parti

pi ei ) pi + compri

Equations 11.7-11.9 adhere to the formulation of the book’s Equations 8.5, 8.7 and 8.8(see pages 583, 585 and 586). This is a clear indication that existing analysis results could be applicable for such a design, by adapting the model parameters.

212

CHAPTER 11. LOAD BALANCING

10. Use the DLTlib library to calculate the optimum partitioning for a “query processing” application, where the communication cost is independent of the workload, consisting of a query during distribution and the result during the collection. You can assume that b = d = 1, the workload consists of L = 106 items, and the parallel platform is made of 10 compute nodes connected in a single level tree, with pi = (i + 1) · 0.01sec/item ∀i ∈ [0, 9] and l = 0.001sec/item.

What would happen if the communication was 10 times slower?

Answer The following listing illustrates how the appropriate DLTlib function can be called and the results printed-out: 1 2 3

#i n c l u d e <t i m e . h> #i n c l u d e <s t d i o . h> #i n c l u d e <i o s t r e a m >

4 5

u s i n g namespace s t d ;

6 7 8

9 10 11 12

//−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− // DLTlib s p e c i f i c d e f i n i t i o n s t h a t need t o be u s e d by t h e ←֓ library . long global_random_seed ; #d e f i n e MAX NODE DEGREE 10 #i n c l u d e ” d l t l i b . cpp ” //−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

13 14 15 16 17 18 19 20 21

i n t main () { double p = 0 . 0 1 ; double l = 0 . 0 0 1 ; long int L = 1000000; double b = 1 , d = 1 ; // STEP 1 Network platform ; platform

// o b j e c t r e p r e s e n t i n g

p a r a l l e l ←֓

22 23 24

25 26

27 28 29 30 31

32

// STEP 2 // i n s e r t one−by−one t h e n o d e s t h a t make up t h e machine . P0 by ←֓ default p a r t i c i p a t e s in // t h e c o m p u t a t i o n p l a t f o r m . I n s e r t N o d e ( ( c h a r * ) ”P0” , p , 0 , ( c h a r * ) N U L L , l , ←֓ true ) ; f o r ( i n t i = 1 ; i < 1 0 ; i++) { char buff [ 1 0 ] ; s p r i n t f ( b u f f , ”P%i ” , i ) ; p l a t f o r m . I n s e r t N o d e ( b u f f , ( i + 1 ) * p , 0 , ( c h a r * ) ”P0” , l ←֓ , true ) ; }

33 34 35

36

// STEP 3 // S o l v e t h e p a r t i t i o n i n g problem f o r 1−p o r t , b l o c k −t y p e ←֓ computation platform . SolveQuery (L , b , d ) ;

37 38 39 40 41

// p r i n t o u t t h e r e s u l t s , i f t h e s o l u t i o n i s v a l i d i f ( p l a t f o r m . v a l i d == 1 ) { c o u t << ” P r e d i c t e d e x e c u t i o n t i m e : ” << p l a t f o r m . ←֓ S i m u l a t e Q u e r y ( b , d , 0 ) << e n d l ;

42 43 44

45

46 47 48 49

// STEP 4 // Compute n o d e s a r e s t o r e d i n a p u b l i c l i n k e d − l i s t t h a t ←֓ allows // r e a r r a n g e m e n t t o t h e o r d e r o f d i s t r i b u t i o n and ←֓ collection c o u t << ” S o l u t i o n i n t e r m s o f l o a d p e r c e n t : \ n” ; Node *h = platform . head ; w h i l e ( h != N U L L ) {

213 // S i n g l e i n s t a l l m e n t c a s e . . . c o u t << h−>n a m e << ” \ t ” << h−>p a r t << e n d l ; h = h−>n e x t _ n ;

50 51 52

} } else c o u t << ” S o l u t i o n c o u l d n o t be f o u n d \n” ; return 0;

53 54 55 56 57 58

}

A noteworthy addition to the sample provided in the book, is line 10 that changes the maximum number of child nodes of a machine. The default of the library is only four, far less than the problem requirement of nine, which if left unchanged would cause the program to fail during execution with a protection fault.

This program can be compiled and run as follows: g++ D L T e x e r c i s e . c p p −o D L T e x e r c i s e −I . . / . . / D L T l i b −l g l p k ./ DLTexercise Predicted execution time : 3414.18 S o l u t i o n i n terms of load p e r c e n t : P0 0.341418 P1 0.170709 P2 0.113806 P3 0.0853543 P4 0.0682834 P5 0.0569028 P6 0.0487738 P7 0.042677 P8 0.0379351 P9 0.0341416

$

$

Finally, testing for a 10 times slower communication medium is as simple as changing the l variable value of line 17 to 0.01. The reported results are almost identical to the previous ones, as the communication cost is negligible in comparison to the computation: Predicted execution time : 3414.22 S o l u t i o n i n terms of load p e r c e n t : P0 0.341422 P1 0.17071 P2 0.113806 P3 0.085354 P4 0.0682828 P5 0.056902 P6 0.0487729 P7 0.042676 P8 0.037934 P9 0.0341404

11. Derive the equivalent of Equations 8.47 and 8.48 for arbitrary a and c, and implement them as part of a partitioning function.

Answer The question is essentially to solve the N-port, block-type, single-installment distribution problem (i.e. what Section 8.3.3.1 covers) for the case when P0 is not the load originating node. As such, P0 has to incur communication cost both for the distribution and the result collection phases.

The solution is identical to the one in Exercise 3. The only difference is that while in Exercise 3 we assume that P0 is not participating in the load processing, in this exercise we assume that P0 does not hold the load originally, i.e. there is another master node that is only distributing the

214

CHAPTER 11. LOAD BALANCING data. We can reuse the solution of Exercise 3, with a simple change of the indices, so that all parti are expressed as functions of part0 , i.e.:

parti = part0 and part0 =

l0 (a + c) + p0 p0 e0 − pi ei + (l0 − li )(b + d) + li (a + c) + pi L(li (a + c) + pi )

(11.10)

PN −1 pj ej −p0 e0 +(lj −l0 )(b+d)

(11.11)

1 + L−1

j=1

(l0 (a + c) + p0 )

lj (a+c)+pj

PN −1

j=0 (lj (a + c) + pj )

−1

The following C++ code implements these two Equations, with function NportBlockSingleInst returning the parti and total execution time: 1

#i n c l u d e <i o s t r e a m >

2 3 4 5

6 7

8 9 10

u s i n g namespace s t d ; // ********************************************** d o u b l e N p o r t B l o c k S i n g l e I n s t ( d o u b l e * p , d o u b l e * e , d o u b l e * l , ←֓ d o u b l e a , d o u b l e b , d o u b l e c , d o u b l e d , i n t N , l o n g L , ←֓ double * part ) { double lacp [ N ] ; // s h o u l d be r e p l a c e d by dynamic a l l o c a t i o n i f ←֓ N i s big double sumTerm [ N ] ; double totalTime ; double nomin , denom ;

11

f o r ( i n t i = 0 ; i < N ; i++) lacp [ i ] = l [ i ] * ( a + c ) + p [ i ] ;

12 13 14

f o r ( i n t i = 1 ; i < N ; i++) s u m T e r m [ i ] = p [ i ] * e [ i ] − p [ 0 ] * e [ 0 ] + ( l [ i ] − l [ 0 ] ) * ( b +←֓ d) ;

15 16

17

nomin = 0; f o r ( i n t i = 1 ; i < N ; i++) n o m i n += s u m T e r m [ i ] / l a c p [ i ] ; nomin = 1 + nomin / L ;

18 19 20 21 22

denom = 0; f o r ( i n t i = 0 ; i < N ; i++) d e n o m += ( 1 / l a c p [ i ] ) ; d e n o m *= l a c p [ 0 ] ;

23 24 25 26 27

part [ 0 ] = nomin / denom ;

28 29

f o r ( i n t i = 1 ; i < N ; i++) p a r t [ i ] = p a r t [ 0 ] * l a c p [ 0 ] / l a c p [ i ] − s u m T e r m [ i ] / ( L * ←֓ lacp [ i ] ) ;

30 31

32

t o t a l T i m e = p [ 0 ] * ( p a r t [ 0 ] * L + e [ 0 ] ) + l [ 0 ] * ( a * p a r t [ 0 ] * ←֓ L + b ) + l [ 0 ] * ( c * part [ 0 ] * L + d ) ; return totalTime ;

33

34 35

}

36 37 38 39 40 41 42 43 44 45

// ********************************************** i n t main () { double p [ 4 ] = { 1 , 2 , 3 , 4 } ; double e [ 4 ] = { 1 , 1 , 1 , 1 } ; d o u b l e b [ 4 ] = { 1 0 0 0 , 1 0 0 0 , 1 0 0 0 , 1000 } ; double l [ 4 ] = { 1 , 1 , 1 , 1 } ; double part [ 4 ] ;

46 47

48

c o u t << ” t : ” << N p o r t B l o c k S i n g l e I n s t ( p , e , l , 1 , 0 , 1 , 0 , 4 , ←֓ 1 0 0 0 0 0 0 , p a r t ) << e n d l ; f o r ( i n t i = 0 ; i < 4 ; i++)

215 c o u t << p a r t [ i ] << ” ” ; c o u t << e n d l ;

49 50 51

return 0;

52 53

}

12. The DLT examples presented in Sections 8.3.3.1 and 8.3.3.2, do not address an aspect of the problem which is significant : what is the optimum subset of nodes to use to process a load? Implement the heuristic algorithm you came up with in Exercise 2, to derive such a set.

Hint: nodes that should not be part of this set, get a negative load assignment parti .

Answer The effects of node choice and order are more significant when a one-port configuration is used. When N-port configurations are used, one needs to deal with the node selection problem, only when the communication cost to a node can exceed the total execution time on the other nodes.

In the following listing we implement the heuristic outlined in Exercise 2.

The program tests for a maximum number of nodes, as specified in the command line, and reports the optimum number of nodes to be used and associated partitioning. The nodes are sorted (actually they are generated in this fashion) in descending order of their computing power.

1 2

#i n c l u d e <i o s t r e a m > #i n c l u d e < s t d l i b . h>

3 4

u s i n g namespace s t d ;

5 6 7

8 9

10

// ********************************************** d o u b l e O n e p o r t B l o c k S i n g l e I n s t ( d o u b l e * p , d o u b l e * e , d o u b l e l , ←֓ double *b , i n t N , long L , double * part ) { d o u b l e p l [ N ] ; // s h o u l d be r e p l a c e d by dynamic a l l o c a t i o n i f N ←֓ i s big double bSum [ N ] ;

11 12 13

double totalTime ; double nomin , denom ;

14 15 16

f o r ( i n t i = 0 ; i < N ; i++) pl [ i ] = p [ i ] + l ;

17 18 19 20

bSum [ 0 ] = 0; f o r ( i n t i = 1 ; i < N ; i++) bSum [ i ] = b [ i ] + bSum [ i − 1 ] ;

21 22 23 24 25 26 27 28

nomin = 0; f o r ( i n t i = 1 ; i < N ; i++) n o m i n += l / p l [ i ] * b S u m [ i ] ; n o m i n /= L ; n o m i n ++; f o r ( i n t i = 1 ; i < N ; i++) n o m i n += l / p l [ i ] ;

29 30 31 32 33 34

denom = 0; f o r ( i n t i = 0 ; i < N ; i++) d e n o m += 1 / p l [ i ] ; d e n o m *= p l [ 0 ] ;

35 36

part [ 0 ] = nomin / denom ;

37 38

f o r ( i n t i = 1 ; i < N ; i++)

216

CHAPTER 11. LOAD BALANCING p a r t [ i ] = p a r t [ 0 ] * p l [ 0 ] / p l [ i ] − l / ( L * p l [ i ] ) * b S u m [ i ] ←֓ − l / pl [ i ] ;

39

40

totalTime = p [ 0 ] * part [ 0 ] * L ; return totalTime ;

41 42 43

}

44 45 46 47 48

// ********************************************** i n t m a i n ( i n t a r g c , c h a r ** a r g v ) { i n t N = atoi ( argv [ 1 ] ) ;

49

double p [ N ] ; double e [ N ] ; double b [ N ] ; double l = . 1 ; double part [ N ] ; long L = 100;

50 51 52 53 54 55 56

for ( int { p[i] e[i] b[i] }

57 58 59 60 61 62

i = 0 ; i < N ; i++) = ( i + 1) * 1 ; = 0.1; = 1;

63

double bestT = p [ 0 ] * L + e [ 0 ] , bestN = 1 ; f o r ( i n t i = 1 ; i < N ; i++) { d o u b l e t = O n e p o r t B l o c k S i n g l e I n s t ( p , e , 1 , b , i + 1 , L , ←֓ part ) ; bool valid = true ; f o r ( i n t j = 0 ; j < i + 1 && v a l i d ; j++) i f ( part [ j ] < 0) valid = f a l s e ; c o u t << i + 1 << ” ” << t << ” ” << v a l i d << e n d l ; ←֓ // d e b u g g i n g o u t p u t i f ( t < bestT ) // v a l i d d o e s n o t need t o be c h e c k e d a s p e r ←֓ t h e comments i n E x e r c i s e 2 { bestN = i + 1; bestT = t ; } }

64 65 66 67

68 69 70 71 72

73

74 75 76 77 78 79

c o u t << ” B e s t t i m e ” << b e s t T << ” , b e s t N ” << b e s t N << e n d l ; O n e p o r t B l o c k S i n g l e I n s t ( p , e , 1 , b , bestN , L , part ) ; // s o l v e ←֓ again to obtain p a r t i f o r ( i n t i = 0 ; i < b e s t N ; i++) c o u t << p a r t [ i ] << ” ” ; c o u t << e n d l ;

80 81

82 83 84 85

return 0;

86 87

}

A sample of the generated output is shown below: . / h e u r i s t i c s D L T 50 2 80.2 1 3 73.4615 1 4 70.039 1 5 67.9655 1 6 66.5785 1 7 65.5904 1 8 64.8559 1 9 64.2931 1 10 6 3 . 8 5 2 4 1 ...

23 6 2 . 1 4 8 6 1 24 6 2 . 1 3 9 4 1 25 6 2 . 1 3 7 5 1 26 6 2 . 1 4 2 2 0 27 6 2 . 1 5 2 6 0 28 6 2 . 1 6 8 3 0 $

217 ...

50 6 3 . 2 0 9 3 0 B e s t t i m e 6 2 . 1 3 7 5 , b e s t N 25 0 . 6 2 1 3 7 5 0 . 0 7 7 5 8 3 5 0 . 0 5 5 6 8 7 6 0 . 0 4 2 5 5 0 1 0 . 0 3 3 7 9 1 7 0 . 0 2 7 5 3 5 8 ←֓ 0 . 0 2 2 8 4 3 8 0 . 0 1 9 1 9 4 5 0 . 0 1 6 2 7 5 0 . 0 1 3 8 8 6 4 0 . 0 1 1 8 9 5 9 0 . 0 1 0 2 1 1 6 ←֓ 0 . 0 0 8 7 6 7 8 9 0 . 0 0 7 5 1 6 7 0 . 0 0 6 4 2 1 9 0 . 0 0 5 4 5 5 9 1 0 . 0 0 4 5 9 7 2 5 ←֓ 0 . 0 0 3 8 2 8 9 7 0 . 0 0 3 1 3 7 5 2 0 . 0 0 2 5 1 1 9 3 0 . 0 0 1 9 4 3 2 0 . 0 0 1 4 2 3 9 3 ←֓ 0.000947934 0.000510017 0.000105786