A Parallel Implementation of blockMesh for Quick

A Parallel Implementation of blockMesh for Quick Generation of Huge Meshes Venugopalan S.G. Raghavan, Dominic Chandar, Harish Gopalan

24-Oct-2018 www.esi-group.com

Copyright © ESICopyright Group, 2018. © ESI AllGroup, rights reserved. 2018. All rights reserved.

1

Contents • • • • • • • • • •

Rationale/Background Current Limitations Challenge Strategy Implementation Scaling Test Example Case Studies Capabilities & Limitations Future Development Conclusion

www.esi-group.com

Copyright © ESI Group, 2018. All rights reserved.

2

Rationale • Large urban simulations – precinct level, district level, city level • Issues in homogenity with refinement regions in snappy, particularly with TKE

Grading based Mesh (G Method)

Refinement-region based Mesh (R Method)

www.esi-group.com


3

Rationale

Ideally: Profiles of U & νt same at inlet & outlet Reality: Discrepancies present. Greatest deviation R-method. G-method shows minimal deviation & hence preferred Solution: Use graded blockMesh as much as possible www.esi-group.com


4

Rationale • Another motivation: Applications such as green building assessment preferring pure hexahedral meshes

Source: Pg 167 @ https://www.bca.gov.sg/GreenMark/others/GM_RB2016_Technical_Guide_Requirements.pdf www.esi-group.com


5

Current Limitations • blockMesh currently runs only in serial • For large meshes, slow to generate blockMesh • Eg: Half of Singapore • blockMesh cell count = ~430 million • blockMesh meshing time = 2h 20min

www.esi-group.com


6

Challenge • No mesh information available (has to be generated first) • Cannot make use of the parallelization tools in OpenFOAM

www.esi-group.com


7

Strategy • • • • •

Simplify the problem – start with single box mesh to be generated Split mesh into discrete number of pieces Generate each piece of mesh separately Benefit: Embarrassingly parallel problem  no communication required Trick: Change the dictionary on the fly…

Mesh to be generated

Split to different pieces

Generate each piece independently www.esi-group.com


8

Implementation

Split to different pieces y

• Minimal changes to the dictionary • Cut the box specified in blockMeshDict into blocks • Cutting done only in one direction!

x

Cut along X

y x

Cut along Y

Mesh to be generated

Z x

Cut along Z (default) www.esi-group.com


9

Implementation • Internally create a class for creating individual meshes • Each instance of a class corresponds to a particular processor • Change the local copy of the blockMeshDict that the processor has • Values for regionName, myRank, meshDict  local to processor • createMesh() performs meshing • Is a wrapper around the existing functionality in blockMesh.C / blockMeshApp.C for each processor

• Multi-grading support • Computes points in direction of cutting and assigns (as far as possible) equal number of cells to each processor

www.esi-group.com


10

Implementation • This is a sample blockMeshDict that was used in order to create a blockMesh via parallelBlockMesh • Structure of the blockMeshDict similar to regular blockMesh • Changes are only in two locations: • simpleGrading: • For each of directions, must be written within (). • The first number must be the absolute length • The second number must be the absolute number of cells www.esi-group.com


11

Implementation • boundary: • The last two patches must be the ones with the same normal as the mesh cutting direction. • In this example, the mesh will be cut along the “Z” direction. The “Bottom” and “Top” patches have normals pointing in the same direction as the cutting direction. • The order of “Bottom” and “Top” should be followed. The lower surface must be specified first followed by the upper surface.

www.esi-group.com


12

Implementation • Internally depending on the processor ID, either “Bottom” or “Top” or both get converted to “processor” patches

Top

Bottom

Top

Top

Bottom Top

proc3to2 proc2to3

Bottom Top Bottom Top

proc2to1 proc1to2 proc1to0 proc0to1

Bottom

Bottom

proc*to* are processor patches www.esi-group.com


13

Implementation • xmin, xmax, ymin, ymax, zmin and zmax re-computed locally • Written into the local copy of the blockMeshDict file • If the cutting of the mesh is along the Z direction and using 56 processors, the 156 cells in the Z direction will be split among the 56 processors • zmin and zmax different for each processor. • Following holds true: zmini = zmax(i-1) www.esi-group.com


14

Implementation Example: Assume 56 processors, cutting along Z direction dz = 400 /156 = 2.564 nz = 156/56 = 2.78 proc0 to proc43 = 3 cells in z-dir proc44 to proc55 = 2 cells in z-dir

proc1: … proc55:

Z = 15.384

zmin = 7.692 zmax = 15.384

z

Z = 7.692 Z = 400

zmin = 394.856 zmax = 400.0 xmin = -500 xmax = 14027 ymin = -500 ymax = 9539

z

Z=0

z

Z = 394.856

Y = 9539

y X = 14027

All procs:

zmin = 0 zmax = 7.692

X = -500

proc0:

Z = 7.692

Y = -500 www.esi-group.com


x

15

Scaling Test • Different types of scaling tests performed to determine performance • Performance of parallelBlockMesh seems to fall after certain number of processors • Two machines: • Machine 1: Intel® Xeon® CPU E5-2697 v4 @ 2.3GHz; Effective cores = 72 (with hyperthreading) • Machine 2: Compute node at National Supercomputing Center Singapore: Intel® Xeon® CPU E5-2690 v3 @ 2.6GHz; Effective cores = 24

www.esi-group.com


16

Scaling Test • Regular Scaling (100 million cells) [Machine 1]

www.esi-group.com


17


www.esi-group.com


18


www.esi-group.com


19

Scaling Test • Weak Scaling I [Machine 1] • On 1 processor 3.2 million cells generated • Subsequently, double mesh count and double number of procs

www.esi-group.com


20

Scaling Test • Weak Scaling II # Procs

Mesh count

Median Mesh Time

Speedup (w.r.t 2 procs)

Ideal speedup

Proc Factor

# Procs

Mesh count

Median Mesh Time

Speedup (w.r.t 12 procs)

Ideal speedup

Proc Factor

2

6.4 x 106

17.08

1.00

1

1

12

15.36

1.00

1

1

4

12.8 x 106

18.10

0.94

1

2

3.84 x 105

24

17.33

0.88

1

2

8

25.6 x 106

19.78

0.86

1

4

7.68 x 105

48

18.53

0.82

1

4

16

51.2 x 106

22.93

0.74

1

8

15.36 x 105

96

19.77

0.77

1

8

32

102.4 x 106

30.63

0.56

1

16

30.72 x 105

Runs performed on Machine 1

Runs performed on Machine 2 www.esi-group.com


21

Scaling Test • Weak Scaling II [Machines 1 & 2]

www.esi-group.com


22

Scaling Test

blockMesh with billion (109) cells created on 800 processors in 56.8s

blockMesh with 8 billion (8 x 109) cells created on 8000 processors in 96.3s www.esi-group.com


23

Example Case Studies • Urban scenario 1

Mesh count: 123.12 million cells Domain Size: 7.2km x 7.2km x 0.18km

# Procs = 56 Time taken = 41.2s www.esi-group.com


24

Example Case Studies • Urban scenario 2: Amsterdam (Solar)

Amsterdam cityGML source: https://3d.bk.tudelft.nl/opendata/3dfier/ www.esi-group.com


25

Example Case Studies • Urban scenario 2: Amsterdam (Solar)

Mesh Count (blockMesh) = 104 million www.esi-group.com


26

Example Case Studies • Urban scenario 2: Amsterdam (Wind)

Mesh count: 195.76 million cells Domain Size: 14km x 1km x 0.4km

# Procs = 56 Time taken (mesh) = 106s www.esi-group.com


27


Mesh count: 195.76 million cells Domain Size: 14km x 1km x 0.4km

# Procs = 56 Time taken (mesh) = 106s www.esi-group.com


28


Mesh count = 195.76 million cells # Procs = 56

Time taken (mesh) = 106s Time taken (total) = 200s

www.esi-group.com


29

Example Case Studies • Urban scenario 3: Combine with overset Background mesh count = 50 x 106 cells # of Procs = 50 Time (mesh) = 45s Total number of regions = 231 Total mesh count = > 200 x 106 cells (Automated mesh generation for 230 building regions)

www.esi-group.com


30

Example Case Studies • Wind Turbine Scenario 1: Generate meshes for use with SOWFA

With pBM

Default case www.esi-group.com


31

Example Case Studies • Wind Turbine Scenario 1: Generate meshes for use with SOWFA

With pBM

Default case

www.esi-group.com


32

Capabilties & Limitations

• Capabilities • Only minor changes in blockMeshDict for parallel • Quick generation of mesh • No changes to blockMesh source code (except for the blockMeshApp.C)

• Limitations • • • • •

Currently can only mesh single block Cannot handle curved edges at the moment Mesh can be sliced in only one direction Number of cells in slicing direction > Number of procs Does not seem to scale linearly with processors

www.esi-group.com


33

Future Development

• Handle multiple blocks • Handle curved edges • Improve scalability (?)

www.esi-group.com


34

Conclusion

• Preliminary version of parallelBlockMesh developed and applied to cases • Further work needs to be done to improve current limitations

www.esi-group.com


35

Acknowledgments

• This work was undertaken as part of the project “Modeling of Air Flow, Thermal and Chemical Gas Dispersion Towards Next Generation Port (Tuas Maritime Hub)” funded by Singapore Maritime Institute (SMI-2016-MA-04)

www.esi-group.com


36

THANK YOU


37

www.esi-group.com

37