SpeedIT Plugin for OpenFOAM Installation guide to run multiGPU calculations ver. 2.2
speed-it.vratis.com vratis.com
Vratis 2012
Vratis Ltd. 2012
Table of contents Introduction..........................................................................................................................................3 Requirements........................................................................................................................................3 Software dependencies.........................................................................................................................3 OpenFOAM.....................................................................................................................................3 CUDA..............................................................................................................................................3 OpenMPI..........................................................................................................................................3 Speedit.............................................................................................................................................4 Cuwrap.............................................................................................................................................4 Plugin Installation............................................................................................................................4 Plugin use.........................................................................................................................................4 Example of decomposeParDict........................................................................................................6 Example of fvSolution: ...................................................................................................................6
2/7
Introduction SpeedIT plugin for OpenFOAM is a set of libraries which allows you to use SpeedIT in OpenFOAM environment. SpeedIT will release the computational power dreaming in NVIDIA Graphics Processing Unit (GPU) that supports CUDA technology. The SpeedIT library provides a set of accelerated solvers and functions for sparse linear systems of equations which are: • • • • • •
Preconditioned Conjugate Gradient Preconditioned Stabilized Bi-Conjugate Gradient Accelerated Sparse Matrix-Vector Multiplication Accelerated Dense Matrix-Vector Multiplication Optimized Memory Management Diagonal Preconditioner
Requirements •
Operating system: Linux 32/64bit
•
GPU: CUDA compatible GPU with compute capability >= 2.0
•
GCC: >=4.4.3
•
CUDA 4.0:
http://developer.nvidia.com/cuda-downloads
•
OpenMPI 1.4.3:
http://www.open-mpi.org
•
OpenFOAM 1.7.1
http://www.openfoam.com/download/
•
SpeedIT 2.2
http://speed-it.vratis.com/
Software dependencies OpenFOAM OpenFOAM is an environment in which SpeedIT plugin is working. You can download it from page http://www.openfoam.com/download/. Install OpenFOAM by following the instructions on the OpenFOAM page. IMPORTANT: Make sure you have done step which sets the OpenFOAM environment variables
CUDA Download CUDA library from http://developer.nvidia.com/cuda-downloads and install it. Add CUDA include directory to your PATH variable: PATH=$PATH:/path/to/cuda/include Depends on your system (32/64-bit) add CUDA lib or lib64 to LD_LIBRARY_PATH i.e: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/cuda/lib:/path/to/cuda/lib64
OpenMPI OpenMPI is an open implementation of the Message Passing Interface. You can learn about it by visiting http://www.open-mpi.org/. This library is used to exchange information between domains during parallel computations.
3/7
Speedit This version of OpenFOAM plugin require SpeedIT 2.2 version to work which allows you to perform distributed computations using many GPU devices in OpenFoam environment.. You can buy it by visiting page: http://speedit.vratis.com
Cuwrap Cuwrap is an intermediate library which achieve compatibility between CUDA and OpenFOAM interfaces. It is distributed with the OpenFOAM plugin. You can find it in the folder cuwrap. It is necessary to build this library if you want to use SpeedIT with OpenFoam. To build this library first in cuwrap folder open Makefile file. Depends on your configuration set proper paths to CUDA environment. For 32-bit suystem and default CUDA installation the header of the file should looks following: CUDA_HOME="/usr/local/cuda" CUDA_LIB="$CUDA_HOME/lib" NVIDIA_CURRENT="/usr/lib" CUDA_INC="$CUDA_HOME/include" CUDA_BIN="$CUDA_HOME/bin"
For 64-bit systems:
CUDA_HOME="/usr/local/cuda" CUDA_LIB="$CUDA_HOME/lib64" NVIDIA_CURRENT="/usr/lib64" CUDA_INC="$CUDA_HOME/include" CUDA_BIN="$CUDA_HOME/bin"
After setting paths from the cuwrap folder run make command. It shall build the library. If you have openFoam configured properly the library should be put inside $FOAM_USER_LIBBIN folder.
Plugin Installation 1. Create directory $HOME/OpenFOAM 2. Create additional directories by typing: mkdir $WM_PROJECT_USER_DIR | mkdir $FOAM_RUN. 3. Compile cuwrap library by calling make from libcuwrap directory 4. Compile plugin by typing from OpenFOAM_SpeedIT_plugin folder : wmake libso 5. If compilation will be successfully completed you should have new file libexternalsolv.so in $FOAM_USER_LIBBIN directory.
Plugin use 1. Copy (or make symbolic links)following libraries to $FOAM_USER_LIBBIN directory: • libmpi.so • libmpi_cxx.so • libcublas.so • libcudart.so • libcuwrap.so • libspeedit.so 4/7
libmpi.so, libmpi_cxx.so are from OpenMPI library libcublas.so, libcudart.so, are from NVIDIA CUDA toolkit libcuwrap.so is distributed with Speedit plugin. NOTE: Remember to use proper version of libraries depends on your system architecture. 32-bit library in 32-bit operating systems and 64-bit library for 64-bit operating systems 2. Go in to the directory with your FOAM case, i.e. $FOAM_RUN/tutorials/incompressible/icoFoam/cavity 3. Append libs (
“libmpi.so” “libmpi_cxx.so” "libcudart.so" "libcublas.so" "libcuwrap.so" "libspeedit.so” "libexternalsolv.so"
);
to the end of your system/controDict file for every FOAM case, for which you want to use external, accelerated solvers. 4. In file system/fvSolution change solver names for solvers, for which you are going to enable acceleration. Remember to use proper names for accelerated solvers. You may replace: PBiCG with SI_PBiCG PCG with SI_PCG
5. For accelerated solvers choose an appropriate preconditioner in file system/fvSolution. • • • • •
SI_DIAGONAL – Diagonal preconditioner SI_AMG – Algebraic Multigrid preconditioner SI_AINV – Approximate Inverse preconditioner SI_AINV_SC – Approximete Inverse Scaled preconditioner SI_AINV_NS – Approximate Inverse Non-Symmetric preconditioner
6. When accelerated solvers are used you have to specify additional keyword “matrix” in solver definition. It can take 2 values CMR or CSR which stands for: CSR – Compressed Sparse Row format. CMR – Compressed Multi-Row format In multiGPU calculations only CSR matrix is supported 7. Run blockMesh 8. Prepare decomposeParDict file in $your_case/system directory which describe how to divide your case into subdomains 9. From case directory run “decomposePar” command to distribute field among subdomains 10. Run: “mpirun – np X icoFoam -parallel” from 5/7
$FOAM_RUN/tutorials/incompressible/icoFoam/cavity. Accelerated solvers should be used from now.
Example of decomposeParDict /**\ | ========= | | | \\ / F ield | OpenFOAM: The Open Source CFD Toolbox | | \\ / O peration | Version: 1.3 | | \\ / A nd | Web: http://www.openfoam.org | | \\/ M anipulation | | \**/ FoamFile { version 2.0; format ascii; root ""; case ""; instance ""; local ""; class dictionary; object decomposeParDict; } // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * // numberOfSubdomains 2; method
simple;
simpleCoeffs { n (1 1 2); delta 0.001; } // ************************************************************************* //
Example of fvSolution: /** C++ **\ | ========= | | | \\ / F ield | OpenFOAM: The Open Source CFD Toolbox | | \\ / O peration | Version: 1.7.1 | | \\ / A nd | Web: www.OpenFOAM.com | | \\/ M anipulation | | \**/ FoamFile { version 2.0; format ascii; class dictionary; location "system"; object fvSolution; } // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * // solvers { p {
6/7
solver SI_PCG; preconditioner SI_AMG; matrix CSR; tolerance 1e06; relTol 0; } U { solver PBiCG; preconditioner DILU; matrix CSR; tolerance 1e05; relTol 0; } } PISO { nCorrectors 2; nNonOrthogonalCorrectors 0; pRefCell 0; pRefValue 0; } // ************************************************************************* //
7/7