SpeedIT OpenCL Plugin for OpenFOAM Installation guide to run OpenCL accelerated calculations ver. 1.0
speed-it.vratis.com vratis.com
Vratis 2013
Vratis Ltd. 2013
Table of contents Introduction..........................................................................................................................................3 Requirements........................................................................................................................................3 Software dependencies.........................................................................................................................3 OpenFOAM.....................................................................................................................................3 OpenMPI..........................................................................................................................................3 Speedit OpenCL...............................................................................................................................3 Plugin Installation............................................................................................................................4 Plugin use.........................................................................................................................................4 Example of decomposeParDict........................................................................................................5 Example of fvSolution:....................................................................................................................6
2/6
Introduction SpeedIT OpenCL plugin for OpenFOAM is a set of libraries which allows you to use SpeedIT OpenCL in OpenFOAM environment. This version of SpeedIT utilizes the power of OpenCL framework, which allows to use computational power of suitable GPUs. The SpeedIT OpenCL library provides a set of accelerated solvers and functions for sparse linear systems of equations which are: • • • •
Preconditioned Conjugate Gradient solver Preconditioned Stabilized Bi-Conjugate Gradient solver Accelerated Sparse Matrix-Vector Multiplication Preconditioners: ◦ Jacobi diagonal ◦ Row scaling with norms l1, l2 and l∞ ◦ ILU0 – Incomplete LU with 0 filling
Requirements •
Operating system: Linux 64bit
•
GPU: compatible with OpenCL
•
OpenCL: installed with GPU vendor drivers
•
GCC: >=4.4.3
•
OpenMPI >=1.4.3:
http://www.open-mpi.org
•
OpenFOAM 1.7.1 – 2.1.1
http://www.openfoam.com/download/
•
SpeedIT OpenCL
http://speed-it.vratis.com/
Software dependencies OpenFOAM OpenFOAM is an environment in which SpeedIT OpenCL plugin is working. You can download it from page http://www.openfoam.com/download/. Install OpenFOAM by following the instructions on the OpenFOAM page. IMPORTANT: Make sure you have done step which sets the OpenFOAM environment variables (sourcing bashrc etc.)
OpenMPI OpenMPI is an open implementation of the Message Passing Interface. You can learn about it by visiting http://www.open-mpi.org/. This library is used to exchange information between domains during parallel computations.
Speedit OpenCL This version of OpenFOAM plugin require SpeedIT OpenCL to work which allows you to perform 3/6
distributed computations using many GPU devices in OpenFoam environment. You can buy it by visiting page: http://speedit.vratis.com
Plugin Installation 1. Create directory $HOME/OpenFOAM 2. Create additional directories by typing: mkdir $WM_PROJECT_USER_DIR | mkdir $FOAM_RUN. 3. Compile plugin by typing from OpenFOAM_SpeedIT_plugin folder : wmake all 4. If compilation will be successfully completed you should have new file libSICLExternal.so in $FOAM_USER_LIBBIN directory.
Plugin use 1. Make sure that copies (or symbolic links) of the following libraries are in $FOAM_USER_LIBBIN directory: • libmpi.so • libmpi_cxx.so • libOpenCL.so • libspeeditCL.so • libSICLExternal.so libmpi.so, libmpi_cxx.so are from OpenMPI library libOpenCL.so should be installed with your device OpenCL driver libspeeditCL.so is delivered after purchase of SpeedIT OpenCL libSICLExternal.so is distributed with SpeedIT OpenCL plugin. NOTE: Remember to use proper version of libraries depends on your system architecture (64-bit library for 64-bit operating systems) 2. Go in to the directory with your FOAM case, i.e. $FOAM_RUN/tutorials/incompressible/icoFoam/cavity 3. Append libs
(
);
“libmpi.so” “libmpi_cxx.so” "libOpenCL.so" "libspeeditCL.so” "libSICLExternal.so"
to the end of your system/controDict file for every FOAM case, for which you want to use external, accelerated solvers. 4. In file system/fvSolution change solver names for solvers, for which you are going to enable acceleration. Remember to use proper names for accelerated solvers. You may replace:
4/6
PBiCG with SICL_BiCG PCG with SICL_CG
5. For accelerated solvers choose an appropriate preconditioner in file system/fvSolution. • • • • •
SICL_JACOBI SICL_RS SICL_RS1 SICL_RS2 SICL_ILU0
6. Run blockMesh 7. If there is only one GPU available, you can run the computations with accelerated solvers by invoking icoFoam command from $FOAM_RUN/tutorials/incompressible/icoFoam/cavity directory. Otherwise follow the instructions below: 8. Prepare decomposeParDict file in $your_case/system directory which describes how to divide your case into subdomains 9. From case directory run “decomposePar” command to distribute field among subdomains 10. Run: “mpirun – np X icoFoam -parallel” from $FOAM_RUN/tutorials/incompressible/icoFoam/cavity , where X is a number of subdomains. Accelerated solvers should be used from now.
Example of decomposeParDict /**\ | ========= | | | \\ / F ield | OpenFOAM: The Open Source CFD Toolbox | | \\ / O peration | Version: 1.3 | | \\ / A nd | Web: http://www.openfoam.org | | \\/ M anipulation | | \**/ FoamFile { version 2.0; format ascii; root ""; case ""; instance ""; local ""; class dictionary; object decomposeParDict; } // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * // numberOfSubdomains 2; method
simple;
simpleCoeffs { n (1 1 2); delta 0.001; } // ************************************************************************* //
5/6
Example of fvSolution: /** C++ **\ | ========= | | | \\ / F ield | OpenFOAM: The Open Source CFD Toolbox | | \\ / O peration | Version: 2.1.1 | | \\ / A nd | Web: www.OpenFOAM.com | | \\/ M anipulation | | \**/ FoamFile { version 2.0; format ascii; class dictionary; location "system"; object fvSolution; } // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * // solvers { p { solver SICL_CG; preconditioner SICL_JACOBI; tolerance 1e06; relTol 0; } U { solver SICL_BiCG; preconditioner SICL_JACOBI; tolerance 1e05; relTol 0; } } PISO { nCorrectors 2; nNonOrthogonalCorrectors 0; pRefCell 0; pRefValue 0; } // ************************************************************************* //
6/6