SpeedIT Plugin to OpenFOAM速 Installation Guide ver. 2.0
speed-it.vratis.com vratis.com
Vratis 2012
Table of contents Introduction .......................................................................................................................................... 3 Requirements ....................................................................................................................................... 3 Software dependencies ......................................................................................................................... 4 OpenFOAM ................................................................................................................................ 4 CUDA......................................................................................................................................... 4 Speedit ........................................................................................................................................ 4 Cuwrap ....................................................................................................................................... 4 Plugin Installation ...................................................................................................................... 5 Plugin use ................................................................................................................................... 7
2/8
Introduction SpeedIT plugin to OpenFOAM is a set of libraries which allows you to use SpeedIT in OpenFOAM® environment. SpeedIT releases the computational power of NVIDIA Graphics Processing Units (GPU) that support CUDA technology. The SpeedIT library provides a set of accelerated solvers and functions for sparse linear systems of equations which are: ñ ñ ñ ñ ñ ñ
Preconditioned Conjugate Gradient Preconditioned Stabilized Bi-Conjugate Gradient Accelerated Sparse Matrix-Vector Multiplication Accelerated Dense Matrix-Vector Multiplication Optimized Memory Management Diagonal Preconditioner
Requirements ñ Operating system: Linux 32/64bit ñ GPU: CUDA compatible GPU with compute capability >= 2.0 ñ GCC: 4.4.3 ñ CUDA 3.2 / 4.0:
http://developer.nvidia.com/cuda-downloads
ñ OpenFOAM® 1.7.1 / 2.0.x
http://www.openfoam.com/download/
ñ SpeedIT 2.0
http://speedit.vratis.com/
3/8
Software dependencies OpenFOAM SpeedIT plugin works inside OpenFOAM® environment. You can download it from http://www.openfoam.com/download/. Install OpenFOAM® by following the instructions on the OpenFOAM® page. IMPORTANT: Make sure to set OpenFOAM® environment variables.
CUDA Download CUDA library from http://developer.nvidia.com/cuda-downloads and install it. Add CUDA include directory to your PATH variable: PATH=$PATH:/path/to/cuda/include ; export PATH Depends on your system (32/64-bit) add CUDA lib or lib64 to LD_LIBRARY_PATH i.e: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/cuda/lib:/path/to/cuda/lib64 export LD_LIBRARY_PATH
;
You can also edit .bashrc in your home directory.
SpeedIT SpeedIT Plugin to OpenFOAM 2.0 requires SpeedIT 2.0 version to work. You can buy this version at: http://speed-it.vratis.com SpeedIT Plugin to OpenFOAM 1.x support SpeedIT Classic and SpeedIT 1.x.
Cuwrap Cuwrap is an intermediate library which ensures compatibility between CUDA and OpenFOAM® interfaces. It is distributed with the OpenFOAM plugin. You can find it in the folder cuwrap. It is necessary to build this library if you want to use SpeedIT with OpenFoam. In order to build this library: 1. Go to cuwrap folder and edit Makefile. 2. Set proper paths to CUDA environment depending on your system configuration as follows. Example for a 32-bit system and default CUDA installation: CUDA_HOME="/usr/local/cuda" CUDA_LIB="$CUDA_HOME/lib" NVIDIA_CURRENT="/usr/lib" CUDA_INC="$CUDA_HOME/include" CUDA_BIN="$CUDA_HOME/bin"
Example for a 64-bit system and default CUDA installation: CUDA_HOME="/usr/local/cuda" CUDA_LIB="$CUDA_HOME/lib64" NVIDIA_CURRENT="/usr/lib64" CUDA_INC="$CUDA_HOME/include" CUDA_BIN="$CUDA_HOME/bin"
4/8
3. run make to build the library. If you have OpenFOAM® configured properly the library should be installed inside $FOAM_USER_LIBBIN folder.
Plugin Installation 1. Create directory $HOME/OpenFOAM 2. Create additional directories by typing: mkdir $WM_PROJECT_USER_DIR mkdir $FOAM_RUN 3. Compile cuwrap library 4. Edit solver source files PBiCG_accel.C, PCG_accel.C to make SpeedIT solver functions available to use (see the example below). ñ Check header files. By default functions from SpeedIT are available in speedit.h ñ Choose deviceID of your target GPU card where calculations will be performed. DeviceID = -1 will cause that SpeedIT will automatically search for the best available device taking into account number of cuda cores and compute capability. ñ Check if proper solver functions are enabled. Go to line 177. If you have OpenFOAM in single precision then a code snipet from #if defined(WM_SP) will be executed, otherwise from #if defined(WM_DP). 5. Compile plugin by typing from SpeedIT_Plugin_to_OpenFOAM folder : wmake libso 6. If compilation is successfully completed you should have a new file libexternalsolv.so in $FOAM_USER_LIBBIN directory.
5/8
Example: … //#include “si_classic.h” #include "speedit.h" . . . /* INSERT SOLVER FUNCTION HERE */ /* You are able to create any type of matrix handler, by default we are using SpeedIT CMR format because in most cases is the fastest. In some cases CSR might have better performance but that need to be checked by the user. */ #if defined(WM_SP) // scalar is float //Create CMR matrix handler on GPU base SI_CMR_FLOAT_HANDLE handle = si_shcmr ( p_vals, p_c_idx, p_r_idx, n_rows, n_rows, nnz, HOST2GPU); //execute solver solver_result = si_gshcmrcg( handle, pgpu_X, pgpu_B, precond, &n_iter, &eps) ; //release matrix si_shreleasecmr(&handle); #elif defined(WM_DP) // scalar is double //Create matrix handler on GPU base SI_CMR_DOUBLE_HANDLE handle = si_dhcmr (p_vals, p_c_idx, p_r_idx, n_rows, n_rows, nnz, HOST2GPU); //execute solver solver_result = si_gdhcmrcg( handle, pgpu_X, pgpu_B, precond, &n_iter, &eps) ; // release matrix si_dhreleasecmr(&handle); #endif
6/8
Plugin use 1. Copy (or make symbolic links) following libraries to $FOAM_USER_LIBBIN directory: ñ libcublas.so ñ libcudart.so ñ libcuwrap.so ñ libspeedit.so libcublas.so, libcudart.so, are from NVIDIA CUDA toolkit libcuwrap.so is distributed with SpeedIT plugin to OpenFOAM®. NOTE: Remember to use proper version of libraries depending on your system architecture, i.e. 32-bit library for 32-bit operating systems and 64-bit library for 64-bit operating systems 2. Go into the directory with your FOAM case, i.e. $FOAM_RUN/tutorials/incompressible/icoFoam/cavity 3. Append libs ( "libcudart.so" "libcublas.so" "libcuwrap.so" "libspeedit.so” "libexternalsolv.so" );
to the end of your system/controDict file for every OpenFOAM® case for which you want to use external, accelerated solvers. 4. In file system/fvSolution change solver names for solvers which you want to accelerate. Remember to use proper names for accelerated solvers. You may replace: PBiCG with PbiCG_accel PCG with PCG_accel
5. For accelerated solvers choose an appropriate preconditioner in file system/fvSolution. You may use only “diagonal” at the moment. 6. Run icoFOAM from $FOAM_RUN/tutorials/incompressible/icoFoam/cavity. Accelerated solvers should be used from now.
7/8
Disclaimer Please note that SpeedIT Plugin to OpenFOAM速 has been created in the spirit of the OpenSource development model. In particular, this offering is not approved or endorsed by Silicon Graphics International Corp., the producer of OpenFOAM速 software and owner of the OpenFOAM速 trademark!
8/8