louhi.csc.fi (Cray XT4/XT5)
Here you find information about the system https://www.csc.fi/english/research/Computing_services/computing/servers/louhi.
Note
These instructions are up-to-date as of August 28th 2012.
GPAW
The recent operating system releases for Cray XT4/5 (CLE 2.2 UP01 and later) supports dynamic libraries which simplifies GPAW installation significantly.
These instructions for GPAW installation use Python 2.6.5 compiled with GNU compiler suite, see the end of this page for instructions for compiling Python.
First, load the Python module and set XTPE_LINK_TYPE
environment
variable for dynamic linking:
module load python
module load hdf5-parallel
setenv XTPE_LINK_TYPE dynamic
GPAW can now be build with a minimal customize.py
#User provided customizations for the gpaw setup
compiler = 'cc'
mpicompiler = 'cc'
mpilinker= 'cc'
extra_compile_args = ['-std=c99']
libraries = []
scalapack = True
hdf5 = True
define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
define_macros += [("GPAW_ASYNC",1)]
Currently, there is a small bug in the Cray libraries which results in
failure when trying to build the _gpaw.so
library. As the library
is not really needed in parallel calculations, the problem can be
circumvented by creating the file after running the setup.py script:
python setup.py build_ext
...
touch build/lib.linux-x86_64-2.6/_gpaw.so
python setup.py install --home=...
Python and Numpy
Python can be compiled with PGI compiler as follows:
setenv XTPE_LINK_TYPE dynamic
./configure --prefix=path_to_install CC=cc CXX=cc OPT=-fastsse LINKFORSHARED=-Wl,--export-dynamic
make install
In order to use optimized BLAS with Numpy one has to first build a CBLAS which is linked with Cray’s optimized BLAS routines. First, download the CBLAS source from netlib:
wget https://www.netlib.org/blas/blast-forum/cblas.tgz
tar -xzf cblas.tgz
Change to the CBLAS directory and copy Makefile.LINUX
to
Makefile.in
. Add correct compiler commands and paths to
Makefile.in
:
...
PLAT = louhi
#-----------------------------------------------------------------------------
# Libraries and includs
#-----------------------------------------------------------------------------
BLLIB =
CBDIR = $(HOME)/CBLAS
CBLIBDIR = $(CBDIR)/lib/$(PLAT)
CBLIB = $(CBLIBDIR)/libcblas.a
#-----------------------------------------------------------------------------
# Compilers
#-----------------------------------------------------------------------------
CC = cc
FC = ftn
LOADER = $(FC)
#-----------------------------------------------------------------------------
# Flags for Compilers
#-----------------------------------------------------------------------------
CFLAGS = -O3 -DADD_ -fPIC
FFLAGS = -O3 -fPIC
...
Finally, build CBLAS:
make alllib
You are now ready to build Numpy with the newly created CBLAS
library. The standard Numpy tries to use only the ATLAS BLAS, and in
order to use different BLAS one has to manually edit the file
numpy/core/setup.py
. Comment out an if statement as follows:
def get_dotblas_sources(ext, build_dir):
if blas_info:
# if ('NO_ATLAS_INFO',1) in blas_info.get('define_macros',[]):
# return None # dotblas needs ATLAS, Fortran compiled blas will not be sufficient.
return ext.depends[:1]
Then, add the correct libraries and paths to the file site.cfg
:
[blas]
blas_libs = cblas
library_dirs = /home/csc/jenkovaa/CBLAS/lib/louhi
[lapack]
lapack_libs = sci
library_dirs = /opt/xt-libsci/10.3.8/pgi/lib
Now, one should be able to build Numpy as usual:
python setup.py install