vsc.univie
The vsc.univie machine is a cluster of dual socket, hexa-core Intel Xeon X5650 2.67 GHz processors with 2 GB of memory per core.
Instructions assume bash, installation under ${HOME}/opt
.
Setup the root directory:
mkdir -p ${HOME}/opt
cd ${HOME}/opt
Set the versions:
export nose=0.11.3
# Warning: version 1.6.0 seems inconsistent about C-, Fortran-contiguous
# https://mail.scipy.org/pipermail/numpy-discussion/2011-July/057557.html
export numpy=1.5.1
export scipy=0.9.0
export acml=4.0.1
export ase=3.5.1.2175
export gpaw=0.8.0.8092
export setups=0.8.7929
and create sh startup script:
cat <<EOF > ${HOME}/opt/campos.sh
#!/bin/sh
#
export GPAW_PLATFORM=`python -c "from distutils import util, sysconfig; print(util.get_platform()+'-'+sysconfig.get_python_version())"`
#
export LD_LIBRARY_PATH=\${HOME}/opt/acml-${acml}/gfortran64/lib:\${LD_LIBRARY_PATH}
export LD_LIBRARY_PATH=\${HOME}/opt/CBLAS.acml-${acml}/lib:\${LD_LIBRARY_PATH} # if cblas used
#
export PYTHONPATH=\${HOME}/opt/nose-${nose}-1/usr/lib/python2.4/site-packages:\${PYTHONPATH}
export PATH=\${HOME}/opt/nose-${nose}-1/usr/bin:\${PATH}
#
export PYTHONPATH=\${HOME}/opt/numpy-${numpy}-1/usr/lib64/python2.4/site-packages:\${PYTHONPATH}
export PATH=\${HOME}/opt/numpy-${numpy}-1/usr/bin:\${PATH}
#
export PYTHONPATH=\${HOME}/opt/scipy-${scipy}-1/usr/lib64/python2.4/site-packages:\${PYTHONPATH}
export PATH=\${HOME}/opt/scipy-${scipy}-1/usr/bin:\${PATH}
#
export PYTHONPATH=\${HOME}/opt/python-ase-${ase}:\${PYTHONPATH}
export PATH=\${HOME}/opt/python-ase-${ase}/tools:\${PATH}
#
export GPAW_SETUP_PATH=\${HOME}/opt/gpaw-setups-${setups}
#
export GPAW_HOME=\${HOME}/opt/gpaw-${gpaw}
export PYTHONPATH=\${GPAW_HOME}:\${PYTHONPATH}
export PYTHONPATH=\${GPAW_HOME}/build/lib.${GPAW_PLATFORM}:\${PYTHONPATH}
export PATH=\${GPAW_HOME}/build/bin.${GPAW_PLATFORM}:\${PATH}
export PATH=\${GPAW_HOME}/tools:\${PATH}
EOF
Download and install acml:
acml-${acml} # download
cd acml-${acml}
tar zxf acml-*.tgz && tar zxf contents-acml-*.tgz
Note: numpy with acml dotblas Segmentation Faults (well, for some
versions on numpy, etc?) for
gpaw/test/numpy_core_multiarray_dot.py
or
gpaw/test/gemm.py
. Still there is no performance improvement
for gpaw/test/gemm.py
(if it works), even if case of dynamic
linking of cblas/acml - check with ldd that _dotblas.so is linked to
both acml and cblas.
This is how you can download and install cblas:
wget https://www.netlib.org/blas/blast-forum/cblas.tgz
tar zxf cblas.tar.gz && mv -f CBLAS CBLAS.acml-${acml} && cd CBLAS.acml-${acml}
cp -p Makefile.LINUX Makefile.in
# fix Makefile.in
export PLAT=LINUX
export BLLIB=${HOME}/opt/acml-${acml}/gfortran64/lib/libacml.a
export CC=gcc
export FC=gfortran
export CFLAGS='-O3 -funroll-all-loops -DADD_ -fPIC'
export FFLAGS='-O3 -funroll-all-loops -DADD_ -fPIC'
#
sed -i "s<^PLAT =.*<PLAT = ${PLAT}<" Makefile.in
sed -i "s<^BLLIB =.*<BLLIB = ${BLLIB}<" Makefile.in
sed -i "s<^CC =.*<CC = ${CC}<" Makefile.in
sed -i "s<^FC =.*<FC = ${FC}<" Makefile.in
sed -i "s<^LOADER =.*<LOADER = \$(FC) -lpthread<" Makefile.in
sed -i "s<^CFLAGS =.*<CFLAGS = ${CFLAGS}<" Makefile.in
sed -i "s<^FFLAGS =.*<FFLAGS = ${FFLAGS}<" Makefile.in
# for dynamic library add the following to the Makefile (before cleanall:)
# Remember TAB in the Makefile!
shared: alllib
( mkdir tmp && cd tmp && cp $(CBLIB) . && ar x $(CBLIB) && $(CC) -shared -o libcblas.so.1.0.0 *.o -Wl,-soname=libcblas.so.1 && cp -p libcblas.so.1.0.0 ../lib && cd ../lib && ln -s libcblas.so.1.0.0 libcblas.so.1 && ln -s libcblas.so.1.0.0 libcblas.so )
make all 2>&1 | tee make_all.log
make shared 2>&1 | tee make_shared.log
# create link: numpy needs all the libraries in one directory
# separate directories in site.cfg do not work
cd lib && ln -s cblas_${PLAT}.a libcblas.a
ln -s ${HOME}/opt/acml-${acml}/gfortran64/lib/libacml.a .
ln -s ${HOME}/opt/acml-${acml}/gfortran64/lib/libacml_mv.a .
# if dynamic library needed
ln -s ${HOME}/opt/acml-${acml}/gfortran64/lib/libacml.so .
ln -s ${HOME}/opt/acml-${acml}/gfortran64/lib/libacml_mv.so .
cd ../..
Build nose/numpy/scipy:
wget --no-check-certificate https://downloads.sourceforge.net/project/numpy/NumPy/${numpy}/numpy-${numpy}.tar.gz
wget --no-check-certificate https://downloads.sourceforge.net/project/scipy/scipy/${scipy}/scipy-${scipy}.tar.gz
wget https://python-nose.googlecode.com/files/nose-${nose}.tar.gz
tar zxf nose-${nose}.tar.gz
tar zxf numpy-${numpy}.tar.gz
tar zxf scipy-${scipy}.tar.gz
cd nose-${nose}
python setup.py install --root=${HOME}/opt/nose-${nose}-1 2>&1 | tee install.log
use the following site.cfg
to build numpy without cblas (that’s safer):
cat <<EOF > ${HOME}/opt/numpy-${numpy}/site.cfg
[DEFAULT]
library_dirs = /usr/lib64:${HOME}/opt/acml-${acml}/gfortran64/lib
include_dirs = ${HOME}/opt/acml-${acml}/gfortran64/lib/../include:/usr/include/suitesparse
[blas]
libraries = acml
library_dirs = ${HOME}/opt/acml-${acml}/gfortran64/lib
[lapack]
libraries = acml, gfortran
library_dirs = ${HOME}/opt/acml-${acml}/gfortran64/lib
EOF
and this one with cblas based on acml:
cat <<EOF > ${HOME}/opt/numpy-${numpy}/site.cfg
[DEFAULT]
library_dirs = /usr/lib64:${HOME}/opt/CBLAS.acml-${acml}/lib
include_dirs = ${HOME}/opt/acml-${acml}/gfortran64/lib/../include:/usr/include/suitesparse:${HOME}/opt/CBLAS.acml-${acml}/include
[blas]
libraries = acml
library_dirs = ${HOME}/opt/CBLAS.acml-${acml}/lib
[lapack]
libraries = acml, gfortran
library_dirs = ${HOME}/opt/CBLAS.acml-${acml}/lib
EOF
Note: the site.cfg
file is used only to specify directories, libraries
from site.cfg
are ignored by numpy. Moreover numpy needs all the libraries in one directory,
separate directories in site.cfg do not work.
continue with:
cd ../numpy-${numpy}
# force numpy to use internal blas for dotblas + acml, note the double quotes!
sed -i "s/_lib_names = \['blas'\]/_lib_names = ['acml']/g" numpy/distutils/system_info.py
sed -i "s/_lib_names = \['lapack'\]/_lib_names = ['acml']/g" numpy/distutils/system_info.py
# or with dotblas acml (through cblas) - seems not working or Segmentation Faults
sed -i "s<_lib_mkl = .*<_lib_mkl = ['acml','cblas']<" numpy/distutils/system_info.py
sed -i "s<\['mkl_lapack32','mkl_lapack64'\]<['acml','gfortran']<" numpy/distutils/system_info.py
sed -i "s<l = 'mkl'<l = 'acml'<" numpy/distutils/system_info.py
# avoid "Both g77 and gfortran runtimes linked in lapack_lite !" setting --fcompiler=gnu95
python setup.py build --fcompiler=gnu95 2>&1 | tee build.log
python setup.py install --root=${HOME}/opt/numpy-${numpy}-1 2>&1 | tee install.log
cd ..
source ${HOME}/opt/campos.sh
python -c "import numpy; numpy.test()"
cd scipy-${scipy}
python setup.py config_fc --fcompiler=gfortran install --root=${HOME}/opt/scipy-${scipy}-1 2>&1 | tee install.log
cd ..
python -c "import scipy; scipy.test()"
Make sure that you have the right mpicc:
which mpicc
/usr/mpi/qlogic/bin/mpicc
Install ASE/GPAW:
wget https://wiki.fysik.dtu.dk/ase-files/python-ase-${ase}.tar.gz
wget https://wiki.fysik.dtu.dk/gpaw-files/gpaw-${gpaw}.tar.gz
wget https://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-${setups}.tar.gz
tar zxf python-ase-${ase}.tar.gz
tar zxf gpaw-${gpaw}.tar.gz
tar zxf gpaw-setups-${setups}.tar.gz
mkdir testase && cd testase && testase.py 2>&1 | tee ../testase.log
wget https://svn.fysik.dtu.dk/projects/gpaw/trunk/doc/platforms/Linux/customize_vsc_univie.py
cd ../gpaw-${gpaw}
python setup.py --remove-default-flags --customize=../customize_vsc_univie.py build_ext 2>&1 | tee build_ext.log
The customize_vsc_univie.py
looks like:
scalapack = False
compiler = 'gcc'
extra_compile_args += [
'-O3',
'-funroll-all-loops',
'-fPIC',
]
libraries = ['gfortran', 'util']
blas_lib_path = '/home/lv70174/gpaw/opt/acml-4.0.1/gfortran64/lib/'
lapack_lib_path = blas_lib_path
extra_link_args = [
blas_lib_path+'libacml.a',
lapack_lib_path+'libacml.a',
]
GPAW tests gpaw-test
can be submitted like this:
qsub run.sh
where run.sh
looks like this:
#!/bin/sh
#$ -pe mpich 8
#$ -V
#$ -M my.name@example.at
#$ -m be
#$ -l h_rt=00:50:00
if [ -z "${PYTHONPATH}" ]
then
export PYTHONPATH=""
fi
source ${HOME}/opt/campos.sh
export OMP_NUM_THREADS=1
mpirun -m $TMPDIR/machines -np $NSLOTS gpaw-python `which gpaw-test`
Please make sure that your jobs do not run multi-threaded, e.g. for a
job running on node02
do from a login node:
ssh node02 ps -fL
you should see 1 in the NLWP column. Numbers higher then 1 mean multi-threaded job.
It’s convenient to customize as described on the Parallel runs page.