uranus
The uranus machine is a cluster of dual socket, quad-core AMD Opteron 2354 CPUs, 2.2 GHz processors with 2 GB of memory per core.
Instructions assume bash, installation under ${HOME}/opt. Build the unoptimized numpy/scipy:
mkdir ${HOME}/opt
cd ${HOME}/opt
wget https://dfn.dl.sourceforge.net/sourceforge/numpy/numpy-1.3.0.tar.gz
wget https://dfn.dl.sourceforge.net/sourceforge/scipy/scipy-0.7.0.tar.gz
wget https://python-nose.googlecode.com/files/nose-0.11.0.tar.gz
tar zxf nose-0.11.0.tar.gz
tar zxf numpy-1.3.0.tar.gz
tar zxf scipy-0.7.0.tar.gz
cd nose-0.11.0
python setup.py install --prefix=${HOME}/opt/python 2>&1 | tee install.log
cd ../numpy-1.3.0
python setup.py install --prefix=${HOME}/opt/python 2>&1 | tee install.log
export PYTHONPATH=${HOME}/opt/python/lib64/python2.4/site-packages:${HOME}/opt/python/lib/python2.4/site-packages
cd ..
python -c "import numpy; numpy.test()"
wget https://www.netlib.org/blas/blas.tgz
tar zxf blas.tgz
export BLAS_SRC=${HOME}/opt/BLAS
wget https://www.netlib.org/lapack/lapack.tgz
tar zxf lapack.tgz
export LAPACK_SRC=${HOME}/opt/lapack-3.2.1
cd scipy-0.7.0
python setup.py config_fc --fcompiler=gfortran install --prefix=${HOME}/opt/python 2>&1 | tee install.log
cd ..
python -c "import scipy; scipy.test()"
Set these environment variables in the .bashrc
file:
if [ -z "${PYTHONPATH}" ]
then
export PYTHONPATH=""
fi
export PYTHONPATH=${HOME}/opt/python/lib64/python2.4/site-packages:${HOME}/opt/python/lib/python2.4/site-packages:${PYTHONPATH}
export OMPI=/usr/local/ompi-ifort
export OPAL_PREFIX=${OMPI}
export OMP_NUM_THREADS=1
if [ -z "${PATH}" ]
then
export PATH=${OMPI}/bin
else
export PATH=${OMPI}/bin:${PATH}
fi
if [ -z "${LD_LIBRARY_PATH}" ]
then
export LD_LIBRARY_PATH=${OMPI}/lib
else
export LD_LIBRARY_PATH=${OMPI}/lib:${LD_LIBRARY_PATH}
fi
Make sure that you have the right mpicc:
which mpicc
/usr/local/ompi-ifort/bin/mpicc
and build GPAW (python setup.py build_ext 2>&1 | tee build_ext.log
)
with this customize.py
file
(Note: instructions valid from the 5232 release):
scalapack = True
compiler = 'gcc'
extra_compile_args += [
'-O3',
'-funroll-all-loops',
'-fPIC',
]
libraries= []
mkl_lib_path = '/opt/intel/mkl/10.0.4.023/lib/em64t/'
extra_link_args = [
mkl_lib_path+'libmkl_intel_lp64.a',
mkl_lib_path+'libmkl_sequential.a',
mkl_lib_path+'libmkl_core.a',
mkl_lib_path+'libmkl_blacs_openmpi_lp64.a',
mkl_lib_path+'libmkl_scalapack.a',
mkl_lib_path+'libmkl_blacs_openmpi_lp64.a',
mkl_lib_path+'libmkl_intel_lp64.a',
mkl_lib_path+'libmkl_sequential.a',
mkl_lib_path+'libmkl_core.a',
mkl_lib_path+'libmkl_intel_lp64.a',
mkl_lib_path+'libmkl_sequential.a',
mkl_lib_path+'libmkl_core.a',
]
define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
Note: is case of problems similar to those found on akka.hpc2n.umu.se static linking is required.
A gpaw script test.py
can be submitted like this:
qsub -l nodes=1:ppn=8 -l walltime=00:30:00 -m abe run.sh
where run.sh
looks like this:
#!/bin/sh
#PBS -m ae
#PBS -M email@email.com
#PBS -q long
#PBS -r n
#PBS -l nodes=1:ppn=8
cd $PBS_O_WORKDIR
echo Running on host `hostname` in directory `pwd`
NPROCS=`wc -l < $PBS_NODEFILE`
echo This jobs runs on the following $NPROCS processors:
cat $PBS_NODEFILE
export PYTHONPATH=${HOME}/opt/gpaw-0.7.5232:${HOME}/opt/python-ase-3.1.0.846:${PYTHONPATH}
export PYTHONPATH=${HOME}/opt/python/lib64/python2.4/site-packages:${HOME}/opt/python/lib/python2.4/site-packages:${PYTHONPATH}
export PATH=${HOME}/opt/gpaw-0.7.5232/build/bin.linux-x86_64-2.4:${PATH}
export GPAW_SETUP_PATH=${HOME}/opt/gpaw-setups-0.5.3574
export OMP_NUM_THREADS=1
mpiexec gpaw-python test.py
Please make sure that your jobs do not run multi-threaded, e.g. for a
job running on node02
do from a login node:
ssh node02 ps -fL
you should see 1 in the NLWP column. Numbers higher then 1 mean multi-threaded job.
It’s convenient to customize as described on the Parallel runs page.