carbon
Here you find information about the system https://www.top500.org/system/9025.
The Carbon machine is a cluster of dual socket, quad-core Intel Xeon 5355 CPUs, 2.66 GHz processors with 2 GB of memory per core.
Warning: numpy build instructions have not been tested recently and may not work. Please use the (unoptimized) system default numpy.
To build (python setup.py install --home=~/numpy-1.0.4-1
)
numpy-1.0.4 add these lines to site.cfg
:
[DEFAULT]
library_dirs = /usr/local/lib:/opt/intel/mkl/10.0.2.018/lib/em64t
include_dirs = /usr/local/include:/opt/intel/mkl/10.0.2.018/include
and, in numpy/distutils/system_info.py
change the line:
_lib_mkl = ['mkl','vml','guide']
into:
_lib_mkl = ['mkl','guide']
and the line:
lapack_libs = self.get_libs('lapack_libs',['mkl_lapack32','mkl_lapack64'])
into:
lapack_libs = self.get_libs('lapack_libs',['mkl_lapack'])
Set these environment variables in the .bashrc
file:
export OMPI_CC=gcc
export OMP_NUM_THREADS=1
export PYTHONPATH=${HOME}/gpaw:${HOME}/ase3k:${HOME}/numpy-1.0.4-1/lib64/python:
export GPAW_SETUP_PATH=${HOME}/gpaw-setups-0.4.2039
export LD_LIBRARY_PATH=/usr/lib64/openmpi:/opt/intel/mkl/10.0.2.018/lib/em64t
export PATH=${HOME}/gpaw/tools:${HOME}/ase3k/tools:/usr/share/openmpi/bin64:${PATH}
if [ $PBS_ENVIRONMENT ]; then
cd $PBS_O_WORKDIR
export PYTHONPATH=${PBS_O_WORKDIR}:${PYTHONPATH}
return
fi
and build GPAW (python setup.py build_ext
) with this
customize.py
file:
scalapack = True
extra_compile_args += [
'-O3'
]
libraries= []
mkl_lib_path = '/opt/intel/mkl/10.0.2.018/lib/em64t/'
extra_link_args = [
mkl_lib_path+'libmkl_intel_lp64.a',
mkl_lib_path+'libmkl_sequential.a',
mkl_lib_path+'libmkl_core.a',
mkl_lib_path+'libmkl_blacs_openmpi_lp64.a',
mkl_lib_path+'libmkl_scalapack.a',
mkl_lib_path+'libmkl_blacs_openmpi_lp64.a',
mkl_lib_path+'libmkl_intel_lp64.a',
mkl_lib_path+'libmkl_sequential.a',
mkl_lib_path+'libmkl_core.a',
mkl_lib_path+'libmkl_intel_lp64.a',
mkl_lib_path+'libmkl_sequential.a',
mkl_lib_path+'libmkl_core.a',
]
define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
Note: due to linking problems similar to those found on akka.hpc2n.umu.se static linking is required.
A gpaw script gpaw-script.py
can be submitted like this:
qsub -l nodes=1:ppn=8 -l walltime=02:00:00 \
-m abe run.sh
where run.sh
looks like this:
cd $PBS_O_WORKDIR
mpirun -machinefile $PBS_NODEFILE -np 8 -x OMP_NUM_THREADS \
$HOME/gpaw/build/bin.linux-x86_64-2.4/gpaw-python gpaw-script.py
Please make sure that your jobs do not run multi-threaded, e.g. for a
job running on n090
do from a login node:
ssh n090 ps -fL
you should see 1 in the NLWP column. Numbers higher then 1 mean multi-threaded job.
It’s convenient to customize as described on the Parallel runs page.