bwgrid

The BWgrid is an grid of machines located in Baden-Württemberg, Germany. The installation in Freiburg is a cluster containing 139 dual socket, quad-core Intel Xenon E5440 CPUs, 2.83GHz processors with 2 GB of memory per core, 16 dual socket, quad-core Intel Xenon X5550 CPUs, 2.67GHz processors with 3 GB of memory per core and eight dual socket, six-core Intel Xenon X5650 CPUs, 2.66GHz processors with 2 GB of memory per core. For more information visit https://www.bfg.uni-freiburg.de/doc/bwgrid.

Building GPAW with Intel compiler

Use the compiler wrapper file icc.py

#!/usr/bin/python
"""icc.py is a wrapper for the Intel compiler,
   converting/removing incompatible gcc args.   """

import sys
from subprocess import call

args2change = {"-fno-strict-aliasing":"",
               "-fmessage-length=0":"",
               "-Wall":"",
               "-std=c99":"-qlanglvl=extc99",
               "-fPIC":"",
               "-g":"",
               "-D_FORTIFY_SOURCE=2":"",
               "-DNDEBUG":"",
               "-UNDEBUG":"",
               "-pthread":"",
               "-shared":"-qmkshrobj",
               "-Xlinker":"",
               "-export-dynamic":"",
               "-Wstrict-prototypes":"",
               "-dynamic":"",
               "-O3":"",
               "-O3":"",
               "-O2":"",
               "-O1":""}

fragile_files = ["test.c"]

cmd = ""
fragile = False
for arg in sys.argv[1:]:
    cmd += " "
    t = arg.strip()
    if t in fragile_files:
        fragile = True
    if t in args2change:
        cmd += args2change[t]
    else:
        cmd += arg

flags = "-w -O3 -std=c99"
cmd = f"mpicc {flags} {cmd}"

call(cmd, shell=True)

Instructions assume bash, installation under $HOME/opt. Load the necessary modules:

module load devel/python/2.7.2
module load compiler/intel/12.0
module load mpi/impi/4.0.2-intel-12.0
module load numlib/mkl/10.3.5
module load numlib/python_numpy/1.6.1-python-2.7.2

Internal libxc

Before revision 10429 libxc was internal. The customize.py had to be changed to customize_bwgrid_icc.py

compiler = './icc.py'
mpicompiler = './icc.py'
mpilinker = 'MPICH_CC=gcc mpicc'

scalapack = True

define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
define_macros += [("GPAW_ASYNC",1)]

libraries += ['mkl_intel_lp64' ,'mkl_sequential' ,'mkl_core',
        'mkl_lapack95_lp64',
        'mkl_scalapack_lp64', 'mkl_blacs_intelmpi_lp64',
        'pthread'
        ]

External libxc

After svn revision 10429 libxc has to be included as external library (see also the libxc web site). To install libxc we assume that MYLIBXCDIR is set to the directory where you want to install:

$ module load compiler/intel/12.0
$ cd $MYLIBXCDIR
$ wget https://www.tddft.org/programs/octopus/down.php?file=libxc/libxc-2.0.2.tar.gz
$ tar -xzvf libxc-2.0.2.tar.gz
$ cd libxc-2.0.2/
$ mkdir install
$ ./configure CFLAGS="-fPIC" --prefix=$PWD/install -enable-shared
$ make |tee make.log
$ make install

This will have installed the libs $MYLIBXCDIR/libxc-2.0.2/install/lib and the C header files to $MYLIBXCDIR/libxc-2.0.2/install/include.

We have to modify the file customize.py to customize_bwgrid_icc_libxc.py

compiler = './icc.py'
mpicompiler = './icc.py'
mpilinker = 'MPICH_CC=gcc mpicc'

scalapack = True

define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
define_macros += [("GPAW_ASYNC",1)]

libraries += ['mkl_intel_lp64' ,'mkl_sequential' ,'mkl_core',
        'mkl_lapack95_lp64',
        'mkl_scalapack_lp64', 'mkl_blacs_intelmpi_lp64',
        'pthread', 'xc'
        ]

libraries += ['xc']
# change this to your installation directory
LIBXCDIR='/home/mw767/gridpaw/libxc-2.0.2/install/'
library_dirs += [LIBXCDIR + 'lib']
include_dirs += [LIBXCDIR + 'include']

Note that the location of the external libxc on runtime has to be enabled by setting:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MYLIBXCDIR/libxc-2.0.2/install/lib

Building and running GPAW

To build GPAW use:

python setup.py build_ext 2>&1 | tee build_ext.log

and ignore some intermediate warnings.

A gpaw script test.py can be submitted to run on 8 cpus like this:

> gpaw-runscript test.py 8
using bwg
run.bwg written
> qsub run.bwg