Difference between revisions of "HowTo:adf"

From CAC Wiki
Jump to: navigation, search
(ADF)
(ADF)
 
(48 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= ADF =
+
== ADF ==
  
 
This is a short help file on how to use the computational Quantum Chemistry code "ADF" on our clusters. We require all users of this software to sign a statement. The software can only be accessed by persons who belong to a specific Unix group. See details below.  
 
This is a short help file on how to use the computational Quantum Chemistry code "ADF" on our clusters. We require all users of this software to sign a statement. The software can only be accessed by persons who belong to a specific Unix group. See details below.  
 +
 +
'''Important: This software is no longer licensed on our systems. Compute Canada users can still [https://docs.computecanada.ca/wiki/ADF run ADF on Graham].'''
 +
  
 
{|  style="border-spacing: 8px;"
 
{|  style="border-spacing: 8px;"
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#e1eaf1; border-radius:7px" |  
+
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#e1eaf1; border-radius:7px" |
== Features ==
+
 
 +
=== Features ===
 
ADF stands for "'''A'''msterdam '''D'''ensity '''F'''unctional" and denotes a package of programs that uses Density Functional Theory (DFT) for electronic and molecular structure calculations. The package is geared towards chemists and physicists with an interest in the structure of molecules and solids.
 
ADF stands for "'''A'''msterdam '''D'''ensity '''F'''unctional" and denotes a package of programs that uses Density Functional Theory (DFT) for electronic and molecular structure calculations. The package is geared towards chemists and physicists with an interest in the structure of molecules and solids.
  
Line 19: Line 23:
 
ADF handles geometry optimizations, transition states, reaction paths, and infrared frequencies. It allows the calculation of a variety of properties, ranging from UV spectra (requiring the treatment of excited states) to NMR chemical shifts and spin-spin couplings (where the use of Slater-type bases is of great use). The BAND code can be used for calculations on polymers, surfaces and bulk solids.
 
ADF handles geometry optimizations, transition states, reaction paths, and infrared frequencies. It allows the calculation of a variety of properties, ranging from UV spectra (requiring the treatment of excited states) to NMR chemical shifts and spin-spin couplings (where the use of Slater-type bases is of great use). The BAND code can be used for calculations on polymers, surfaces and bulk solids.
  
== Location of the program and setup ==
+
=== Location of the program and setup ===
  
The present version of ADF is 2016.101. The programs in the ADF package reside in /opt/adf. To use ADF on our machines, it is required that you read our licensing agreement and [http://www.hpcvl.org/sites/default/files/ADF_statement.pdf sign a statement]. You will then be made a member of a Unix group adf, which enables you to run the software.
+
The present version of ADF is 2017.108. The programs in the ADF package reside in /global/software/adf. To use ADF on our machines, it is required that you read our licensing agreement and [http://cac.queensu.ca/wp-content/files/cac-adf-statement.pdf sign a statement]. You will then be made a member of a Unix group adf, which enables you to run the software.
  
 
ADF requires the sourcing of a setup script to function properly:
 
ADF requires the sourcing of a setup script to function properly:
  
 
<pre>
 
<pre>
source /opt/adf/adf2016.101/adfrc.sh
+
source /global/software/adf/adf2017.108/adfrc.sh
 
</pre>
 
</pre>
  
Line 33: Line 37:
 
The above settings is best applied through a call to usepackage on our system. Issuing the command
 
The above settings is best applied through a call to usepackage on our system. Issuing the command
  
<pre>use adf</pre>
+
<pre>module load adf</pre>
  
 
will take care of this, as well as including the $ADFBIN directory in the $PATH. This command may also be placed into a login shell setup file (.bash_profile).
 
will take care of this, as well as including the $ADFBIN directory in the $PATH. This command may also be placed into a login shell setup file (.bash_profile).
  
== Scratch files ==
+
=== Scratch files ===
  
 
One of the settings is the environment variable '''SCM_TMPDIR''' which is required to redirect the temporary files that ADF uses to the proper scratch space, presently  
 
One of the settings is the environment variable '''SCM_TMPDIR''' which is required to redirect the temporary files that ADF uses to the proper scratch space, presently  
  
<pre>/scratch/hpcXXXX</pre>
+
<pre>
 +
/global/scratch/hpcXXXX
 +
</pre>
  
where hpcXXXX stands for your username. If for some reason ADF does not terminate normally (e.g. a job gets cancelled), it leaves behind large '''scratch files''' which you may have to delete manually. To check if such files exist, type
+
where hpcXXXX stands for your username. If for some reason ADF does not terminate normally (e.g. a job gets cancelled), it leaves behind large '''scratch files''' which you may have to delete manually. To check if such files exist, use the ls command. Usually the scratch files are in sub-directories that start with '''kid_''':
  
<pre>ls -lt /scratch/hpcXXXX</pre>
+
<pre>
 +
$ ls $SCM_TMPDIR
 +
kid_0.ax49ZF  kid_0.d9VDRE  kid_0.LQQtwj  kid_0.VLH2Ek  kid_0.Zjn0mZ
 +
kid_0.BBskVK  kid_0.DIHNlB  kid_0.m1wEx2  kid_0.vMalpJ  kid_1.1LZdFQ
 +
kid_0.cdUtDi  kid_0.fnWCrB  kid_0.pyWQ5u  kid_0.wElmpK  kid_2.mFNeFQ
 +
kid_0.cK4iVj  kid_0.HVRsbX  kid_0.QLOJsE  kid_0.XHby8B  kid_3.5t9REQ
 +
kid_0.AcBCde  kid_0.cmepvE  kid_0.kJ85iE  kid_0.sX0Ddm  kid_0.Y5xTMN
 +
</pre>
  
Usually the scratch files are in sub-directories that start with '''kid_'''. Once you have determined that the scratch files are no longer needed (because the program that used them is not running any more), you can delete them by typing
+
Once you have determined that the scratch files are no longer needed (because the program that used them is not running any more), you can delete them:
  
<pre>rm -r /scratch/hpcXXXX/kid_*</pre>  
+
<pre>
 +
$ rm -r $SCM_TMPDIR/kid_*
 +
$ ls $SCM_TMPDIR
 +
$
 +
</pre>  
  
 
Cleaning up the scratch space is the user's responsibility. If it is not done regularly, it can cause jobs to terminate, and much work to be lost.
 
Cleaning up the scratch space is the user's responsibility. If it is not done regularly, it can cause jobs to terminate, and much work to be lost.
Line 56: Line 73:
 
{|  style="border-spacing: 8px;"
 
{|  style="border-spacing: 8px;"
 
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#f7f7f7; border-radius:7px" |
 
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#f7f7f7; border-radius:7px" |
== Running ADF from a command line==
+
 
 +
=== Running ADF from a command line===
  
 
The following instructions assume that you are a member of the Unix group "adf". The instructions in this section are only useful if you need to run test jobs of a short duration. If you want to run a production job, please refer to to instructions on how to start a ADF batch job.
 
The following instructions assume that you are a member of the Unix group "adf". The instructions in this section are only useful if you need to run test jobs of a short duration. If you want to run a production job, please refer to to instructions on how to start a ADF batch job.
Line 68: Line 86:
 
Instructions about the job are read from standard input, which has been redirected from a file in in the above command lines. Commonly an input file will be constructed to specify what calculation is to be run. The output of the program(s) goes to "standard output" and has been redirected to an output file out above. Note that the output of these programs is commonly thousands of lines long and should therefore be redirected in any case.
 
Instructions about the job are read from standard input, which has been redirected from a file in in the above command lines. Commonly an input file will be constructed to specify what calculation is to be run. The output of the program(s) goes to "standard output" and has been redirected to an output file out above. Note that the output of these programs is commonly thousands of lines long and should therefore be redirected in any case.
  
The construction of a proper input file for ADF is an involved process, and is outside the scope of this help file. Detailed instructions can be found in the ADF User's Guide, which should be studied before the program can be used properly. As an initial hint, here is a  sample input file:
+
The construction of a proper input file for ADF is an involved process, and is outside the scope of this help file. Detailed instructions can be found in the [https://www.scm.com/doc/ADF/index.html ADF User's Guide], which should be studied before the program can be used properly. As an initial hint, here is a  sample input file:
  
 
<pre>
 
<pre>
Line 117: Line 135:
 
'''Note:''' It is absolutely essential to have a good idea about the size and complexity of your calculations before you start a ADF job. Many of the methods have terrible scaling properties, i.e. the computational cost grows very quickly with the number of electrons, degrees of freedom, or number of basis functions used. We suggest you start with a small basis set and a cheap method, and then slowly increase those parameters.
 
'''Note:''' It is absolutely essential to have a good idea about the size and complexity of your calculations before you start a ADF job. Many of the methods have terrible scaling properties, i.e. the computational cost grows very quickly with the number of electrons, degrees of freedom, or number of basis functions used. We suggest you start with a small basis set and a cheap method, and then slowly increase those parameters.
  
== Submitting (parallel) ADF jobs ==
+
=== Submitting (parallel) ADF jobs ===
  
In most cases, you will run ADF in batch mode.
+
In most cases, you will run ADF in batch mode through a scheduler. This is '''mandatory''' for production jobs, i.e. jobs of more than about 1 hr duration that use multiple processes, or multiple jobs submitted as part of a project. Production jobs that are submitted outside of the load balancing software will be terminated by the system administrator.
  
Production jobs are submitted to our systems via the Grid Engine, which is a load-balancing software. To obtain details, read our Grid Engine FAQ. For an ADF batch job, this means that rather than issuing the above commands directly, you wrap them into a Grid Engine batch script. Here is an example for such a batch script:
+
The advantage to submit jobs via a load balancing software is that the software will automatically find the resources required and put the job onto a node that has a low load. This will help executing the job faster.
 +
 
 +
Production jobs on Frontenac are submitted via [https://slurm.schedmd.com/overview.html SLURM]. To obtain details, read our [[SLURM|SLURM help file]]. For an ADF batch job, this means that rather than issuing the commands directly, you wrap them into a batch script. Here is an example for such a script:
  
 
<pre>
 
<pre>
#! /bin/bash
+
#!/bin/bash
#$ -S /bin/bash
+
#SBATCH --job-name=ADF_test
#$ -V
+
#SBATCH --mail-type=ALL
#$ -cwd
+
#SBATCH --mail-user=myEmail@whatever.com
#$ -M MyEmailAdress@whatever.com
+
#SBATCH --output=STD.out
#$ -m be
+
#SBATCH --error=STD.err
#$ -o STD.out
+
#SBATCH --nodes=1
#$ -e STD.err
+
#SBATCH --ntasks=8
#$ -pe shm.pe 12
+
#SBATCH --cpus-per-taskl=1
adf -n $NSLOTS <sample.adf >sample.log
+
#SBATCH --time=30:00
 +
#SBATCH --mem=1000
 +
module load adf
 +
export SCM_TMPDIR=$TMPDIR
 +
adf -n $SLURM_NTASKS < testsys.adf
 
</pre>
 
</pre>
  
This script needs to be altered by replacing all the relevant items. It sets all the necessary environment variables (make sure you issued a "use adf" statement before using this), and then starts the program. The lines in the script that start with #$ are interpreted the Grid Engine load balancing software as directives for the execution of the program.
+
This script needs to be altered by replacing all the relevant items. It sets all the necessary environment variables through the "module load adf" command, and then starts the program. The lines in the script that start with #SBATCH are interpreted SLURM as directives for the execution of the program.
  
For instance the line "#$ -m be" tells the Grid Engine to notify the user via email when the job has started and when it is finished, while the line beginning with "#$ -M" tells the Grid Engine about the email address of the user.
+
For instance the line "#SBATCH --mail-type=ALL" tells SLURM to notify the user via email when the job starts, finishes, or is cancelled. The line beginning with "#SBATCH --mail-user=" tells the scheduler about the email address of the user.
  
The -o and -e lines determine whence the standard input and the standard error are to be redirected. Since the job is going to be executed in batch, no terminal is available as a default for these.  
+
The --output and --error lines determine whence the standard input and the standard error are to be redirected. Since the job is going to be executed in batch, no terminal is available as a default for these.  
  
The ADF package is able to execute on several processors simultaneously in a distributed-memory fashion. This means that some tasks such as the calculation of a large number of matrix elements, or numerical integrations may be done in a fraction of the time it takes to execute on a single CPU. For this, the processors on the cluster need to be able to communicate. To this end ADF uses the MPI (Message Passing Interface), a well-established communication system.
+
The ADF package is able to execute in parallel using MPI. This means that some tasks may be done in a fraction of the time it takes to execute on a single CPU.  
  
Because ADF uses a specific version of the parallel system MPI (ClusterTools 7), executing the use adf command will also cause the system to "switch" to that version, which might have an impact on jobs that you are running from the same shell later. To undo this effect, you need to type use ct8 when you are finished using ADF and want to return to the production version of MPI (ClusterTools 8).
+
ADF parallel jobs contains several lines that determine the number of parallel processes to be used. The --nodes line determines how many nodes a involved (at present this needs to be 1), while -ntasks relates how many tasks (MPI processes) are being used. This number is crucial, as it is being re-used as an option in the "adf" command line through the internal variable SLURM_NTASKS. In the example we chose 8. Finally, then --cpus-per-task option sets the number of cores per task to 1. This should be set to 1 for ADF jobs.
  
ADF parallel jobs that are to be submitted to Grid Engine will use the MPI parallel environment and queues already defined for the user.
+
Once properly modified, the script (let's call it "adf.sh") can be submitted by typing
  
Our sample script contains a line that determines the number of parallel processes to be used by ADF. The Grid Engine will start the MPI parallel environment (PE) with a given number of slots that you specify by modifying that line:
+
<pre>sbatch adf.sh</pre>
  
<pre>#$ -pe shm.pe ''number of processes''</pre>
+
|}
  
where ''number of processes'' must be replaced (for instance, by 12 in our example above). It then determines the value of the environment variable NSLOTS which is used in the "adf" line of the sample script. This way, the system allocates exactly the number of processors that are used for the adf run, and no mismatch can occur.
+
{|  style="border-spacing: 8px;"
 +
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#e1eaf1; border-radius:7px" |
  
Once properly modified, the script (let's call it "adf.sh") can be submitted to the Grid Engine by typing
+
=== Migration SW -> Frontenac ===
  
<pre>qsub adf.sh</pre>
+
The following is a list of differences concerning ADF usage on tghe SW cluster vs the new Frontenac cluster. The main impact comes from the different scheduler.
  
The advantage to submit jobs via a load balancing software is that the software will automatically find the resources required and put the job onto a node that has a low load. This will help executing the job faster. Note that the usage of Grid Engine for all production jobs on HPCVL clusters is mandatory. Production jobs that are submitted outside of the load balancing software will be terminated by the system administrator.
+
{| class="wikitable" style="float:left; margin-right: 25px;"
 
+
!colspan="3"| '''Changes when migrating from SW cluster (SGE) to Frontenac (SLURM)'''
Luckily, there is an easier way to do all this: We are supplying a small perl script called that can be called directly, and will ask a few basic questions, such as the name for the job to be submitted and the number of processes to be used in the job. Simply type
+
|-
 
+
|
<pre>ADFSubmit</pre>
+
|'''SW (swlogin1)'''
 
+
|'''Frontenac (caclogin01)'''
and answer the questions. The script expects a ADF input file with "file extension" .adf to be present and will do everything else automatically. This is meant for simple ADF job submissions. More complex job submissions are better done manually.
+
|-
 +
| '''Version'''
 +
| 2016.01
 +
| 2017.01
 +
|-
 +
| '''Scheduler'''
 +
| Sun Grid Engine (SGE)
 +
| SLURM
 +
|-
 +
| '''GUI available'''
 +
| yes
 +
| yes
 +
|-
 +
| '''Queue specification <br> in submit script'''
 +
| none
 +
| none
 +
|-
 +
| '''Login node for <br> submission'''
 +
| swlogin1
 +
| caclogin01
 +
|-
 +
| '''Setup command'''
 +
| use adf
 +
| module load adf
 
|}
 
|}
  
{|  style="border-spacing: 8px;"
+
=== Licensing ===
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#e1eaf1; border-radius:7px" |
+
== Licensing ==
+
  
 
ADF is a licensed program. The license held by the Centre for Advanced Computing is limited to our computers at our main site. That means that any of our users can use the program on our machines (but nowhere else), whether they are located at Queen's or not.
 
ADF is a licensed program. The license held by the Centre for Advanced Computing is limited to our computers at our main site. That means that any of our users can use the program on our machines (but nowhere else), whether they are located at Queen's or not.
  
We require users of ADF to [http://www.hpcvl.org/sites/default/files/adf-statement.pdf sign a statement] in which they state that they are informed about the [http://www.hpcvl.org/sites/default/files/adf-licence.pdf terms of the license] to be included in the Gaussian user group named "adf". Please fax the completed statement to (613) 533-2015 or scan/email to [mailto:cac.admin@queensu.ca cac.admin@queensu.ca].
+
We require users of ADF to [http://cac.queensu.ca/wp-content/files/cac-adf-statement.pdf sign a statement] in which they state that they are informed about the [http://cac.queensu.ca/wp-content/files/cac-adf-license.pdf terms of the license] to be included in the ADF user group named "adf". Please fax the completed statement to (613) 533-2015 or scan/email to [mailto:cac.admin@queensu.ca cac.admin@queensu.ca].
  
== Where can I get more detailed information ? ==
+
=== Help ===
  
* To learn the basics about Gaussian input and output, refer to the [https://www.scm.com/documentation/ADF/index/ ADF 2016 Manual].  
+
* To learn the basics about Gaussian input and output, refer to the [https://www.scm.com/documentation/ADF/index/ ADF 2017 Manual].  
 
* For templates, and to get many examples, check out [https://www.scm.com/documentation/ADF/Examples/Examples/ https://www.scm.com/documentation/ADF/Examples/Examples/].
 
* For templates, and to get many examples, check out [https://www.scm.com/documentation/ADF/Examples/Examples/ https://www.scm.com/documentation/ADF/Examples/Examples/].
* The [http://www.gaussian.com Gaussian web page] contains a lot of information.  
+
* The [https://www.scm.com/adf-modeling-suite/ ADF web page] contains a lot of information.  
 
* '''Send [mailto:cac.help@queensu.ca|email to cac.help@queensu.ca]'''. We're happy to help.
 
* '''Send [mailto:cac.help@queensu.ca|email to cac.help@queensu.ca]'''. We're happy to help.
 
|}
 
|}

Latest revision as of 16:34, 11 May 2018

ADF

This is a short help file on how to use the computational Quantum Chemistry code "ADF" on our clusters. We require all users of this software to sign a statement. The software can only be accessed by persons who belong to a specific Unix group. See details below.

Important: This software is no longer licensed on our systems. Compute Canada users can still run ADF on Graham.


Features

ADF stands for "Amsterdam Density Functional" and denotes a package of programs that uses Density Functional Theory (DFT) for electronic and molecular structure calculations. The package is geared towards chemists and physicists with an interest in the structure of molecules and solids.

The ADF package consists of two main components:

  • ADF for molecular calculations
  • BAND for calculations on solids

Unlike most other molecular/solid/electronic structure codes, ADF employs "Slater-type" basis sets, ie, functions that have an exponential behaviour, which are more suitable for the description of chemical systems than the more commonly employed "Gaussian type" ones. The downside of this are computational difficulties that may be circumvented by numerical integration. Since DFT depends largely on numerical integration anyhow, the "Slater approach" is particularly well-suited for DFT code.

ADF is arguably the best DFT code available at this time for transition metal compounds and solids.

ADF handles geometry optimizations, transition states, reaction paths, and infrared frequencies. It allows the calculation of a variety of properties, ranging from UV spectra (requiring the treatment of excited states) to NMR chemical shifts and spin-spin couplings (where the use of Slater-type bases is of great use). The BAND code can be used for calculations on polymers, surfaces and bulk solids.

Location of the program and setup

The present version of ADF is 2017.108. The programs in the ADF package reside in /global/software/adf. To use ADF on our machines, it is required that you read our licensing agreement and sign a statement. You will then be made a member of a Unix group adf, which enables you to run the software.

ADF requires the sourcing of a setup script to function properly:

source /global/software/adf/adf2017.108/adfrc.sh

This script sets environment variables that are necessary for proper program execution and are used for the system to find executables and data files such as basis sets. Among others, SCMLICENSE is used by the license manager of the program to find a machine specific license file.

The above settings is best applied through a call to usepackage on our system. Issuing the command

module load adf

will take care of this, as well as including the $ADFBIN directory in the $PATH. This command may also be placed into a login shell setup file (.bash_profile).

Scratch files

One of the settings is the environment variable SCM_TMPDIR which is required to redirect the temporary files that ADF uses to the proper scratch space, presently

/global/scratch/hpcXXXX

where hpcXXXX stands for your username. If for some reason ADF does not terminate normally (e.g. a job gets cancelled), it leaves behind large scratch files which you may have to delete manually. To check if such files exist, use the ls command. Usually the scratch files are in sub-directories that start with kid_:

$ ls $SCM_TMPDIR
kid_0.ax49ZF  kid_0.d9VDRE  kid_0.LQQtwj  kid_0.VLH2Ek  kid_0.Zjn0mZ
kid_0.BBskVK  kid_0.DIHNlB  kid_0.m1wEx2  kid_0.vMalpJ  kid_1.1LZdFQ
kid_0.cdUtDi  kid_0.fnWCrB  kid_0.pyWQ5u  kid_0.wElmpK  kid_2.mFNeFQ
kid_0.cK4iVj  kid_0.HVRsbX  kid_0.QLOJsE  kid_0.XHby8B  kid_3.5t9REQ
kid_0.AcBCde  kid_0.cmepvE  kid_0.kJ85iE  kid_0.sX0Ddm  kid_0.Y5xTMN

Once you have determined that the scratch files are no longer needed (because the program that used them is not running any more), you can delete them:

$ rm -r $SCM_TMPDIR/kid_*
$ ls $SCM_TMPDIR
$

Cleaning up the scratch space is the user's responsibility. If it is not done regularly, it can cause jobs to terminate, and much work to be lost.

Running ADF from a command line

The following instructions assume that you are a member of the Unix group "adf". The instructions in this section are only useful if you need to run test jobs of a short duration. If you want to run a production job, please refer to to instructions on how to start a ADF batch job.

Once program usage is set up through the "use" command, the program(s) can be run:

adf <in >out 

Instructions about the job are read from standard input, which has been redirected from a file in in the above command lines. Commonly an input file will be constructed to specify what calculation is to be run. The output of the program(s) goes to "standard output" and has been redirected to an output file out above. Note that the output of these programs is commonly thousands of lines long and should therefore be redirected in any case.

The construction of a proper input file for ADF is an involved process, and is outside the scope of this help file. Detailed instructions can be found in the ADF User's Guide, which should be studied before the program can be used properly. As an initial hint, here is a sample input file:

title benzene BP/SZ bondorders tol=0.05

define
 cc=1.38476576
 ccc=120.0
 dih=0.0
 hc=1.07212846
 hcc= 120.0
 dih2=180.0
end

atoms Z-matrix
 C  0 0 0
 C  1 0 0  cc
 C  2 1 0  cc ccc
 C  3 2 1  cc ccc dih
 C  4 3 2  cc ccc dih
 C  5 4 3  cc ccc dih
 H  2 1 3  hc hcc dih2
 H  3 2 4  hc hcc dih2
 H  4 3 5  hc hcc dih2
 H  5 4 3  hc hcc dih2
 H  6 5 4  hc hcc dih2
 H  1 2 3  hc hcc dih2
end

basis
 Type SZ
 Core None
end

symmetry NOSYM

xc
  gga becke perdew
end

bondorder tol=0.05 printall

noprint sfo

The input consists of several units, separated by blank lines, starting with a keyword, and ending with the statement END. For instance, the atoms in a molecules may be specified by issuing the keyword atoms, followed by one line with the atom name and "Z-matrix" relative coordinates for each atom, and closing with end (case insensitive).

Note: It is absolutely essential to have a good idea about the size and complexity of your calculations before you start a ADF job. Many of the methods have terrible scaling properties, i.e. the computational cost grows very quickly with the number of electrons, degrees of freedom, or number of basis functions used. We suggest you start with a small basis set and a cheap method, and then slowly increase those parameters.

Submitting (parallel) ADF jobs

In most cases, you will run ADF in batch mode through a scheduler. This is mandatory for production jobs, i.e. jobs of more than about 1 hr duration that use multiple processes, or multiple jobs submitted as part of a project. Production jobs that are submitted outside of the load balancing software will be terminated by the system administrator.

The advantage to submit jobs via a load balancing software is that the software will automatically find the resources required and put the job onto a node that has a low load. This will help executing the job faster.

Production jobs on Frontenac are submitted via SLURM. To obtain details, read our SLURM help file. For an ADF batch job, this means that rather than issuing the commands directly, you wrap them into a batch script. Here is an example for such a script:

#!/bin/bash
#SBATCH --job-name=ADF_test
#SBATCH --mail-type=ALL
#SBATCH --mail-user=myEmail@whatever.com
#SBATCH --output=STD.out
#SBATCH --error=STD.err
#SBATCH --nodes=1
#SBATCH --ntasks=8
#SBATCH --cpus-per-taskl=1
#SBATCH --time=30:00
#SBATCH --mem=1000
module load adf
export SCM_TMPDIR=$TMPDIR
adf -n $SLURM_NTASKS < testsys.adf

This script needs to be altered by replacing all the relevant items. It sets all the necessary environment variables through the "module load adf" command, and then starts the program. The lines in the script that start with #SBATCH are interpreted SLURM as directives for the execution of the program.

For instance the line "#SBATCH --mail-type=ALL" tells SLURM to notify the user via email when the job starts, finishes, or is cancelled. The line beginning with "#SBATCH --mail-user=" tells the scheduler about the email address of the user.

The --output and --error lines determine whence the standard input and the standard error are to be redirected. Since the job is going to be executed in batch, no terminal is available as a default for these.

The ADF package is able to execute in parallel using MPI. This means that some tasks may be done in a fraction of the time it takes to execute on a single CPU.

ADF parallel jobs contains several lines that determine the number of parallel processes to be used. The --nodes line determines how many nodes a involved (at present this needs to be 1), while -ntasks relates how many tasks (MPI processes) are being used. This number is crucial, as it is being re-used as an option in the "adf" command line through the internal variable SLURM_NTASKS. In the example we chose 8. Finally, then --cpus-per-task option sets the number of cores per task to 1. This should be set to 1 for ADF jobs.

Once properly modified, the script (let's call it "adf.sh") can be submitted by typing

sbatch adf.sh

Migration SW -> Frontenac

The following is a list of differences concerning ADF usage on tghe SW cluster vs the new Frontenac cluster. The main impact comes from the different scheduler.

Changes when migrating from SW cluster (SGE) to Frontenac (SLURM)
SW (swlogin1) Frontenac (caclogin01)
Version 2016.01 2017.01
Scheduler Sun Grid Engine (SGE) SLURM
GUI available yes yes
Queue specification
in submit script
none none
Login node for
submission
swlogin1 caclogin01
Setup command use adf module load adf

Licensing

ADF is a licensed program. The license held by the Centre for Advanced Computing is limited to our computers at our main site. That means that any of our users can use the program on our machines (but nowhere else), whether they are located at Queen's or not.

We require users of ADF to sign a statement in which they state that they are informed about the terms of the license to be included in the ADF user group named "adf". Please fax the completed statement to (613) 533-2015 or scan/email to cac.admin@queensu.ca.

Help