|
If you have any specific questions
relating to running AMBER at SDSC please contact
consulting@sdsc.edu.
General questions concerning AMBER should be directed to the
AMBER mailing list (amber@scripps.edu).
Amber 9 Installation / Available Codes
The recommended version of AMBER to run on the
Bluegene architecture is AMBER 9. At present this is the
only installed version. AMBER 9 is installed in
/usr/local/apps/amber9 In
here you will find an exe
directory containing the executables and a
dat directory containing the
force field files. The architecture of the IBM Bluegene
system is such that only PMEMD in parallel mode is
available. This restricts the possible calculations to
explicit solvent PME simulations and Generalized Born
Implicit solvent simulations.
| Executable (aliases) |
Description |
| pmemd.MPI (pmemd) |
Only available executable. For running Molecular
Dynamics simulations in parallel on IBM Bluegene. Supports Particle Mesh Ewald (PME)
and Generalized Born (GB) simulations. |
Amber 9 Performance and Scaling
The scaling behaviour will very much
depend on the type and size of your job. In order to
effectively utilize the Bluegene architecture it is
imperative that you problem scales to at least 512 cpus.
This typically means that for PME simulations you will
require 100,000+ atoms and for GB simulations you will
require 512+ residues. The graph below shows
the expected scaling for two PME simulations (FactorIX [91K atoms]
and Cellulose [408K atoms]).
Ps/day
 |
Speedup
 |
| click image for
larger view |
As you can see from the graph all simulations have a region
where the scaling is acceptable and then where it tends to
tail off. Caution: Going to very large numbers of cpus can
often result in your code taking longer. The exact scaling
you see will depend on the size and type of job you are
running so before burning to much cpu time you should test
the scaling with the simulation you plan to run. Typically
the optimum point on the Bluegene machine is between 256 and
512 cpus but if your simulation is small you may need to use
less cpus. Required Environment Options
The Amber 9 installation in /usr/local/apps/amber9
was compiled with xlf90 for the 440d architecture but
at -O3 and without the SIMD options. Using the SIMD options
of -O > 3 currently gives incorrect answers. This is an
issue with the IBM compiler that we are hoping will be fixed
in later releases.
Example Job Submission Scripts
The following is an
example job submission script for a PMEMD run (the \'s act
as line continuation characters. If you want you can put all
of the options on a single line):
|
pmemd_bluegene_256cpu.x |
|
| Script |
Explanation |
#!/usr/bin/ksh
#@ environment = COPY_ALL;
#@ job_type = BlueGene
#@ class = parallel
#@ bg_partition = bot128-2
#@ output = file.$(jobid).out
#@ error = file.$(jobid).err
#@ notification = complete
#@ notify_user = youremail@sdsc.edu
#@ wall_clock_limit = 00:30:00
#@ account_no = accountcode
#@ queue
mpirun -mode VN -np 256 \
-exe /usr/local/apps/amber9/exe/pmemd \
-cwd /users/username/files/ \
-args "-O -i mdin -c inpcrd -p prmtop -o mdout"
|
bg_patition : be sure to selected an
appropriate machine partition for the number of
cpus you are requesting. Refer to the
Bluegene userguide for more info on this.
walltime=0:30:00 : set this to
slightly longer than you think your job will
take to run. Maximum is 18:00:00 = 18 hours.
Smaller values can get your jobs run sooner due
to backfill opportunities.
account_no : make sure you replace
accountcode with the account you want to be
charged.
-mode VN : select VN for virtual node
mode and CO for co-processor mode. Benchmarks
above were run in virtual node mode. Refer to
the
Bluegene userguide for more info on this.
-np 256 : change this to the number of
cpus you want to use depending on the partition
you selected.
|
You can submit this job to the queue using llsubmit.
KNOWN LIMITATIONS
Only mpi jobs using PMEMD are available due to
restrictions with the Bluegene architecture.
| Return to Main Page |
| 
