LAMMPS Replica Exchange

Prep Work

Download LAMMPS as explained in the Compiling LAMMPS section.
Prepare the packages and files specific to your type of simulation (Go-Model; 3SPN2).
Add the replica package before building the executable. That can be done by traveling to the src folder and typing the following command:

make yes-replica

Build the executable as explained in the Compiling LAMMPS section. (This will be a file named something similar to "lmp_mpi".)
Prepare the input file (the file usually ending with ".in" that contains the simulation instructions)
Prepare data files if they are used (the files usually ending with ".data" that contain long lists of data).
Write a submission script and make sure it works before continuing.

Running Replica Exchange

Replica Exchange works similarly to a normal LAMMPS simulation. The only difference is that you are having several copies of the simulation run on other processors, with the temperatures swapping between the copies. There are two places where changes need to be made in order for this to happen. The first is in the LAMMPS input file (usually ending in .in), where you specify that the simulation will have multiple temperatures and be a "temper" simulation. The second place is in the submission script, where you need to request all the additional processors needed for the replica exchange simulation.

LAMMPS Input File Additions

There are five commands that need to be included in the input file in order to properly run a replica exchange:

A variable definition for the temperature variable array
A fix command to fix the temperature
A fix command to correct momentum drift
A velocity command to create initial velocities
A temper command to start the actual replica exchange

Note that, depending on what input file you are converting over to replica exchange, some or all of these commands may already be present.

Temperature Array Definition

The format of the temperature array definition is as follows:

variable [variable_Name] world [list_of_temperatures]

variable is one of the allowed LAMMPS commands. It does what one might expect: it defines a variable. The rest of the line are the arguments required for this command.
[variable_Name] should be replaced with a user-chosen name. It does not need to be prefaced with any special characters here, but elsewhere in the input script it will usually be prefaced with a dollar sign.
world indicates that this type of variable is a multi-value variable with each value assigned to a different processor.
[list_of_temperatures] should be replaced with a list of numbers, each number corresponding to a temperature (usually in Kelvins) for one of the replica exchange boxes. Separate each number with a space.

Here is an example of what a completed temperature array definition would look like:

variable temperature world 350 352 354 356 358 360 362 364

Temperature Fixing

Some kind of temperature fixing is required for replica exchange. LAMMPS contains several different styles of temperature fixing, most of which are compatible with replica exchange. This section will use, as an example, a NVT temperature fix.

The format is as follows:

fix [fix_name] all nvt temp [temperature_variable] [temperature_variable] [dampening_amount] tchain [chain_length]

fix is one of the allowed LAMMPS commands. It is used for setting some kind of restraint on the simulation. The rest of the line are the arguments required for this command.
[fix_name] should be replaced with a user-chosen name. Any name will do, but be descriptive.
all indicates that all atoms are affected.
nvt indicates that this will be a fix where the Number (n) Volume (v) and Temperature (t) are fixed. If you were doing a different type of temperature fixing instead of nvt, that type would go here.
temp indicates the next three arguments are related to the temperature.
[temperature_variable] should be replaced with the variable name you used in the temperature array definition, prefaced by a dollar sign and inside curly braces (braces are optional if it is only 1 character long). This argument tells where the Temperature starts at when the fix activates.
[temperature_variable] should be replaced with the variable name you used in the temperature array definition, prefaced by a dollar sign and inside curly braces (braces are optional if it is only 1 character long). This argument tells where the Temperature stops at when the fix activates.
[dampening_amount] should be replaced with a time unit. This controls how forcefully the simulation will yank the temperature.
tchain indicates the next argument will be the length of the thermostat chain.
[chain_length] should be replaced with a number specifying the length of the thermostat chain.

Here is an example of what a completed temperature array definition would look like:

fix NoseHoverTemperatureFixing all nvt temp ${temperature} ${temperature} 100.0 tchain 3

Momentum Drift Correction

If the temperature is being corrected then the method of temperature correction could cause the entire collection of atoms to start drifting or rotating. This corrects the drift or rotation and keeps the total drift/rotation to zero.

The format is as follows:

fix [fix_name] all momentum [Timestep] linear 1 1 1 rescale angular

fix is one of the allowed LAMMPS commands. It is used for setting some kind of restraint on the simulation. The rest of the line are the arguments required for this command.
[fix_name] should be replaced with a user-chosen name. Any name will do, but be descriptive.
all indicates that all atoms are affected.
momentum indicates that this fix affects the momentum.
[Timestep] should be replaced with a number indicating the number of timesteps between momentum drift corrections.
linear indicates that the next three arguments are flags that say whether to include to exclude specific directions in the momentum correction.
1 indicates that the momentum in the x-direction should be corrected.
1 indicates that the momentum in the y-direction should be corrected.
1 indicates that the momentum in the z-direction should be corrected.
rescale indicates that the Kenetic Energy should be conserved by adjusting the velocities after fixing the momentum.
angular indicates that the angular momentum should be corrected.

Here is an example of what a completed momentum drift correction would look like:

fix MomentumDriftCorrection all momentum 100000 linear 1 1 1 rescale angular

Velocity Initialization

The velocity of the different particles in the simulation needs to be initialized at the start of the simulation. The velocity command does this.

The format is as follows:

velocity all create [temperature_variable] [random_seed] dist [distribution]

velocity is one of the allowed LAMMPS commands. It is used for manipulating the velocity of the particles in some way.
all indicates that all atoms are affected.
create indicates that this velocity manipulation is one that sets initial velocities.
[temperature_variable] should be replaced with the variable name you used in the temperature array definition, prefaced by a dollar sign and inside curly braces (the braces are optional if it is only 1 character long). This argument tells what value the temperature should be initialized to.
[random_seed] should be replaced with a number that will serve as the random seed for computing the velocities.
dist indicates that the next argument will describe how to distribute the velocities.
[distribution] should be replaced with the word "uniform" (without quotes), or the word "gaussian" (without quotes). A uniform distribution creates velocities that are evenly spaced between a maximum an a minimum while a gaussian distribution will have a gaussian probability curve for the velocities.

Here is an example of what a completed velocity initialization would look like:

velocity all create ${temperature} 12345678 dist uniform

Temper Command

The temper command is the key command that starts the simulation. It should be the last line in the file. Normally the command to start is simulation is run, but if you want to do a replica exchange, you need to use temper instead.

The format is as follows:

temper [length_of_simulation] [timesteps_between_swaps] [temperature_variable] [temperature_fixing] [seed1] [seed2]

temper is one of the allowed LAMMPS commands. It starts and runs a replica exchange simulation.
[length_of_simulation] should be replaced with a number indicating how many timesteps to run the simulation.
[timesteps_between_swaps] should be replaced with a number indicating how many timesteps to go between swaps.
[temperature_variable] should be replaced with the variable name you used in the temperature array definition, prefaced by a dollar sign and inside curly braces (braces are optional if it is only 1 character long). This argument controls the temperatures the boxes will be set to.
[temperature_fixing] should be replaced with the name you assigned to the fix that will fix the temperature.
[seed1] should be replaced by a random number that will serve as the seed for randomly swapping the temperatures.
[seed2] should be replaced by a random number that will serve as the seed for the Boltzmann factor in Metropolis swap.

Here is an example of what a complete temper command would look like:

temper 40000000 2000 ${temperature} NoseHoverTemperatureFixing 3847 58728

Submission Script Additions

The submission script must be set to run as many cores as there are temperature variables. The number of cores is the number of nodes times the processors per node count.

Here is the format of the line of the submission script that runs the simulation:

mpirun -np [partitionCount] [executable] -partition [partitionCount]x1 -in [inputFile]

mpirun stands for Message Passing Interface Run and is a command to start a parallel computation that runs on several different partitions.
-np means the next argument will state the number of processes to run
[partitionCount] should be replaced with the number of different boxes you want in your replica exchange simulation.
[executable] should be replaced with the name of the executable you compiled (usually named something like lmp_mpi).
-partition indicates the next argument will tell how to partition the simulation.
[partitionCount]x1 is a statement of how to partition the simulation. If you have eight partitions, you would write 8x1, if you have eighty partitions, you would write 80x1, etc..
-in indicates the next argument will specify the input file
[inputFile] should be replaced with the name of your simulation's input file. This is the file commonly ending in ".in".

Here is an example of a complete line:

mpirun -np 8 lmp_mpi -partition 8x1 -in Protein_Simulation.in

Here is an example of what a full submission script looks like on Watt:

#!/bin/bash
#PBS -l nodes=2:ppn=4,pmem=400MB,walltime=24:00:00
#PBS -N ProteinSimulation
mpirun -np 8 lmp_mpi -partition 8x1 -in Protein_Simulation.in

Here is an example of what a full submission script looks like on the supercomputer:

#!/bin/bash
#SBATCH --time=24:00:00
#SBATCH --ntasks=8
#SBATCH --mem-per-cpu=400M
#SBATCH -J "ProteinSimulation"
mpirun -np 8 lmp_mpi -partition 8x1 -in Protein_Simulation.in