2. Reference to gracos Executable

This chapter documents gracos executable, - our main product.

N-body simulations tend to be massive; however before any computationally large problem is submitted to a cluster it is more than useful to test the same problem for a reduced problem size. The script examples presented throughout this reference are just those small test cases. In order to extend them to larger problems just increase the number of processes, the gridsize and make sure no part of the script is intrinsically unscalable. This chapter assumes the successful completion of the installation procedure, See section Installation Procedure.

2.1 Introduction to Interface

Sophisticated interface is used to extend gracos capabilities and make it easier to document the simulations. This section provides necessary introduction.

2.1.1 Running gracos

gracos can be run both in parallel and serially; typing gracos --help or gracos -? yields a useful help message.

Usage: gracos [OPTION...]
A parallel program for N-body simulations and data analysis.

  -c, -e, --command=string   Read and execute commands from the string
  -f                         Forward the standard output of the head 
                             process to an so-0 file
  -i, --input=FILE           Read and execute commands from FILE
  -t, --term                 Read and execute commands from standard input
  -v, --verb=integer         Set the verbosity level
  -?, --help                 Give this help list
      --usage                Give a short usage message
  -V, --version              Print program version

Mandatory or optional arguments to long options are also mandatory or optional
for any corresponding short options.

Report bugs to Alexander Shirokov.

The user interacts with gracos by passing the commands within the gracos scripting session. The commands can be entered from standard input (option -t or the default), a file (option -i), or passed in the command line (option -e). The commands are executed using the Tcl embedded interpreter, See Commands for more detail.

To enter the commands through the standard input, simply type gracos or, for a parallel run on Nproc logical processes type

SHELL$ mpirun -np Nproc gracos

You may use any integer number Nproc of processes greater than zero on any machine, however it is best to accommodate the agreement between the number Nproc of logical processes and the number of physical processes and cores available on the system.

The gracos% token that will appear prompts the user to enter a new command from the standard input in the newly opened gracos scripting session. As can be seen from the following example by a user familiar with Tcl, indeed gracos uses the facilities provided by the Tcl computing language:

SHELL$ mpirun -np 4 gracos
/---------------------------------------------------\
|            GRACOS, VERSION: 1.0.1a10              |
\___________________________________________________/
gracos% puts "Hello"
Hello
gracos% foreach i { 1 2 3 4 } {
.......    puts $i
....... }
1
2
3
4
gracos% 

The input is distinguished from the output by the presence or absence of the prompts ("gracos% " and "....... " - for incomplete commands). In the above example, $i refers to a Tcl variable, and foreach is a Tcl built-in command. As soon as the complete command is entered within the scripting session, its copies are sent to the remaining MPI processes and the command is processed with the embedded Tcl interpreter individually on each process, See Commands for more information.

In order to master gracos, the user should learn some basics of Tcl. Fortunately, Tcl is usually found easy to learn. A number of resources are available online; we refer the users to the Tcl reference, the Tcl manual and the Tcl tutorial for version 8.5. In gracos we avoid the use of any advanced Tcl feature, when possible. For those readers whose access to the above online resources is restricted, let us just say that the Tcl built-in commands set and puts are used to manage the Tcl variables (setting a variable and displaying its value); the square brackets '[' and ']' are used for an in-line expansion of a Tcl expression; the curly brackets '{' and '}' as well and the double quotation marks are used for quoting in slightly different contexts (mainly: the expansion of subexpressions is allowed in double quoted expressions but is not allowed in the curly bracket quotations); the expr Tcl built-in command is used for arithmetic expression evaluation, the exec is for external shell expression evaluation, and the eval is for the evaluation of the arguments passed to the command in the current scripting session. Finally cd can be used to change the current working directory path.

In addition to the more generic commands intrinsic to the Tcl, gracos has a number of commands that are only specific to the gracos executable; their implementation forms the trunk of the gracos C-code.

As soon as gracos executable is ran, a number of useful files appear in the working directory. File gracos_history contains the current listing of all the executed commands; files so-1 … so-(Nproc-1) show the standard output on non-server processes. The standard output on process zero appears on the screen, it is not redirected unless the --f command line option is used.

To execute the same commands as in the example above by reading them from a file, instead of standard input, we create a temporary file with the mktemp Linux command, fill it with the (identical) commands to be executed and pass the filename with the -i option to gracos

SHELL$ TmpFile=`mktemp tmp.XXXXXX` 
SHELL$ cat >  $TmpFile
puts "Hello"
foreach i { 1 2 3 4 } {
   puts $i
}
SHELL$ mpirun -np 4 gracos -i $TmpFile
gracos% puts "Hello"
Hello
gracos% foreach i { 1 2 3 4 } {
   puts $i
}

1
2
3
4
SHELL$

Now, instead of having to type the commands in through the shell session, we can place them into a new file, make the file executable and run. Shown below is the copy of the working script, also provided at gracos-examples/hello.sh

#!/bin/bash
# File: gracos-examples/hello.sh

# GRACOS Session
TmpFile=`mktemp tmp.XXXXXX`
cat <<EOF >  $TmpFile
puts "Hello"
foreach i { 1 2 3 4 } {
   puts \$i
}
EOF

mpirun -np 4 gracos -i $TmpFile
rm $TmpFile

Typing

chmod +x ./hello.sh
./hello.sh

yields the expected result. The important detail to notice in the last example above is using the backslash character (\) necessary to avoid the conflict between the interpretations of the '$' character, which is used for variable dereferencing by both Tcl and bash. If you do not use the backslash character in the example above the variable $i is interpreted as a bash shell variable and, according to the bash syntax, be expanded to empty value because no such bash variable is defined in the script. Because of the use of backslash character, this variable is interpreted as a Tcl variable which is set in the preceeding foreach Tcl statement. One just has to be careful using backslashes to avoid the similar misinterpretations.

2.1.2 gracos-examples

Each case considered in this reference is presented as a gracos script ready to be executed in your Unix-flavored shell, any time following the GRACOS installation procedure, See section Installation Procedure. These scripts are organized within the so called gracos-examples directory whose archive is one of the products of GRACOS installation. In order to make a local copy of the directory, simply type:

SHELL$ cp `gracos-config info pkgdatadir`/gracos-examples.tar.gz .
SHELL$ tar zxf gracos-examples.tar.gz

in your Unix flavored shell. The procedure creates the gracos-examples directory which contains all the example scripts presented in this reference. For example, the hello.sh script presented in Running gracos is included as gracos-examples/hello.sh; you may execute it any time following the installation procedure.

The user may become confused by the large number of files automatically generated as the examples are ran; type gracos-examples -r to clean the gracos-examples directory and return to the initial pristine state.

2.1.3 Command Line Options

The full list of command line options is provided in Running gracos.

If you use a batch system for job submission we recommend using option -f. Job submission systems spool extra information into their standard output file, making it difficult to extract the standard output of process zero from the standard output of the job, the latter produced only at the end of the run also. Do not use this option for interactive sessions as you will not be able to see the prompt.

Regarding the option -t, note that one of the previously released versions of LAM MPI are known to show an erroneous behavior on standard input: the interactive session would terminate as soon as it is started without prompting the user. Even though the bug was corrected in the subsequent versions of LAM, it may be a good idea to always use a non-interactive session where possible (copy your input to a file and pass the using the -i option to gracos).

Option -v controls the verbosity associated with the tverb internal gracos bitwise variable. Use this option to set "on" the memory allocation verbosity when debugging memory leaks. Using -v-1 sets verbosity to the maximum (all bits are "on" for -1 in its bitwise representation).

2.1.4 Commands

gracos interacts with the user via its own extendable command line-based language

The complete list of gracos commands is given in gracos Commands.

gracos uses the Tcl (Tool Command Language) as the embedded interpreter; which means that any command processed within the gracos scripting session is parsed and executed by the Tcl (version 8.4) interpreter. This interpreter is extended within gracos to include new gracos-specific commands and procedures. Tcl is currently chosen as the interpreter due to its ease of use, portability, light weight, and the open source license. Typing tclsh in your Unix-type prompt brings you to the default Tcl scripting session (in the same way as typing python for example brings you to the Python scripting session). We refer the users who are new to Tcl to the online Tcl reference, Tcl manual and Tcl (version 8.5) tutorial. The book Tcl and Tk Toolkit 1994, by John K. Ousterhout is also recommended for its coverage of embedding; however it covers a much earlier version of Tcl. The gracos interpreter differs from the default Tcl interpreter exactly by the presence of the specific Tcl variables, procedures defined in the init.tcl startup file, and the new commands linked to the internal gracos C-functions using the Tcl_CreateCommand function. File init.tcl is sourced each time gracos is started. This file contains definitions for custom gracos Tcl procedures and variables. Tcl commands always originate on process zero for parallel runs with MPI. As soon as a complete Tcl command is entered on process zero its copies are immediately sent to the remaining MPI processes. The identical commands are then synchronously interpreted on all the remaining MPI processes.

2.1.5 Variables

In addition to the usual Tcl variables, gracos operates with the so called gracos variables (those variables are bound to the pointers inside the C-code).

The complete list of the gracos variables is given in Reference to gracos Executable.

gracos variables are operated by varset, varunset, varputs, vars, and varchmod gracos commands, See section gracos Commands. In addition, they are automatically set or updated within the source code, at any point in the source code where such setting is rational.

gracos variables are linked to the internal source code variables so that whenever a value of a gracos variable is changed in the script the corresponding variable within the source code is also immediately changed. gracos variables are not associated with Tcl variables in any way and are referred to in this reference as simply variables. The list of all gracos variables for this version of gracos is given in gracos Variables.

Commands, varset, varunset, varputs, varchmod and vars are used to manage the gracos variables from within the gracos scripting session. C-macros varset, varget are used to manage gracos variables within the C-code. A gracos variable should be carefully distinguished from a regular Tcl variable; the easiest way to do it is by keeping in mind that the '$' character is always used for referring to a Tcl variable, but is never used for a gracos variable.

Intentionally, no new gracos variables can be introduced in the runtime. Each variable is assigned a unique key and a type which do not change in the runtime, See gracos Variables for the complete list of gracos variables with their keys and types; however, as we already mentioned, a given variable can be either set or unset, that is - its status may change in the runtime.

gracos variables provide us with an important method of error control intrinsic to the gracos scripts. In particular, any attempt to access a variable whose current status is unset results in termination with an error message, regardless of the value of the memory space associated with the variable. The termination behavior can be avoided if necessary, by setting the unvar mode of tetol bitwise variable, See Bitwise Variables.

The list of all the currently set gracos variables is displayed by typing vars within the gracos scripting session (use the vars -a for the complete list, including uninitialized variables). The first three columns in the output of this command are each one character wide. The first column says if the variable is set (' ') or unset ('?'); the second column says if the variable is currently writable (' ') or not ('r'); the third column says if the variable is an array ('A') or a scalar variable (' '). The last two columns of vars output show respectively the variable name and its value or values (for the array variables).

Command varchmod can be used to switch write protection status of a variable and is useful for such purposes, as keeping the verbosity constant while loading a datafile which may or may not contain verbosity assignments. The syntax is varchmod {+w|-w} KEY where KEY is the name of the gracos variable. The usage of other varcommands is explained in the next subsection.

In the following example, we illustrate the use of the gracos variables versus usual Tcl variables. We first display the list of the gracos variables using vars command twice: with and without the -a flag. Then we set a gracos variable nstep to a specific legal value using the varset command; then we set a Tcl variable to a different value choosing the same variable name. Then we display their values by using varputs for gracos variable and puts for Tcl variable. We observe that the variables have kept their originally assigned values and are therefore indeed distinct.

#!/bin/bash
# File: gracos-examples/interface/vars.sh

# GRACOS Session
TmpFile=`mktemp  ./tmp.XXXXXXXX`
cat > $TmpFile <<EOF

# Display the list of the currently set application variables
vars 

# Display the list of all application variables
# (notice that variable nstep is currently unset)
vars -a


# Set an application variable nstep to value 2
varset nstep 2

# Check to see that the status of the variable is indeed changed
vars 

# Display the value of the application variable now
varputs nstep


# Define a new Tcl variable with the same name (nstep)
# and set its value to 100
set nstep 100

# Show the value of Tcl variable nstep (the Tcl variable 
# dereferencing character is used)
puts \$nstep

# The application variable with this name has kept its original value
varputs nstep

# Application variables are entirely separate from the usual Tcl variables
# even if they are assigned identical  names

EOF
mpirun -np 4 gracos -i $TmpFile
rm $TmpFile

2.1.6 Bitwise Variables

Bitwise variables are technically just regular gracos 4 byte integer variables; they are used to control miscellaneous bitwise ("yes" or "no"s) modes. The varset and varputs commands applicable to the gracos variables in general can be used to manage values of bitwise variables as well. Each bitwise variables can currently be assigned no more than 32 modes.

Type bitvar -vars to see the list of all bitwise variables; bitvars is a Tcl procedure defined in init.tcl, See section Commands.

In bitwise string representation bitwise variable is shown as a string of zeros and ones.

The in integer representation of bitwise variables is more compact than bitwise string representation and is convenient in certain situations. Setting a bitwise variable to -1 enables all the modes because all bits of the -1 integer are "yes". Setting the integer to zero disables all the modes at once. It is not convenient however to operate with integer values if one wants to set some specific bit of a bitwise variable.

Token representation uses tokens to specify values of each mode. The user conveniently operates with tokens, combining them with the OR ('|'), AND ('&') and NOT ('~') bitwise operators. Section gracos Variables lists all tokens with brief description for each bitwise variable.

The bitvar procedure defined in init.tcl can be used to manipulate bitwise variables in their token representations. The highly commented example below illustrates different ways of manipulating bitwise variables on the example of tverb bitwise variable.

#!/bin/bash
# File: gracos-examples/interface/bitvars.sh

# GRACOS Session
TmpFile=`mktemp  ./tmp.XXXXXXXX`
cat > $TmpFile <<EOF

# DISPLAY all bitwise variables
bitvar -vars

# tverb is a bitwise variable controlling the
# gracos technical verbosity modes.

# DISPLAY variable tverb
varputs tverb

# Store its current value
set storedvalue [varputs tverb]

# DISPLAY its value as a string of '0's and '1's
eval [bitvar -bits tverb]

# DISPLAY the its complete list of modes
eval [bitvar -toks tverb]

# We allow unconventional order of the first two flags
eval [bitvar tverb -toks]

# DISPLAY its token representation
eval [bitvar -see tverb]

# TURN ON the allocation (alc) and MPI use (mpi) modes
eval [bitvar -add tverb {alc | mpi}]

# TURN OFF both allocation and MPI use verbosities
eval [bitvar -del tverb {alc | mpi}]

# Set verbosity to allocation (alc) and MPI use (mpi) only
eval [bitvar -set tverb {alc}]

# Revert to the previously stored value
varset tverb \$storedvalue

puts "bye!"

EOF
mpirun -np 4 gracos -i $TmpFile
rm $TmpFile

2.1.7 Runtime Administration

Runtime administration technique is the perfect tool for doing things on the fly (as the code runs).

The user requests runtime administration by creating specifically named runtime administration file(s) within the working directory either before or during the run. The runtime administration files are read by gracos as soon as the control reaches any of the runtime administration checkpoints within the source code (functions admin_checkpoint)) with a unique label admin_label passed as an argument, See Runtime Administration for their complete list.

Runtime administration can be invoked explicitly, using the admin command. There are a few kinds of runtime administration described in the following text.

There are two important points to keep in mind

• Avoid partial updates of the runtime administration files If you edit runtime administration file during the run, casual saving an incomplete version of file may lead to errors if the checkpoint is reached at the time.
• Avoid possibility of infinite recursion loop. If your runtime administration method invokes one of the commands that themselves contain a runtime administration checkpoint the infinite recursion loop is avoided by restricting the scope of administration to particular checkpoints; See Example: Tabulated Transfer Function for example.

The runtime administration files do not need to have the executable file permissions.

The following is the list of all runtime administration checkpoint ids.

integ

Invoked each timestep of the particle integration run, See section integ: Running an N-body simulation.

integ_pre

Invoked at the beginning in the integ command, just before the main timestepping loop of the integration of particle trajectories starts.

image

Invoked just after the image data is projected to form a numerical matrix-plate but before it is converted to produce a `gif' file, during the image command, See section image: Particle Data Imaging. This administration checkpoint can be used to tune the dynamic range covered by the image to the brightness of the dimmest and the brightest spots on the image plate.

admin

Invoked when command admin is executed in the interpreter.

env

Invoked when command env is executed in the interpreter.

2.1.7.1 One Time Administration

File inc.tcl, if found at the runtime administration checkpoint, is interpreted as an input gracos script and is then moved to inc.tcl~, unless the new admin_keep variable is set to 1 while processing that script.

Tcl variable admin_label, temporarily defined as the label of the current administration checkpoint, may be used to restrict the scope of one time administration to a particular administration checkpoint in order to avoid infinite recursion. An example of one time administration with such restricted scope of administration is presented in Example: Tabulated Transfer Function.

As the most simple example, try executing the following bash shell command within the working directory during any of the GRACOS runs to see the run stop:

SHELL$ echo end > inc.tcl

The newly created file inc.tcl contains just a single gracos command end to request an end of the gracos session as soon as possible. One should not worry about infinite recursion in this example because, as we know from Runtime Administration the command end does not itself contain another runtime administration checkpoints.

2.1.7.2 Repeatable Administration

The gracos variable admin_path, if set, defines the name of the repeatable administration file, which gets processed at each administration checkpoint. If the repeatable administration file is defined but does not exist the code shall terminate with the error message. By contrast with one time administration file, the path of the repeatable administration file is not fixed and the file is not removed after getting executed, thus getting processed each administration checkpoint for as long as it is defined and the run is continued.

The content of the repeatable administration file is processed according to its filename extension.

A file whose extension is .tcl is interpreted in the same way as described in Section One Time Administration.

A file whose extension is .py is processed under Python interpreter. The details of the intrinsic procedure are as follows. First an automatically generated header Python script is processed in which a number of variables are defined. Type

gracos -e env

to see what this Pythons header roughly looks like (the real header will differ only by the runtime generated values in the assignments). First, all the GRACOS variables are set under their original GRACOS names (floating point numbers may lose significant bits in this assignment); those GRACOS variables that are unset are initialized to the Pythons None. Next, an unrestricted choice of variables are defined by convenience, according to their setting within the source code. The admin_label is also defined and can be used to restrict the scope of the runtime administration to a particular checkpoint. The content of the repeatable administration file is then processed in the same Python session. Should the user define the admin_out Python variable, the content of this variable is sourced as an input Tcl script within the GRACOS scripting session.

Note that Python interpreter is invoked by using the C-code system(...) call, which is known to have portability issues on some rare systems (the Infiniband networking systems is the only such system, as far as we currently know). You must have configured GRACOS without --disable-system-calls configure option for the Python runtime administration to work; See Sections Installation Procedure, and ibtest: System Call Portability Test for more details.

2.1.7.3 Scheduled Administration

Scheduled administration method is useful for scheduling certain tasks to be executed at pre-specified time.

File `sched.txt', if present in the working directory, must be the text file containing the integer (of may be floating point number) representing the number of seconds since 1970-01-01 00:00:00 UTC at which to schedule executing a command: the standard of output of C-command time(NULL) or shell command date +%s.

The command, which is the content of variable cmd_sched, will be executed as soon as control reaches one of the scheduled administration checkpoints and time exceeds the number written in file `sched.txt'. The content of the variable is subject to the same substitution rules as those applicable to the output control variables described in Checkpoint Control Variables.

Below is the example of scheduled administration

#!/bin/bash
# File: gracos-examples/problems/sched.sh

input_file=`gracos-config info pkgdatadir`/ParticleData/p3m-0.8_`uname -m`.dat

# Setup scheduled administration in 5 seconds from now
sched_sec=5
echo `date +%s`+$sched_sec | bc > sched.txt

# GRACOS Session
TmpFile=`mktemp ./tmp.XXXXXXXX`
cat > $TmpFile <<EOF

# Load particle data from the input file
dataman -mode=load -fmtid=p3mdat -masseq -path=$input_file

# Specify restart output
varset out_restart "dataman @MODE@ -fmtid=udf -spec=nfio:1 -path=out-@LABEL@.udf-"

# Define scheduled administration command: 
# "checkpoint using the restart output and exit"
varset cmd_sched "checkpoint --labels=r; end"

# Run integration with the target expansion factor '1' for restart
# output.  Note: due to scheduled administration it will be interrupted
# in 5 seconds rather than actually finishing at this expansion factor.
integ -aouts=1.:r
EOF
mpirun -np 4 gracos -i $TmpFile
rm $TmpFile

2.1.8 Primary Output Files

A run with gracos executable produces a number of primary output files within the working directory. These files contain useful information about the run and the used executable.

• File gracos_history: The list of commands submitted to gracos interpreter so far. This file is is created on process zero and is updated during the run as soon as a new command starts being interpreted. If you look at this file during the runtime, the line at the bottom indicates the currently processed command or procedure.
• Files so-Rank: The standard output of process whose MPI rank is Rank (an integer). The files are created, one for each process, and are updated as the run progresses. The amount of data written into these files depends on the settings of the verbosities, controlled by the bitwise variables tverb and sverb See section Bitwise Variables.
• File gracos_info: Structured document containing the version- and run-specific information. This file is updated only a few times during the run and is created on process zero only.

  The file is structured as an XML document, and is thus easily parseable by any of the standard XML parsers, such as xml.dom in Python.

  The first XML node configure contains the configuration and version information on the executable. The second node run contains information specific to each particular run: the time at which run was started in two formats (time and date), the number of MPI processes associated with the run (nproc), and the process specific information for each process (processes): the rank, the hostname of the hosting node as the current process, the process ID on hosting node, and the multiplicity - the total number of processes assigned to the same hosting node as the current process. The greater than one value of the multiplicity indicates that the resources of the node are shared with the other processes associated with the same run.

  Shown below is the example of this file, when created by the run on four processes in the hello.sh example in Running gracos

  <configure> <property key="PACKAGE" value="gracos"/> <property key="PACKAGE_VERSION" value="1.0.1a10"/> <property key="CONFIG_CFLAGS" value="-Wall -O3 -fno-strict-aliasing"/> <property key="CONFIG_CPPFLAGS" value="-I/home/shirokov/local/arch/i686/tcl8.4.14/include -I/home/shirokov/local/arch/i686/sfftw-2.1.5/include "/> <property key="CONFIG_LDFLAGS" value="-L/home/shirokov/local/arch/i686/tcl8.4.14/lib -L/home/shirokov/local/arch/i686/sfftw-2.1.5/lib "/> <property key="CONFIG_DATECFG" value="Sat Apr 26 14:12:36 EDT 2008"/> <property key="CONFIG_USERCFG" value="shirokov"/> <property key="CONFIG_ABS_TOP_SRCDIR" value="/home/shirokov/gracos-1.0.1a10-use"/> </configure> <run status="initial"> <property nproc="4"/> <property time="1209233667"/> <property date="Sat Apr 26 14:14:27 2008"/> <processes> <process rank="0" hostname="procion" pid="26747" multiplicity="4"> <process rank="1" hostname="procion" pid="26748" multiplicity="4"> <process rank="2" hostname="procion" pid="26749" multiplicity="4"> <process rank="3" hostname="procion" pid="26750" multiplicity="4"> </processes> </run> <run status="final"> <property runtime="0"> </run>

2.1.9 Error Behavior

If gracos halts before having reached the return statement of main C-function it does so abnormally. These failures may be due to a variety of reasons. Excepting the batch job resource limit and

Internal test failures

Internal test failures always trigger a call to one of the C-macros stop or stopm("ERROR MESSAGE") defined in zero.h. The macros produce message(s) similar to these

Exit (gracos.c:278) rank=rank or simply ERROR MESSAGE Exit (gracos.c:278) rank=rank

in both the standard output and error outputs of the process(es) where the internal test failure(s) occurred. These messages show the location of error in the source code and the rank of the process that triggered the error message. The ERROR MESSAGE, if present, usually provides some hints on the nature of the problem; e.g. that a particular variable is used but not initialized.

Warnings

Rare warnings produce messages similar to those above but do not result in program termination.

Memory leaks

A memory leak always indicates the presence of a bug, and resolving it is a very high priority. In GRACOS all allocated memory is carefully tracked to ensure against memory leaks. If it does occur, the following message appears at the end of standard output and error outputs of process zero:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! WARNING: memory leak on at least one process: cntsum=4 memsum=20

if you observe a memory leak message please report it to gracos authors.

2.2 File Input and Output with dataman

GRACOS supports a number of formats for primary (e.g. suitable for a whole restart of the simulation) file input-output; they all are implemented with the dataman command managing high level file input-output. In Supported Data Formats we list and all necessary details for the available data formats supported by dataman.

While input/output the data is automatically converted when needed from the the units particular to the datafile format into the units used by gracos, See section Physical Units. Sometimes, as in case considered in Section Santa Barbara Cluster Project such unit conversion requires same gracos variables to be set; if so, not having them set will result in an error message indicating which of the required parameters are missing.

2.2.1 Options for dataman

Below is the full description of the options.

dataman OPTIONS....

Load or save the data from or into a datafile(s).

The following table lists the available OPTIONS.

-mode=string

Mandatory flag used to specify the required action. The possible values of the string argument are save,load, or delete, indicating file output, input or deletion.

-fmtid=string

This is a mandatory flag. The data formats used for file I/O are identified by their Format ID, the argument of -fmtid specifies the Format ID (See below for the list of all the supported formats). In the absence of this option, the code tries to determine the format using the extension of the path given by the -path flag argument. If both methods fail the run stops with an error message, since the machine does not know which format to use for file I/O.

-path=string

This is a mandatory flag; its argument provides the longest path prefix that is a prefix of all the paths used for I/O in the current instance of file input-output. The final file pathname will be appended when necessary: for example, in parallel I/O the path is appended with MPI ID of the process performing the I/O; in case when many file paths are used on one process a proper extension used to distinguish the path is added; See also the -suf and -prefixes options.

-suf=string

Optional flag, used to append a string to the end of the file path.

-prefixes=string

When local disks are cross-mounted, this option can be used to modify the leading parts of -path. The argument of -prefixes is a text file, readable on process zero, and is the list of the prefixes in order of their process ids. The leading portion of the -path argument is appended individually for each process whose process id is rank with the content of the row number rank in this file. If the file has less than rank rows the path is reset to NULL for this process. The file should end with a newline.

-spec=string

The argument provides the format specification string and is used in conjunction with the format ids for which such format specification is applicable; see the description for each format below for documentation.

-masseq

If masses of all particles are the same it is not efficient to store them in a datafile. If masses of particles are not stored in a datafile used for particle data input, this option should be used to initialize them to the same value.

-sort

Sort particles with respect to their particle ids before writing them to disk. This option is applicable only for data output (-fmtid=save) using serial data file formats and is silently ignored when used in all other cases.

2.2.2 Supported Data Formats

A data file format can be static or dynamic. The static datafile format allows only a predefined number and type of the simulation parameters be stored; while the dynamic format allows more freedom. In addition, a data file format can be distributed and serial. In the distributed File I/O format, more than one process are allowed to write data on disks. For the serial datafile format the whole file I/O procedure is always performed on one process only. In the parallel runs the processes exchange messages with the process using MPI.

2.2.2.1 binpart: The Simplest Format

In this dynamic serial format (binpart signifies the Format ID) the particle data are stored in a single file in the most simple possible way, controlled by the user. No header is used, and the data is read or written in a simple sequence of records, one for each particle in the simulation volume.

The user must provide the definition for the layout of data in the record using the format specification option -spec to the dataman command. The syntax for the -spec argument string is a comma-separated list of tokens given below

x, y , or z

The x,y, or z- components of the comoving particles coordinates, in Megaparsec.

vx, vy, or vz

The x,y, or z- components of the proper peculiar velocity, in km/s.

vxc, vyc, or vzc

The x,y, or z- components of the comoving peculiar velocity, in km/s.

m

The mass of the particle in the solar masses.

id

The particle id, currently a 4 byte integer.

xs, ys, zs

vxs, vys, or vzs

ms

gxs, gys, or gzs

The coordinates, velocities, masses and gravitational accelerations, in the GRACOS code units, See section Physical Units.

If the id specifier is not supplied on input, the particle ids are assigned sequentially in order of their appearance in the file, starting from 1.

The binpart format is already used widely due to its simplicity. For example The Santa Barbara Cluster project has adopted a variation of this format to publish their particle data. Section Santa Barbara Cluster Project shows how, using the dataman to load the particle data in their format directly into gracos and start N-body simulation in the same session.

This format does not have any built-in header for holding any variables. Some additional parameters are generally required if one uses this format for input, e.g. for the domain decomposition. One has to manually supply the missing information with varset before the dataman command can be called successfully. If a required variable is unset the run terminates with an error message indicating the name of the required variable. The npartall variable is set automatically based on the input file size and the format specification.

2.2.2.2 udf: Unified Datafile Format

This dynamic distributed format (Format ID: udf) is the perfect choice for easy simulation restart and runtime backups automatically stores all the necessary gracos variables and data, being also very fast due to the usage of local disk space (some networking systems however do not have local disk spaces). The udf format is however somewhat frequently changed between different gracos versions and is hard to be interpreted by other software. However, if the data is written in this format one can always restart the simulation and convert the data to other easily interpretable formats such as binpart, See section binpart: The Simplest Format.

The udf format is customizable and self-describing. It allows the option for the arbitrary number of the output files not exceeding the number of running MPI processes. Each process writes at most one output file. This format is portable between the machines of the same CPU-endian types. The datafiles are currently designed to store all the code variables, particle data, and the refinement numbers of the chaining mesh cell.

The -spec argument is used for file output only. The argument of -spec option is a string: a comma-separated list of tokens with their subarguments, following the below syntax

eq

Request equal sized output files. The cost is slightly slower completion due to the necessary interprocessor communications. This option is useful for use in clusters with local disc spaces on each node; it ensures that the disc space gets used up uniformly.

nfio:int32

Integer (the default value is Nproc); option results in the specified number of the output files. If option -eq is given, these files will be equal sized. The actual data is not moved while their copies are moved between the processes in the failure tolerant way so as to accommodate the specified number of the output files. This option should be used if one wants to reduce the number of output files without compressing the data to the bottom processes. if nfio is less than nshr (below) then the option -eq is automatically set. If nshr is less than nfio then nfio=nshr is set automatically.

nshr:int32

Integer, less than or equal to Nproc; the option allows user the ability to later restart the simulation on fewer number of processes than the running number of processes (Nproc). Repartitioning of domain decomposition is performed so that the whole domain gets divided evenly (weighed by the workload per cell) in the result between the nshr (as opposed to Nproc as it normally is) bottom processes before data is saved into the datafiles. The simulation can be restarted in a separate gracos session using a number of processes (that is equal or higher than nshr). Do not use this option if you do not plan to restart the same simulation on fewer number of processes.

2.2.2.3 tpm: Tree-Particle-Mesh format

The static serial data format (Format ID: tpm) adopted by the TPM code version 1.2, See http://www.astro.princeton.edu/~bode/TPM is composed of 6 binary files with the same basename and extensions `.px', `.py', `.pz' and `.vx', `.vy', `.vz'. They contain identical information in the header and the corresponding component of position or velocities of the particles in their main bodies.

2.2.2.4 p3mdat: Fortran p3m Datafile Format

The static serial format (Format ID: p3mdat) used by the p3m serial Fortran 77 codes. The data is saved onto one file consisting of the header and the main body. The number of particles that can be saved is limited in the format to the maximum numbed held by an integer (INT_MAX as defined in the standard `limits.h' C-header). The error message will appear and the run will stop if this limit is exceeded.

Particle IDs are not stored. However one may use particle sorting procedure to store particle IDs implicitly. Pass -sort option to dataman and particles will be written into the disk in the order of increasing particle IDs. While loading from a p3mdat datafile, particle ids are automatically set to the sequential number (starting from one) in which the particle appears in the datafile. In this way particle IDs are easily recovered.

The masses of particles are also not directly stored. The -masseq option must be used to initialize the masses to the equal value. The p3mdat format is supported by the Serial Fortran 77 codes that we are using for tests and verification against the older p3m2_93 codes.

The following table lists the data items stored in a p3m.dat header in the order of their appearance in case.

long unsigned

Fortran 77 conventional mark for the start of a binary File I/O record.

npartall

Same as npartall from gracos Variables, but here constrained to 32 bits. The number of particles can not exceed INT_MAX in this format.

nx, ny, nz

See section gracos Variables.

dx, epsilon

Expressed in the comoving Megaparsecs.

a, omegam, omegav, h0, dtsin, etat, and nstep

See section gracos Variables.

ekin, egrav, and egint

Same as in Section gracos Variables, but expressed in the physical units; the conversion factor is given in Section Physical Units.

long unsigned

End of the Fortran 77 record.

2.2.2.5 mwhite: Martin White format

This format has at some point been adopted by Martin White for his codes. Its original description follows

The phase space data files have to be in a specific format. First there is a 4-byte integer, which should be 1. Then an integer 20. Then a header which contains: number of DM particles, number of SPH particles (0 in our case), number of star particles (0), scale-factor of the output, spline softening length in units of the box size. The first 3 entries are 4 byte ints, the last 2 are 4 byte floats. If you know a Plummer softening, the spline softening is a factor 2.8 larger. Then all of the particle positions, 3x 4-byte floats per particle, in units of the box length. So all coordinates should run [0,1) and the order is x0,y0,z0,x1,y1,z1, etc. After the positions are the velocities vx0,vy0,vz0,vx1,vy1,vz1,... These are stored in units of the Hubble velocity across the box. If you have them in km/s then divide by a.H.Lbox. Finally the particle ID number for each particle, as a 4-byte integer. And that's it.

This format allows the following specification with -spec option, whose argument should be a comma-separated list of tokens given in table below

nfh:int32

Allows one to use nfh files instead of just one (the default); the files are still written/read on process zero. Files written with some value of the argument of nfh should be read with the same argument to nfh.

2.2.3 Physical Units

The particle data are normally expressed within gracos source code in so called gracos code units: the particle positions are comoving, in the units of dx; the velocities are proper, in the units of (h0*dx/a), See section gracos Variables for the definitions of h0, dx, and a. The units of mass are chosen so that the total mass of all the particles in the simulation box is normalized to equal the number of PM density/force mesh grid points nx*ny*nz. For example, each particle has a unit weight in code units, in the case of one particle per each PM-mesh cell and all identical particles.

2.2.4 Examples

2.2.4.1 Datafile Manipulations

N-body problems generally require storing many kinds of data on a disk for various purposes such as to be able to restart the simulation from a midpoint, or perform data analysis on particles in a separate run. New datafile formats are routinely devised in the N-body community in order to fit particular problems.

gracos offers a file interface that supports a few data formats, See section File Input and Output with dataman. The interface is implemented through the dataman command, allowing the user to load, save, or delete the data stored in datafiles, or make a conversion between the supported formats. The list of the supported file formats will be updated based on the need.

Most of the examples in this reference use dataman in one way or another. Any of the supported datafile formats can be used, for example, to start a full N-body simulation with gracos in the same session. The example in Santa Barbara Cluster Project shows how to load data files using the format adopted by the Santa Barbara Cluster Project.

In the densely commented example below we perform a few conversion between various supported formats listed in File Input and Output with dataman.

#/bin/bash
# File: gracos-examples/interface/fio.sh

# Terminate shell session on any errors
set -e

# Specify the input datafile
InputFile=`gracos-config info pkgdatadir`
InputFile=${InputFile}/ParticleData/p3m-0.8_`uname -m`.dat


# Running gracos on 8 processes
TmpFile=`mktemp  ./tmp.XXXXXXXX`
cat > $TmpFile <<EOF

# Load data from a p3m.dat file, assign equal masses to all 
# the loaded particles. 
dataman -mode=load -fmtid=p3mdat \
 -masseq \
 -path=$InputFile

# Compute acceleration to find cell workloads
accel

# Save data in three files with possibility to restart 
# on four or more processes.
dataman -mode=save -fmtid=udf \
 -spec=nshr:4,nfio:3 \
 -path=a.udf-

EOF
mpirun -np 8 gracos  -i $TmpFile


# Restarting gracos on the smaller number (4) of processes
cat > $TmpFile <<EOF

# Read the output of the previous run
dataman -mode=load \
 -fmtid=udf \
 -masseq \
 -path=a.udf-

# Save into four files of equal size
dataman -mode=save -fmtid=udf \
 -spec=eq \
 -path=b.udf-

# Save particle data into one file in the most simple binary 
# format possible: the sequence of binary numbers representing the 
# particle positions (x,y,z), velovities (vx,vy,vz), the mass (m)
# and the particle id (id) for each particle in the simulation volume.
dataman -mode=save -fmtid=binpart \
 -spec=x,y,z,vx,vy,vz,m,id \
 -path=t1.bin

#
# Save particle data similarly to the above case, now using 
# other choice for units of coordinates and velocities;
# and the produced sequence of particles in the file is now 
# sorted with particle ids.
#
dataman -mode=save -fmtid=binpart \
 -spec=xs,ys,zs,vxs,vys,vzs,id \
 -sort \
 -path=t2.bin

# Again, a similar procedure
dataman -mode=save -fmtid=binpart \
 -spec=x,y,z,vx,vy,vz \
 -path=t3.bin

EOF
mpirun -np 4 gracos -i $TmpFile



# Restart gracos on a larger number (10) of processes 
# and read the output produced by the previous run.
cat > $TmpFile <<EOF

# Set the necessary missing parameters
varset epsilon 0.1
varset nx 32
varset ny 32
varset nz 32
varset dx 1.0
varset h0 50
varset a 0.8
varset omegam 1.0

# Load particle data from the file written in the above
# gracos session.
dataman -mode=load -fmtid=binpart \
 -spec=x,y,z,vx,vy,vz -masseq \
 -path=t3.bin

# Save particles
dataman -fmtid=binpart -suf=.bin \
 -spec=xs,ys,zs,vxs,vys,vzs,ms \
 -mode=save  -path=a-0 


vars

EOF
mpirun -np 10 gracos -i $TmpFile
rm $TmpFile

2.2.4.2 Santa Barbara Cluster Project

In our next example, we start the simulation with the Santa Barbara Cluster Project (SBCP) initial conditions. These particle data initial conditions must first be downloaded and unachieved using the following Unix-flavored shell commands

SHELL$ wget http://t8web.lanl.gov/cosmic/ic_sb128.tar
SHELL$ tar xvf ic_sb128.tar
particles_ic_sb128
input_sb

The data in the unpacked data files is written using the following format, as quoted from the SBCP webpage:

… There are two files in the archives: particles_name, which has the following content: x[Mpc], v_x[km/s], y[Mpc], v_y[km/s], z[Mpc], v_z[km/s], particle mass[M_sol], particle tag. This file is written in single precision and binary format. … All data are given in comoving units …

Note that the velocities are expressed in comoving km/s, which is unusual.

In order to load the datafiles into gracos, we use the above description to set the -spec flag appropriately in the example below, See section binpart: The Simplest Format.

The example below is a script that should be executed in the same directory as the above download procedure. The important thing to notice here is that the particle datafile alone does not completely specify the simulation parameters in this case. Additional simulation parameters must be set in order for GRACOS to read the particle datafile; this is because part of the necessary information is supplied in the text file `input_sb', some more information (the initial redshift or the starting expansion factor a, etc) are provided in the SBCP web page; additionally, GRACOS uses its own units for data and domain decomposition which depend on a few more simulation parameters such as epsilon and etat. It is easy to find the list of the necessary parameters since skipping any of the settings inevitably results in a crash with the appropriate error message indicating which of the parameters is missing. Another important thing is the use of the -spec option to dataman. The format specification is made consistent with the description quoted above.

Next to loading the particle data from the SBCP initial conditions data, the script executable below generates the power spectrum, and the images of the density distribution simulation box as commented in the script. According to the setting for img_slabs code variable, four images are produced in slabs along the line of sight, See image: Particle Data Imaging for more examples and explanations on the image parameters.

We encourage the users to execute the below script (after finishing the above procedure for downloading the SBPC data) to reproduce the results.

#!/bin/bash
# File: gracos-examples/problems/sbarb.sh

# Make bash session halt on any errors
set -e
TmpFile=`mktemp tmp.XXXXXXXX`

# Files downloaded from the Santa Barbara Cluster Project webpage
SB_header=input_sb
SB_particles=particles_ic_sb128
if [ ! -f $SB_header ] || [ ! -f $SB_particles ]; then
    echo "Missing one or both files: \"$SB_header\", \"$SB_particles\""
    exit 2
fi
# Extract more simulation parameters from the header datafile.
# This defines: Lb, Om, Ov, h0
sed 's/ //g' input_sb > $TmpFile
. $TmpFile


z=63.;
Ngrid=128;
astart=$(echo "1./(1.+$z)" | bc -l)


dx=$(echo "$Lb/$Ngrid" | bc -l)


# GRACOS Session
cat > $TmpFile <<EOF

# Set the necessary parameters
varset nx $Ngrid
varset ny $Ngrid
varset nz $Ngrid
varset dx $dx
varset h0 $H0
varset a $astart
varset omegam $Omega_m

# Specify the Plummer softening distance and
# Load the Santa Barbara Cluster Project datafiles into GRACOS.
varset epsilon 0.1
dataman -mode=load \
  -fmtid=binpart \
  -spec=x,vxc,y,vyc,z,vzc,m,id \
  -path=$SB_particles


# Measure the power spectrum of density perturbations;
# display power spectrum with the "pmpower" MATLAB command.
pmpower

# Produce the images of the simulation box: 

# First, Set the image parameters "reasonably", without producing 
#        the actual image yet
image --set-defaults -no-image

# Second, Set a few selected image parameters to more preferred values
varset img_rbl 0.95
varset img_rbr 1.4
# Make four image snapshots in slabs, along the line of sight
varset img_slabs 4
varset img_pixw 500
# Increase the default img_rsmoo by a factor of 1.5
varset img_rsmoo [expr 1.5*[varputs img_rsmoo]]

# Finally, produce the actual images
image -path=sbarb.gif

EOF
mpirun -np 8 gracos -i $TmpFile
rm $TmpFile

The image below shows the power spectrum measurement for matter distribution in the particle data volume. The image data are generated as the result of the pmpower call in the above example; the data are written into file pmpower.dat and the image itself is generated by typing the same pmpower (in MATLAB) in the same directory, See section pmpower: Power Spectrum Estimation. The lower half of error bars is longer than the upper one due to the non-linearity of logarithm while we use linear values power spectra for variance estimators.

The following four images appear as the result of the image call in the above example; the number of images is defined by the img_slabs variable setting the number of image slab along the line of sight.

2.3 integ: Running an N-body simulation

The integ command starts the N-body simulation on the currently loaded particle data.

Load balancing is performed automatically each timestep, See section lbal: Load Balancing.

Forces are computed each time step. One has the freedom to choose between the long range (PM) forces only or full P3M force. This is controlled by the mode acc a bitwise variable whose default value is "full P3M acceleration", the equivalent of typing eval [bitvar -set acc { PM | PP | FPM | FPP } ]. If you need PM forces only, just type eval [bitvar -set acc PM] before running integ.

Timesteps are using the usual time stepping criterion during an N-body simulation; the positions and velocities of the particles are made synchronized at the certain points in time. The output expansion factors specified using the -aouts option will coincide with the expansion factors corresponding to the particle data synchronization points. This is achieved by truncating the last timestep in the sequence leading to the target expansion factor. The truncation can be disabled using the disable_timestep_truncation sub-option.

Variable etat must be set before calling integ. A reasonable value for this variable is 0.05; setting this variable to lower values can be used to increase time resolution of the simulation.

The following two options are available

-aouts

Mandatory option, a string containing the comma separated list of output specifiers each formatted as A[:B], where A is either the floating point number setting the output expansion factor at which to produce data output or the word INF to indicate unlimited integration of particle trajectories. Optional column separated modifier B (the default value is r) may have the following mnemonic values

r

Restart output, using the expansion of variable out_restart, See section Checkpoint Control Variables.

a

Analysis output, using the expansion of variable out_analysis, See section Checkpoint Control Variables.

b

Backup output, using the expansion of variable out_backup, See section Checkpoint Control Variables.

n

No output

For example, command integ -aout=0.1:a,1:ra indicates that the analysis output will be executed at expansion factor 0.1; both the restart and analysis outputs will be executed at expansion factor 1.

The user may define actions performed at the restart, backup and analysis outputs by setting the variables out_restart, out_backup and out_analysis, See section Checkpoint Control Variables.

-mode

Optional argument. The default value is "-mode=scheme:DKD"

scheme:string

The scheme specifies the time integration scheme used for particles. The allowed values are KDK for Kick-Drift-Kick integration scheme and DKD for Drift-Kick-Drift integration scheme; the default is DKD.

reset_econ

Reset the energy conservation measure (egrav+ekin-egint) to zero at the beginning of the run, by modifying egint. The status of energy conservation is normally something that accumulates from the initial conditions produced by the particle initial conditions generator. The use of this option leads to resetting the energy conservation history to zero, so that any accumulated value during the integration run will have resulted from energy computation occurred during the current run as opposed to the cumulative value.

disable_timestep_truncation

By default, this option is disabled. The use of this option disables timestep truncation needed to match the output expansion factors set by the -aouts flag. For each output expansion factor the data is saved at the end of the current timestep without timestep truncation. This leads to preserving the natural, non-truncated values for all time steps in the simulation at the inconvenience of having slightly higher output expansion factors than those specified by the user.

break:nsteps

This option should not be normally used. The mandatory positive integer argument nsteps of this option gives number of time steps for the simulation, after which the integration loop is exited even if the aimed expansion factors specified by the aout are not reached.

mult

Use multiple timestepping scheme (this option is not currently available).

2.3.1 Checkpoint Control Variables

The expanded values of output control variables

out_restart

out_analyze

out_backup

yield the command to be executed at a checkpoint, See section integ: Running an N-body simulation.

In addition to usual Tcl expansion, the following pattern expansion takes place

@MODE@

Expands to -mode=save, this pattern should be used in the dataman commands.

@OUTID@

In case of integ, expands to the id of the current output specifier given in -aouts flag, See section integ: Running an N-body simulation.

In all other cases expands to "outid".

@LABEL@

expands to "r", "a", "b", or "n" for restart, analysis, backup or no outputs, respectively.

2.3.2 checkpoint Command

If you wish to checkpoint at an arbitrary time, independently of the outputs specified by the -aouts argument to integ, use the checkpoint command; Its usage is as follows

checkpoint -labels=LABELS

where LABELS is a string containing any combination of "r", "a", and "b", See section Checkpoint Control Variables.

2.3.3 Automatic Backup Option

Variable nchkpt (See gracos Variables), if set, indicates the backup period, in seconds. For large integration times it may it may be necessary to leverage the system failure risk with this option. The timer is started at the beginning of a run with gracos. Once nchkpt seconds are passed, variable out_backup is expanded and its content is executed. No backups are performed if nchkpt variable is unset.

Given extremely large integration times, many instances of the backup are executed. To avoid unnecessary file accumulation, old backups are removed at each new backup output, at this point the @MODE@ pattern is expanded into -mode=delete and @OUTID@ expands to the value OUTID-1.

2.4 accel: Computing Particle Accelerations

This command is similar to integ, except for instead of computing forces and evolving particle data in position and velocity space it just computes the particle accelerations once. Accelerations can then be retrieved, for example, by using the gxs, gys, or gzs format specifiers within the binpart format, See section binpart: The Simplest Format.

2.5 lbal: Load Balancing

This command performs optimization procedure aimed at efficient (load balanced) computation of forces. Because load balancing procedure is invoked automatically each integration time step (see integ: Running an N-body simulation), one normally does not need to use this command.

Our load balancing procedure uses the workloads associated with each cell of the computation volume. While workloads are computed cumulatively during the integration procedure and one should normally not worry about them, in some cases, the workloads are unknown. This can happen, for example, if you just loaded particle data from a file that does not include workload specification for each cell. In fact, of all the formats described in Supported Data Formats, only the udf format normally stores cell workloads, in addition to other types of particle data. Therefore, if you loaded data from a non-udf format, especially, if particles are highly clustered, we advise using the lbal -empiric command before any invoking accel or integ. Formal description of this option follows.

-empiric

Setup workloads and refinement numbers for each cell using an empiric model (Figure 3-18 of A. Shirokov Thesis), before proceeding with actual load balancing.

2.6 image: Particle Data Imaging

image is a gracos command implementing the imaging of particles with various choices for the projection volume, direction and other imaging parameters. The maximum dynamic range of 256 colors is currently allowed for imaging. The color 0 in the palette corresponds to the lowest threshold value below which no contrast can be discerned from the image. Similarly, the color 255 corresponds to the highest threshold in the plotted value.

A number of code variables must be set before this command is called. One can avoid having to set them all manually by using the -set-defaults flag:

gracos% image -set-defaults

to generate a reasonable guess to all the required variables from the current data, See Section Example: Particle Data Imaging or Example: Tabulated Transfer Function for the examples of the use of this command.

The following is the complete list of the required variables

img_cmap

An integer within the range 0-4, specifying the colormap to be used for images. We found the colormaps 1 and 4 to be most useful. The colormap 4 produces black and white images.

img_dim

An integer 1,2 or 3, stands for the direction (x,y or z) of the axis used for the projection of the simulation box. The plane of the generated image will span the remaining two dimensions as shown in the following figure.

img_cx, img_cy, img_cz

Specify the center of the main image box within the simulation volume.

img_bx, img_by, img_bz

Specify the size of the main image box.

img_slabs

An integer greater than zero, setting the number of images in the output of the image command. If this number is 1 (the default), then the content of the whole main image box, whose dimensions are given by (img_cx, img_cy, img_cz), and (img_bx, img_by, img_bz) is projected into the image plane. If this number is greater than one, then the main image box is split into the same number of slabs of equal thickness and the output images are the projections for each individual slab in the sequence.

img_pixw

Specify the output image width, in pixels. The height is found automatically so as to observe isotropic scaling in all directions within the image plane.

img_rsmoo

Image smoothing length.

img_rbl, img_rbr

The two floating point variables set the contrast.

2.6.1 Example: Particle Data Imaging

In the example below, we first load the particle data from a sample datafile (supplied along with the source code distribution) using dataman, then we produce the images of the distribution of particles in the simulation box using different colormaps, projection dimensions, image sharpness and image dimensions.

#!/bin/bash
# File: gracos-examples/problems/imaging.sh

# Specify input datafile
pkgdatadir=`gracos-config info pkgdatadir`
InputFile=$pkgdatadir/ParticleData/p3m-0.8_`uname -m`.dat

# GRACOS Session
TmpFile=`mktemp ./tmp.XXXXXXXX`
cat > $TmpFile <<EOF

# Load particles from a sample datafile
dataman -mode=load -fmtid=p3mdat -masseq  -path=$InputFile

# Set image parameters "reasonably", do not actually write the image
image --set-defaults -no-image

# Produce four images with various choices on image parameters:
# Image 1: the default parameters, set by "image --set-defaults"
image -path=im1.gif

# Image 2: change the projection dimension (project along the Y-axis)
#          change to another colormap
varset img_cmap 4
varset img_dim 2
image -path=im2.gif

# Image 3: Recenter the frame to the halo at the upper right edge
#          zoom in, sharpen the resolution scale by the factor of two,
#          increase the upper brightness limit range by a factor of two.
varset img_cmap 0
varset img_cx [expr [varputs img_cx]+0.5*[varputs img_bx]]
varset img_cz [expr [varputs img_cz]-0.2*[varputs img_bz]]
varset img_bx [expr 0.3*[varputs img_bx]]
varset img_bz [expr 0.3*[varputs img_bz]]
varset img_rsmoo [expr 0.5*[varputs img_rsmoo]]
image -path=im3.gif

# Image 4: Invoke Runtime Administration
varset admin_path imaging-admin.py
image -path=im4.gif
varunset admin_path

EOF

mpirun -np 4 gracos -i $TmpFile
rm $TmpFile

The following runtime administration file must be placed into the same directory under the name imaging-admin.py, See section Runtime Administration.

# Runtime administration file for imaging.sh
# File: gracos-examples/problems/imaging-admin.py

if admin_label == "image":
    admin_out = "varset img_rbr %E" % readonly_cmlup

The four images below have been produced by running the above script. They are originated in the left-to-right, top to bottom order as files im1.gif, im2.gif, im3.gif, and im4.gif.

Note that the third image is particularly overexposed. This is because, the default value (10.) for the img_rbr code variable, giving the upper threshold for the brightness range sampled by the available colormap, is too low for this overdense halo; so that most pixels on the image fall above the threshold (this default value is set by the image --set-defaults in the above script). In order to tune the threshold to the brightness distribution of the pixels we have to use runtime administration technique described in Repeatable Administration because it is generally not possible to determine the correct value for the upper brightness threshold before the actual data is projected onto the image plate. The instance of runtime administration is invoked for the last image, at the image checkpoint in which the upper brightness threshold is set exactly to the value of the brightness threshold Out_cmlup exceeded by a low fraction (0.003) of the total number of pixels in the image. This is an example where runtime administration technique is useful.

2.7 pmpower: Power Spectrum Estimation

Power spectrum is a very frequently used statistics of density field with many analytical fits and expressions given in the literature and is directly related to two point correlation function.

Typing in gracos prompt

pmpower [--path=FILE ] [--nbins=NBINS]

yields the binned power spectrum of the currently loaded particle data written in the text file FILE (the default is 'pmpower.dat'); NBINS is the number of bins used within the range of the wavevectors of the main Brillouin zone.

The data can be visualized by typing the same command in MATLAB:

>> pmpower [(FILE)]

using the same default name for FILE.

In Santa Barbara Cluster Project and Example: Tabulated Transfer Function we show an example of thus produced image of the power spectrum.

The internal procedure for computing the power spectrum is as follows. First, particle mass is interpolated on the grid using the TSC density assignment scheme; then the Fourier transform is performed. The value of the power spectrum is estimated for each grid point in the Fourier space as proportional to the square of the Fourier transform of the density at the same gridpoint. The binned values of the power spectrum are found by averaging of these values for each wave vector bin, up to the Nyquist frequency. The rms deviation (the error bars) of the binned values are found by dividing the rms deviation of the measurements by the square root of the number of measurements for each bin (the Central Limit Theorem).

2.8 Initial Condition Generators

In general, any data simulation or data analysis task is preceded by the specification of the current state of the system. In our case, we specify the data such as the particle positions, velocities etc; followed by the parallel domain decomposition.

This section shows the examples of running a large simulation with GRACOS for various choices of the initial conditions; these examples are presented in the form of readily executable scripts. Cosmological initial conditions are currently generated using the Zeldovich approximation with four methods of specifying the initial power spectrum of density perturbations. This section lists all the currently available methods of automatically generating the initial conditions; both rectangular and glass initial particle grid can be used.

Once the initial conditions are generated: the data can be analyzed or evolved, and saved into the datafiles using one of the methods described in Section File Input and Output with dataman. Once saved into datafiles, the data can be loaded evolved and analyzed again in a separate GRACOS scripting session. In general, to start a cosmological simulation, the user is only required to provide the cosmological and other simulation parameters. GRACOS starts with those parameters in the input and finishes with the complete realization of the required data, usually the particle data at the given expansion factors.

2.8.1 grafic: the Initial Conditions Generator

This command implements an extension of the grafic1 initial conditions generator originally presented as an Fortran77 code by E. Bertschinger, See section Serial Fortran N-body Codes.

Section @ref{sec_cgf} provides us with the test scripts that should be used to verify the compatibility of the result of the gracos command with the output of original Fortran codes.

The following gracos variables must be set before this command is called

nx,ny,nz,dx

Mandatory

epsilon

Unless the option -nodom is used

omegam, omegav and h0

Unless the option -lingerdat is used

a

If this variable is set, it defines the starting expansion factor of the universe. Either set this variable or use the -sigstart option described below; error message terminates the run when attempting to use both ways.

The grafic command accepts the following options

-icase=int32

Select the type of initial power spectrum (matter transfer function T(k)). The Integer takes the following values:

0

Use tabulated values for the power spectrum. See the -table option.

1

Use the transfer command tabulated in a `linger.dat' file. See the -lingerdat option. Set the omegam, omegav and h0 code variables from the `linger.dat' file header.

2

Use the BBKS analytical approximation for the T(k).

3

Use T(k) = 1 (the scale-free case).

-pnorm=float32

Mandatory option. A floating point number, sets the desired normalization at a = 1:

for icase = 0

Mandatory option. pnorm equals -sigma_8 at the present epoch (only negative pnorm is currently allowed).

for icase = 1,2

if pnorm > 0

pnorm equals Qrms-ps/micro-K

if pnorm < 0

pnorm equals -sigma_8 at the present epoch.

for icase = 3

if pnorm > 0

pnorm equals k^3 * P(k), where P(k) is an unfiltered value, e.g. with the Gaussian or Hanning filters, of the initial power spectrum, and k = pi/dx (or k = pi/rsmoo, if the -rsmoo option (down the list) is set and rsmoo > dx).

if pnorm < 0

pnorm equals -sigma_8 at the present epoch.

-lingerdat

Provide the name of the lingerdat file (for -icase value 1 only).

-enable-baryons

GRACOS currently does not have the capability for gas N-body simulations. By using this option, however, one generates the velocities for baryonic matter, just as they would be generated with the original grafic1 code, See section Serial Fortran N-body Codes; this data, however is not used for CDM or any other particle initializations. Using this option in conjunction with -ic-vel-files allows one to save baryonic data into files and analyze it.

-offvelc=int32

-offvelb=int32

Integer parameters for offsetting baryon and CDM velocity fields by 0.5 grid spacing. The argument should be +/-1 or 0; the default value is +1, just as the default in the grafic1 code. The origin of the baryon velocity field will be offvelb*(0.5,0.5,0.5)*dx and similarly for offvelc.

-sigstart=float32

This option is mandatory unless the a code variable is set before calling grafic.

sigstart sets the initial linear density fluctuation amplitude on the ultimate sub grid scale (or the approximately starting expansion factor). You must either set this option or preset the a code variable before calling grafic.

If the power spectrum is sufficiently steep so that k^2 P(k) converges slowly at low wavenumbers (consider for example the scale-free free power spectrum with ns=-3) the internal Romberg integrator fails; use the -trlowk option to avoid this problem.

-ns=float32

Long-wave spectral index (scale-invariant is ns = 1.), a floating point number; defines the shape of the primordial power spectrum. This option is mandatory when -fnl is used or -table is used with choice:transfer_function.

-seed=uns32

Mandatory option. The random number seed, - a positive integer smaller than 2^32-5=4294967291.

-power-plot=kmin:kmax

Use this option to make a plot of the power spectrum used to generate the initial conditions. Floating point numbers kmin and kmax define the range of the wave vectors (in the units of inverse comoving Megaparsecs) covered by the plot. The text file power.dat is produced in the working directory as the initial conditions are generated. The first row in this file shows ns and pnorm; the remaining rows are

k/h, P(k)*h^3, and 4*pi*P(k)*k^3

where h equals h0/100; k is the wave vector in the units of inverse comoving Megaparsecs; and P(k) is the power spectrum in comoving Megaparsecs cubed.

In addition to the data we described each rows may contain additional data appended at the end of each line. If this is the case, the specification for the additional data is outside the scope of this reference.

-unscalable

Using this option gives the advantage that the particle realization does not depend on the number of processes used while running gracos. Thus one is able to completely replicate the initial particle data realization of the original grafic-2_101 code, See section Serial Fortran N-body Codes, @xref{sec_cgf}. The expense is somewhat slower performance because the last process has to replay the complete sequence of random numbers almost for the entire box in order to arrive to its starting position in the sequence of random numbers.

Random number generators are very fast, and we would generally recommend using this option for any runs smaller than 800^3 particles.

-ic-vel-files

Generate files ic_deltab, ic_velbx, ic_velby, ic_velby, ic_deltac, ic_velcx, ic_velcy, ic_velcy; replicating the output of the original grafic1 code. If gracos is ran in parallel, then for each of the file of the above list, a set of files file-rank are generated instead, where rank goes from 0 to Nproc-1. This option may be useful if you are familiar with the traditional grafic1 output and, perhaps, have some data analysis routines that take these file as the input. It may as well be useful as a way to extract the generated baryonic initial conditions in case you have chose option -enable-baryons. Note if you are using the -fnl or -glass extensions (See the following table): their effect does not extend to the generated ic_* files.

-disable-domain-decomposition

After having completely computed the overdensities, and velocities while calling the grafic command, gracos by default uses this information to completely initialize the particle data array and perform the parallel volume domain decomposition, sending particles between the processes; the latter procedure results in the complete setup, such that the commands pmpower or integ may be used, for example without much additional work. The domain decomposition may take a considerable amount of time for very large problem sizes, but is necessary for the followup with most of the other gracos commands. Using the -disable-domain-decomposition option will disable particle domain decomposition.

This option combined with the -ic-vel-files and -unscalable for icase>0 will just result in the very close replication of the result as computed by the serial grafic1 code, See section Serial Fortran N-body Codes.

If only the above options are used (excluding the case icase = 0) the produced result will be identical to the roundoff error to the equivalent serial run with the grafic-2_101 code, See section Serial Fortran N-body Codes. The following list adds important optimization options and a few new options compared to those provided by the original grafic-2_101 code.

-table=path:string,choice:string

This option is applicable for the case icase=0 only. Both path and choice suboptions are mandatory. path gives the name of the two column ASCII file, whose first column is the wave vector expressed in inverse comoving Megaparsecs. The interpretation of the second column depends on the other suboption. If choice equals "power_spectrum" then the second column gives the primary power spectrum to be used to setup the initial conditions; if choice instead equals "transfer_function" then the second column yields the transfer function. The normalization of the data in the second column is irrelevant as the power spectrum is normalized intrinsically.

The rows in file path can be given in an arbitrary order. However, the wavevector values given in the table should cover the complete wavevector range used in the code, since no automatic extrapolation in the wavevector space is performed. A table with a limited wavevector range should be manually appended with one or two entries to extend the wavevector range as needed at both high and low wavevector values.

-disable-hanning

Disable the use of Hanning filter, particular to the grafic1 code.

-fnl=float32

Apply non-Gaussianity correction to the generated initial conditions, fnl is the second order correction to the primordial proper potential phi . The motivation for the fnl model of initial conditions, pioneered, among others, by E. Komatsu is to explore the possibility that the initial conditions in our universe are slightly non-gaussian, in which the true initial conditions potential phi can be expanded in terms of the gaussian potential phi0 as phi = phi0 + fnl ( phi^2 - < phi0^2 > ) when expressed in the units of the speed of light squared. The -ns option to grafic must also be set.

-rsmoo=float32

Apply the gaussian filter; the power spectrum used for the generation of the initial conditions is multiplied by exp(-k^2 rsmoo^2) ; rsmoo is expressed in comoving Mpc.

-trlowk

Limit the integration to the frequencies above pi/(dx*nx), while computing power spectrum normalization to find starting expansion factor using -sigstart option; see the description above for the -sigstart option.

-glass=seed:uns32,nexp:float

Generate glass initial conditions, instead of those on the regular grid. The seed suboption is mandatory and is used to initialize the initial Poisson distribution, necessary for making the glass particle distribution. The nexp suboption is the number of expansion factors to evolve the Poisson distribution with repulsive gravity requested to arrive to the target glass distribution. The quality of the latter improves with the increase nexp; if no option is given, this number is automatically initialized to the theoretical estimate nmax^12 , where nmax equals the maximum of nx, ny and nz; even though the theoretically estimated numerical value of nexp is rather large, the nbody simulation will proceed rapidly under the repulsive gravity. One may run the power spectrum tests like those shown in Example: generating the glass Distribution in order to show that the generated glass is reliable.

2.8.1.1 Example: Tabulated Transfer Function

In this section, we show how to start an arbitrarily large simulation starting from the values of cosmological parameters and tabulation for the power spectrum P(k) of matter density perturbations; we present the readily executable gracos script below.

Let us assume that we are having a table of the power spectrum P(k): the values of P_i , tabulated at the wave numbers k_i ; where P is expressed in arbitrary units (consistent for different i ) and k_i is expressed in the units of h/(Mpc comov), where h is the present time Hubble constant, in the units of 100*km/s/Mpc. The power spectrum table should cover the entire range of the wave vectors used during the gracos run because gracos does not do any extrapolation of power spectrum in this case. The GRACOS installation provides a sample of such power spectrum table for use in this example of the gracos-examples, but you should use your own power spectrum table.

At the end of this subsection we provide the complete details on how we generated this power spectrum table using CMBFAST).

First, we define the WMAP3 cosmological parameters (we remind that the "." at the beginning of the line in bash means sourcing the path as if it were included into the script). The gracos-config executable is a part of GRACOS installation and is used to locate files provided with GRACOS installation; type

cat `gracos-config info pkgdatadir`/CosmoParams/WMAP3

in your shell prompt to see the WMAP3 parameters used.

The following line in the script defines the path to the power spectrum file that we generated using CMBFAST, also the part of the installation for the purpose of this reference; again, the user should generally use their own power spectrum table as the power spectrum used in our example is in not suitable for a general situation (as we see at the end of this section).

Next, we specify the random number seed seed used to generate the sequence of random numbers used by the initial conditions generator and the size of the particle mesh Ngrid the user is free to change this number. The Ngrid parameter is low for the purpose of this example; the user is free to change this number as needed to change the resolution of the N-body simulation (if you significantly increase this number you should also increase the number of processes for the mpirun command line prefix). Next, we arbitrarily set the simulation boxsize Lbox, expresses in comoving Megaparsec (not the Megaparsec/h!) and compute the cell spacing dx using bc (the bc is a simple calculator, usually provided with any standard Unix distribution).

Next, we set the Plummer force softening length epsilon (expressed in the units of dx) and the integration timestepping parameter etat; the 0.05 is a good value; using lower value will result in more timesteps needed to complete the simulation. Finally, we provide the comma separated list of of output specifiers with aouts (only one in our example); each formatted as aout[:modifier], where aouts is the output expansion factor and the modifier is a character specifying what to do at the given expansion factor, See integ: Running an N-body simulation for more explanations.

Now that we have defined all the simulation parameters, we begin writing the body of the gracos script into a temporary file whose name is generated with a standard mktemp Unix command.

First we set the necessary simulation parameters for gracos with a set of varset commands, See Variables, Variables and gracos Variables. The list of all the necessary parameters for this run can easily be found by the trial and error method (this is indeed how we knew which parameters are needed to be set): gracos terminates with a useful error message any time a required variable is uninitialized.

Next, we generate the particle initial conditions by calling the grafic command using the icase=0 selection, See section grafic: the Initial Conditions Generator. The arguments of this command completely specify all the additional information needed to generate the particle realization of the given power spectrum. The -power-plot flag provides the option to output the power spectrum plot in the given range of wave vectors.

Next, we call image to generate and image of the simulation box, See section image: Particle Data Imaging. The -set-defaults flag automatically sets up a default selection for all the image parameters. The result is a new GIF file `i.gif' in the current working directory.

Next, using dataman (See File Input and Output with dataman) we save the initial conditions data into file using the distributed file input-output, See section udf: Unified Datafile Format; call integ to start an N-body simulation, make an image using image, and call dataman again to save the data in the evolved state.

Finally, we make another image of the simulation box, producing file `f.gif'.

The entire gracos session is written into a temporary file $TmpFile, whose name is passed as the argument to the -i option to gracos; gracos is ran in parallel with mpirun command on four processes.

The user is free to increase the problem size by increasing Ngrid and the number of processes, up to the system resource limitation.

#!/bin/bash
# File: gracos-examples/problems/pktable.sh

# Define cosmlogical parameters: H0,Om,Ob,Ov,Oc,ns,sig8,taulss
. `gracos-config info pkgdatadir`/CosmoParams/WMAP3

# Produce the transfer function table using 
# the Eisenstein and Hu fitting formula
table=table.dat
gracos-ehu \
 --hubble=`echo "$H0/100." | bc -l` \
 --krange=1.E-5:300:100. \
 --omega_baryon=$Ob \
 --omega_lambda=$Ov \
 --omega_matter=$Om \
 --redshift=0 \
 > $table
# Extend the wavevector range covered by the table manually
# with one k_low, T(k_low) pair, borrowing the value of the transfer
# function from the first entry of the already existing table
# (since T=const at low k).
table_ext=table-ext.dat
awk 'NR == 1 {print 1.0E-50, $2}' $table > $table_ext
cat $table >> $table_ext


# Set simulation Parameters for GRACOS
# Random number seed 
seed=123456789
# Number of particles and PM-grid points per one side
Ngrid=32
# Simulation boxsize, comoving Megaparsec
Lbox=200.
# Grid spacing
dx=$(echo "$Lbox/$Ngrid" | bc -l)

# Integration timestep parameter
etat=0.05
# Plummer force softening length (in units of dx)
epsilon=0.1

# GRACOS Session
TmpFile=`mktemp ./tmp.XXXXXXXX`
cat > $TmpFile <<EOF

varset epsilon $epsilon 

varset nx $Ngrid
varset ny $Ngrid
varset nz $Ngrid
varset omegam $Om 
varset omegav $Ov 
varset h0 $H0
varset dx $dx
 
grafic -icase=0 \
 -table=path:$table_ext,choice:transfer_function \
 -offvelc=1 -ns=$ns \
 -sigstart=0.2  \
 -pnorm=-$sig8 -seed=$seed \
 -power-plot=1E-5:10

# Measure the power spectrum from the box.
pmpower -nbins=100

# Make an image
image --set-defaults -path=ini.gif

# Integration parameter
varset etat $etat

# Measure the initial conditions power spectrum
pmpower -path=pmpower-ini.dat

# Optionally pass extra commands from users SHELL environment
$gracos_extra_commands

# Saving data using the Unified Data Format
dataman  -mode=save -fmtid=udf -path=ini.udf-

# Start the N-body simulation
# The -aouts argument is a comma-separated list of output specifiers
integ -aouts=1.:n

# Make an image
image --set-defaults -path=fin.gif

# Measure the evolved state power spectrum
pmpower -path=pmpower-fin.dat

# Again, saving data using the Unified Data Format
dataman -mode=save -fmtid=udf -path=fin.udf-
EOF
mpirun -np 4 gracos -i $TmpFile
rm $TmpFile

Imaging

Shown below are the images in files ini.gif and fin.gif (the latter is at redshift zero) produced by the image command just before and after the particles are evolved with the integ GRACOS command as shown in the script above. If we makes a sequence of these images as we evolve the box we can actually observe that the large scale structure is smoothly transformed from that of shown on the left panel to that on the right (please See Runtime Administration for the available options on executing arbitrary commands, such as image for making images, in the runtime).

An Example of Run Time Administration

It is generally possible to make the images on the fly, using the technique described in See section Runtime Administration. Here we demonstrate an example of this. By copying the example below into file inc.tcl within the working directory during the run, we execute the imaging command (see the example) as soon as the runtime control reaches the integ checkpoint; the current images are then created and this file is automatically erased. We do not show the image generated; because they apparently depend on the exact time at which the inc.tcl file is created. You should try this copy command while running the above script.

# Author: Alexander V. Shirokov
# File: gracos-examples/problems/admin-image.tcl

# File: gracos-examples/problems/admin-image.tcl

if { $admin_label eq "integ" } {
    # Make the images as soon as the 'integ' checkpoint 
    # is reached
    # This file is deleted because 'admin_keep' is unset.
    image -set-defaults
} else {
    # Keep this file in case of other checkpoints
    set admin_keep 1
}

The Power Spectrum

Now, typing

pmpower('pmpower-fin.dat')
pmpower('pmpower-ini.dat')
icpower

in MATLAB in the same directory yields the plot below of the power spectrum of density perturbations as a function of the wave number, spanning the range of [2 pi /(nx * dx) ,pi/dx)

as shown by the two vertical dash-dotted green lines.

The red line at the bottom is the power spectrum used for the generation of particle initial conditions. The oscillations of this curve above the Nyquist frequency are caused by the use of the Hanning filter and should not worry the reader, as they are outside the wavevector range used. Passing the -disable-hanning option to grafic disables the Hanning filter; the black dashed line shows the unfiltered power spectrum.

The lower broken blue line is the measurement of the power spectrum based on the unevolved particle positions (See the first pmpower call in the GRACOS script above). The deviations from the red curve are due to the following effects: 1) Cosmic variance at low wavenumbers; 2) the effect of the randomness of the current particle realization given the power spectrum; 3) coarse graining near the Nyquist frequency.

The upper blue broken line is the power spectrum of matter perturbations in the evolved state (the second pmpower call) and the black dash-dotted line shows the initial power spectrum given by the red line corrected multiplied by the linear growth rate characteristic for the expansion from the starting expansion factor to redshift zero, the final point of our simulation. The systematic deviation of the black dash dotted line with the upper blue broken line is due to the nonlinear evolution.

2.8.1.2 Example: lingers Initial conditions

In this section we show how to start an arbitrarily large simulation of CDM with gracos directly, starting from the values of the cosmological parameters using the included lingers package, a subpackage by C. Ma, E. Bertschinger, 1995 included into GRACOS, See section Serial Fortran N-body Codes. lingers is automatically installed during the GRACOS Package Installation. The transfer function tabulated in both redshift and k-space is included into the linger.dat file automatically generated using lingers.

As we know, the baryonic, CDM and matter transfer functions agree to within the percent level close to z=0. For the power spectrum normalization we use sigma8 of the power spectrum of all matter linearly evolved to z=0. sigma8 is measured as an integral with respect to k over the power spectrum at a given a. Usually it is assumed that the power spectrum grows as D+ . With lingers however this dependence is intrinsic and is not factored out.

If we plot the power spectrum at a=0.02 (normalized at z=0), then we go back by the intrinsic growth factor and by the D+ in the two cases. In the result we get a good agreement between the curves. The ehu curve goes right through the middle of the matter power spectrum oscillations. But the CDM particle displacements are generated with the CDM power spectrum, not matter. The CDM and MATTER power spectra, while almost matching exactly each other at z=0, are very far apart at a=0.02 (about 12% or dlog10 = 0.05).

Now, if we assume that Om=Oc+Ob, we imply that the power spectra of Oc and Ob exactly match each other at a=0.02. The correct thing to do would be to generate the displacements using the matter power spectrum at astart.

The evolution factor of CDM is different than that for MATTER.

#!/bin/bash
# File: gracos-examples/problems/lingers.sh

# Define cosmlogical parameters: H0,Om,Ob,Ov,Oc,ns,sig8,taulss
. `gracos-config info pkgdatadir`/CosmoParams/WMAP3

# Run lingers from the included grafic1 package to generate transfer 
# function and store it in 'linger.dat' file.
tcmb=2.726; yhe=0.24
kmin=1.E-5; kmax=10.; nk=121
zend=0.; case=1
TmpFile=`mktemp ./tmp.XXXXXXXX`
cat > $TmpFile <<EOF
$Ob $Oc $Ov 0
$H0 $tcmb $yhe 0
$kmin $kmax $nk $zend
$case
EOF
cat $TmpFile | lingers 

# Set the simulation Parameters for GRACOS
# Random number seed 
seed=123456789
# Number of particles and PM-grid points per one side
Ngrid=32
# Simulation boxsize, comoving Megaparsec
Lbox=200.
# Grid spacing
dx=$(echo "$Lbox/$Ngrid" | bc -l)

# Integration timesteping parameter
etat=0.05
# Plummer force softening length (in units of dx)
epsilon=0.1

# GRACOS Session
cat > $TmpFile <<EOF

varset epsilon $epsilon
varset dx $dx
varset nx $Ngrid
varset ny $Ngrid
varset nz $Ngrid

# Generate Initial Conditions with grafic
grafic -icase=1 \
 -offvelc=1 \
 -sigstart=0.2  -ns=$ns -lingerdat=linger.dat \
 -pnorm=-$sig8 -seed=$seed \
 -power-plot=$kmin:$kmax \
 $grafic_flags_extras

# Measure the power spectrum; display with pmpower MATLAB command.
pmpower

# Make an image
image --set-defaults -path=ini.gif

# Optionally pass extra commands from users SHELL environment
$gracos_extra_commands

# Integration parameter
varset etat $etat

# Saving data using the Unified Data Format
dataman  -mode=save -fmtid=udf -path=ini.udf-

# Start the N-body simulation
# The -aouts argument is a comma-separated list of output specifiers
integ -aouts=1.:n

# Make an image
image --set-defaults -path=fin.gif

# Again, saving data using the Unified Data Format
dataman -mode=save -fmtid=udf -path=fin.udf-
EOF
mpirun -np 8 gracos -i $TmpFile
rm $TmpFile

Typing icpower and pmpower now, in MATLAB, within the working directory yields the power spectrum plot. The resulting plot is distinguishable from the plot in the previous section, by the presence of the baryonic oscillations.

2.8.1.3 Example: BBKS Initial Conditions

An analytical approximation to the initial power spectrum by Bardeen, Bond, Kaiser and Szalay named thus the BBKS approximation is directly applicable in gracos.

File gracos-examples/problems/bbks.sh In this section we show how to start an arbitrarily large cosmological N-body simulation starting with values for a few cosmological and simulation parameters, and assuming that the initial power spectrum is described by the BBKS approximation.

The user should be familiar with the content of the script shown below from the discussion in the two previous sections. The script below does not make calls to executables rather than gracos: using the BBKS approximation is reserved for icase=2:

Typing icpower in MATLAB within the working directory yields the power spectrum plot below. As we see by comparison with the power spectrum plot generated with CMBFAST and lingers, the BBKS approximation is slightly underestimates power for lower wave vectors, while slightly exaggerating for larger k's.

2.8.1.4 Example: Scale-Free Initial Conditions

Sometimes it is useful to start a simulation using scale-free initial conditions with P(k) proportional to k^ns.

This section shows how, given a few required parameters, to start an arbitrarily large simulation with scale-free power spectrum of matter perturbations. The icase=3 option to grafic is dedicated to scale free initial conditions. The script below is very similar to the one in the previous section, which, similarly to the present case, uses an analytical expression for the initial power spectrum of CDM density perturbations.

#!/bin/bash
# File: gracos-examples/problems/scalefree.sh

# Define cosmlogical parameters: H0,Om,Ob,Ov,Oc,ns,sig8,taulss
. `gracos-config info pkgdatadir`/CosmoParams/WMAP3

# Set simulation Parameters for GRACOS
# Random number seed 
seed=123456789
# Number of particles and PM-grid points per one side
Ngrid=32
# Simulation boxsize, comoving Megaparsec
Lbox=200.
# Grid spacing
dx=$(echo "$Lbox/$Ngrid" | bc -l)

# Integration timestep parameter
etat=0.05
# Plummer force softening length (in units of dx)
epsilon=0.1

# GRACOS Session
TmpFile=`mktemp ./tmp.XXXXXXXX`
cat > $TmpFile <<EOF

varset epsilon $epsilon

varset nx $Ngrid
varset ny $Ngrid
varset nz $Ngrid
varset omegam $Om 
varset omegav $Ov 
varset h0 $H0
varset dx $dx

grafic -icase=3 \
 -seed=$seed \
 -offvelc=1 \
 -sigstart=0.2 -ns=$ns -pnorm=-$sig8 \
 -power-plot=1.E-5:10 \
 $grafic_flags_extras


# Make an image
image --set-defaults -path=ini.gif

# Integration parameter
varset etat $etat

# Saving data using the Unified Data Format
dataman  -mode=save -fmtid=udf -path=ini.udf-

# Start the N-body simulation
# The -aouts argument is a comma-separated list of output specifiers
integ -aouts=1.:n

# Make an image
image --set-defaults -path=fin.gif

# Again, saving data using the Unified Data Format
dataman -mode=save -fmtid=udf -path=fin.udf-
EOF
mpirun -np 4 gracos -i $TmpFile
rm $TmpFile

2.8.2 poisson: Poisson Distribution

This routine may be a good starting point if you want to write your own initial conditions generator that can not be expressed as an extension of grafic.

This command generates N Poisson-distributed particles across the box. The syntax is

poisson -seed=seed

where seed is a random number seed.

Variables nx, ny and nz must be set. If npartall code variable is unset, then the total number of generated particles along with the npartall code variable are set to nx*ny*nz; otherwise N equals npartall.

The internal procedure is as follows. The volume domain decomposition is performed so that approximately equal volumes are assigned to each process. The Poisson random number generator is called on each process with exactly the same random number seed so that each process scans the identical sequence of randomly distributed particles. If a particle belongs to a given process, that process assigns that particle to itself; the entire sequence of particles is thus assigned to all Nproc processes.

The Poisson random number generation is relatively fast. However, the procedure described above, is not scalable. In principle, by generating particles with varying random number seed on each process, one can make this procedure scalable: each process would try to assign each particle in the sequence by a certain mapping from the uniform distribution within simulation volume into the irregular volume particular to each process. The Jacobian of the mapping must equal to one; we are not certain at this point however on how to setup such a mapping, suggestions are welcome however. Hence, we are currently employing the above non-scalable procedure which is safe against any tiling effects.

2.8.2.1 Example: Poisson Distribution

In the example below we generate the power spectrum of the Poisson particle distribution (See poisson: Poisson Distribution) and compare it with the analytic slope (zero).

#!/bin/bash
# File: gracos-examples/tests/powersp/poisson.sh

Ngrid=128

# GRACOS Session
TmpFile=`mktemp ./tmp.XXXXXXXX`
cat > $TmpFile <<EOF

# Gridsize and particle number
varset nz $Ngrid
varset ny $Ngrid
varset nx $Ngrid

# Needed for domain decomposition 
varset epsilon 0.1

varset dx 1.0

# Generate Poisson particle distribution
poisson -seed=123456789 

# Measure the power spectrum. Use pmpower MATLAB command to visualize.
pmpower

dataman -mode=save -fmtid=binpart -spec=xs,ys,zs -path=poisson.bin

EOF

mpirun -np 4 gracos -i $TmpFile
rm $TmpFile

Typing

pmpower
yline( log10(1/(2*pi)^3) )

in MATLAB yields the image below. The theoretical power spectrum of the Poisson distribution is the constant 1/(2*pi)^3, shown with the horizontal slash-dotted line; this prediction is mostly observed in the plot generated. The large error bars on large scales (low wavevectors) are due to cosmic variance or low statistical sample of these modes; while at the very high wavenumbers there is a characteristic upturn, whose nature is not entirely obvious. The plot shows the bounds of the validity of the power spectrum generated with pmpower.

2.8.2.2 Example: generating the glass Distribution

This command generates N Glass-distributed particles across the box. The syntax is

glass -seed=seed -nexp=nexp

where seed is a random number seed and nexp is the number of expansion factors requested to evolve glass. Further in this Section Example: generating the glass Distribution we provide an example of using this command.

In GRACOS, the procedure for making glass distribution is defined as follows: first we distribute particles uniformly across the box, using the Poisson distribution; then we evolve the particles with repulsive gravity. By measuring the power spectrum, we find that the power spectrum, initially dominated by the shot noise, settles down to form the single power law power spectrum characteristic for glass P(k) .

Glass distribution for particles, introduced by S. White, is similar to the distribution of molecules in a liquid, (we know from the statistical mechanics background that the glass material is liquid). GRACOS provides the utility for generating glass distribution and glass initial conditions, See Example: generating the glass Distribution for more details.

As a good test of GRACOS one can run a simulation to produce the glass structure and measure the power spectrum during the inverted gravity run. Shown below, as usual is the GRACOS script that does the job. This is also, the first script in this reference that makes use of the runtime administration technique to measure the power spectrum on the fly (See further below for more details).

#!/bin/bash
# File: gracos-examples/tests/powersp/glass/generate.sh

TmpFile=`mktemp ./tmp.XXXXXXXX`
cat > $TmpFile <<EOF

# Set simulation parameters
varset nz 32
varset ny 32
varset nx 32
varset omegam 1.0
varset omegav 0.0
varset dx 1.
varset a 0.00001
varset epsilon 0.1
varset etat 0.05

# Request a repeated runtime administration 
# to measure power spectrum on the fly.
varset admin_path generate-admin.py

# Generate glass particle distribution by expanding the 
# uniform poisson initial particle distribution with inverse gravity
# for the 1.E15 expansion factors.
glass -seed=123456789 -nexp=1.E15

# Invoke an additional instance of runtime administration explicitly
admin

# Save the data into one file in the xs,ys,zs binary format
dataman -mode=save -fmtid=binpart -spec=xs,ys,zs -path=glass.bin
EOF
mpirun -np 4 gracos -i $TmpFile
rm $TmpFile

The above script as is will just generate the assumed glass particle distribution. But in order to test whether a reliable glass distribution is indeed generated we must measure the power spectrum and compare it with the power spectrum of glass distribution which is shown theoretically to be a fourth power low of the wave number k. We can just place the pmpower command at the end of the above script. But it would be useful to see how the power spectrum really evolves as we evolve particles with repulsive gravity for this purpose we make use of the repeated runtime administration method described in Section Repeatable Administration. Shown below is the Python runtime administration file that should be placed into the same directory as the script above. The runtime administration script conditionally invokes a set of GRACOS commands to execute in a GRACOS subsession. The script mostly consists of the useful comments but is essentially just a few lines long.

#!/usr/bin/python
# File: gracos-examples/tests/powersp/glass/generate-admin.py

# Runtime administration file to measure the power spectrum
# at the specified timesteps and on any explicit administration call.

# First, restrict the administration to the specified checkpoints

if admin_label == 'integ_pre' or admin_label == 'integ' or admin_label == 'admin':

    # The condition to measure the power spectrum:
    # Specify the timesteps at which to measure the power spectrum
    # 1. Measure the power spectrum at the specified timesteps
    # 2. Measure the power spectrum each 'admin' administration checkpoint
    # independently  of the timestep

    specific_nsteps = [0, 22, 62, 116, 149, 194]
    tag = None
    # Check to see if the current administration checkpoint is labeled 'admin'
    if admin_label == 'admin':
        tag = "fin"
    # Check to see if the current timestep is one of the specified in the list
    elif nstep in specific_nsteps:
        tag = "%d" % nstep

    # Conditionally measure the power spectrum. The admin_out variable,
    # when set in this script, yields the GRACOS script to execute.
    if tag:
        admin_out = [
            '# ::::::: Instance of Runtime Administration at timestep %d"' % nstep
            , 'pmpower --path=glass-%s.dat' % ( tag )
            , '# :::::: Runtime administration finished'
            ]

Using the Pythons flexibility it is quite easy to make any other selection in the above script, say, for example, to make power spectrum measurements each specified number of timesteps, make images of the particle distribution, save the data into files, or any other operation allowed by the set of the GRACOS commands. One can modify the content of this file during the run to change the administration mode as needed. Using runtime administration thus yields a very powerful method of control.

Now, if you run the bash shell script at the beginning of this section, and issue the following commands in MATLAB (in the same directory)

>> pmpower('glass-0.dat')
>> pmpower('glass-22.dat')
>> pmpower('glass-62.dat')
>> pmpower('glass-116.dat')
>> pmpower('glass-149.dat')
>> pmpower('glass-194.dat')
>> pmpower('glass-fin.dat')

you will see the following plot of the power spectra at the timesteps specified in the runtime administration script.

The upper curve on the plot just shows the white noise of the initial Poisson distribution (glass generation starts with uniformly distributed random Poisson particle distribution, See Example: generating the glass Distribution for more details). The white noise curve is in agreement with the theoretically predicted zero white noise power slope. As we evolve matter with the repulsive gravity to form glass distribution the matter becomes less and less chaotic as particles move under the repulsive force from each other; the power spectrum thus settles down on all scales. Since the theoretical slope of the glass distribution is approximately the fourth power of the wave vector, the modes with the highest wave number hit their theoretical glass power spectrum curve, in agreement with the behavior shown, where for the timestep 62, for example, the small scale perturbations are smoothly distributed along the theoretical curve and the largest scale perturbations are still settling down. Finally, as we evolve matter for 15 powers of the expansion factor, the matter settles down on all scales sampled by the simulation resolution. We have thus reached the accurate distribution on all scales.

2.8.3 Glass Extension

This option is under development; please, email GRACOS authors if you have any specific suggestions.

2.9 halo_finder: Halo Finder

This command is currently under development; email GRACOS authors to indicate your interest in this command or any suggestions you may have.

2.10 Tables of Reference

This section provides the tables of all the available gracos commands, variables and modes. A short description (a docstring) is supplied for each entry; the detailed descriptions may be found in more specialized sections of this reference.

2.10.1 gracos Commands

The following table lists all the available gracos commands and options. The internal Tcl procedures and the procedures sourced in the gracos startup file init.tcl (See Commands) are not listed here.

help

Display Reference Tables in XML format

Location: gracos.c:324

ARGUMENTS

-o string

end

Request an end to the scripting session as soon as possible.

Location: cdcom.c:409

inc

Process commands given in the file.

Location: cdcom.c:391

ARGUMENTS

-path string

admin

Invoke an instance of runtime administration now

Location: gracos.c:82

ARGUMENTS

-i string

env

cvars , Print values of environment variables

Location: cdcom.c:507

ARGUMENTS

-xml int32

-o string

varset

Set a given variable to a given value.

Location: cdvar.c:423

varunset

Unset a variable

Location: cdvar.c:354

varputs

Print the value of a specified variable

Location: cdvar.c:570

vars

Show current values for a code variable

Location: cdvar.c:610

ARGUMENTS

-a int32

-v string

varchmod

Set write permission for a given variable.

Location: cdvar.c:395

dataman

Manage data and datafiles

Location: dataman.c:204

ARGUMENTS

-prefixes string

-suf string

-mode string

-fmtid string

-masseq int32

-path string

-spec string

-sort int32

grafic

Generate particle cosmological initial conditions

Location: grafic.c:234

ARGUMENTS

-lingerdat string

-ns float32

-ic-vel-files int32

-unscalable int32

-glass string

-sigstart float32

-power-plot string

-trlowk int32

-disable-hanning int32

-pnorm float32

-end int32

-rsmoo float32

-seed long unsigned

-enable-baryons int32

-icase int32

-table string

-fnl float32

-offvelc int32

-offvelb int32

glass

Generate glass particle distribution

Location: glass.cc:107

ARGUMENTS

-nexp float32

-seed long unsigned

poisson

Generate Poisson particle distribution

Location: poisson.cc:154

ARGUMENTS

-seed long unsigned

integ

Start an N-body simulation

Location: integ.c:323

ARGUMENTS

-aouts string

-mode string

checkpoint

Checkpoint your simulation

Location: misc.cc:273

ARGUMENTS

-labels string

accel

Compute particle accelerations

Location: singular.c:130

ARGUMENTS

-setrfm string

image

Visualization

Location: wimage.c:276

ARGUMENTS

-path string

-no-image int32

-set-defaults int32

pmpower

Density power spectrum estimation using, PM interpolation.

Location: pmpower.c:244

ARGUMENTS

-path string

-nbins int32

lbal

Repartitioning: Calculate current loads and adjust partitions.

Location: lbal.c:718

ARGUMENTS

-empiric int32

hc_partit_set

Set partitions to the given state.

Location: repartit.cc:1369

hc_shrink

Shrink particle data to the bottom processes

Location: repartit.cc:1508

ARGUMENTS

-np int32

hc_adj

Move the top partition of the specified process by a specified number of cells

Location: repartit.cc:1194

hc_adj_shift

Shift all the partitions by the given number calls along the SFC.

Location: repartit.cc:1403

hc_loadplot

Output ASCII data files with information on current workload distribution.

Location: lbal_loads.cc:727

compare_particles_serial

Analyze particle data in data files

Location: gtool.c:49

ARGUMENTS

-load1 string

-load2 string

-include string

sam_selectbox

Select all particles within the given box. By marking them with negative ids.

Location: sampling.c:102

ARGUMENTS

-xc float32

-yc float32

-b float32

-zc float32

sam_unselectall

Unselect particles by setting their ids to be positive.

Location: sampling.c:126

sam_traj

Update an ASCII datafile the trajectories of the selected particles

Location: sampling.c:233

an

Given the particle id, show the process it resides on and its x,m,v,g.

Location: gtoolp.cc:101

ARGUMENTS

-id int32

an_idsort

Sort particles by their ids, placing them contiguously and destroying the HC structure.

Location: gtoolp.cc:148

an_compare

Compare in parallel, id sorted sets of particles in two datafiles at given paths

Location: gtoolp.cc:527

ARGUMENTS

-path2 string

-path1 string

euler_planes

Output some eulerian space analysis.

Location: sampling.c:492

ARGUMENTS

-what string

-id int32

2.10.2 gracos Variables

The following table lists of all the gracos variables for version 1.0.1a10. The description of bitwise variables also lists the available modes.

version

Version number, when last configured

Format: string

Initialization value: 1.0.1a10

nproc

The number of processes

Format: int32

Initialized to the current number of MPI processes

tverb

Global verbosity

Format: int32

Initialization value: 8451

(which is, in token representation: prm | eval | sci | dat)

Variable "tverb" is bitwise, the following table lists all the allowed modes.

prm

Show Prompt

eval

Show all Tcl commands being processed

progr

Show progress updates in the position and potentially lengthy routines

timers

Timer operations

mpi

Show where MPI Commands start and finish

fftw

Show where FFTW starts and finishes

alc

Memory allocation and release

paths

Show file I/O paths

sci

Show some science data

pars

Parsing results for options and suboptions

var

Variable management

mk

Selected data on all processes

mk1

Extension of mk (large, use with caution)

dat

Technical data in a human readable form

dat1

Extension of dat: arbitrary form of output

dat2

Extension of dat1 for more verbose outputs

dump

Debug data dumps (large, use with caution)

tmode

Global mode

Format: int32

Initialization value: 1

(which is, in token representation: warn)

Variable "tmode" is bitwise, the following table lists all the allowed modes.

warn

Show warning messages

deb2

Run heavy error checking routines along the way

wraps

Enable misc. (e.g. MPI and FFTW) wrappers

tetol

Global error tolerance mode

Format: int32

Initialization value: 0

Variable "tetol" is bitwise, the following table lists all the allowed modes.

var

Ignore any uses of a variable with unknown name

fun

Ignore calls unknown function calls

sys

Ignore system calls that return error

smode

gracos-specific mode

Format: int32

Initialization value: 0

Variable "smode" is bitwise, the following table lists all the allowed modes.

thorough

Do some extra interesting science computations along the way

speedup

Speedup by disabling some inexpensive error checks

fftw_work

Enable the FFTW work buffer allocation while computing FFT's..

origmode

Mode to immitate the Fortran codes

Format: int32

Initialization value: 0

Variable "origmode" is bitwise, the following table lists all the allowed modes.

old_tschm

Do not use new scheme for timesteps

old_pmtab

Do not use new pmtables

old_spl

Do not use splines for short range forces

old_crmax

Do not use new method for crmax

old_rfmt

Do not select refinement number based on timing

old_lintab

Do not use linear table at inner radii

only_dkd

Halt, if trying to use not the original (DKD, single timestepping) integration scheme.

orig_filter

Do not use A.V.S. Hanning filter use correction.

lowk_bug

Do not apply low k interpolation bugfix in linger.dat input.

lingers_norm

Do not fix lingers expansion factor normalization bug

grafic1

Do not apply various grafic1 fixes

admin_path

The default name of the script to process to invoke subinterpreter.

Format: string

Not initialized

h0

Present value of the Hubble constant, expressed in (km/s)/Mpc

Format: float32

Not initialized

a

Universe expansion factor

Format: float32

Not initialized

omegam

Universe matter density parameter (CDM + baryonic + cold neutrinos)

Format: float32

Not initialized

omegav

Universe vacum energy density parameter

Format: float32

Not initialized

dx

The density and force mesh cell spacing, expressed in comoving Megaparsecs.

Format: float32

Not initialized

rmax

Maximum separation for short range force correction in a pair, in the units of dx.

Format: float32

Not initialized

nx

Simulation volume size along the X-dimension, in PM-cells

Format: int32

Not initialized

ny

Simulation volume size along the Y-dimension, in PM-cells

Format: int32

Not initialized

nz

Simulation volume size along the Z-dimension, in PM-cells

Format: int32

Not initialized

fm_cf

FPM Cf-parameter

Format: float32

Initialization value: 15

fm_lmax

FPM maximum alias number

Format: int32

Initialization value: 0

pm_eta

PM kernel size, in code units

Format: float32

Initialization value: 3.29999995

pm_lmax

Parameter lmax in PM summation

Format: int32

Initialization value: 2

epsilon

Plummer force softening length.

Format: float32

Not initialized

ntab

The number of elements (separation ranges) in force table

Format: int32

Initialization value: 20000

pm_work

Set to non-zero value to enable *work array allocation for FFTW

Format: int32

Initialization value: 1

acc

Acceleration mode

Format: int32

Initialization value: 15

(which is, in token representation: PM | PP | FPM | FPP)

Variable "acc" is bitwise, the following table lists all the allowed modes.

PM

Enable PM force

PP

Enable PP force

FPM

Enable FPM force

FPP

Enable FPP force

nstep

The current timestep number

Format: int32

Not initialized

etat

Integration timestep parameter

Format: float32

Not initialized

nchkpt

Checkpoint each nchkpt seconds

Format: int32

Not initialized

dt_sing

The value of timestep suitable for singular timestepping scheme

Format: float32

Not initialized

egrav

Potential Energy, at the last synchronization point

Format: float32

Not initialized

egint

The integral term in the Lazer-Irvine equation

Format: float32

Not initialized

ekin

Kinetic energy, at the last synchronization point

Format: float32

Not initialized

out_restart

output flags for restart output

Format: string

Not initialized

out_backup

output flags for backup output

Format: string

Not initialized

out_analyze

output flags for analysis output

Format: string

Not initialized

cmd_sched

Scheduled administration command

Format: string

Not initialized

ufp

Particle type

Format: string

Initialization value: xvi

ufc

Saved (udf) members of chaining mesh structured.

Format: string

Initialization value: r

pa_incr

Extra margin allocation parameter

Format: float32

Initialization value: 1.005

npartall

The total number of particles in the simulation.

Format: long long unsigned

Not initialized

nploc

Local Number of particles

Format: unsigned

Not initialized

hc_rawb

The index of the first cell in this process domain

Format: long long unsigned

Not initialized

hc_rawn

The number of space filling curve cells within this process

Format: long long unsigned

Not initialized

lbal_tol

Load imbalance tolerance to initiate repartitioning

Format: float32

Initialization value: 0

lbal_fudge

Workload estimator parameter

Format: float32

Initialization value: 0.699999988

img_cmap

Image colormap, integer within the range 0-4

Format: int32

Not initialized

img_dim

Projection dimension, integer ranged 1-3

Format: int32

Not initialized

img_slabs

Image the specified number of slices along the line of sight

Format: int32

Not initialized

img_pixw

Image width, in pixels

Format: int32

Not initialized

img_cx

X-coordinate of the center of image

Format: float32

Not initialized

img_cy

Y-coordinate of the center of image

Format: float32

Not initialized

img_cz

Z-coordinate of the center of image

Format: float32

Not initialized

img_bx

Size of the whole image box in X-dimension

Format: float32

Not initialized

img_by

Size of the whole image box in Y-dimension

Format: float32

Not initialized

img_bz

Size of the whole image box in Z-dimension

Format: float32

Not initialized

img_rsmoo

Image smoothing length

Format: float32

Not initialized

img_rbl

Black color threshold

Format: float32

Not initialized

img_rbr

White color threshold

Format: float32

Not initialized

testi

Redundant variable for passing test integer values (should not be used normally)

Format: int32

Not initialized

testf

Redundant variable for passing floating point test values (should not be used normally)

Format: float32

Not initialized

This document was generated by Alexander Shirokov on April, 26 2008 using texi2html 1.76.