David Waterman 2012-03-26

Introduction
============

CCP4Dispatchers is a project currently under development to provide
automatically generated Python dispatchers to wrap the executables distributed
with the CCP4 Suite.

The motivation for providing this is to lift the requirement for explicitly
sourcing a shell environment set up file (such as ccp4.setup-sh or
ccp4.setup-csh) on Linux and Mac systems, and for centrally set environment
variables on Windows. Instead, the correct environment is set by the dispatcher
just prior to program execution.  In this way the environment is encapsulated,
avoiding problems caused by clashes of incompatible environment definitions
with non-CCP4 software. In addition, this approach requires less maintenance,
as generation and use of the dispatchers works in the same way on all systems
with a compatible Python interpreter.

A generated dispatcher can be used from the command line as a drop-in
replacement for the executable it wraps, with no change to syntax.
Alternatively, the package CCP4Dispatchers can be imported within Python and
used to run CCP4 software as subprocesses. This provides a simple mechanism for
Python scripting of tasks using CCP4 programs.

Dispatcher generation
=====================

Generating dispatchers is a two step process. First, an environment definition
file must be created, which contains the variable names and their values
defined, with one line per definition in one of the following formats:

VARIABLE=VALUE
export VARIABLE=VALUE
setenv VARIABLE VALUE

The reason for choosing these formats is that the environment definition file can
then double up as a script that may be sourced on systems with an
appropriate shell to obtain the same environment set by the dispatchers. Any
lines that do not adhere to this format are ignored.

The example file 'ccp4-env_win.txt' is provided as a template that may be
edited to obtain the correct environment definition file for a Windows
installation of the Suite.

To make the process of determining the correct environment definition easy for
systems with a Bourne-compatible shell, the Python program envExtractor may be
used. This program writes out a definition file based on the result of sourcing
an existing ccp4.setup-sh script in a sanitised environment by /bin/sh.

Once an environment definition file has been created, called for example
'ccp4-env.sh', it may be passed to the dispatcherGenerator, along with the path
to the directory containing CCP4 executables, to generate the CCP4Dispatchers
Python package. By default, that package will be created under the current
directory. In addition, symbolic links (or .bat files, on Windows) pointing to
the dispatchers will be written. These have the same names as the programs
wrapped by the dispatchers. This allows use of the dispatchers to be
transparent; by adding the location of the links to PATH the dispatchers can
then be used from the command line exactly as if the programs were being used
directly.

Further usage instructions for envExtractor and dispatcherGenerator are printed
when they are run at the command line without further arguments. This is useful
when, for example, it is desired to write the CCP4Dispatchers package elsewhere
than the current working directory, or to separate the directory of Python
modules from the symbolic links (or .bat files).

Using CCP4Dispatchers within Python
===================================

The CCP4Dispatcher directory created by dispatcherGenerator is a Python
package, and the individual modules that set up Dispatcher classes for each
wrapped program may be not only run from the command line, but also imported
and used from an existing Python process. For this to work, the CCP4Dispatcher
directory must be on the Python search path. A typical way to achieve that is
to add its parent directory to the PYTHONPATH environment variable.

Then the following lines of Python code demonstrate instantiation and use of a Dispatcher
object:

from string import Template
import CCP4Dispatchers.refmac5
d = refmac5.Dispatcher()
cmd = Template("HKLIN $CEXAM/rnase/rnase18.mtz " + \
               "HKLOUT $CCP4_SCR/rnase_simple_out.mtz " + \
               "XYZIN $CEXAM/rnase/rnase.pdb " + \
               "XYZOUT $CCP4_SCR/rnase_simple_out.pdb")
cmd = cmd.substitute(os.environ)
d.set_cmd_args(cmd)
d.set_keywords("""LABIN FP=FNAT SIGFP=SIGFNAT FREE=FreeR_flag
NCYC 10
END
""")
d.call()
for line in d.stdout_data: print line
sys.exit(d.call_val)

In this simple code snippet the refmac5 dispatcher module was imported, which
set the CCP4 environment automatically, and a Dispatcher for refmac5 was
instantiated. It is possible to re-set the CCP4 environment any time later if
needed using d.set_env().

The command line string cmd for d was then set by calling the Dispatcher method
set_cmd_args(). Here we used the convenient Python Template strings to
substitute the values for CCP4 environment variables CEXAM and CCP4_SCR from
os.environ.

The keyword records, which are passed to the STDIN of refmac5, are set using
the method set_keywords(). This method accepts either a single string (as
shown) or a list of strings, with each element a separate line of input.

The call() method handles dispatch to the refmac5 binary, passing the command
line arguments and keywords, and returns the exit code once the process
completes. As a side-effect, the attributes stdout_data and stderr_data of d
are set, containing the STDOUT and STDERR output from the process respectively.

In some cases it is necessary to have more control over the running
sub-process. For those instances there is a different interface provided by the
call() method. If

d.call(wait=False)

is used, then control is returned immediately to the main Python program rather
than blocking on the call() expression. After this, each time the Dispatcher
method monitor() is called a tuple is returned containing a single line of
output from both STDOUT and STDERR of the process (or None is none is
available). As a side-effect monitor also fills d.stdout_data and d.stderr_data
so that all of the lines can inspected after the program completes.

With this scheme it is incumbent upon the client code to call monitor() enough
times that all of the output of the program is read. One way to ensure this is
to repeatedly call monitor() while the attribute d.isRunning is True. This is
safe because this attribute is set to False by monitor() only when the sub-
process has completed execution AND there are no more lines of output left to
read from either STDOUT or STDERR. The basic structure of a monitoring loop is
simply

while d.isRunning:
    d.monitor()

As this approach allows live progress monitoring of a running program, an
additional method, abort(), is provided to cleanly stop a job if there is no
reason to let it run to completion.

Some example scripts using the dispatchers to run demonstration jobs are
provided in the 'example_scripts' subdirectory.