English | Deutsch

How does file staging work?

When the submit host and the execution host do not have access to a common, shared file system or are simply using different root directories, the necessary job data must be copied (via rcp or a similar utility) to the execution host, and the job results must be copied back to the submit host.

Because such a file staging mechanism must be very flexible and fully configurable, it is desirable to let the actual copying be handled by the system itself, such as by a script written by the administrator, so that one can focus on just the necessary data. Because the actual work of copying is delegated to the system, i.e. the script, the term "delegated file staging" is used. With delegated file staging in the Grid Engine 6.0 release two scripts are needed. The first is called in the prolog and copies the input data to the selected execution host. The second is called in the epilog and copies the results to the submit host.

What do the delegated file staging scripts do?

The scripts must, at a minimum, copy the stdin, stdout, and stderr files either from the submit host to the execution host or vice versa.

The additional needed information is obtained by the script from command-line parameters, which can be set in the cluster configuration using the "qconf -mconf" command. Under "prolog", after the path to the prolog script(s), the following variables can be given to be replaced automatically by the DRM with the appropriate, corresponding values when the prolog is called:

$fs_stdin_file_staging

The values of these variables at the time the prolog is called will either be "1" or "0", according to whether file staging is enabled for that particular data type.

$fs_stdout_file_staging
$fs_stderr_file_staging
$fs_stdin_host

The values of these variables at the time the prolog is called will be the names of the hosts from which the data is to be copied.

$fs_stdout_host
$fs_stderr_host
$fs_stdin_path

The values of these variables at the time the prolog is called will be the path (relative to the respective hosts) to the data to be copied.

$fs_stdout_path
$fs_stderr_path
$fs_stdin_tmp_path

The values of these variables at the time the prolog is called will be the path to where the data is to be copied.

$fs_stdout_tmp_path
$fs_stderr_tmp_path

A typical call to a prolog script, i.e. what would appear in "qconf -mconf" would look like:

and a typical call to an epilog script would look like:

The following must then appear in the prolog script:

and similarly in the epilog script:

And then it works?

Almost. The administrator must add the following entry to the cluster configuration:

Now delegated file staging is available for use.

And how does one use the thing?

From this point forward, we are talking specifically about using delegated file staging with DRMAA. For more information on DRMAA, see the DRMAA howto.

Because drmaa_transfer_files is an optional attribute in DRMAA, the first thing an application must do is use drmaa_get_attribute_names() to determine if the attribute is supported by the DRMAA implementation in use. The DRMAA implementation provided with Grid Engine 6.0 only supports the drmaa_transfer_files attribute when delegated_file_staging true is enabled in the cluster configuration. (See the previous section.)

If the drmaa_transfer_files attribute is available, the application can set this attribute's value to one of following: "i", "o", "e", or a concatinated combination of the three, such as "oi". Each letter, if present in the attribute's value indicated that the corresponding data source should be copied using delegated file tranferal.

Where does one find more information?

Example scripts are available:

A general review of file staging can be found in the general file staging howto.