Grid Engine Configuration Recipes

Set shell_start_mode to unix_behavior in your queue configurations to get the normally-expected behaviour of starting job scripts with the shell specified in the initial #! line.

A side-effect of unix_behaviour is usually not getting the normal login environment, specifically not with the module command available for those sites that use environment modules. At least for use with the bash shell, add the following to the site sge_request file to avoid users having to source modules.sh etc. in job scripts, assuming sessions from which jobs are submitted have modules available:

This may not work with other shells.

Suppose you have various sets of compute nodes over which you want to run parallel jobs, but each job must be confined to a specific set. Possible reasons are that you have

Then you’ll want to define multiple parallel environments and host groups. There will typically be one PE and one host group (with possibly an ACL) for each node type or island. The PEs will all be the same, unless you want a different fixed allocation_rule for each, but with different names. The names need to be chosen so that you can conveniently use wildcard specifications for them. Normally the names will all have the same name base, e.g. mpi-…​. As an example, for different architectures, with different numbers of cores which get exclusive use for any tightly integrated MPI:

with corresponding host groups @quadcore and @hexcore for each type of dual-processor box. Those PEs are assigned one-to-one to host groups, ensuring that jobs can’t run across the groups (since parallel jobs are always granted a unique PE by the scheduler, whereas they can be split across queues).

Now the PE naming comes in useful, since you can submit to a wildcarded PE, -pe mpi-*, if you’re not fussy about the PE you actually get. See Wildcarded PEs for the next step.

Suppose you want to retain the possibility of running across all the PEs (assuming they’re not isolated). Then you can define an extra PE, say allmpi, which isn’t matched by the wildcard.

When you would use a wildcarded PE as above, for convenience and abstraction, you can use a JSV to write the wildcard pattern. This JSV fragment from jsv_on_verify in Bourne shell re-writes -pe mpi to -pe mpi-*:

Suppose you want to retain the possibility of running across all the PEs (assuming the groups aren’t isolated). Then you can define an extra PE, say allmpi, which isn’t re-written by the JSV.

Users sometimes transfer job scripts from MS Windows systems in binary mode, so that they end up with CRLF line endings, which typically fail, often with a rather obscure failure to execute if shell_start_mode is set to unix_behavior. The following fragment from jsv_on_verify in a shell script JSV prevents submitting such job scripts

To change scheduling so that hosts in different host groups are preferentially used in some defined order, set queue_sort_method to seqno:

and define the ordering in the relevant queue(s) as required with seq_no):

It is possible to use seqno, for instance, to schedule serial jobs preferentially to one `end' of the hosts and parallel jobs to the other `end'.

To schedule preferentially to hosts which are already running a job, as opposed to the default of roughly round robin according to the load level, change the load_formula:

assuming the slots consumable is defined on each node.

Reasons for compacting jobs onto a few nodes as possible include avoiding fragmentation (so that parallel jobs which require complete nodes have a better chance of being fitted in), or being able to power down complete unused nodes.

Since the load formula is used to determine scheduling when hosts are equal according to queue_sort_method, you can schedule to the preferred host groups by seqno as above, and still compact jobs onto the nodes using slots in the load formula, as above, i.e. with this configuration:

To avoid “starvation” of larger, higher-priority jobs by smaller, lower-priority ones (i.e. the smaller ones always run in front of the larger ones) enable resource reservation by setting max_reservation to a reasonable value (maybe around 100), and arrange that relevant jobs are submitted with -R y, e.g. using a JSV. Here is a JSV fragment suitable for client side use, to add reservation to jobs over a certain size, assuming that PE slots is the only relevant resource:

To monitor reservations, set MONITOR=1 in sched_conf(5) params and use qsched(1) after running process_scheduler_log; qsched -a summarizes all the current reservations.

It is often required to provide a fair share of resources in some sense, whereby heavier users get reduced priority. There are two SGE policies for this. The share tree policy assigns priorities based on historical usage with a specified lifetime, and the functional policy only takes current usage into account, i.e. is similar to the share tree with a very short decay time. (It isn’t actually possible to use the share tree with a lifetime less than one hour.) Use one of the other, but not both to avoid confusion.

With both methods, ensure that the default scheduler parameters are changed so that weight_ticket is a lot larger than weight_urgency and weight_priority or set the latter two to zero if you don’t need them. Otherwise it is possible to defeat the fair share by submitting with a high priority (-p) or with resources with a high urgency attached. See sge_priority(5) for details.

You may also want to set ACCT_RESERVED_USAGE in execd_params to use effectively `wall clock' time in the accounting that determines the shares.

You normally want to prevent over-subscription of cores on execution hosts by limiting the slots allocated on a host to its core (or actually processor) count — where ``processors'' might mean hardware threads. There are multiple ways of doing so, according to taste, administrative convenience, and efficiency.

If you only have a single queue, you can get away with specifying the slot counts in the queue definition (qconf -mconf), e.g. by host group

but with multiple queues on the same hosts, you may need to avoid over-subscription due to contributions from each queue.

An easy way for an inhomogeneous cluster is with the following RQS (with qconf -arqs), although it may lead to slow scheduling in a large cluster:

This would probably be the best solution if num_proc, the processor count, is variable by turning hardware threads on and off.

Alternatively, with a host group for each hardware type, you can use a set of limits like

which will avoid the possible scheduling inefficiency of the $num_proc dynamic limit.

Finally, and possibly the most foolproof way in normal situations is to set the complex on each host, e.g.

Normally it is advisable to prevent jobs swapping. To do so, make the h_vmem complex consumable, and give it a default value that is (probably slightly less than) the lowest memory/core that you have on execution hosts, e.g.:

(See complex(5) and the definition of memory_specifier.)

Also set h_vmem to an appropriate value on each execution host, leaving some head-room for system processes, e.g. (with bash-style expansion):

Then single-process jobs can’t over-subscribe memory on the hosts—​at least the jobs on their own—​and multi-process ones can’t over-subscribe long term (see below).

Jobs which need more than the default (2000m per slot above) need to request it at job submission with -l{nbsp}h_vmem=..., and may end up under-subscribing hosts' slots to get enough memory in total.

Each process is limited by the system to the requested memory (see setrlimit(2)), and attempts to allocate more will fail. If it is a stack allocation, the program will typically die; if it is an attempt to malloc(3) too much, well-written programs should report an allocation failure. Also, the qmaster tracks the total memory accounted to the job, and will kill it if allocated memory exceeds the total requested.

These mechanisms are not ideal in the case of MPI-style jobs, in particular. The rlimit applied is the h_vmem request multiplied by the slot count for the job on the host, but it is for each process in the job—​the limit does not apply to the process tree as a whole. This means that MPI processes, for instance, can over-subscribe in the PDC_INTERVAL before the execd notices, and out-of-memory system errors may still occur if a job can fill swap fast enough. Future use of memory control groups will help address this on Linux. However, if you happen to use Open MPI 1.7 or later, you could use a JSV to set OMPI_MCA_opal_set_max_sys_limits consistent with h_vmem.

For managing Flexlm licence tokens, see Olesen’s method. This could be adapted to similar systems, assuming they can be interrogated suitably. There’s also the licence juggler for multiple locations.

If any of a job’s processes detach themselves from the process tree under the shepherd, they are not killed directly when the job terminates. Use ENABLE_ADDGRP_KILL to turn on finding and killing them at job termination. It will probably be on by default in a future version.

Binding processes to cores (or `CPU affinity') is normally important for performance on `modern' systems (in the mainstream at least since the SGI Origin). Assuming cores are not over-subscribed, a good default (since SGE 8.0.0c) is to set a default in sge_request(5) of

The allocated binding is accessible via SGE_BINDING in the job’s environment, which can be assigned directly to GOMP_CPU_AFFINITY for the benefit of the GNU OpenMP implementation, for instance.

If you happen to use Open MPI, a good default in openmpi-mca-params.conf (at least for Open MPI 1.6), matching the SGE -binding is:

Binding is the default in Open MPI 1.8, but to sockets, not cores, in most cases. That can be changed with, say,

which also prefers binding adjacent-numbered ranks to adjacent cores.

Open MPI will respect the core binding from SGE, i.e. it won’t try to bind to cores outside SGE_BINDING.

To use Infiniband via openib, processes need to be able to `register' a large amount of memory, and the limit for jobs is typically too small by default (see ulimit -l). It is typically necessary to add H_MEMORYLOCKED=infinity to execd_params (see sge_conf(5)). This may be necessary with other transports too.

To reboot execution hosts, you need to ensure they’re empty and avoid races with job submission. Thus, submit a job which requires exclusive access to the host and then does a reboot. Since you want to avoid root being able to run jobs for security reasons, use sudo(1) with appropriate settings to allow password-less executions of the commands by the appropriate users. You want to comment out Defaults requiretty from /etc/sudoers, add !requiretty to the relevant policy line, or use -pty y on the job submission. It is cleanest to shut down the execd before the reboot.

The job submission parameters will depend on what is allowed to run on the hosts in question, but assuming you can run SMP jobs on all hosts (some might not be allowed serial jobs), a suitable job might be

where $node is the host in question, and we try to ensure the job runs early by using a resource reservation and a high priority.

For a mass reboot it may be better to submit an array job of a size equal to the number of nodes to boot. It will have to request a complex set via a load sensor with a value reflecting a state before a reboot, such as the old kernel version for reboots into a new kernel.