Setting Up A Load Sensor in Grid Engine

The example below illustrates how to set up a load sensor to track the amount of /tmp space on each Grid Engine host. Load sensors to monitor any desirable value can be written by using this as a template. Once a load sensor is added, the new resource can be used as a load threshhold, or consumable resource. The steps for adding a load sensor are as follows:

Step 1: Define the resource
Step 2: Configure the resource
Step 3: View/Verify the resource
Step 4: Request the resource

Step 1: Define the resource attributes in the host complex

First, modify the complex called "host" or "global", depending on the type of resource, host-specific or clusterwide. To modify the complex, click on "Complexes Configuration" on the main toolbar of qmon. Select "host" or "global", then click on "Modify". If the global host does not exist, it must be created by clicking on "Add"(the name, global, must be used). In the example below, host-specific resources are added to the "host" complex.

   name   shortcut   type     value  relop requestable consumable default
   ----------------------------------------------------------------------
   tmpfree tmpfree   MEMORY    0       <=    YES         YES        0 
   tmptot  tmptot    MEMORY    0       <=    YES         NO         0 
   tmpused tmpused   MEMORY    0       >=    NO          NO         0

This says: There is a complex attribute called "tmpfree" with the shortcut "tmpfree" of type memory. The "value" is supplied by the load sensor. It is requestable ("yes"), and it is consumable ("yes"). The "default" should be set to 0.

When using qmon, do not forget to press the "Add" button after entering each line. When all the lines are entered (all are in the table below), press the "Ok" button to close this window.

The complex may be viewed at the command line by running the following:

% qconf -sc host
   name      shortcut   type     value relop requestable consumable default
   ------------------------------------------------------------------------
   tmpfree   tmpfree    MEMORY   0     <=    YES         YES        0    
   tmptot    tmptot     MEMORY   0     <=    YES         NO         0    
   tmpused   tmpused    MEMORY   0     >=    NO          NO         0  

Step 2: Configure the "global" host in the cluster configuration

In this case, the load sensor should run on all hosts (each host will return a different value). Therefore, the pseudo host "global" needs to be configured. In the case of a floating license, a single host would be configured to run our load sensor (since the value would be the same and all hosts returning this same value would be redundant).

In the main qmon window, click on "Cluster Configuration". Highlight "global", then click on "Modify". On the "General Settings" tab, add the path and name of the load sensor program to the load sensor box. As an example/template, tmpspace.sh is included below. Once OK is pressed, the load sensor will be automatically started on each host. This may take several minutes.

Note: The resource names output by the load sensor must be the same as the names added to the host or global complex. Grid Engine takes the smaller of the two values when determining the current value.

Step 3: View the new global resources

The new global resource can be viewed by running the following:

   % qhost -F tmpfree,tmptot,tmpused 

   HOSTNAME ARCH      NPROC LOAD  MEMTOT   MEMUSE   SWAPTO   SWAPUS  
   ----------------------------------------------------------------
   BALROG   solaris6  2     1.47  1.0G     974.0M   150.0M   130.0M  
       Host Resource(s):      hl:tmpfree=337.744000M
       hl:tmptot=338.808000M
       hl:tmpused=1.014709M

See qhost(1) for more information.

Step 4: Requesting a resource

Include the -l switch on the command line to request a resource:

 % qsub -l tmpfree=100M myjob.sh

This will dispatch the job only to those machines whose tmpfree value is greater than or equal to 100 MiB.

Note on Using a Load Sensor for Floating Licenses

In order to track the number of floating licenses used outside of Grid Engine, a load sensor may be used in conjunction with a consumable resource (e.g. http://olesenm.github.com/flex-grid/). The lesser of the Consumable Resources or the load sensor value will be used to prevent license oversubscription. The load sensor in this case only needs to run on a single host. To do this, output the string 'global' in place of the machine name in the load sensor.

# ----------------< tmpspace.sh >-----------------------------
#!/bin/sh
# Grid Engine will automatically start/stop this script on exec hosts, if
# configured properly. See the application note for configuration
# instructions.
# fs to check
FS=/tmp
# Get POSIX df on SunOS
[ -d /usr/xpg4/bin ] && PATH=/usr/xpg4/bin:$PATH

if [ "$SGE_ROOT" != "" ]; then
   root_dir=$SGE_ROOT
fi

# invariant values
myarch=`$root_dir/util/arch`
myhost=`$root_dir/utilbin/$myarch/gethostname -name`

ende=false
while read input; do

   if [ "$input" = quit ]; then
      break
   fi

   # ---------------------------------------- 
   # send mark for begin of load report
   # NOTE: for global consumable resources not attached
   # to each machine (ie. floating licenses), the load
   # sensor only needs to be run on one host. In that case,
   # echo the string 'global' instead of '$myhost'.
   echo "begin"

   # -P ensures filesystem info all on one line with long fields (e.g.
   # long server names).
   dfoutput=`df -P -k $FS | tail -1`
   tmpfree=`echo $dfoutput | awk '{print $4}'`
   tmptot=`echo $dfoutput | awk '{print $2}'` 
   tmpused=`echo $dfoutput | awk '{print $3}'`

   echo "$myhost:tmpfree:${tmpfree}k"
   echo "$myhost:tmptot:${tmptot}k"
   echo "$myhost:tmpused:${tmpused}k"

   echo end
done

#----------------------< CUT HERE >--------------------------------