remote_startup.5
NAME
remote_startup - the Grid Engine remote startup mechanism
DESCRIPTION
Grid Engine supports several commands to facilitate interactive
commands or remote startup of a tightly integrated parallel job. Each
command can be set up with qconf(1) (option -sconf) to use different
daemons and commands to start the final session. Different startup
methods can therefore contain different daemons and commands, and are
not related to other startup methods in any way, although it is often
desirable to have the same communication method for all startup
methods.
Each method requires a separate instance of the communication daemon to
be started by sge_shepherd(8) for each job, which thus must use a
randomly-chosen port, to which the client will try to connect. This is
necessary to support tight integration, ensuring that everything for a
given job stays in the same process tree and can be properly controlled
and accounted by Grid Engine.
QLOGIN
An interactive qlogin session invoked by qlogin(1) will show up in
qstat with the default name QLOGIN unless changed by the -N name
option. The two entries qlogin_daemon and qlogin_command are
responsible for establishing the communication to start such a session.
The default is the value builtin, which will trigger an SGE internal
communication method:
qlogin_command builtin
qlogin_daemon builtin
In cases where you want a different communication method, it can also
be set up for the formerly-used communication method based on telnet.
Despite the fact that telnet is used, there is no need to have telnetd
running all the time - SGE will start a unique one for each job, even
when it's from the same user, and telnet can stay disabled as a system
service (whether under init or inetd). Only the file /etc/hosts.equiv
needs to contain the name of the machines from where a qlogin(1) should
be allowed. This is often the head node of a cluster, or particular
submit machines. Using telnet, the traffic is not encrypted, but that
may be reasonable on a private subnet for the cluster, especially if
SGE's CSP security isn't used to secure the system generally.
To achieve this:
qlogin_command /usr/bin/telnet
qlogin_daemon /usr/sbin/in.telnetd
must be defined in sge_conf(5). The defined qlogin_command will then
be called with two additional parameters: 'HOST', and 'PORT' in exactly
that order, which refer to the machine the qlogin_command should
address and the port to be used.
This can also be used to set up secure communication using SSH (which
can also provide X and credential forwarding, as well as compression).
In this case a small wrapper must be implemented, whose sole purpose is
to swap the two given arguments and prepend -p to the port argument. A
suitable one is installed as
$SGE_ROOT/util/resources/wrappers/qlogin_wrapper:
qlogin_command /opt/sge/util/resources/wrappers/qlogin_wrapper
qlogin_daemon /usr/sbin/sshd -i
QRLOGIN
An interactive qrlogin session invoked by qrsh(1) without a command
will show up in qstat having the default name QRLOGIN unless changed by
the -N name option. The two entries rlogin_daemon and rlogin_command
are responsible for establishing the command to start such a session.
The default is the value builtin, which will trigger an SGE internal
communication method:
rlogin_command builtin
rlogin_daemon builtin
In cases where you want a different communication method, it can follow
the formerly-used communication method based on rlogin. As for telnet,
rlogin can stay disabled as a system service (whether under init or
inetd). Only the file /etc/hosts.equiv needs to contain the name of
the machines from where a qrsh(1) should be allowed. This is often the
head node of a cluster, or particular submit machines. Using rlogin,
the traffic is not encrypted, but that may be reasonable on a private
subnet for the cluster, especially if SGE's CSP security isn't used to
secure the system generally.
To achieve this:
rlogin_command $SGE_ROOT/utilbin/$ARC/rlogin
rlogin_daemon /usr/sbin/in.rlogind
must be defined in sge_conf(5). The value of $SGE_ROOT must be
replaced by the root of the installation, and $ARC must be replaced by
the particular platform architecture, as use of environment variables
is not implemented for these entries. When the cluster is homogeneous,
it can be set to e.g. 'lx-amd64' or 'lx-x86'. In a heterogeneous
cluster local configurations need to be defined, where preferably the
minority of machines will get local configurations.
The defined rlogin_command will then be called with three additional
parameters: '-p','PORT', and 'HOST' in exactly that order, which refer
to the machine the rlogin_command should address, and the port to be
used.
This can also be used to set up a secure communication using SSH:
rlogin_command /usr/bin/ssh
rlogin_daemon /usr/sbin/sshd -i
QRSH
An interactive session for a remote command invoked by qrsh(1) with a
command will show up in qstat by default with name of the command
issued, unless changed by the -N name option. The two entries
rsh_daemon and rsh_command are responsible for establishing the
communication to start such a session. This startup method will also
be used by the master task of a tightly integrated parallel job to
start slave processes on other granted exechosts.
The default is the value builtin, which will trigger an SGE internal
communication method:
rsh_command builtin
rsh_daemon builtin
In cases where you want a different communication method, it can also
be set up for the formerly-used communication method based on rsh. As
for telnet, rsh can stay disabled as a system service (whether under
init or inetd). Only the file /etc/hosts.equiv needs to contain the
name of the machines from where a qrsh(1) should be allowed. This is
often the head node of a cluster, or particular submit machines. Using
rsh, the traffic is not encrypted, but that may be reasonable on a
private subnet for the cluster, especially if SGE's CSP security isn't
used to secure the system generally.
To achieve this:
rsh_command $SGE_ROOT/utilbin/$ARC/rsh
rsh_daemon $SGE_ROOT/utilbin/$ARC/rshd -l
must be defined in sge_conf(5). The value of $SGE_ROOT must be
replaced by the root of the installation, and $ARC must be replaced by
the particular platform architecture, as use of environment variables
is not implemented for these entries. When the cluster is homogeneous,
it can be set to e.g. 'lx-amd64' or 'lx-x86'. In a heterogeneous
cluster local configurations need to be defined, where preferably the
minority of machines will get local configurations.
The defined rsh_command will then be called with four additional
parameters: '-n', '-p', 'PORT', and 'HOST' in exactly that order, which
refer to the machine the rsh_command should address and the port to be
used.
This can also be used to set up a secure communication using SSH:
rsh_command /usr/bin/ssh
rsh_daemon /usr/sbin/sshd -i
Again, this is independent of SSH as a system service, which can remain
disabled.
LOCAL CONFIGURATIONS OF EXECHOSTS
It is important to note that the communication method set up for one
particular startup method must match at each end. This can either be
achieved by using only a global configuration, or carefully setting up
local configurations for the exechosts involved. Whether or not local
configurations exist, which must be taken care of, can be checked with
qconf -sconfl.
As a general rule, for setting up a communication method between a
machine A (where the command is issued) and a machine B (where the
daemon is started) it must be guaranteed that the:
setup communication method for the command on machine A
(either global configuration from sge_conf(5) or local
configuration qconf -sconf A of machine A)
matches
setup communication method for the daemon for machine B
(either global configuration from sge_conf(5) or local
configuration qconf -sconf B of machine B)
This way it is also possible to use different communication methods,
depending whether a connection from A to B is invoked, or from B to A.
RESTRICTIONS
For all three communication methods, a direct connection between the
target and the source machine where the particular command was issued
must exist. This can also be implemented using TCP/IP forwarding, but
will usually fail if one machine is behind NAT which will mangle the
machines' addresses. The communication methods won't work with simple
firewalling of the exec hosts since the methods use a random port. It
may be possible to set up application-specific firewalling, if
necessary, or to wrap the methods and start an SSH tunnel on the port
specified for each communication instance.
The builtin method does not support forwarding of X graphics from the
compute nodes, or GSSAPI tokens to them. If you need that for any of
the remote methods, you will want to set up SSH communication instead.
SSH AUTHENTICATION
To allow the SSH setup explained above to work, the user must be
authenticated without the use of a passphrase. While entering a
passphrase would still work for interactive commands, it will fail in
case of a tightly integrated parallel job, where the master process
tries to start a slave process on another exechost.
You can set up passphraseless SSH keys, although this is discouraged. A
simpler and global working setup is to use host-based authentication
<http://arc.liv.ac.uk/SGE/howto/hostbased-ssh.html> for the machines
inside the cluster.
SSH TIGHT INTEGRATION
To have a tight integration of SSH into SGE, the started sshd needs an
additional group ID to be attached. With this additional group ID, SGE
is able to record the resource consumption and computing time in a
correct way. Also a qdel of such a job will be able to succeed.
Such a tight SSH integration can be achieved by two means:
Use of PAM
The easiest way on supported platforms (at least GNU/Linux): a
pam(7) module pam_sge-qrsh-setup(8) is available for use with
the system ssh; it attaches the necessary additional group ID to
the started process to provide tight integration. See also the
workshop paper <http://gridengine.org/assets/static/ws2007/
K5SGE.pdf>.
Recompile Grid Engine with ./aimk -tight-ssh ...
The source of Grid Engine contains the necessary additions to
compile a modified sshd, which will honor the additional group
ID and attach it also to the started process. It's necessary to
provide the source of OpenSSH in the directory 3rd_party inside
$SGE_ROOT having a plain name 'openssh'. Inside this directory
the file sshd.c needs to be patched:
in main():
init_rng();
#ifdef SGESSH_INTEGRATION
sgessh_readconfig();
#endif
in privsep_postauth():
/* Drop privileges */
#ifdef SGESSH_INTEGRATION
sgessh_do_setusercontext(authctxt->pw);
#else
do_setusercontext(authctxt->pw);
#endif
See the original documentation <http://gridengine.org/assets/
static/ws2007/SGE-openSSHTightIntegration.RonChen.pdf>.
RESTRICTING ACCESS
With the builtin method in use, there is no need to allow direct access
for normal users to compute nodes with ssh etc. However, you may want
to allow users to access the nodes for debugging. If you don't want to
over-subscribe the nodes, so that qrsh etc. can be used for access, you
can use PAM to restrict access for a user only to the nodes on which
they have a running job, so as to minimize interference with other
others.
There are two possible ways. The cleanest uses pam_sge_authorize(8).
Otherwise you can use generic PAM modules, such as pam_limits(8) or
pam_access(8), with modifications to their configuration set up and
taken down in the job prolog and epilog respectively. See, for
instance, a user list message <http://gridengine.markmail.org/message/
mu3i7haeshlevu6q?q=282211>, and other examples of similar prolog/epilog
scripts provided with locking in the pam_authuser contribution in the
Torque distribution.
SECURITY
See the notes above concerning security of the communication channel.
EXAMPLES
Using SSH with the PAM module, forcing tty allocation, and preventing
the delegation of GSSAPI credentials to the compute nodes:
rsh_daemon /opt/sge/util/rshdwrapper
rsh_command ssh -tt -o GSSAPIDelegateCredentials=no
qlogin_daemon /opt/sge/util/rshdwrapper
qlogin_command ssh -tt -o GSSAPIDelegateCredentials=no
rlogin_daemon /opt/sge/util/rshdwrapper
rlogin_command ssh -tt -o GSSAPIDelegateCredentials=no
Old-style method, using telnet and rlogin:
qlogin_command /usr/bin/telnet
qlogin_daemon /usr/sbin/in.telnetd
rlogin_command /opt/sge/utilbin/lx-x86/rlogin
rlogin_daemon /usr/sbin/in.rlogind
rsh_command /opt/sge/utilbin/lx-x86/rsh
rsh_daemon /opt/sge/utilbin/lx-x86/rshd -l
FILES
$SGE_ROOT/util/resources/wrappers/qlogin_wrapper
SSH-based wrapper for qlogin (see above)
SEE ALSO
qconf(1), qlogin(1), qrsh(1), sge_conf(5), pam_sge_authorize(8),
pam_sge-qrsh-setup(8), Grid Engine-specific remote programs <http://
arc.liv.ac.uk/repos/darcs/sge/source/3rdparty/remote/remote.html>.
AUTHOR
Man page written by Reuti, partly based on Sun material. Some
additions by Dave Love.
COPYRIGHT
See sge_intro(1) for a full statement of rights and permissions.
SGE 8.1.3pre 2010/11/22 20:58:24 remote_startup(5)
Man(1) output converted with
man2html