frogcircus.org

Remote Execution and GNU Screen

by Charles Fry
Janurary 2004

screen can be very helpful if you run certain classes of remote server processes. It is a simple matter to ssh into a remote machine and to run a program from the command line. The risk with this, however, is that by default, if your ssh connection is broken then the executing program will be sent a HUP signal, which in most situations will result in the program's termination.

An initial attempt to solving this problem might be to run the remote program under nohup, making it immune to hangups. This has the potentially undesirable side-effect that output must be sent to a non-tty, which is problematic for interactive programs, and can be inconvenient for any program. A second, and improved, solution would be to run the remote process inside of screen. This would also make it immune to hangups, with the added advantage of direct program interaction. The drawback of this approach is that it requires a fair amount of manual setup, preventing it from scaling to large numbers of machines.

It should come as no surprise that screen, in its infinite wisdom, already contains several mechanisms which will significantly simplify the process of remote execution in such a way that it can effortlessly scale to many machines. First, screen accepts some optional parameters that are rarely used, but which are especially relevant in this case. From the man page:

screen [ -options ] [ cmd [ args ] ]

When screen and its options are followed by a command name, with optional arguments, the specified command is executed in the new screen's first window. The first potential difficulty that could be encountered with this solution is that when the executed command terminates, screen will also terminate, and the command's output could become unaccesible. This can be countered in two different ways. While screen, by default, closes windows when their contained processes terminate, it also can be instructed to leave them open with the zombie command. This can be done by adding a line like the following to your .screenrc:

zombie ko

An added benefit of doing this is that it provides a simple mechanism by which the terminated command can be reinvoked. Another option, which can be combined with zombie, is logging, which can be turned on by:

deflog on

It may also be desirable to specify the file to which the log is written. When running related processes on a large number of machines, it can be especially helpful to store all of the logfiles in a central, shared directory. In this case it would be desirable for the logfile names to indicate which machine they came from. All of this can be accomplished with the logfile command:

logfile $LOG_DIR/screenlog.%H

With screen now fully configured, it is a simple matter to execute a command in screen after having remotely logged into a machine. However, by default screen must be attached to a tty, flustering invocations such as:

ssh <hostname> screen <cmd> <args>

Once again, screen comes to the rescue with its ability to start in detached mode, which can be requested with the options -d -m:

ssh <hostname> screen -d -m <cmd> <args>

This can now be easily wrapped inside of a small script and invoked automatically on a whole cluster of machines (dsh might also come in handy). If such scripts need to wait for execution to complete in order to perform some postprocessing, just specify -D -m instead:

ssh <hostname> screen -D -m <cmd> <args>

Once a remote (or local) screen session has been started, it can always be controlled by attaching to it and performing the required operation. Conveniently, it is also possible to remotely control a screen session using the -X flag, which sends a command to a running screen session:

ssh <hostname> screen -X <cmd>

The command sent with -X can be any of the commands listed in the screen manual. One especially convenient one that I have found is quit, which terminates a screen session after killing all of its windows.

Under certain circumstances it may be desirable to run multiple processes on the same remote machine. While screen always allows explicit control of its sessions through the use of the process identifier, it can often be more convenient to associate an explicit label with each screen session. Each session can be given its own name with the -S flag. As long as the session's name is not the prefix for another session's name, it can be used to refer to that session, without requiring knowledge of the session's process identifier. For example, one could start a remote screen with an explicit session name, and then use that name to send commands to that session:

ssh <hostname> screen -S <sessionname> -d -m <cmd> <args>
ssh <hostname> screen -S <sessionname> -X <cmd>

As a final note, remember that the -ls option will list all active screen sessions, providing the process identifier, the sessionname, and the Attached/Detached status of each session.