A Console Server Thomas A. Fine Steven M. Romig The Ohio State University Abstract Most computers nowadays include some sort of ``console terminal'' from which one can perform certain special administrative tasks, such as booting the machine, or running diagnostics. Some things also write error message to the console terminal. Unfortunately, these consoles take up valuable space, often in rooms with special environments where space is at a premium. Many people use CRTs as console terminals, which usually means that you can not make a record of the messages that have appeared before the latest 24 lines. Finally, the location of the console terminals tends to be inconvenient. We had 30 console terminals piled on desks in our machine room, which took up a considerable amount of space. We replaced most of these console terminals with a spare Sun 3/180 outfitted with 32 serial ports and a custom software package called ``the console server''. The console server saves us space in our machine room, logs output from all of the console ports, and provides convenient access to the console ports on the machines that it serves. In this paper we will describe the reasons for undertaking this project, and the design and implementation of the software. 1 Introduction Our environment at Ohio State has been growing rapidly over the past three years. We found ourselves faced with several problems relating to the console terminals in our machine room. The first problem we faced was that we were running out of space in the machine room. Most of our servers are Suns, and there is no convenient place to put the terminals on the Sun cabinets (we were using h19 terminals). We had 30 of these terminals piled four high in the machine room, which took up about 80 square feet, with no room left for any new machines. This arrangement also made it difficult to use most of the consoles because they were either on the floor, or at shoulder height. Another problem we wanted to solve was the lack of logging with the h19 terminals. Often a machine would crash, or some other problem would occur, and by the time someone got into the machine room, any relevant messages were already scrolled off of the screen. We did not want to use printing terminals, since they would have made our space problem much worse (they can not be stacked), and they are prone to periodic mechanical failures. Of course, syslog does provide some logging, but it is by no means complete. A third problem was convenience. As mentioned above, the arrangement in the machine room made it difficult to use many of the terminals. Their location was also very inconvenient, because of the need to run to the machine room every time a problem arose. We could not just move the consoles because many times you need to be near the machine while at the console. We could have added consoles with Y connectors, but this would have been a cabling nightmare, and would not have helped either our space or logging problems. Our solution was to replace all the terminals with a single machine servicing all the console terminal lines, and software which provides logging and direct, secure network access. The machine, the console server, is a Sun 3/180 with 32 additional serial ports (two alm2 cards). The software consists of a server program which does the logging and handles connections, and a client program used to connect to any desired terminal. This solves all of our problems nicely. We immediately freed up about 60 square feet in our machine room (enough space for at least six more servers). The logging provided is complete, as long as the server is up. The improvement in convenience is enormous, providing access to our file server consoles and log files from anywhere in our network, as well as from home (with a modem). 2 Environment Our environment is primarily a network of Sun workstations, served by 23 Sun file servers which are hooked to our console server. We also have four Pyramids, an HP file server, an Encore Multimax, and a BBN Butterfly attached to the console server. The console server itself does not depend on any other machines, and only handles administrative tasks. 3 Design The software consists of a server program and a client program. The server program logs all the activity from the consoles, and handles network connections from the client program. The client program is used to request a connection to a console from anywhere on the network. The client program requests a given console line by the name of the file server, and can request one of four commands on that console line: 1. Attach (two-way connection) 2. Spy (read-only connection) 3. Force (two-way connection, superseding an existing two-way connection) 4. Who (returns who is connected to the requested console line) In addition to the ``who'' command above, you can request a list of all connections everywhere, by requesting connection to a server named ``who''. This last was added as an afterthought, and does not use the normal connection protocol used for other connections. Escape sequences are provided for controlling the state of the connection. This includes switching between read-only and two-way connections, toggling flow control, and disconnecting. An escape sequence is also provided to halt Sun file servers. All of the escape sequences are handled by the server program. A simple configuration file is read when the console server is started. This file contains file server names, their log file, the serial port they are attached to, and a group number which determines which servers will be handled by each server process (described below). 3.1 Server Program The server program forks off several children, with each child process handling a small subset of the consoles. This was done primarily because a single process would run out of open files quickly (each terminal uses two files, and each connection to a terminal uses one). Alternatively, with a separate process for each console, we were worried that the excessive number of processes would bog down the machine. In our current configuration, we use eight child processes, each handling four or five console lines. The parent process remains to serve as a port lookup for the other processes. Each of the children listens on a separate port for connections, but only the port used by the parent process is known to the client program. The client must first connect to the parent process and request the port number for the desired server, then connect to the child process at that port. The server program handles all connection states, keeping track of which connection is two-way for each of the console lines. It is also responsible for breaking connections when requested with an escape sequence from the client program (typed by the user). 3.2 Client Program The client program is a very straight-forward piece of software. It connects to the server software as described below, and if it succeeds in connecting, it simply passes all data from the keyboard to the console server, and all data from the console server to the screen, until the console server breaks the connection. 3.3 Connection Protocol The following is a simple version of the connection protocol: 1. The client program connects to the parent process of the server program, and sends the name of the server requested by the user. 2. The server program responds with either a port number (sent in ascii) or the message ``Server not found'', and breaks the connection. 3. If the client program receives the error message, it displays it, and exits. Otherwise, it requests a password from the user, connects to the new port number, and transmits the user's name, the password, the source machine, the command (two-way, read only, forced, who), and the file server name. 4. The server program (child process) checks the password, and if valid, it completes the connection, sending ``ok'' to the client program. If it is not valid, the server returns the message ``Sorry'' and breaks the connection. Note that if the command is who, password checking is skipped. 3.4 Security The console terminals can do special things on servers (like halting, booting in single user mode) so we were somewhat concerned about security. The password required is a yellow pages password, so the level of security provided by this method is as good (or as bad) as the rest of the system. We would eventually like to use Kerberos to authenticate users in a more secure fashion. 4 Implementation Issues The initial effort required for the software was minimal. The first version (with few features) was written in just a few days. Fine tuning has been a much longer process which is not yet (and may never be) complete. The following covers problems we encountered and features added (in no particular order). We immediately added flow control, because we found that there was significant data loss when the output was going to a slow terminal (like the Sun frame buffer terminal emulator). Faster terminals (a real CRT, or an X windows terminal emulator) rarely had a problem, but the flow control was still necessary. An escape sequence was also added to toggle flow control on and off, primarily so emacs could be used without rebinding control-S. The majority of our file servers are Sun 3s. These machine do not have an ascii sequence used to halt the machine. Instead (if the console is on a serial port) they use a terminal line break. We had to add and escape sequence that would tell the console server to generate a line break. That was simple enough, but this also presented another problem: we wanted to be able to remove the cables from the console server at various times. Unfortunately, the suns also interpret a drop in the Data Terminal Ready line as a request to halt the server. We had to modify the file server end of the cable to maintain DTR even when the cable was unplugged on the console server end. Four of our file servers are Pyramids. Unfortunately, these machines are convinced that their terminals are Wyse terminals (it is built into their microcode). There was no good way around this, so we have Wyse terminals scattered around in convenient places. We also wrote a program that will filter out the worst of the Wyse sequences, so that if a Wyse terminal is not available, you can scrape by with whatever you have. We realized that it would be convenient to see the last few lines of output from a console's logfile after connecting. We added an option to display the last twenty lines from the log after connecting. This is sometimes not enough, and we may add escape sequences to replay arbitrary amounts of the log file. It is possible to create a loop of two or more console connections, which will continually circulate the connection message through the loop (and into all involved log files). If a disconnect sequence is then sent by the user, it only reaches the first connection, but leaves the loop intact. There is currently nothing to solve this problem, but it has only occurred (accidentally) once. The log files can sometimes become rather unwieldy. We have them on a file system with lots of space, so that has not been a problem, but it is hard to get useful information out of a 24 Meg log file. Ordinarily the log files grow slowly, but when there is a problem (like a root file system becomes full), the file server can generate an amazing volume of messages in a short period of time. We haven't gotten around to providing automatic truncation of these files, but it should not be a big problem. We needed the log files to contain time stamping. We used a simple shell script that wakes up every hour and prints the date in each logfile. This provides time stamping without interrupting activity on the console. We were initially worried about the speed of the console server during excessive use (many connections, or many machines rebooting) but we have not seen a problem like this so far. If the console server ever crashes badly, and we have a hard time getting it back up, we will have to deal with all the other file servers by plugging cables into a spare terminal. Since a bad crash is likely to come from a power outage, this could be a serious catastrophe, as most of the servers would probably need immediate attention. It has not happened yet (knock on wood). 5 Conclusion The console server really works well. It has done everything we hoped it would, and has not had any serious problems. The improved convenience has been the biggest payoff from the project. The ability to fix a problem from home (or from Anaheim, California) has saved a great deal of time, gasoline, and sleep. And in the office it enables staff members to become even more desk-bound. There have been a couple of other benefits that have not been mentioned above. Since the console server software does not heavily load the machine itself, it is an ideal machine to handle other administrative tasks (yellow pages master, backups, etc.). With all of these uses, the console server is a very cheap alternative to CRT terminals.