I have a lot of devices which use serial consoles for my job, most of them with USB connectors but some with traditional 9-pin serial ports as well. Juggling connections for all of these can be a chore, but there are ways to simplify the process.

I have tried several different tactics over the years but the best solution I’ve found so far is to use a Raspberry Pi as a dedicated serial console multiplexer (MUX). The title seemed fitting as it takes multiple inputs (serial consoles) and manages a single output (screen) though the user controls the active input in this case.

Supplies

  • Raspberry Pi (or equivalent)

    Though this post is geared toward using a Raspberry Pi with Raspbian as a serial console server, many of the techniques used are general and could be adapted to a variety of platforms.

  • USB serial hub with lots of ports

    Like this 13-port USB 2.0 hub. Linux doesn’t seem to get along with lots of USB 3.0 ports and there isn’t any compelling reason to use USB 3.0 for serial consoles anyhow.

  • USB serial devices

    Varies by device, some have USB consoles, some have serial ports with 9-pin or RJ45 connectors. Some may require null modem adapters. This is mostly beyond the scope of what I’m trying to cover here. I’m going to assume you can already talk to the devices with USB serial in some way.

    Speaking of USB serial, I like these USB serial adapters for devices with regular non-USB serial ports.

  • GNU screen

    GNU screen is capable of talking to serial ports directly, so there is no need to use a separate terminal emulator program such as minicom. Though if you prefer to use a different terminal program in each screen or maybe using tmux, similar tactics should work there, too.

Setup the hardware

Setting up the Pi is out of scope here, I’m going to assume you have a working pi connected to your network.

Using a wired connection is best for real-time interactive work in SSH, but depending on your AP and wireless setup it may still work OK that way.

Connect all of the USB serial devices to the hub and connect the hub to the Pi. To ensure things are probed in the correct and expected order, rebooting the Pi is a useful measure at this point.

Lots of USB serial connections in one hub

From here on, everything is done over SSH connected to the Pi. Use the SSH program of your choice (e.g. OpenSSH, SecureCRT, PuTTY).

Map the serial ports to consistent names

Before setting up the consoles let’s take the time to map each port to a consistent and useful name. Rather than having them numbered, we can set it up so that each port is labeled after the device. This makes keeping the screen names and purposes easier to manage over time. If you are lucky and each one has a unique set of properties, then it won’t even matter which order or ports they plug into or what order they probe in.

Gather device attributes

Look at the connected USB ports:

$ ls -l /dev/ttyUSB*
crw-rw---- 1 root dialout 188,  0 Mar 29 11:36 /dev/ttyUSB0
crw-rw---- 1 root dialout 188,  1 Mar 29 11:42 /dev/ttyUSB1
crw-rw---- 1 root dialout 188,  2 Mar 29 11:53 /dev/ttyUSB2
crw-rw---- 1 root dialout 188,  3 Mar 14 16:08 /dev/ttyUSB3
crw-rw---- 1 root dialout 188,  4 Mar 11 14:04 /dev/ttyUSB4

For each of these, go through the device information and find useful and unique properties which identify a given device. This will give us enough information to craft udev rules to setup the port names. The command probes the entire stack to reach the device, but the one you are interested in is usually a few entries into the output and will look similar to what is shown below.

Replace x in the command with the device number, and repeat the command for each ttyUSB device.

$ sudo udevadm info --name=/dev/ttyUSBx --attribute-walk
[...]
  looking at parent device '/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1.3/1-1.1.3.2':
[...]
    KERNELS=="1-1.1.3.2"
    ATTRS{manufacturer}=="Silicon Labs"
    ATTRS{idProduct}=="ea60"
    ATTRS{devpath}=="1.1.3.2"
[...]
    ATTRS{serial}=="xxxxxx01"
[...]
    ATTRS{idVendor}=="10c4"

This is an ideal case. The device has a unique vendor ID (idVendor), product ID (idProduct), and serial number (serial). Not every device will have a serial number and multiple devices may even have the same serial number, which complicates matters.

The trickier cases may have to rely on the device path (devpath) which means they have to remain plugged into the same USB port.

Setup udev rules

Create or edit /etc/udev/rules.d/99-usb-serial.rules and add rules based on the attributes for each device along with its name.

$ sudo vi /etc/udev/rules.d/99-usb-serial.rules
# These two devices are similar but have unique serial numbers.
SUBSYSTEM=="tty", \
	ATTRS{idVendor}=="10c4", \
	ATTRS{idProduct}=="ea60", \
	ATTRS{serial}=="xxxxxx01", \
	SYMLINK+="serial_edge"
SUBSYSTEM=="tty", \
	ATTRS{idVendor}=="10c4", \
	ATTRS{idProduct}=="ea60", \
	ATTRS{serial}=="xxxxxx02", \
	SYMLINK+="serial_server"

# These two devices don't have a serial number,
# so we have to use the device path
SUBSYSTEM=="tty", \
	ATTRS{idVendor}=="067b", \
	ATTRS{idProduct}=="2303", \
	ATTRS{devpath}=="1.1.3.7.4", \
	SYMLINK+="serial_switch1"
SUBSYSTEM=="tty", \
	ATTRS{idVendor}=="067b", \
	ATTRS{idProduct}=="2303", \
	ATTRS{devpath}=="1.1.3.7.6", \
	SYMLINK+="serial_switch2"

# This device has a unique vendor and product ID, but use the serial
# number anyhow in case a similar device is added in the future.
SUBSYSTEM=="tty", \
	ATTRS{idVendor}=="0403", \
	ATTRS{idProduct}=="6001", \
	ATTRS{serial}=="xxxxxx03", \
	SYMLINK+="serial_adapter"

Reload and test

Now reload the rules and re-trigger the device detection:

$ sudo udevadm control --reload-rules
$ sudo udevadm trigger --attr-match=subsystem=tty

Now look at the mapped devices with our nice names:

$ ls -ld /dev/serial*
drwxr-xr-x 4 root root 80 Oct 24 15:09 /dev/serial
lrwxrwxrwx 1 root root  7 Oct 24 15:09 /dev/serial1 -> ttyAMA0
lrwxrwxrwx 1 root root  7 Oct 24 15:09 /dev/serial_edge -> ttyUSB2
lrwxrwxrwx 1 root root  7 Oct 24 15:09 /dev/serial_server -> ttyUSB1
lrwxrwxrwx 1 root root  7 Oct 24 15:09 /dev/serial_switch1 -> ttyUSB4
lrwxrwxrwx 1 root root  7 Oct 24 15:09 /dev/serial_switch2 -> ttyUSB3
lrwxrwxrwx 1 root root  7 Oct 24 15:09 /dev/serial_adapter -> ttyUSB0

These will come back on reboot and when devices are unplugged and replugged.

It’s a good idea to reboot at this point to ensure the custom names return as expected.

Check privileges

Ensure that your user can access the serial ports. By default in Raspbian the user is pi and it is a member of the dialout group, and the dialout group has permission to access the ttyUSB* devices. If you use another user, add it to the dialout group.

$ sudo usermod -a -G dialout myuser

Alternately, you can run screen through sudo, but that is typically unnecessary with proper permissions and group memberships.

GNU screen

GNU screen allows a user to run multiple concurrent terminal sessions from a single login session. This is much easier to manage than multiple SSH connections to the same server, with the added benefit that screen can “detach” and leave the terminals connected in the background while the user does other things or even logs out. Later, the user can resume the screen session and it is as if they never left.

Many devices will output critical messages to the serial console and if the user isn’t watching the console when they happen, they can be easily missed. Using screen with a large scrollback buffer means the user can monitor these devices and look back at recent output, search it for specific strings, or even write it out to a file.

Setup screen configuration

Create a configuration file for this dedicated instance of screen:

$ vi .consoles.screenrc 

Add in some useful configuration options along with definitions for each of the serial terminals.

# Handy terminal defaults
termcapinfo xterm ti@:te@
termcapinfo xterm-color ti@:te@
termcap xterm 'is=\E[r\E[m\E[2J\E[H\E[?7h\E[?1;4;6l'
terminfo xterm 'is=\E[r\E[m\E[2J\E[H\E[?7h\E[?1;4;6l'
vbell off

# Status line -- careful the second line is quite long!
hardstatus alwayslastline
hardstatus string '%{gk}[%{G}%H%{g}][%= %{wk}%?%-Lw%?%{=b kR}(%{W}%n*%f %t%?(%u)%?%{=b kR})%{= kw}%?%+Lw%?%?%= %{g}]%{=b C}[%m/%d/%y %C %A]%{W}'

# Misc other defaults
shell -$SHELL
logtstamp on
logtstamp after 1

# No need for the intro screen
startup_message off

# 50000 lines of scrollback per window
defscrollback 50000

# Custom session name
sessionname consoles

# Definitions for each separate screen
screen -t 'Edge'     /dev/serial_edge    115200
screen -t 'Server'   /dev/serial_server  115200
screen -t 'Switch-1' /dev/serial_switch1 115200
screen -t 'Switch-2' /dev/serial_switch2 115200
screen -t 'Adapter'  /dev/serial_adapter 115200

Replace the names, serial devices, and serial port speeds as needed to match your own setup.

Using GNU screen

Overview

When screen is running it will display a nice status bar with screen numbers and the short names we set above with -t <name>.

Screen session with multiple windows as configured for this example.

Start screen

This command starts screen using our custom configuration file:

$ screen -c ~/.consoles.screenrc

Resume a screen session that is already running

This command resumes our specially named session (consoles) in multi-display mode.

$ screen -S consoles -x

In multi-display mode the user can remain connected to the screen from multiple separate SSH login sessions simultaneously.

Screen commands

The following commands are entered in screen by pressing Ctrl-A (Hold the Ctrl key and tap the a key), then release both and press the next key. For brevity, this will be abbreviated like ^A-d for pressing Ctrl-A then d.

^A-d - Detach
Leaves the session running in the background and returns the user to the shell.
^A-DD (Shift+D twice) - Power detach
Detach and logout in a single step.
^A-a - Toggle next/last screen
Changes between the current screen and the last used screen. If on the first screen, it switches to the next one. Press again to swap back and forth.
^A-<num> - Switch to a specific screen number
Changes to a specific numbered screen 0-9
^A-n - Next screen
Changes to the next screen in order
^A-p - Previous screen
Changes to the previous screen in order
^A-<esc> - Scrollback buffer mode
Enters scrollback history mode. Moving through the buffer depends on the terminal program and settings. It may move with arrow keys, hjkl, or even the mouse scroll wheel. Type ?<string> to search backward, /<string> to search forward. Press Esc again to exit.
^A-\ - Quit
Terminates screen.
^A-: - Screen command prompt
Accepts a text command from the user. For example, to write the contents of the entire scrollback buffer to a file, use :hardcopy -h outfile.txt.
^A-? - Help
Displays a help page with a list of commands and other information.

Finish up

With this in place, you can detach from the screen session and leave it running indefinitely and reattach as needed. This is great for monitoring consoles in case something goes wrong and prints logs or console messages for debugging.

Updated: