Red Hat / Fedora :: Pseudo-ttys - Both BSD And SVR5 Versions Seem To Be Incomplete

I'm trying to complete the porting to RHEL 5 of a commercial application that we've had running on AIX and SCO for decades. It originally ran on dumb-terminals, and even now runs on dumb-terminal emulators that run on Windows. We have thousands of customers, many of whom we'd like to migrate to Linux. We're pretty much done with the port, with the exception of this little glitch we're having with pseudo-ttys...

Our app utilizes pseudo-ttys, such that sub-portions of the application run under a screen manager that provides for screen-switching of multiple instances of the sub-apps on non-GUI terminals, some of which are still running serial. In this regard, it is not unlike the GNU "screen" program. However, it also supports file transfers over serial links using zmodem, and requires the ability to switch in and out of a fully 8-bit transparent mode where the screen switching keyboard commands must be ignored.

On AIX, we're using the BSD TIOCUCNTL (UIOCCMD) capability to send commands from the sub-app to the screen manager. TIOCUCNTL provides custom user ioctls and is an ideal way to solve the problem. It is not implemented on Linux, and the "alternative" of TIOCPKT does not provide arbitrary ioctls so at best we'd have to commandeer some of the existing TIOCPKT ioctls for our own use if that's even possible. TIOCPKT seems like a hack, it apparently was implemented not as a general mechanism but to solve problems specific to rlogin/rlogind.

SCO didn't support TIOCUCNTL either, so years ago we used the SVR5 alternative of putmsg/getmsg. This is a streams-based mechanism of sending control information over streams separate from the data. While RHEL 5 seems to have man pages indicating these commands exist, and in fact you can link programs using them to libc without errors, they are apparently stubbed out and return "function not implemented." This seems to be, according to this Wiki, due to certain Linux factions deeming that streams are "technically inadequate":

Certainly, I would agree that a streams implementation that is incomplete is undoubtedly "technically inadequate."

At the same time, it would appear that the SVR5 method of pseudo-tty is now the recommended implementation, the BSD one being "deprecated" (not to mention, also incomplete on Linux, given TIOCUCNTL is missing):

So here I am trying to figure out how to solve the problem. I have found a library that apparently implements the getmsg/putmsg commands for Linux, called libLiS. However, according to IBM, it has a problem with SELinux:

We've had to disable SELinux for other reasons anyway, so this is not a show stopper for us, and I am actively exploring using it as our solution. However, I've never been exactly crazy about the idea of depending on disabling security features to make something work. But, there's something to be said for the compatibility it would offer for our apps.

However, it occured ot me that we are probably not the first to have this problem, and was wondering if perhaps there is another solution that we should consider.

It has occured to me to use named pipes, but this would be rather ugly--we'd have to maintain a directory full of hundreds of named pipes, probably named after the associated ptys that the sub-apps could use to issue control commands to their screen manager, and the manager would have to monitor these pipes for commands, etc.-- certainly possible, but it seems like it might be re-inventing the wheel a bit, or at the very least, is a somewhat kludgy work-around to the apparent lack of any user-customizable control mechanisms in the pseudo-ttys. And we'd have to support it in addition to the code for the other methods since those platforms aren't going away anytime soon (SCO death-throes notwithstanding).