Author Topic: make tmux & screen able to survive socket failure during session attached  (Read 2573 times)

0 Members and 1 Guest are viewing this topic.

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 4475
  • Country: gb
Usually if a socket error occurs when you re-attach a session you lose what you were doing.
Code: [Select]
# screen -R development
do stuff
CTRL A D  ---> de-attach ... (the process runs the background)
...
later
...
# screen -r development ---> re-attach
=====Zaap socket error========

Socket error might occur due to loss of wifi signal, either because the router or switch is turned off, or because there are serious noises, such as to cause packets to be lost and the tcp/ip timeout to be triggered.

To partially solve this, I am using a RS422 <==> optical-fiber connection between the remote server and the local terminal.
So I am working with GNU/Screen attached to /dev/ttyS3 (uart).

This way my working session survives a socket failure

Is there to do something similar with { screen, tmux }, on /dev/pts ? under ssh?

I can patch/modify/recompile { screen, tmux, whatever ... } from sources  :D
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 4475
  • Country: gb
( GNU/Linux running on both local terminal and remote server )
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline JohanH

  • Frequent Contributor
  • **
  • Posts: 703
  • Country: fi
Usually if a socket error occurs when you re-attach a session you lose what you were doing.

I don't understand your use case. I've lost connection hundreds of times without losing any work/data, that's the whole point with running screen or tmux on the remote server. I have some vague memory that some socket error would have happened at some point, but it must be more than 10 years ago.
 

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 4475
  • Country: gb
I don't understand your use case. I've lost connection hundreds of times without losing any work/data, that's the whole point with running screen or tmux on the remote server. I have some vague memory that some socket error would have happened at some point, but it must be more than 10 years ago.

I never noticed it before because I've never had such a poor connection as these days  :-//
As I said:
No problem with detached session. Which is the point of using screen. You put a process in the background.
The problem I have is when a re-attached something and then the ssh-connection is lost.
Mobaxterm on Windows { XP, 10 } exposes the same problem.

The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 4475
  • Country: gb
the error rate is up to 40% packets lost  :o :o :o
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 4475
  • Country: gb
i did another experiment
I unplugged the router cable between the terminal and the server, with a session attached
Code: [Select]
macmini2-intel # screen -R dev2
macmini2-intel # nano test.txt
macmini2-intel # CTRL A D
macmini2-intel # screen -r dev2
macmini2-intel # screen -list
There are screens on:
        16461.dev1     (Detached)
        19195.dev2     (Attached) <---------- I am here
2 Sockets in /system/screen/S-root.
waited 1 minut, which should be beyond the tcp timeout
replugged the router cable
Code: [Select]
macmini2-intel # screen -list
There are screens on:
        16461.dev1     (Detached)
        19195.dev2     (Attached)
2 Sockets in /system/screen/S-root.
macmini2-intel # screen -r dev2

no problem, nothing lost  :-// :-// :-//

So what the hell happens duing a serious network error?!?
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online Whales

  • Super Contributor
  • ***
  • Posts: 2148
  • Country: au
    • Halestrom
This does not sound like a network connectivity issue.  screen and tmux should not forget your session for any network related reason, that's not how they work.  That would be like saying you lost the contents of your apartment because there was an issue with the roads.

Can you provide the exact error message that mentions sockets?  All I can find in your post is this and I assume it's something you have hand written:

> =====Zaap socket error========

Keep in mind that "network sockets" are only one type of "socket", there are many non-network variants of sockets.  Tmux & screen will be using local (file or unix sockets) for some of their comms; if these are breaking then you might have more serious issues on your host that will make other software misbehave too.
« Last Edit: January 14, 2025, 11:46:58 pm by Whales »
 
The following users thanked this post: JohanH

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 4475
  • Country: gb
Can you provide the exact error message that mentions sockets? 

On GNU/Linux, there is any error message when the connection is lost.
On Windows, mobaxterm only says "network error".

If there was an error message, I would have at least Googled it.
Instead nothing, and dmesg doesn't say anything either.
The kernel doesn't show any clues.

When I reconnect over ssh,  screen -list continues to list all the various entries
but if the network connection has dropped on an attached-session it will be "zombied"
I mean it's there but I cannot be re-attached as the PID of the process is not even present in the running processes.
It's gone.

Last week, with a wired-p2p-connection between the server and the terminal,
both machiens were problem-free, with an uptime of 23 days, 24 hours a day.
« Last Edit: January 15, 2025, 12:05:07 am by DiTBho »
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 4475
  • Country: gb
Luckily I have a pair of fiber optic transceivers
which are seen as serial we are talking about 100m of optical cable
between where I am, and where the server is

And luckily it works at 119200bps
enough to work more than decently with nano, vim, gcc ... etc.
All in text console.

Anyway, it should be fixed, because it's very annoying  :o
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online Whales

  • Super Contributor
  • ***
  • Posts: 2148
  • Country: au
    • Halestrom
On GNU/Linux, there is any error message when the connection is lost.
On Windows, mobaxterm only says "network error".

If there was an error message, I would have at least Googled it.
Instead nothing, and dmesg doesn't say anything either.
The kernel doesn't show any clues.

Are you describing what happens when you see the initial disconnection, or when you see "socket error"?  Where did you get the words "socket error" from?  The exact error messages and their exact context are very important, they help us greatly in understanding what has gone wrong.


Quote
When I reconnect over ssh,  screen -list continues to list all the various entries
but if the network connection has dropped on an attached-session it will be "zombied"

Please post a full copy of screen -list when this happens.


Quote
I mean it's there but I cannot be re-attached as the PID of the process is not even present in the running processes.
It's gone.

I am not sure what you are trying to say.
(1) Screen lists a PID
(2) You can't re-attach to this PID (have you tried "screen -r -d ID" ?  The -d is important)
(3) The PID that screen lists is not longer on your system?  (that's a serious bug if true)
« Last Edit: January 15, 2025, 03:53:24 am by Whales »
 

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 4064
  • Country: us
Re: make tmux & screen able to survive socket failure during session attached
« Reply #10 on: January 15, 2025, 04:14:24 am »
Yes if a connection is dropped without being reset or closed, the screen client can stick around for a long time.  The default behavior of screen is to reattach to the first available disconnected screen instance.  You need to use the -d or -D option to detach the zombie client.
 
The following users thanked this post: JohanH, DiTBho

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 4475
  • Country: gb
Re: make tmux & screen able to survive socket failure during session attached
« Reply #11 on: January 15, 2025, 03:43:09 pm »
Code: [Select]
Network error: Software caused connection abort
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 4475
  • Country: gb
Re: make tmux & screen able to survive socket failure during session attached
« Reply #12 on: January 15, 2025, 03:56:42 pm »
Yes if a connection is dropped without being reset or closed, the screen client can stick around for a long time.
The default behavior of screen is to reattach to the first available disconnected screen instance.
You need to use the -d or -D option to detach the zombie client.

GNU/screen was emerged without -d/-D options.
Re-emerged. Now, it sucessfully resumes sessions!


However ...
... using MobaXterm (professional edition v21.5) as terminal on Windows 10 ... there's still something wrong.
In this case, something kills the attached process.

e.g. if under MabaXterm, connected to the remote GNU/Linux server via ssh, I run screen -R dev1, and then nano file.txt, ... if the network drops, the process GNU/nano is killed.

This is really weird,

I just found out that this only happens with a mobaxterm terminal on Windows.

I will try later to grab more information and post here. I can't access that laptop at the moment.
« Last Edit: January 15, 2025, 06:48:09 pm by DiTBho »
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 7320
  • Country: fi
    • My home page and email address
Re: make tmux & screen able to survive socket failure during session attached
« Reply #13 on: January 15, 2025, 04:03:03 pm »
The error is from MobaXterm, with the TCP/IP connection dropped due to inactivity.  Enable Settings > Configuration > SSH > SSH Keepalive in MobaXterm.
 
The following users thanked this post: DiTBho

Online Whales

  • Super Contributor
  • ***
  • Posts: 2148
  • Country: au
    • Halestrom
Re: make tmux & screen able to survive socket failure during session attached
« Reply #14 on: January 15, 2025, 07:43:18 pm »
e.g. if under MabaXterm, connected to the remote GNU/Linux server via ssh, I run screen -R dev1, and then nano file.txt, ... if the network drops, the process GNU/nano is killed.

This is really weird,

That's super bad.  That's not what you want at all.

SSH keepalives should be unrelated to this problem.

What distro are you on?  Maybe it's one of the many systemd antifeatures?  https://superuser.com/questions/1372963/how-do-i-keep-systemd-from-killing-my-tmux-sessions

Offline madires

  • Super Contributor
  • ***
  • Posts: 8333
  • Country: de
  • A qualified hobbyist ;)
Re: make tmux & screen able to survive socket failure during session attached
« Reply #15 on: January 15, 2025, 08:10:20 pm »
Have you disabled TCPKeepAlive in /etc/ssh/sshd_config at the server side? The default setting is 'yes' and would be the right choice.
 
The following users thanked this post: DiTBho

Online radiolistener

  • Super Contributor
  • ***
  • Posts: 4180
  • Country: 00
Re: make tmux & screen able to survive socket failure during session attached
« Reply #16 on: January 16, 2025, 08:52:19 am »
e.g. if under MabaXterm, connected to the remote GNU/Linux server via ssh, I run screen -R dev1, and then nano file.txt, ... if the network drops, the process GNU/nano is killed.

When the connection is unexpectedly dropped, the `nano` process on the remote machine is not terminated, it continues running in the background. The disconnection only affects your terminal SSH session. You can reconnect by starting a new terminal SSH session and attaching to the running `tmux` session that contains your `nano` process. All your work in nano is not lost, because it continue to run and you can finish your work from a new ssh session. The same all stdout/stderr output is not lost, it is stored and will be displayed when you reconnect again.

« Last Edit: January 16, 2025, 08:55:58 am by radiolistener »
 

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 4475
  • Country: gb
Re: make tmux & screen able to survive socket failure during session attached
« Reply #17 on: January 16, 2025, 11:05:38 am »
e.g. if under MabaXterm, connected to the remote GNU/Linux server via ssh, I run screen -R dev1, and then nano file.txt, ... if the network drops, the process GNU/nano is killed.

When the connection is unexpectedly dropped, the `nano` process on the remote machine is not terminated, it continues running in the background. The disconnection only affects your terminal SSH session. You can reconnect by starting a new terminal SSH session and attaching to the running `tmux` session that contains your `nano` process. All your work in nano is not lost, because it continue to run and you can finish your work from a new ssh session. The same all stdout/stderr output is not lost, it is stored and will be displayed when you reconnect again.

It shouldn't but it is! That's precisely the problem  :o :o :o

Using MobaXterm (Professional, pay-version) as ssh-terminal, when the connection is unexpectedly dropped(1), the GNU/nano process on the remote machine IS terminated!
  • ps -A | grep nano doesn't show GNU/Nano running
  • its pid is no more present in /proc/<pid>
  • launched as nano file.txt, I see a backup copy file.txt.save
Something somehow terminates it!

It's fixed using a GNU/Linux laptop as ssh terminal.
In this case the GNU/nano process was not terminated. I was still able to see its running PID, but before enabling the -d/-D options in GNU/Screen I was unable to resume the session.

What happens with MobaXterm seems much more drastic, because seriously GNU/Nano crashes, and it also happens with VIM, Joe, ... I tried other text editors. They are all terminated.

And you also have to manually delete the file that screen created for them (Screen says "defunct"), and that remains in ls /tmp/screen/$user like a corpse



server side (GNU/Linux), changes:
Code: [Select]
/etc/ssh/sshd_config
- TCPKeepAlive no
+ TCPKeepAlive yes

terminal side (GNU/Linux), changes:
recompiled app-misc/screen with -d/-D options.




(1) edit:
it happens with
a: 802.11 wireless stack, default rate control Minstrel algorithm, connections
b: wireless BATMAN-network mesh connections
c: wired/copper ethernet, 100-baseT connections

never happened with
d: wired/optical ethernet, 100-baseSX connections
« Last Edit: January 16, 2025, 11:34:16 am by DiTBho »
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 4475
  • Country: gb
Re: make tmux & screen able to survive socket failure during session attached
« Reply #18 on: January 16, 2025, 11:14:37 am »
Code: [Select]
# screen -list
There are screens on:
        25443.idp       (Detached)
        16461.nano      (Defunct)
        19195.deve      (Detached)
3 Sockets in /tmp/screen/S-toor

Code: [Select]
# file /tmp/screen/S-toor/*
/tmp/screen/S-toor/16461.nano:  socket
/tmp/screen/S-root/19195.deve:  socket
/tmp/screen/S-toor/25443.idp:   socket

Code: [Select]
# ps -A | grep nano

so, the socket is still there, but there is no GNU/Nano running .... perplexed  :-//
« Last Edit: January 16, 2025, 11:32:59 am by DiTBho »
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online radiolistener

  • Super Contributor
  • ***
  • Posts: 4180
  • Country: 00
Re: make tmux & screen able to survive socket failure during session attached
« Reply #19 on: January 16, 2025, 11:34:54 am »
It shouldn't but it is! That's precisely the problem  :o :o :o

Using MobaXterm (Professional, pay-version) as ssh-terminal, when the connection is unexpectedly dropped(1), the GNU/nano process on the remote machine IS terminated!

Technically, this is possible if your terminal application (such as MobaXterm or another terminal you are using) sends unauthorized commands, like Ctrl+X, in response to network errors. However, this is not an issue with tmux, it is a problem with your terminal client. If the terminal application sends unauthorized commands to the remote computer, the remote system cannot prevent this and is forced to execute them.

Did you tried other terminal clients, like PuTTY, Alacritty, etc?

It's fixed using a GNU/Linux laptop as ssh terminal.

It confirms, that the issue with your terminal MobaXterm application. Just use normal termial application, for example Alacritty, or check its settings.

In this case, there are no issues with tmux or sockets, the problem lies in the faulty behavior of your terminal client, which sends command to terminate your console app with no your request.
« Last Edit: January 16, 2025, 11:42:46 am by radiolistener »
 
The following users thanked this post: DiTBho

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 4475
  • Country: gb
Re: make tmux & screen able to survive socket failure during session attached
« Reply #20 on: January 16, 2025, 12:20:29 pm »
The error is from MobaXterm, with the TCP/IP connection dropped due to inactivity. 
Enable Settings > Configuration > SSH > SSH Keepalive in MobaXterm.

Applied, relaunched!
This solved the problem  :o :o :o
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 4475
  • Country: gb
Re: make tmux & screen able to survive socket failure during session attached
« Reply #21 on: January 16, 2025, 12:30:09 pm »
Technically, this is possible if your terminal application (such as MobaXterm or another terminal you are using) sends unauthorized commands, like Ctrl+X, in response to network errors.

It seems "SSH Keepalive" in MobaXterm options solves the problem.
This probably prevents unauthorized commands, such as Ctrl+X, from being sent in response to network errors  :-//


The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online radiolistener

  • Super Contributor
  • ***
  • Posts: 4180
  • Country: 00
Re: make tmux & screen able to survive socket failure during session attached
« Reply #22 on: January 16, 2025, 12:50:44 pm »
Even if this option solves the issue with disconnections, it doesn't solve the problem with your SSH client app, which sends unrequested commands to the remote host during the connection loss event. So, I strongly suggest using a different SSH client that handles connection loss properly, without sending unwanted commands.
 

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 4475
  • Country: gb
Re: make tmux & screen able to survive socket failure during session attached
« Reply #23 on: January 17, 2025, 02:26:23 pm »
Code: [Select]
Network error: Software caused connection abort

Today, again, network error on MobaXterm.
Wifi connection lost for 2 minutes.
Reconnect ... this time, thanks to the new "SSH Keepalive" option in MobaXterm, it worked!

Code: [Select]
macmini2-intel ~ # screen -list
There are screens on:
        25443.idp       (Attached)
        16461.suppa     (Detached)
        19195.deve      (Detached)
3 Sockets in /tmp/screen/S-root.

I was attached to session 25443.idp

Code: [Select]
macmini2-intel ~ # screen -d idp
[25443.idp detached.]
macmini2-intel ~ # screen -r idp
{
    app();

    return 0;


macmini2-intel new # mygcc-beauty
 + app.c: ok
 + app.h: ok
macmini2-intel new # make
- generating interface public for app ... done
- generating interface private for app ... done
- compiling app ... done
- linking to test


Great! Problem solved  :-+


edit:
I paid 60 euro for my MobaXterm Professional Edition, 1 license; it works very well with both vt-term and x11 stuff, and it's one of the few software accepted in various companies that put restrictions on the software you can/cannot install on your work Windows laptop, so I would have been very annoyed to have to replace it with something else.
« Last Edit: January 17, 2025, 02:33:02 pm by DiTBho »
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 7320
  • Country: fi
    • My home page and email address
Re: make tmux & screen able to survive socket failure during session attached
« Reply #24 on: January 17, 2025, 08:32:34 pm »
This probably prevents unauthorized commands, such as Ctrl+X, from being sent in response to network errors  :-//
More specifically, Linux uses the standard Unix/POSIX termios, where (by default)
  • EOT (ASCII 004, Ctrl+D; termios .c_cc[VEOF]) flushes the outgoing buffer; and if sent at the beginning of a line, read() will return 0, signifying end-of-file/end-of-stream/end-of-transmission (assuming termios ICANON is set)
  • ETX (ASCII 003, Ctrl+C; termios .c_cc[VINTR]) causes an INT signal to be sent to the processes having this as their controlling terminal (assuming termios ISIG is set)
  • FS (ASCII 034; termios .c_cc[VQUIT]) causes a QUIT signal to be sent to the processes having this as their controlling terminal (assuming termios ISIG is set)
  • SUB (ASCII 032, Ctrl+Z; termios .c_cc[VSUSP]) causes an TSTP signal to be sent to the processes having this as their controlling terminal (assuming termios ISIG is set)
The standard SSH keepalive is sending null packets, essentially transferring no payload data but ensuring the other end is still in contact.

You could compile and run the following stupid program I cobbled together, to log the exact cause of the error to the file you specify as the only command line parameter:
Code: [Select]
// SPDX-License-Identifier: CC0-1.0
// Author: Nominal Animal, 2025
//
// Compile using e.g.
//      gcc -Wall -O2 this-file.c -o executable-name
// and run using e.g.
//      ./executable-name /tmp/log-file-name
//
#define  _POSIX_C_SOURCE  200809L
#define  _GNU_SOURCE
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <signal.h>
#include <string.h>
#include <time.h>
#include <errno.h>

static int wrpp(const int fd, const char *ptr, const char *const end) {
    const int saved_errno = errno;
    while (ptr < end) {
        ssize_t  n = write(fd, ptr, (size_t)(end - ptr));
        if (n > 0) {
            ptr += n;
        } else
        if (n != -1) {
            // This is not actually allowed, so treat it as EIO error
            return EIO;
        } else {
            // Normally we'd retry on errno being EINTR, EAGAIN, or EWOULDBLOCK
            const int retval = errno;
            errno = saved_errno;
            return retval;
        }
    }
    errno = saved_errno;
    return 0;
}

static int wrerr(const char *msg) {
    if (msg && *msg)
        return wrpp(STDOUT_FILENO, msg, msg + strlen(msg));
    else
        return 0;
}

static int wrout(const char *msg) {
    if (msg && *msg)
        return wrpp(STDERR_FILENO, msg, msg + strlen(msg));
    else
        return 0;
}

static int wr(int fd, const char *msg) {
    if (fd == -1)
        return EBADF;
    else
    if (msg && *msg)
        return wrpp(fd, msg, msg + strlen(msg));
    else
        return 0;
}

static volatile sig_atomic_t  caught_int = 0;
static volatile sig_atomic_t  caught_hup = 0;
static volatile sig_atomic_t  caught_term = 0;
static volatile sig_atomic_t  caught_pipe = 0;
static volatile sig_atomic_t  caught_quit = 0;
static volatile sig_atomic_t  caught_tstp = 0;

static void catch_int(int signo)  { (void)signo; caught_int = 1; }
static void catch_hup(int signo)  { (void)signo; caught_hup = 1; }
static void catch_term(int signo) { (void)signo; caught_term = 1; }
static void catch_pipe(int signo) { (void)signo; caught_pipe = 1; }
static void catch_quit(int signo) { (void)signo; caught_quit = 1; }
static void catch_tstp(int signo) { (void)signo; caught_tstp = 1; }

static const struct {
    const char *const name;
    void      (*const handler)(int);
    const int         signo;
} signal_list[] = {
    { "INT",  catch_int,  SIGINT  },
    { "HUP",  catch_hup,  SIGHUP  },
    { "TERM", catch_term, SIGTERM },
    { "PIPE", catch_pipe, SIGPIPE },
    { "QUIT", catch_quit, SIGQUIT },
    { "TSTP", catch_tstp, SIGTSTP },
};
#define  signal_list_count  (sizeof signal_list / sizeof signal_list[0])

static int catch_signal(int signo, void (*catcher)(int)) {
    struct sigaction  act;

    memset(&act, 0, sizeof act);
    sigemptyset(&act.sa_mask);

    act.sa_handler = catcher;
    act.sa_flags = 0;   // In particular, no SA_RESTART

    if (sigaction(signo, &act, NULL) == -1)
        return errno;

    return 0;
}

static const char *timestamp(const char *errmsg) {
    static char      buf[128];
    time_t           now_time;
    struct tm        now;
    size_t           len;

    now_time = time(NULL);
    if (localtime_r(&now_time, &now) != &now) {
        errno = ENOTSUP;
        return errmsg;
    }
    len = strftime(buf, sizeof buf, "%Y%m%d %T", &now);
    if (len < 1 || len >= sizeof buf) {
        errno = ENOMEM;
        return errmsg;
    }

    return (const char *)buf;
}

int main(int argc, char *argv[]) {
    const char *arg0 = (argc > 0 && argv && argv[0] && argv[0][0]) ? argv[0] : "(this)";
    int         logfd;

    // Verify we have exactly two arguments.
    if (argc != 2 || !argv[1] || !argv[1][0] || !strcmp(argv[1], "-h") || !strcmp(argv[1], "--help")) {
        wrerr("\n"
              "Usage: "); wrerr(arg0); wrerr(" [ -h | --help ]\n"
              "       "); wrerr(arg0); wrerr(" LOGFILE\n"
              "\n"
              "This program runs until killed by a signal or loss of terminal,\n"
              "saving the cause in write-only file LOGFILE.\n"
              "\n");
        return EXIT_SUCCESS;
    }

    // Install signal handlers.
    for (size_t i = 0; i < signal_list_count; i++) {
        if (catch_signal(signal_list[i].signo, signal_list[i].handler)) {
            const char *msg  = strerror(errno);
            const char *name = signal_list[i].name;
            wrerr("Cannot install SIG"); wrerr(name); wrerr(" signal handler: "); wrerr(msg); wrerr(".\n");
            return EXIT_FAILURE;
        }
    }

    // Open log file.
    do {
        logfd = open(argv[1], O_WRONLY | O_CREAT | O_TRUNC | O_NOCTTY | O_CLOEXEC, 0666);
    } while (logfd == -1 && errno == EINTR);
    if (logfd == -1) {
        const char *errmsg = strerror(errno);
        wrerr(argv[1]); wrerr(": "); wrerr(errmsg); wrerr(".\n");
        return EXIT_FAILURE;
    }

    // Write started message to standard output and log file.
    {
        const char *const when = timestamp("");
        const char *const what = " Waiting for end of input or terminal to close.\n";
        wrout(when);
        wrout(what);
        wr(logfd, when);
        wr(logfd, what);
    }

    int  err = 0;

    while (1) {
        char  buf[4096];
        ssize_t n;

        n = read(STDIN_FILENO, buf, sizeof buf);
        if (n > 0) {
            continue;
        } else
        if (n == 0) {
            err = 0;
            break;
        } else
        if (n != -1) {
            // Should never happen
            err = EIO;
            break;
        } else {
            err = errno;
            break;
        }
    }

    {
        const char *const when = timestamp("");
        const char *const what = err ? strerror(err) : "End of input";

        wr(logfd, when);
        if (caught_int)  wr(logfd, " SIGINT,");
        if (caught_hup)  wr(logfd, " SIGHUP,");
        if (caught_term) wr(logfd, " SIGTERM,");
        if (caught_pipe) wr(logfd, " SIGPIPE,");
        if (caught_quit) wr(logfd, " SIGQUIT,");
        if (caught_tstp) wr(logfd, " SIGTSTP,");
        wr(logfd, " ");
        wr(logfd, what);
        wr(logfd, ".\n");

        fsync(logfd);
        close(logfd);
    }

    return EXIT_SUCCESS;
}
If you run it under tmux or screen with the MobaXterm SSH Keepalive option disabled, and manage to reproduce the problem, the log file would tell us exactly why the target process died.

The above is not a good program for this at all: it would be much better for it to use sigwait()/sigwaitinfo() to catch all signals and log them to a file with a timestamp, also logging the number of characters received (when received!) from the terminal.  However, I did it the silly-simple way, because DiTBho probably uses a MIPS target machine, and I didn't want to mix in stuff like pthreads and locking.
« Last Edit: January 17, 2025, 08:34:23 pm by Nominal Animal »
 
The following users thanked this post: DiTBho


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf