Mystery process adding "options rotate" to resolv.conf file

[peter-h]

I have a server, centos / nginx, whose content is just static html. No PHP, etc. No interactive stuff. No JS sent to the client browser.

I have a guy who looks after some sites for me. He is pretty familiar with unix stuff and databases...

The server has a cron job which pings a healthcheck site every hour so we can see if the server is down. The websites on it work but this ping stopped a month ago.

Something is messing with the resolv.conf file. It was found to contain no nameserver entries (which presumably is why pinging hc-ping.com was failing) and contains some 80000 lines with "options rotate", one being added every 80 seconds.

But there is no cron job doing this. But there must be a "script" somewhere which is running. But where?

The server is a virtual server and was originally set up about 20 years ago. It was updated to latest OS maybe 10 years ago.

Access (updates) is done with rsync, from a local copy held on a PC. There is also sftp and ftp access.

Does this ring a bell with anyone? It could be an attack (which failed to do much) or it could be some accident.

[madires]

Could be some DHCP automation gone wild. A 'cd /etc' and 'grep -R "options rotate"' might find the source.

[magic]

You could start with something like fuser or lsof to see if any process has this file opened constantly.
Unfortunately, I don't know how to wait for some process to open a file and log which process it was.

You could also make the file read only and see if it helps and see if any daemon logs errors because of it.

I'm a big sys*^&d hater so it surely must be this POS gone rogue, somehow

[peter-h]

Thank you

A partial success: /etc/dhcp/dhclient-exit-hooks. Supposedly it runs whenver dhcp gets or loses IP address. Currently it does echo 'options rotate' >> /etc/resolv.conf. Now why would somebody have set this up?

The file was set to R/O.

Doesn't sound like a hacker because a hacker with that access could have done much more damage.

The DNS IP disappeared from that file around 21 Nov 2024.

Turns out that an image restore was performed on that server on that day. It was done to reverse some experimental work on directory listing privileges. Why the restored image buggered up that DNS IP file, is a mystery.

Anyway, this has provided an excuse for some cleanup work, like log rotation, and removing no longer used services like pop boxes

[Postal2]

... Why the restored image buggered up that DNS IP file, is a mystery. ...

What "mysteries" can there be if you restore the system from an image and it does not boot after that.

It's clear that toothbrushes and plunger are to blame for everything.

[peter-h]

Oh dear, ChatGPT a.k.a. Postal2 (from Russia) is back in action.

[Siwastaja]

Oh dear, ChatGPT a.k.a. Postal2 (from Russia) is back in action.

I wonder what kind of ChatGPT prompt one needs to use to get this output.

[peter-h]

IF the image operation was the cause, it is interesting how it could have caused it, because the virtual hosting company shuts down the server before doing the image backup.

Or at least I think they do! Maybe they don't, because the site would go dead for the few mins it takes to do it. In that case it is not surprising that there are file system inconsistencies. I can also do a manual snapshot and for that I can do a shutdown.

Same as doing a Trueimage backup for example. That does a "shadow copy" (whatever that is). Doing the backup from a boot DVD of TI would avoid that issue.

[Postal2]

... I think they do! Maybe they don't, ....

RAID, second copy stops.

[Nominal Animal]

the virtual hosting company shuts down the server before doing the image backup. Or at least I think they do! Maybe they don't, because the site would go dead for the few mins it takes to do it. In that case it is not surprising that there are file system inconsistencies. I can also do a manual snapshot and for that I can do a shutdown.

I use LVM2 for filesystem snapshots, and that yields true filesystem state (including page cache, i.e. kernel-cached data; not just data on-disk) at that point in time without delaying future writes, because the snapshot is copy-on-write: the first write to the state copies that sector/chunk.  As a result, one needs to keep some disk space in reserve for the snapshots, but that's about it.  Note that it isn't just the data on disk; dirty pages in page cache are included in the snapshot, so it does not matter whether some parts of the filesystem state is in memory or not.  Whether you run sync before snapshotting or not, does not affect the contents of the snapshot.

For the image to be consistent, you want the system to be quiescent: not actively modifying files, with all changes in userspace service/application/library memory buffers written to disk.  Note that you only need it for a fraction of a second, for the duration of creating the snapshot using LVM2 tools; the snapshot itself remains usable afterwards, while the parent filesystem is in normal use.  (There are some tricks, like creating a preliminary snapshot without quieting the system, then updating the snapshot when quiescent, to reduce the real-world impact of the snapshotting.  Most users automate all this into scripts or maintenance panels.)

Some services, like nginx and Apache, are easy to stop for the duration of the snapshot, to ensure quiescent state; some have dedicated options for this for userspace snapshotting.  Some services, like systemd and associated services, are impossible to stop/sync/quieten this way without essentially rebooting the userspace, because they're not designed to support it.

(In addition to experimenting with replacing the standard C library with something more suitable for today, especially including asynchronous I/O support, I've also investigated exactly what kind of init system would allow this, and monitoring service status, without the tendency and vulnerability to single point failures and security problems that systemd type of designs have.  I've also done quite a lot of security-sensitive service development, although I'm no cryptographer (and rely on proper professionals on that front and many others), so if anyone is interested in discussing these further, just drop me a PM or an email, or start a new thread here.  A particular detail is secure Nginx/Apache configuration, when using server-side scripts; also when having a complex hierarchy of web administrators, and their access controls via group membership and Unix access controls.)

When using complete virtual hosts, the VM snapshot includes both the "processor" state and RAM, and the filesystem(s).  When using local area storage with sufficient caching (in the gigabytes), a virtual image can be transferred from one hardware to another within a fraction of a second, limited only by the throughput of the local area network and the size of the RAM image and processor state.  (Networked filesystem state does not need to be transferred.  A single-processor 1GB image using 10G networking can be transferred within a couple of seconds, easily fast enough for not even TCP/IP connections to break, only a couple-second "hiccup" during transfer.)  Because these snapshots includes the processor state and RAM, there is no need to make the system quiescent before snapshotting.  The downside is the small overhead from running a virtualized OS instead of on the bare metal.  When filesystem storage is network-connected also, and you don't use GPGPU or other external hardware computing accelerators, the difference to bare metal is basically neglible.  (I'm not up to date how well Nvidia stuff is virtualized/virtualizable to a specific VM, though.)

[Postal2]

...For the image to be consistent, you want the system to be quiescent: not actively modifying files, ...

I know from my own experience that if you are working in the VC++ 6.0 environment with a project and your program gets a blue screen, then you will be collecting pieces of your code from fragments of chains restored into separate files. That is, the files are open, saved, the system is in a state of rest. You interrupt the computer's power supply and suffer for a long time after that. Accordingly, having made an instant snapshot of the disk under these conditions, you will suffer similarly.

There is a special file system for Linux:
https://www.axllent.org/docs/btrfs-raid1/

[Nominal Animal]

I know from my own experience that if you are working in the VC++ 6.0 environment with a project and your program gets a blue screen, then you will be collecting pieces of your code from fragments of chains restored into separate files. That is, the files are open, saved, the system is in a state of rest. You interrupt the computer's power supply and suffer for a long time after that. Accordingly, having made an instant snapshot of the disk under these conditions, you will suffer similarly.

No.  That may be how Windows works, or how VC++ 6.0 works under Wine, but it definitely isn't how Linux and applications in Linux work.

You do need to issue a syncfs() on the mounted snapshot filesystem (sync -f any-file-on-the-mounted-snapshot) to ensure the snapshot data on the storage medium contains all cached data, but other than additional I/O load, it will not affect the use of the original filesystem.

I do not use VC++ 6.0, so I do not know and have not investigated how it quietens its own state.  All editors I've used in Linux do use temporary files, either in user-specific temporary directory (under /tmp/application-and-username/, /var/tmp/application-and-username/, ~username/.application/, or ~username/.cache/application-or-tempname/) or in a temporary subdirectory (name starting with a . to "hide" it from normal file listing, by convention) to the original file.  This is not kept instantly up to date, but auto-saved on a varying frequency.

All these editors fall into one of the two categories, similar to Nginx/Apache and systemd: either they quieten easily or automatically, or they do not at all.  It is up to the application and service designer/developer to decide whether state is kept in recoverable files, or just in RAM.  It is trivial to detect the loss of focus in graphical environments, and use a timeout for user input in any environment, as a trigger for flushing recoverable state to disk and fsync() it in a helper thread (not hindering the other threads of the process in any way).  If state is only maintained in RAM –– which it often is, since it is easiest, and most developers don't care about weird use cases or snapshots or error situations ––, then you're better off running it in a Virtual Machine, and snapshotting that VM, as snapshotting just the filesystem will not capture the full state.

There is a special file system for Linux:
https://www.axllent.org/docs/btrfs-raid1/

which will do absolutely nothing about the fundamental problem here, and not perform any better wrt. filesystem snapshots compared to LVM2, assuming you use a journaling filesystem like ext4 (default in most Linux distributions) and multi-device storage (RAID1/5/6/10 etc.).

[Postal2]

...  All editors I've used in Linux do use temporary files, either in user-specific temporary directory....

Try turning off the power unexpectedly for the system or removing the battery from the laptop.

Or, for the programmer, send zeros to the RAM controller registers.

[Nominal Animal]

...  All editors I've used in Linux do use temporary files, either in user-specific temporary directory....

Try turning off the power unexpectedly for the system or removing the battery from the laptop.

It's happened to me often enough with my small Linux SBC's.  I know very, very well how to deal with it in Linux.  The only thing that tends to get harmed is if I have to use an SD card for the storage; they really don't like losing power in the middle of a write, generally speaking.

What happens is that data already written to the filesystem journal (I use the default 30 second time limit) is autorecovered on the next boot, and data not written to disk yet is lost.  Applications that use temporary files that they apply fsync() on will be able to recover from the crash.

There is no scenario in Linux where application data would appear in random files on the filesystem.  LVM2 does not affect this, nor does mdraid/dmraid/hardware RAID.

Linux uses the inode model, where filesystem entries (file and directory names) are opaque byte sequences delimited by NUL (0), path components delimited by / (47), and each refers to an inode.  An inode contains both the data and metadata (size, owner user and group, access mode, extended attributes).  An anonymous temporary file is one where the filesystem entry ("name") has been deleted, but the file is open by at least one process.  An unnamed temporary file is one opened using the O_TMPFILE flag, specifying only the directory where the unnamed temporary file may be named later using the linkat() syscall, in Linux.  If the operating system crashes, these will be lost; although a custom invocation of fsck can recover these into named files on many filesystems, including ext4.  If recovered, these will have the contents that were written to disk (or journaled) before the OS crashed.  In no case will the temporary file data be scattered or confused somehow.

Windows, even NTFS, works very differently, so the behaviour between Linux and Windows (including WSL, and when running Windows applications in Linux using Wine), is very different.

As to the resolv.conf mess, I would inspect the resolver part of systemd.  Standard C library reads /etc/resolv.conf at run time when certain POSIX functions are run, so network service managers often replace it dynamically whenever the configuration changes, keeping the actual configuration somewhere else.  (Similarly for nsswitch.conf for name services in general, although services typically do not need to modify this one at run time.)

If you write C code, in Linux you should use POSIX getaddrinfo() for name-to-address translation, and getnameinfo() for address-to-name translation, not gethostbyname()/gethostbyaddr().  The underlying issue is the same as open-coding opendir()/readdir()/closedir(), instead of using the POSIX nftw() or scandir() or glob(), the latter having been available in the same standard C libraries for decades.  (Well, except that gethostbyname() and gethostbyaddr() are actually deprecated already nowadays.)

[DiTBho]

@Nominal Animal
I have observed something similar in the past, in two occasions

• human error due to high stress
  I usually use the tab-key to select a file on the shell, one night I wrongly select the target of the "cat" command
  and I only discovered the error weeks later, when I rebooted the router and found that a server was not starting as usual. Of course it wasn't, one of its configuration files had been overwritten with something that didn't make sense for that server, for example imagine putting "/system/config/ssmp" in /system/config/sshd" ...
• kernel bug
  The HP C3600 workstation had a serious bug, I posted a patch years ago on the Linux ML.
  I remember, I was working on an experimental kernel and testing a PCI sATA HBA card I had just purchased on eBay. It passed all the tests I ran in 24 hours. Too bad there was a remote possibility of catastrophic failure, due to a bug involving both the HPPA2 CPU and the DMA subsystem, all of which in 99.99% of the cases do not happen, but in that 0.01% of probability, when it does happen... the consequences are catastrophic: in the case... parts of the content of files opened in RW ending up in the wrong place because the DMA wrote the wrong thing in the right place, and the kernel did not notice.
  
  The result was... that I opened /etc/profile, and found inside a piece of C code that I was working on in the terminal...
  Of course, being an experimental machine, ram-rootfs. So, no serious damage.
  

 

[DimitriP]

There are few "mysteries" when it comes to "computers going wrong" and it usually involves somone else having done something that they have disclosed to no one other than "there is an issue that cropped up".

As I was "reading the matrix" from what the OP stated,  here is what was underlined, highlighted and blinking:

I have a server,

I have a guy who looks after some sites for me. He is pretty familiar with unix stuff and databases.

The server has a cron job which pings a healthcheck site every hour so we can see if the server is down. The websites on it work but this ping stopped a month ago.

Something is messing with the resolv.conf file. It was found to contain no nameserver entries

But there is no cron job doing this. But there must be a "script" somewhere which is running. But where?

The server is a virtual server and was originally set up about 20 years ago. It was updated to latest OS maybe 10 years ago.

So far the short version of this is "suddenly it started doing wierd things".


...and finally the beans are spilled: 

The DNS IP disappeared from that file around 21 Nov 2024.
Turns out that an image restore was performed on that server on that day.

Aha!

It was done to reverse some experimental work on directory listing privileges.

Suuuuuure....Someone was screwing around, and instead of undoing their work - proabably because they weren't sure they knew what they were doing in the firstplace - decided to "put it back" from "the image"
Then a little bit of hell broke loose and they washed their hands of it  becuase "I restored from the image- everything should be working" . Except it wasn't.

And here we are.

If all these events have the same person in common... you need a new person or keep your fingers crossed for the next time something similar goes wrong;  "suddenly"!

EOT
 

[peter-h]

Hardly.