Mystery process adding "options rotate" to resolv.conf file

[peter-h]

I have a server, centos / nginx, whose content is just static html. No PHP, etc. No interactive stuff. No JS sent to the client browser.

I have a guy who looks after some sites for me. He is pretty familiar with unix stuff and databases...

The server has a cron job which pings a healthcheck site every hour so we can see if the server is down. The websites on it work but this ping stopped a month ago.

Something is messing with the resolv.conf file. It was found to contain no nameserver entries (which presumably is why pinging hc-ping.com was failing) and contains some 80000 lines with "options rotate", one being added every 80 seconds.

But there is no cron job doing this. But there must be a "script" somewhere which is running. But where?

The server is a virtual server and was originally set up about 20 years ago. It was updated to latest OS maybe 10 years ago.

Access (updates) is done with rsync, from a local copy held on a PC. There is also sftp and ftp access.

Does this ring a bell with anyone? It could be an attack (which failed to do much) or it could be some accident.

[madires]

Could be some DHCP automation gone wild. A 'cd /etc' and 'grep -R "options rotate"' might find the source.

[magic]

You could start with something like fuser or lsof to see if any process has this file opened constantly.
Unfortunately, I don't know how to wait for some process to open a file and log which process it was.

You could also make the file read only and see if it helps and see if any daemon logs errors because of it.

I'm a big sys*^&d hater so it surely must be this POS gone rogue, somehow

[peter-h]

Thank you

A partial success: /etc/dhcp/dhclient-exit-hooks. Supposedly it runs whenver dhcp gets or loses IP address. Currently it does echo 'options rotate' >> /etc/resolv.conf. Now why would somebody have set this up?

The file was set to R/O.

Doesn't sound like a hacker because a hacker with that access could have done much more damage.

The DNS IP disappeared from that file around 21 Nov 2024.

Turns out that an image restore was performed on that server on that day. It was done to reverse some experimental work on directory listing privileges. Why the restored image buggered up that DNS IP file, is a mystery.

Anyway, this has provided an excuse for some cleanup work, like log rotation, and removing no longer used services like pop boxes

[Postal2]

... Why the restored image buggered up that DNS IP file, is a mystery. ...

What "mysteries" can there be if you restore the system from an image and it does not boot after that.

It's clear that toothbrushes and plunger are to blame for everything.

[peter-h]

Oh dear, ChatGPT a.k.a. Postal2 (from Russia) is back in action.

[Siwastaja]

Oh dear, ChatGPT a.k.a. Postal2 (from Russia) is back in action.

I wonder what kind of ChatGPT prompt one needs to use to get this output.

[peter-h]

IF the image operation was the cause, it is interesting how it could have caused it, because the virtual hosting company shuts down the server before doing the image backup.

Or at least I think they do! Maybe they don't, because the site would go dead for the few mins it takes to do it. In that case it is not surprising that there are file system inconsistencies. I can also do a manual snapshot and for that I can do a shutdown.

Same as doing a Trueimage backup for example. That does a "shadow copy" (whatever that is). Doing the backup from a boot DVD of TI would avoid that issue.

[Postal2]

... I think they do! Maybe they don't, ....

RAID, second copy stops.

[Nominal Animal]

the virtual hosting company shuts down the server before doing the image backup. Or at least I think they do! Maybe they don't, because the site would go dead for the few mins it takes to do it. In that case it is not surprising that there are file system inconsistencies. I can also do a manual snapshot and for that I can do a shutdown.

I use LVM2 for filesystem snapshots, and that yields true filesystem state (including page cache, i.e. kernel-cached data; not just data on-disk) at that point in time without delaying future writes, because the snapshot is copy-on-write: the first write to the state copies that sector/chunk.  As a result, one needs to keep some disk space in reserve for the snapshots, but that's about it.  Note that it isn't just the data on disk; dirty pages in page cache are included in the snapshot, so it does not matter whether some parts of the filesystem state is in memory or not.  Whether you run sync before snapshotting or not, does not affect the contents of the snapshot.

For the image to be consistent, you want the system to be quiescent: not actively modifying files, with all changes in userspace service/application/library memory buffers written to disk.  Note that you only need it for a fraction of a second, for the duration of creating the snapshot using LVM2 tools; the snapshot itself remains usable afterwards, while the parent filesystem is in normal use.  (There are some tricks, like creating a preliminary snapshot without quieting the system, then updating the snapshot when quiescent, to reduce the real-world impact of the snapshotting.  Most users automate all this into scripts or maintenance panels.)

Some services, like nginx and Apache, are easy to stop for the duration of the snapshot, to ensure quiescent state; some have dedicated options for this for userspace snapshotting.  Some services, like systemd and associated services, are impossible to stop/sync/quieten this way without essentially rebooting the userspace, because they're not designed to support it.

(In addition to experimenting with replacing the standard C library with something more suitable for today, especially including asynchronous I/O support, I've also investigated exactly what kind of init system would allow this, and monitoring service status, without the tendency and vulnerability to single point failures and security problems that systemd type of designs have.  I've also done quite a lot of security-sensitive service development, although I'm no cryptographer (and rely on proper professionals on that front and many others), so if anyone is interested in discussing these further, just drop me a PM or an email, or start a new thread here.  A particular detail is secure Nginx/Apache configuration, when using server-side scripts; also when having a complex hierarchy of web administrators, and their access controls via group membership and Unix access controls.)

When using complete virtual hosts, the VM snapshot includes both the "processor" state and RAM, and the filesystem(s).  When using local area storage with sufficient caching (in the gigabytes), a virtual image can be transferred from one hardware to another within a fraction of a second, limited only by the throughput of the local area network and the size of the RAM image and processor state.  (Networked filesystem state does not need to be transferred.  A single-processor 1GB image using 10G networking can be transferred within a couple of seconds, easily fast enough for not even TCP/IP connections to break, only a couple-second "hiccup" during transfer.)  Because these snapshots includes the processor state and RAM, there is no need to make the system quiescent before snapshotting.  The downside is the small overhead from running a virtualized OS instead of on the bare metal.  When filesystem storage is network-connected also, and you don't use GPGPU or other external hardware computing accelerators, the difference to bare metal is basically neglible.  (I'm not up to date how well Nvidia stuff is virtualized/virtualizable to a specific VM, though.)

[Postal2]

...For the image to be consistent, you want the system to be quiescent: not actively modifying files, ...

I know from my own experience that if you are working in the VC++ 6.0 environment with a project and your program gets a blue screen, then you will be collecting pieces of your code from fragments of chains restored into separate files. That is, the files are open, saved, the system is in a state of rest. You interrupt the computer's power supply and suffer for a long time after that. Accordingly, having made an instant snapshot of the disk under these conditions, you will suffer similarly.

There is a special file system for Linux:
https://www.axllent.org/docs/btrfs-raid1/

[Nominal Animal]

I know from my own experience that if you are working in the VC++ 6.0 environment with a project and your program gets a blue screen, then you will be collecting pieces of your code from fragments of chains restored into separate files. That is, the files are open, saved, the system is in a state of rest. You interrupt the computer's power supply and suffer for a long time after that. Accordingly, having made an instant snapshot of the disk under these conditions, you will suffer similarly.

No.  That may be how Windows works, or how VC++ 6.0 works under Wine, but it definitely isn't how Linux and applications in Linux work.

You do need to issue a syncfs() on the mounted snapshot filesystem (sync -f any-file-on-the-mounted-snapshot) to ensure the snapshot data on the storage medium contains all cached data, but other than additional I/O load, it will not affect the use of the original filesystem.

I do not use VC++ 6.0, so I do not know and have not investigated how it quietens its own state.  All editors I've used in Linux do use temporary files, either in user-specific temporary directory (under /tmp/application-and-username/, /var/tmp/application-and-username/, ~username/.application/, or ~username/.cache/application-or-tempname/) or in a temporary subdirectory (name starting with a . to "hide" it from normal file listing, by convention) to the original file.  This is not kept instantly up to date, but auto-saved on a varying frequency.

All these editors fall into one of the two categories, similar to Nginx/Apache and systemd: either they quieten easily or automatically, or they do not at all.  It is up to the application and service designer/developer to decide whether state is kept in recoverable files, or just in RAM.  It is trivial to detect the loss of focus in graphical environments, and use a timeout for user input in any environment, as a trigger for flushing recoverable state to disk and fsync() it in a helper thread (not hindering the other threads of the process in any way).  If state is only maintained in RAM –– which it often is, since it is easiest, and most developers don't care about weird use cases or snapshots or error situations ––, then you're better off running it in a Virtual Machine, and snapshotting that VM, as snapshotting just the filesystem will not capture the full state.

There is a special file system for Linux:
https://www.axllent.org/docs/btrfs-raid1/

which will do absolutely nothing about the fundamental problem here, and not perform any better wrt. filesystem snapshots compared to LVM2, assuming you use a journaling filesystem like ext4 (default in most Linux distributions) and multi-device storage (RAID1/5/6/10 etc.).

[Postal2]

...  All editors I've used in Linux do use temporary files, either in user-specific temporary directory....

Try turning off the power unexpectedly for the system or removing the battery from the laptop.

Or, for the programmer, send zeros to the RAM controller registers.

[Nominal Animal]

...  All editors I've used in Linux do use temporary files, either in user-specific temporary directory....

Try turning off the power unexpectedly for the system or removing the battery from the laptop.

It's happened to me often enough with my small Linux SBC's.  I know very, very well how to deal with it in Linux.  The only thing that tends to get harmed is if I have to use an SD card for the storage; they really don't like losing power in the middle of a write, generally speaking.

What happens is that data already written to the filesystem journal (I use the default 30 second time limit) is autorecovered on the next boot, and data not written to disk yet is lost.  Applications that use temporary files that they apply fsync() on will be able to recover from the crash.

There is no scenario in Linux where application data would appear in random files on the filesystem.  LVM2 does not affect this, nor does mdraid/dmraid/hardware RAID.

Linux uses the inode model, where filesystem entries (file and directory names) are opaque byte sequences delimited by NUL (0), path components delimited by / (47), and each refers to an inode.  An inode contains both the data and metadata (size, owner user and group, access mode, extended attributes).  An anonymous temporary file is one where the filesystem entry ("name") has been deleted, but the file is open by at least one process.  An unnamed temporary file is one opened using the O_TMPFILE flag, specifying only the directory where the unnamed temporary file may be named later using the linkat() syscall, in Linux.  If the operating system crashes, these will be lost; although a custom invocation of fsck can recover these into named files on many filesystems, including ext4.  If recovered, these will have the contents that were written to disk (or journaled) before the OS crashed.  In no case will the temporary file data be scattered or confused somehow.

Windows, even NTFS, works very differently, so the behaviour between Linux and Windows (including WSL, and when running Windows applications in Linux using Wine), is very different.

As to the resolv.conf mess, I would inspect the resolver part of systemd.  Standard C library reads /etc/resolv.conf at run time when certain POSIX functions are run, so network service managers often replace it dynamically whenever the configuration changes, keeping the actual configuration somewhere else.  (Similarly for nsswitch.conf for name services in general, although services typically do not need to modify this one at run time.)

If you write C code, in Linux you should use POSIX getaddrinfo() for name-to-address translation, and getnameinfo() for address-to-name translation, not gethostbyname()/gethostbyaddr().  The underlying issue is the same as open-coding opendir()/readdir()/closedir(), instead of using the POSIX nftw() or scandir() or glob(), the latter having been available in the same standard C libraries for decades.  (Well, except that gethostbyname() and gethostbyaddr() are actually deprecated already nowadays.)

[DiTBho]

@Nominal Animal
I have observed something similar in the past, in two occasions

• human error due to high stress
  I usually use the tab-key to select a file on the shell, one night I wrongly select the target of the "cat" command
  and I only discovered the error weeks later, when I rebooted the router and found that a server was not starting as usual. Of course it wasn't, one of its configuration files had been overwritten with something that didn't make sense for that server, for example imagine putting "/system/config/ssmp" in /system/config/sshd" ...
• kernel bug
  The HP C3600 workstation had a serious bug, I posted a patch years ago on the Linux ML.
  I remember, I was working on an experimental kernel and testing a PCI sATA HBA card I had just purchased on eBay. It passed all the tests I ran in 24 hours. Too bad there was a remote possibility of catastrophic failure, due to a bug involving both the HPPA2 CPU and the DMA subsystem, all of which in 99.99% of the cases do not happen, but in that 0.01% of probability, when it does happen... the consequences are catastrophic: in the case... parts of the content of files opened in RW ending up in the wrong place because the DMA wrote the wrong thing in the right place, and the kernel did not notice.
  
  The result was... that I opened /etc/profile, and found inside a piece of C code that I was working on in the terminal...
  Of course, being an experimental machine, ram-rootfs. So, no serious damage.
  

 

[DimitriP]

There are few "mysteries" when it comes to "computers going wrong" and it usually involves someone else having done something that they have disclosed to no one other than "there is an issue that cropped up".

As I was "reading the matrix" from what the OP stated,  here is what was underlined, highlighted and blinking:

I have a server,

I have a guy who looks after some sites for me. He is pretty familiar with unix stuff and databases.

The server has a cron job which pings a healthcheck site every hour so we can see if the server is down. The websites on it work but this ping stopped a month ago.

Something is messing with the resolv.conf file. It was found to contain no nameserver entries

But there is no cron job doing this. But there must be a "script" somewhere which is running. But where?

The server is a virtual server and was originally set up about 20 years ago. It was updated to latest OS maybe 10 years ago.

So far the short version of this is "suddenly it started doing wierd things".


...and finally the beans are spilled: 

The DNS IP disappeared from that file around 21 Nov 2024.
Turns out that an image restore was performed on that server on that day.

Aha!

It was done to reverse some experimental work on directory listing privileges.

Suuuuuure....Someone was screwing around, and instead of undoing their work - probably because they weren't sure they knew what they were doing in the firstplace - decided to "put it back" from "the image"
Then a little bit of hell broke loose and they washed their hands of it  because "I restored from the image - everything should be working" . Except it wasn't.

And here we are.

If all these events have the same person in common... you need a new person or keep your fingers crossed for the next time something similar goes wrong;  "suddenly"!

EOT
 

(fixed spelling of someone , probably & because )

[peter-h]

Hardly.

[Postal2]

...  All editors I've used in Linux do use temporary files, either in user-specific temporary directory....

Try turning off the power unexpectedly for the system or removing the battery from the laptop.

It's happened to me often enough with my small Linux SBC's.  I know very, very well how to deal with it in Linux.  ......

Now you know what's going on in Windows, the lost megabytes of code will be in the file123.chk files in small pieces (after checking the disk). And when I found these pieces for the first time, I was very happy.

... If all these events have the same person in common...

I would also draw attention to this.

[5U4GB]

Just as a general observation, if something randomly changes your networking config behind your back and breaks things then 99% of the time it's systemd.  They really, really like screwing around with people's networking configs.

[Siwastaja]

Windows, even NTFS, works very differently

I had this happen to me in Windows (XP, NTFS) over a decade ago: at some point an mp3 music file was corrupted such that it contained random excerpts of other mp3 files. Apparently the mp3 file stream format itself was robust enough that mp3 player software decoded it just fine with minor glitching. It became an interesting collection of music clips of 1-2 seconds each.

Whether it was NTFS or Windows filesystem implementation failing, or some userland application going haywire, or even HW bug, I don't know. It was quite alarming nevertheless; with computers "anything can happen" even if it's not very usual.

I'm sure Linux does better, but one can never completely rule out weird esoteric kernel bugs etc.

[peter-h]

Thanks 5U4GB. I've passed that back.

Fundamentally it must be difficult to make image backups of running systems, especially if the OS uses temporary files to carry interprocess messages. In Windows, whenever I make a Trueimage backup I reboot the PC afterwards, but that is a different issue to restoring such a backup. I currently have a problem elsewhere where a TI backup is not restoring to a running system so I will re-do that with a boot TI CD, to make sure the machine is not running.

FWIW, the work which was being done on this machine prior to the image restore was setting up listing privileges on a particular directory, which I wanted to be invisible in a dir listing which currently is visible on a specific URL. That dir contains just one file which I wanted to keep hidden unless someone had the full path to it. I solved this easily by making it an aes256 encrypted zip. Normally this is done with htaccess but IIRC nginx uses a different system.

That conf file needs to only contain 8.8.8.8 so that's been done and it has been made R/O. That seems to have solved it. Possibly some error log will appear in due course showing what may be trying to write to it

[5U4GB]

As an example of sustemd's messing with things, from some old notes:

As of 18.04 systemd overrides DNS with systemd-resolved.service which tries to plaster over the use of the standard resolver but often just breaks it leading to DNS lookup failures, e.g. on apt-get update, with a typical error message 'unable to resolve host XXXX: Name or service not known'.

followed by something like a full A4 page of all the steps necessary to get it to stop breaking resolv.conf.  One thing that comes up from those is to check whether resolv.conf is a symlink, in this case to ../run/resolvconf/resolv.conf, which means the problem is coming in from there.

See also this link for another possible cause although that depends on use of udhcpc, including the malware-like behaviour where even the nuclear option of using chattr +i was bypassed.

[Nominal Animal]

I have observed something similar in the past, in two occasions

Sure –– but those kinds of bugs can really produce all sorts of weird effects and problems! 

That is, the symptoms match only because of random chance, not because they correlated somehow.

I had this happen to me in Windows (XP, NTFS) over a decade ago: at some point an mp3 music file was corrupted such that it contained random excerpts of other mp3 files.

It really does happen because of the different file system model in the Windows kernel.  In the MS-DOS era, on FAT12/16/32 volumes, this was even more common.  In both cases, it boils down to the file system model considering each allocated sector to be part of a file; whereas Unix/POSIX/Linux inode model combines file data and metadata (but not the name) into a logical object, that occupies a set of sectors.

In the inode model, when a file is extended the kernel grabs some unused sectors, and writes the new or modified data to them, and finally the inode is updated to reflect the modified size and which sectors are allocated to this file.  (Some filesystems also maintain a separate list or bitmap of unused sectors; this is usually updated last. POSIX fdatasync() returns when the data written to the file has been sent to the storage device, and fsync() returns when both the data and the inode updates have been sent to the storage device. In essence, this is integral to the inode model, and not just an implementation detail.

The end result is that if the kernel crashes or power is lost in the middle of a filesystem metadata update, NTFS and FAT will generate unnamed files from the sectors that were allocated and saved but metadata not yet reflected in the file entry (MFT), whereas for inode-based filesystems such data is simply lost: the lost data was essentially written to unallocated sectors.

I am not a filesystem specialist, and do not know exactly how much journaling helps (quantitatively, I mean).  Anecdotally/qualitatively, it seems to significantly reduce the time window for the above to happen, making it much rarer, but cannot completely eliminate it.

Now, the temptation would be to argue whether it is better to make the data recoverable instead of just discarding it.  Fact is, most filesystem research in the last fifty years has used the inode model as a basis, because it has proven more efficient and more robust; the time windows during which crashes lose data are much shorter than they are in simplistic filesystems like NTFS and FAT/exFAT.  (I'm not referring to Linux-originated ones like ext2/3/4 and btrfs, but XFS and ZFS, and research papers on file systems at ACM, and presentations at USENIX.)  To simplify, where FAT and NTFS were designed essentially for single-user workstation use, inode-model filesystems were designed to work for large multi-user concurrent systems where robustness is a key requirement.

All this also means that writing or regularly appending or modifying important data files in a reliable manner differs between Windows and POSIXy OSes. 

Similar to how serial port/terminal-like devices are handled completely differently (POSIX uses termios), these differences make writing portable applications, services and libraries less desirable, because portability requires compromises affecting efficiency and robustness.  (In particular, I've found all cross-platform serial libraries' POSIX implementations pretty much utter crap.  For data files, most developers don't even check their low-level calls for errors anyways ("I'll add them later", or "if that error happens, the user has bigger issues to worry about", and such inanities), so they mostly just throw data at the storage and hope it sticks, on all OSes.)

My solution to this is to ignore Windows compatibility, and limit myself to POSIXy systems.  It annoys me that others see this as dissing Microsoft, while it really is just about not willing to do the compromises required for compatibility between the completely different approaches.  I'm not even saying the Windows approach is any worse, just incompatible with what I like to use unless compromises that I do not want are made! (Although, I am saying NTFS is simplistic and not very good compared to ext4-on-LVM, XFS, or ZFS, for the basic reasons outlined above.)

Apologies for the rant-y post. 

[peter-h]

It was /etc/dhcp/dhclient-exit-hooks doing that .conf file "editing".

We think it started when I restored a backup, but it is not clear why resolv.conf was empty after that. And that silly script just kept adding "options rotate".

[Postal2]

... I am not a filesystem specialist, and do not know exactly how ....

I don't understand either, but I believe that the system will protect its files from failures, at least for Windows it is so. Then it turns out that the system does not consider the file resolv.conf to be its own, the owner of the file is someone else. And we come to the human factor again.