General > General Technical Chat

Server Error Reports

<< < (133/142) > >>

gnif:

--- Quote from: ledtester on August 12, 2023, 03:42:59 am ---
--- Quote from: gnif on August 11, 2023, 11:43:21 am ---
It does seems to be a coincidence, the Wordpress website runs on a different version of PHP which was unaffected and is isolated from the forum. ...

--- End quote ---

Out of curiosity, what is the nature of the isolation?

--- End quote ---

Sorry but we do not divulge details related to the security of the server

gnif:

--- Quote from: kcbrown on August 11, 2023, 10:02:56 pm ---
--- Quote from: gnif on August 11, 2023, 11:43:21 am ---It does seems to be a coincidence, the Wordpress website runs on a different version of PHP which was unaffected and is isolated from the forum. Anything more is just speculation, the processes would have needed to have been run under a debugger with debug symbols to catch the fault and obtain any useful debugging information (which would kill server performance while waiting for it to fault again, if ever it decides to)

--- End quote ---

It depends on the degree to which the optimizer switches affect the performance of the server, among other things (e.g., do you need to set a memory watchpoint?).  It's possible to have debug symbols compiled in while the optimizer switches remain the same, but whether you get anything useful as a consequence when the fault occurs just depends.  Note that there's a huge difference between being able to see stack traces and being able to step through the code in a sane manner, and optimization affects the latter a lot more than the former.  There are some optimizations that could potentially interfere with getting a proper stack trace (e.g., -fomit-frame-pointer) but in reading up on it a bit it seems that it may have limited to no impact on x86-64.  I would expect gdb these days to be able to reconstruct the call stack even if -fomit-frame-pointer was in use, based on what I've read about what it does.

That leaves only the performance impact to the running binary.  Attaching a debugger in and of itself will do nothing to the performance of the running process unless the operating system underneath is horribly coded.  A segfault is something the operating system will detect and an attached debugger will be notified of that as soon as it happens, at which point the debugger can be used to examine the stacks and other things.  The debugger will be idle up until that point, and the running process will proceed as normal up until that point.

The main issue I've seen with debugging segfaults like that is that they often involve smashed stacks, and if that's happening then tracking them down might prove difficult at best.  Just enabling the stack protection in the compiler (-fstack-protector or -fstack-protector-all) could easily have a significant performance impact (I've never experimented with it so I really can't say), and if I understand the mechanisms it uses correctly, it's hit-or-miss anyway (meaning, something can scribble on your stack frame in such a way that it doesn't tickle the canary value), and only tells you which function's stack frame was smashed, not which code performed the smash.

Bottom line: whether or not you get a performance hit will depend on what exactly you're trying to chase down.  The faults you mentioned might be against the heap, or might be against the stack, or might even be against the heap as a result of improper changes to the stack.  There's no way to know without venturing down the rabbit hole.

--- End quote ---

Obviously having compiled symbols in the binary are an option however packages provided by upstream do not do this and as we are running production binaries when this fault occurred we have nothing to go on. As this was a random occurrence and seems to have no regularity to it at this point (it may never even happen again) there is little point building custom binaries with debug symbols enabled making it difficult to keep updated with the latest security releases upstream as they become available.

Also attaching gdb does have a performance hit always, if you write low latency multi-threaded applications you will know that sometimes a bug is harder to reproduce when running under gdb as it slows things down enough to "fix" problems such as race conditions.

The PHP-FPM model uses FastGGI, and forks a set of child PHP processes which have a limited number of requests to serve before they are each restarted, this is done to avoid problems caused by potential memory leaks and is a common FastCGI strategy. The overhead of gdb following forks is quite heavy.

Please note that I have extensive experience in software debugging and development where my primary development IDE is ViM & gdb. I appreciate the input but in this instance at this point the cost vs benefit currently makes this not worth the effort when a simple service upgrade may resolve an issue that may have already been fixed upstream.

gnif:
The server outage this morning is being investigated, there is no need to report this. Cause is as of yet unknown, the database service had stopped responding.

Brumby:

--- Quote from: gnif on August 12, 2023, 09:23:33 am ---Sorry but we do not divulge details related to the security of the server

--- End quote ---

That is the first rule of security for me: Don't provide any information.

Why give anyone a head start for hacking?

(Sorry gnif, I didn't mean to trigger a notification)

Karel:
Security through obscurity... hmmm...

Navigation

[0] Message Index

[#] Next page

[*] Previous page

There was an error while thanking
Thanking...
Go to full version
Powered by SMFPacks Advanced Attachments Uploader Mod