Author Topic: Server Error Reports  (Read 778062 times)

0 Members and 1 Guest are viewing this topic.

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1707
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Server Error Reports
« Reply #650 on: August 11, 2023, 09:50:02 am »
For an unknown reason the PHP process on the server had gone into a bad state throwing hundreds of general protection faults on all HTTP servers in the cluster. Reason for this is as of yet unknown, looks like a bug in the PHP process itself.
 
The following users thanked this post: MK14, beanflying

Offline MK14

  • Super Contributor
  • ***
  • Posts: 4952
  • Country: gb
Re: Server Error Reports
« Reply #651 on: August 11, 2023, 09:54:19 am »
Presumably that is why I haven't been able to get on to this forum, for the last few hours, until now.

Because of errors, which typically looked like this:

Quote
Max connect timeout reached while reaching hostgroup 1 after 10000ms
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1707
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Server Error Reports
« Reply #652 on: August 11, 2023, 09:57:28 am »
Looks like a database fault but it wasn't, the PHP process was in a bad state. I will be scheduling an upgrade tonight to try to resolve this. There should be no interruptions during this upgrade
 
The following users thanked this post: EEVblog, PA0PBZ, MK14

Offline EEVblog

  • Administrator
  • *****
  • Posts: 38720
  • Country: au
    • EEVblog
Re: Server Error Reports
« Reply #653 on: August 11, 2023, 11:28:30 am »
For an unknown reason the PHP process on the server had gone into a bad state throwing hundreds of general protection faults on all HTTP servers in the cluster. Reason for this is as of yet unknown, looks like a bug in the PHP process itself.

Almost certainly no coincidence it happened right after I tried to update Wordpress. Different databases though, and I dont recall Wordpress website problem taking down the forum before?
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1707
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Server Error Reports
« Reply #654 on: August 11, 2023, 11:43:21 am »
For an unknown reason the PHP process on the server had gone into a bad state throwing hundreds of general protection faults on all HTTP servers in the cluster. Reason for this is as of yet unknown, looks like a bug in the PHP process itself.

Almost certainly no coincidence it happened right after I tried to update Wordpress. Different databases though, and I dont recall Wordpress website problem taking down the forum before?

It does seems to be a coincidence, the Wordpress website runs on a different version of PHP which was unaffected and is isolated from the forum. Anything more is just speculation, the processes would have needed to have been run under a debugger with debug symbols to catch the fault and obtain any useful debugging information (which would kill server performance while waiting for it to fault again, if ever it decides to)
 
The following users thanked this post: EEVblog

Offline kcbrown

  • Frequent Contributor
  • **
  • Posts: 896
  • Country: us
Re: Server Error Reports
« Reply #655 on: August 11, 2023, 10:02:56 pm »
It does seems to be a coincidence, the Wordpress website runs on a different version of PHP which was unaffected and is isolated from the forum. Anything more is just speculation, the processes would have needed to have been run under a debugger with debug symbols to catch the fault and obtain any useful debugging information (which would kill server performance while waiting for it to fault again, if ever it decides to)

It depends on the degree to which the optimizer switches affect the performance of the server, among other things (e.g., do you need to set a memory watchpoint?).  It's possible to have debug symbols compiled in while the optimizer switches remain the same, but whether you get anything useful as a consequence when the fault occurs just depends.  Note that there's a huge difference between being able to see stack traces and being able to step through the code in a sane manner, and optimization affects the latter a lot more than the former.  There are some optimizations that could potentially interfere with getting a proper stack trace (e.g., -fomit-frame-pointer) but in reading up on it a bit it seems that it may have limited to no impact on x86-64.  I would expect gdb these days to be able to reconstruct the call stack even if -fomit-frame-pointer was in use, based on what I've read about what it does.

That leaves only the performance impact to the running binary.  Attaching a debugger in and of itself will do nothing to the performance of the running process unless the operating system underneath is horribly coded.  A segfault is something the operating system will detect and an attached debugger will be notified of that as soon as it happens, at which point the debugger can be used to examine the stacks and other things.  The debugger will be idle up until that point, and the running process will proceed as normal up until that point.

The main issue I've seen with debugging segfaults like that is that they often involve smashed stacks, and if that's happening then tracking them down might prove difficult at best.  Just enabling the stack protection in the compiler (-fstack-protector or -fstack-protector-all) could easily have a significant performance impact (I've never experimented with it so I really can't say), and if I understand the mechanisms it uses correctly, it's hit-or-miss anyway (meaning, something can scribble on your stack frame in such a way that it doesn't tickle the canary value), and only tells you which function's stack frame was smashed, not which code performed the smash.

Bottom line: whether or not you get a performance hit will depend on what exactly you're trying to chase down.  The faults you mentioned might be against the heap, or might be against the stack, or might even be against the heap as a result of improper changes to the stack.  There's no way to know without venturing down the rabbit hole.
 

Offline ledtester

  • Super Contributor
  • ***
  • Posts: 3248
  • Country: us
Re: Server Error Reports
« Reply #656 on: August 12, 2023, 03:42:59 am »

It does seems to be a coincidence, the Wordpress website runs on a different version of PHP which was unaffected and is isolated from the forum. ...

Out of curiosity, what is the nature of the isolation?
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1707
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Server Error Reports
« Reply #657 on: August 12, 2023, 09:23:33 am »

It does seems to be a coincidence, the Wordpress website runs on a different version of PHP which was unaffected and is isolated from the forum. ...

Out of curiosity, what is the nature of the isolation?

Sorry but we do not divulge details related to the security of the server
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1707
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Server Error Reports
« Reply #658 on: August 14, 2023, 05:29:59 am »
It does seems to be a coincidence, the Wordpress website runs on a different version of PHP which was unaffected and is isolated from the forum. Anything more is just speculation, the processes would have needed to have been run under a debugger with debug symbols to catch the fault and obtain any useful debugging information (which would kill server performance while waiting for it to fault again, if ever it decides to)

It depends on the degree to which the optimizer switches affect the performance of the server, among other things (e.g., do you need to set a memory watchpoint?).  It's possible to have debug symbols compiled in while the optimizer switches remain the same, but whether you get anything useful as a consequence when the fault occurs just depends.  Note that there's a huge difference between being able to see stack traces and being able to step through the code in a sane manner, and optimization affects the latter a lot more than the former.  There are some optimizations that could potentially interfere with getting a proper stack trace (e.g., -fomit-frame-pointer) but in reading up on it a bit it seems that it may have limited to no impact on x86-64.  I would expect gdb these days to be able to reconstruct the call stack even if -fomit-frame-pointer was in use, based on what I've read about what it does.

That leaves only the performance impact to the running binary.  Attaching a debugger in and of itself will do nothing to the performance of the running process unless the operating system underneath is horribly coded.  A segfault is something the operating system will detect and an attached debugger will be notified of that as soon as it happens, at which point the debugger can be used to examine the stacks and other things.  The debugger will be idle up until that point, and the running process will proceed as normal up until that point.

The main issue I've seen with debugging segfaults like that is that they often involve smashed stacks, and if that's happening then tracking them down might prove difficult at best.  Just enabling the stack protection in the compiler (-fstack-protector or -fstack-protector-all) could easily have a significant performance impact (I've never experimented with it so I really can't say), and if I understand the mechanisms it uses correctly, it's hit-or-miss anyway (meaning, something can scribble on your stack frame in such a way that it doesn't tickle the canary value), and only tells you which function's stack frame was smashed, not which code performed the smash.

Bottom line: whether or not you get a performance hit will depend on what exactly you're trying to chase down.  The faults you mentioned might be against the heap, or might be against the stack, or might even be against the heap as a result of improper changes to the stack.  There's no way to know without venturing down the rabbit hole.

Obviously having compiled symbols in the binary are an option however packages provided by upstream do not do this and as we are running production binaries when this fault occurred we have nothing to go on. As this was a random occurrence and seems to have no regularity to it at this point (it may never even happen again) there is little point building custom binaries with debug symbols enabled making it difficult to keep updated with the latest security releases upstream as they become available.

Also attaching gdb does have a performance hit always, if you write low latency multi-threaded applications you will know that sometimes a bug is harder to reproduce when running under gdb as it slows things down enough to "fix" problems such as race conditions.

The PHP-FPM model uses FastGGI, and forks a set of child PHP processes which have a limited number of requests to serve before they are each restarted, this is done to avoid problems caused by potential memory leaks and is a common FastCGI strategy. The overhead of gdb following forks is quite heavy.

Please note that I have extensive experience in software debugging and development where my primary development IDE is ViM & gdb. I appreciate the input but in this instance at this point the cost vs benefit currently makes this not worth the effort when a simple service upgrade may resolve an issue that may have already been fixed upstream.
 
The following users thanked this post: EEVblog, kcbrown, MK14

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1707
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Server Error Reports
« Reply #659 on: September 13, 2023, 12:29:16 am »
The server outage this morning is being investigated, there is no need to report this. Cause is as of yet unknown, the database service had stopped responding.
 
The following users thanked this post: Ed.Kloonk, PA0PBZ, xrunner, MK14, 2N3055, SiliconWizard, Nominal Animal

Offline Brumby

  • Supporter
  • ****
  • Posts: 12387
  • Country: au
Re: Server Error Reports
« Reply #660 on: September 17, 2023, 04:49:08 am »
Sorry but we do not divulge details related to the security of the server

That is the first rule of security for me: Don't provide any information.

Why give anyone a head start for hacking?

(Sorry gnif, I didn't mean to trigger a notification)
« Last Edit: September 17, 2023, 04:57:41 am by Brumby »
 

Offline Karel

  • Super Contributor
  • ***
  • Posts: 2267
  • Country: 00
Re: Server Error Reports
« Reply #661 on: September 17, 2023, 06:16:55 am »
Security through obscurity... hmmm...
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1707
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Server Error Reports
« Reply #662 on: September 17, 2023, 10:29:09 am »
Security through obscurity... hmmm...

You are misusing the term, the software in use is open source and available to the public as such, public scrutiny. How we have decided to make use of this software and build the EEVBlog infrastructure is kept private as this knowledge makes it easier to infiltrate the system if someone does get a foothold in some software/service. This is called "operational security" and is very different to "security through obscurity".
« Last Edit: September 17, 2023, 10:31:54 am by gnif »
 
The following users thanked this post: Brumby, MK14, 2N3055, bd139

Offline KE5FX

  • Super Contributor
  • ***
  • Posts: 2014
  • Country: us
    • KE5FX.COM
Re: Server Error Reports
« Reply #663 on: September 18, 2023, 06:20:24 am »
The term you're looking for may be "Defense in depth."  A lot of people confuse that with "Security through obscurity," but they aren't the same.
 

Offline beanflying

  • Super Contributor
  • ***
  • Posts: 7375
  • Country: au
  • Toys so very many Toys.
Re: Server Error Reports
« Reply #664 on: October 04, 2023, 12:44:39 pm »
Getting some patchy issues with the Server in the last day or so not being available and just had a prolonged outage.
Coffee, Food, R/C and electronics nerd in no particular order. Also CNC wannabe, 3D printer and Laser Cutter Junkie and just don't mention my TEA addiction....
 
The following users thanked this post: MK14

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1707
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Server Error Reports
« Reply #665 on: October 04, 2023, 01:25:17 pm »
We are aware, debugging an issue with the main website is having an impact on services.
 

Offline wilfred

  • Super Contributor
  • ***
  • Posts: 1263
  • Country: au
Re: Server Error Reports
« Reply #666 on: October 04, 2023, 02:30:43 pm »
Security through obscurity... hmmm...

You are misusing the term, the software in use is open source and available to the public as such, public scrutiny. How we have decided to make use of this software and build the EEVBlog infrastructure is kept private as this knowledge makes it easier to infiltrate the system if someone does get a foothold in some software/service. This is called "operational security" and is very different to "security through obscurity".

According to my dictionary the use of obscurity was spot on. Your desire to keep knowledge of the infrastructure private  and not well known is pretty much the dictionary's definition.

 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1707
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Server Error Reports
« Reply #667 on: October 04, 2023, 03:12:44 pm »
Security through obscurity... hmmm...

You are misusing the term, the software in use is open source and available to the public as such, public scrutiny. How we have decided to make use of this software and build the EEVBlog infrastructure is kept private as this knowledge makes it easier to infiltrate the system if someone does get a foothold in some software/service. This is called "operational security" and is very different to "security through obscurity".

According to my dictionary the use of obscurity was spot on. Your desire to keep knowledge of the infrastructure private  and not well known is pretty much the dictionary's definition.

Say whatever you like, we do not rely on the server being kept secure by omitting information as such this is not "security through obscurity". Limiting operational information to the general public is a no brainier as it gives administrators more time to combat attacks in progress that are trying to probe for such information to progress the attack. Sure the attacker may not be able to break in at all due to the security layers in place, but why risk it? Why hand out a map to the treasure even though the treasure box is padlocked?

Again, we are using open software with industry standard well known and reviewed security implementations. We are not relying on security though some custom logic that is closed and has not been peer reviewed. We are not "secure" because we rely on the fact that people don't know the specifics of how things are configured.

If we are to apply your definition of "security by obscurity" one could argue that the authentication credentials used to access the servers are kept secret and as such is "security by obscurity".
« Last Edit: October 04, 2023, 03:21:19 pm by gnif »
 
The following users thanked this post: EEVblog

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6956
  • Country: fi
    • My home page and email address
Re: Server Error Reports
« Reply #668 on: October 04, 2023, 04:06:48 pm »
According to my dictionary the use of obscurity was spot on. Your desire to keep knowledge of the infrastructure private  and not well known is pretty much the dictionary's definition.
Bullshit.  By that definition, not keeping your password on a post-it note stuck to your monitor would be security-through-obscurity also.

In the industry, "security through obscurity" is used to describe the situation when knowing the technical and implementation details of the system is sufficient to break the security scheme.  "Operational security" is used to describe the situation when the technical details of the system are kept private to limit the attack surface: knowing them does not break the security scheme, but it may help an attacker trying to break the security scheme by providing useful targeting information.

For example, if one accepts incoming SSH connections on a nonstandard port, and keeps that port number private (only telling those who do use those SSH connections), one applies operational security, not security through obscurity.  Revealing the port does not compromise security per se, but it can help attackers.  As security measures go, it is a very weak one, but if combined with diverting incoming connections from addresses that have recently tried to connect to other ports (especially the standard SSH port) to a designated honey pot, it can be a powerful tool for early automated attack detection.  Essentially, it does not provide any added security per se, but it definitely can help in detecting intrusion attempts, and focusing human security monitoring to where it is most likely needed.  Also, it does nothing at all against directed attacks.

Gnif uses the simplest operational security approach there is, one that is literal millenia old: minimize the attack surface by only providing information to those who actually need it.

The reason this is done, is that it makes attacks and intrusion attempts harder and more costly.  Before anyone can attack, they need to know what they are attacking, where, and how.  Even if an attacker uncovers this information, the security is not yet compromised; they still need to do the attack itself.  If an attacker cannot uncover that information, they need to guess what kind of an attack might work, and usually try several ones to have any kind of chance at success, and failed attack attemps tend to be easy to detect.

What benefit would there be for gnif or Dave or other gnif's clients, if they revealed the system security scheme details publicly?  Here, absolutely nothing; it would just make it easier for attackers to design and plan intrusion attempts to these systems.
There are certain forums where specific details and their application and usefulness are discussed, but to reduce the possibility of attackers exploiting the information discussed, they tend to be closed/insular, invitation-only, with real-world identity verification.  Me, I prefer physical meetings and face-to-face discussions, because of body language.
 
The following users thanked this post: gnif, MK14

Offline bd139

  • Super Contributor
  • ***
  • Posts: 23096
  • Country: gb
Re: Server Error Reports
« Reply #669 on: October 05, 2023, 08:20:33 pm »
Sort of.

To be fair it doesn't necessarily increase the attack risk but that depends on who you are. Most of the attack IVs are drive by i.e. from network scans etc. No one is going to read something in this forum and go "yay lets tear eevblog a new asshole". No joking but in the scale of things no one really gives a fuck about EEVblog because the end game has no financial advantage. I mean you might have pissed off an adversary by banning them for being a prick when posting but that's about the only major risk vector. Realistically defending against automated network attacks is probably the only major worry, followed by casual idiots attacking problems with the forum software. Fundamentally the profile depends if you are an interesting target i.e. you have a lot of PII, financial data and/or information to leverage to generate money in some way. Not all endpoints are equal.

But at no point should you make anyone's life easier for them if they are an adversary and that means keeping schtum on configuration, location, versions, everything. A dumbass could work that out. Do you pin a sign on your door with "I have a nice TV and the window is over there?". Nope!
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6956
  • Country: fi
    • My home page and email address
Re: Server Error Reports
« Reply #670 on: October 05, 2023, 09:09:01 pm »
No one is going to read something in this forum and go "yay lets tear eevblog a new asshole".
No, but there are many people that would love to cause Dave and gnif grief.  The kind of people who might frequent the seedy side of the web, looking for nasty scripts and 0day exploits for free or for little money, to use against their many "enemies".

Not real security risks, but annoying assholes.  And like you said, no reason to give them any more targeting information.

Realistically defending against automated network attacks is probably the only major worry, followed by casual idiots attacking problems with the forum software.
Determining and deciding that is something gnif and Dave and gnif's other clients would have to discuss first.  Like I said, gnif's operational security approach is the oldest, simplest one, and it makes sense.  Any other choice or approach must be evaluated; and when you are a responsible sysadmin/provider, such evaluations take time and effort.

Note that I am not claiming you are wrong in your assessment.  I am only saying that gnif's current approach requires no assessment, which makes it the easy, obvious choice.  To choose a different option, someone has to make the assessment and take responsibility.  It's only worth it if there is something weighty in the balance, and I personally only see curiosity on the other side of the balance, and no gain for Dave or gnif really.

But at no point should you make anyone's life easier for them if they are an adversary and that means keeping schtum on configuration, location, versions, everything. A dumbass could work that out. Do you pin a sign on your door with "I have a nice TV and the window is over there?". Nope!
I'd change that slightly, replacing 'if they are an adversary' with 'if you are not absolutely sure they are your friend/partner and reliable'.

Then again, I myself am "paranoid" about giving even myself too much access.  (Because if my account or private keys are compromised, I want to minimize the risk of harm to others.  While I do jokingly call it "paranoia", it too is just sensible operational security.)
 

Offline PlainName

  • Super Contributor
  • ***
  • Posts: 7303
  • Country: va
Re: Server Error Reports
« Reply #671 on: October 06, 2023, 09:00:10 pm »
Ahem! Gnif has previously said he doesn't want this thread being anything but server reports. The discussion of what constitutes security, etc, perhaps should go in the thread I started (to derail exactly that from here) over in:
https://www.eevblog.com/forum/security/kneejerks/
 
The following users thanked this post: EEVblog, gnif, Brumby, 2N3055, Nominal Animal

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1707
  • Country: au
  • Views and opinions are my own
    • AMD
Re: Server Error Reports
« Reply #672 on: October 11, 2023, 08:28:29 am »
Please be aware that we are performing some changes that may disrupt this service for a short period. I will post here again when the work is complete.
 
The following users thanked this post: EEVblog, MK14, Dan123456

Offline tautech

  • Super Contributor
  • ***
  • Posts: 29482
  • Country: nz
  • Taupaki Technologies Ltd. Siglent Distributor NZ.
    • Taupaki Technologies Ltd.
Re: Server Error Reports
« Reply #673 on: October 11, 2023, 08:56:30 am »
Please be aware that we are performing some changes that may disrupt this service for a short period. I will post here again when the work is complete.
LOL, no kidding.
Please try again. If you come back to this error screen, report the error to an administrator.
Avid Rabid Hobbyist.
Some stuff seen @ Siglent HQ cannot be shared.
 
The following users thanked this post: Dan123456

Offline Dan123456

  • Regular Contributor
  • *
  • Posts: 199
  • Country: au
Re: Server Error Reports
« Reply #674 on: October 11, 2023, 10:13:25 am »
Please be aware that we are performing some changes that may disrupt this service for a short period. I will post here again when the work is complete.
LOL, no kidding.
Please try again. If you come back to this error screen, report the error to an administrator.

Lol same  :P

Then it went to a 502 error for half an hour or so  :P Wasn’t sure if it my was on my end or what  :P

All good EEVBlog forum team! Thanks for the post :) Fingers crossed for you all  :)
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf