-
Presumably that is why I haven't been able to get on to this forum, for the last few hours, until now.
Because of errors, which typically looked like this:QuoteMax connect timeout reached while reaching hostgroup 1 after 10000ms
-
Looks like a database fault but it wasn't, the PHP process was in a bad state. I will be scheduling an upgrade tonight to try to resolve this. There should be no interruptions during this upgrade
-
For an unknown reason the PHP process on the server had gone into a bad state throwing hundreds of general protection faults on all HTTP servers in the cluster. Reason for this is as of yet unknown, looks like a bug in the PHP process itself.
Almost certainly no coincidence it happened right after I tried to update Wordpress. Different databases though, and I dont recall Wordpress website problem taking down the forum before? -
For an unknown reason the PHP process on the server had gone into a bad state throwing hundreds of general protection faults on all HTTP servers in the cluster. Reason for this is as of yet unknown, looks like a bug in the PHP process itself.
Almost certainly no coincidence it happened right after I tried to update Wordpress. Different databases though, and I dont recall Wordpress website problem taking down the forum before?
It does seems to be a coincidence, the Wordpress website runs on a different version of PHP which was unaffected and is isolated from the forum. Anything more is just speculation, the processes would have needed to have been run under a debugger with debug symbols to catch the fault and obtain any useful debugging information (which would kill server performance while waiting for it to fault again, if ever it decides to) -
It does seems to be a coincidence, the Wordpress website runs on a different version of PHP which was unaffected and is isolated from the forum. Anything more is just speculation, the processes would have needed to have been run under a debugger with debug symbols to catch the fault and obtain any useful debugging information (which would kill server performance while waiting for it to fault again, if ever it decides to)
It depends on the degree to which the optimizer switches affect the performance of the server, among other things (e.g., do you need to set a memory watchpoint?). It's possible to have debug symbols compiled in while the optimizer switches remain the same, but whether you get anything useful as a consequence when the fault occurs just depends. Note that there's a huge difference between being able to see stack traces and being able to step through the code in a sane manner, and optimization affects the latter a lot more than the former. There are some optimizations that could potentially interfere with getting a proper stack trace (e.g., -fomit-frame-pointer) but in reading up on it a bit it seems that it may have limited to no impact on x86-64. I would expect gdb these days to be able to reconstruct the call stack even if -fomit-frame-pointer was in use, based on what I've read about what it does.
That leaves only the performance impact to the running binary. Attaching a debugger in and of itself will do nothing to the performance of the running process unless the operating system underneath is horribly coded. A segfault is something the operating system will detect and an attached debugger will be notified of that as soon as it happens, at which point the debugger can be used to examine the stacks and other things. The debugger will be idle up until that point, and the running process will proceed as normal up until that point.
The main issue I've seen with debugging segfaults like that is that they often involve smashed stacks, and if that's happening then tracking them down might prove difficult at best. Just enabling the stack protection in the compiler (-fstack-protector or -fstack-protector-all) could easily have a significant performance impact (I've never experimented with it so I really can't say), and if I understand the mechanisms it uses correctly, it's hit-or-miss anyway (meaning, something can scribble on your stack frame in such a way that it doesn't tickle the canary value), and only tells you which function's stack frame was smashed, not which code performed the smash.
Bottom line: whether or not you get a performance hit will depend on what exactly you're trying to chase down. The faults you mentioned might be against the heap, or might be against the stack, or might even be against the heap as a result of improper changes to the stack. There's no way to know without venturing down the rabbit hole.
-
It does seems to be a coincidence, the Wordpress website runs on a different version of PHP which was unaffected and is isolated from the forum. ...
Out of curiosity, what is the nature of the isolation?
-
It does seems to be a coincidence, the Wordpress website runs on a different version of PHP which was unaffected and is isolated from the forum. ...
Out of curiosity, what is the nature of the isolation?
Sorry but we do not divulge details related to the security of the server -
It does seems to be a coincidence, the Wordpress website runs on a different version of PHP which was unaffected and is isolated from the forum. Anything more is just speculation, the processes would have needed to have been run under a debugger with debug symbols to catch the fault and obtain any useful debugging information (which would kill server performance while waiting for it to fault again, if ever it decides to)
It depends on the degree to which the optimizer switches affect the performance of the server, among other things (e.g., do you need to set a memory watchpoint?). It's possible to have debug symbols compiled in while the optimizer switches remain the same, but whether you get anything useful as a consequence when the fault occurs just depends. Note that there's a huge difference between being able to see stack traces and being able to step through the code in a sane manner, and optimization affects the latter a lot more than the former. There are some optimizations that could potentially interfere with getting a proper stack trace (e.g., -fomit-frame-pointer) but in reading up on it a bit it seems that it may have limited to no impact on x86-64. I would expect gdb these days to be able to reconstruct the call stack even if -fomit-frame-pointer was in use, based on what I've read about what it does.
That leaves only the performance impact to the running binary. Attaching a debugger in and of itself will do nothing to the performance of the running process unless the operating system underneath is horribly coded. A segfault is something the operating system will detect and an attached debugger will be notified of that as soon as it happens, at which point the debugger can be used to examine the stacks and other things. The debugger will be idle up until that point, and the running process will proceed as normal up until that point.
The main issue I've seen with debugging segfaults like that is that they often involve smashed stacks, and if that's happening then tracking them down might prove difficult at best. Just enabling the stack protection in the compiler (-fstack-protector or -fstack-protector-all) could easily have a significant performance impact (I've never experimented with it so I really can't say), and if I understand the mechanisms it uses correctly, it's hit-or-miss anyway (meaning, something can scribble on your stack frame in such a way that it doesn't tickle the canary value), and only tells you which function's stack frame was smashed, not which code performed the smash.
Bottom line: whether or not you get a performance hit will depend on what exactly you're trying to chase down. The faults you mentioned might be against the heap, or might be against the stack, or might even be against the heap as a result of improper changes to the stack. There's no way to know without venturing down the rabbit hole.
Obviously having compiled symbols in the binary are an option however packages provided by upstream do not do this and as we are running production binaries when this fault occurred we have nothing to go on. As this was a random occurrence and seems to have no regularity to it at this point (it may never even happen again) there is little point building custom binaries with debug symbols enabled making it difficult to keep updated with the latest security releases upstream as they become available.
Also attaching gdb does have a performance hit always, if you write low latency multi-threaded applications you will know that sometimes a bug is harder to reproduce when running under gdb as it slows things down enough to "fix" problems such as race conditions.
The PHP-FPM model uses FastGGI, and forks a set of child PHP processes which have a limited number of requests to serve before they are each restarted, this is done to avoid problems caused by potential memory leaks and is a common FastCGI strategy. The overhead of gdb following forks is quite heavy.
Please note that I have extensive experience in software debugging and development where my primary development IDE is ViM & gdb. I appreciate the input but in this instance at this point the cost vs benefit currently makes this not worth the effort when a simple service upgrade may resolve an issue that may have already been fixed upstream. -
The server outage this morning is being investigated, there is no need to report this. Cause is as of yet unknown, the database service had stopped responding.
-
Sorry but we do not divulge details related to the security of the server
That is the first rule of security for me: Don't provide any information.
Why give anyone a head start for hacking?
(Sorry gnif, I didn't mean to trigger a notification) -
Security through obscurity... hmmm...
-
Security through obscurity... hmmm...
You are misusing the term, the software in use is open source and available to the public as such, public scrutiny. How we have decided to make use of this software and build the EEVBlog infrastructure is kept private as this knowledge makes it easier to infiltrate the system if someone does get a foothold in some software/service. This is called "operational security" and is very different to "security through obscurity". -
The term you're looking for may be "Defense in depth." A lot of people confuse that with "Security through obscurity," but they aren't the same.
-
Getting some patchy issues with the Server in the last day or so not being available and just had a prolonged outage.
-
We are aware, debugging an issue with the main website is having an impact on services.
-
Security through obscurity... hmmm...
You are misusing the term, the software in use is open source and available to the public as such, public scrutiny. How we have decided to make use of this software and build the EEVBlog infrastructure is kept private as this knowledge makes it easier to infiltrate the system if someone does get a foothold in some software/service. This is called "operational security" and is very different to "security through obscurity".
According to my dictionary the use of obscurity was spot on. Your desire to keep knowledge of the infrastructure private and not well known is pretty much the dictionary's definition.
-
Security through obscurity... hmmm...
You are misusing the term, the software in use is open source and available to the public as such, public scrutiny. How we have decided to make use of this software and build the EEVBlog infrastructure is kept private as this knowledge makes it easier to infiltrate the system if someone does get a foothold in some software/service. This is called "operational security" and is very different to "security through obscurity".
According to my dictionary the use of obscurity was spot on. Your desire to keep knowledge of the infrastructure private and not well known is pretty much the dictionary's definition.
Say whatever you like, we do not rely on the server being kept secure by omitting information as such this is not "security through obscurity". Limiting operational information to the general public is a no brainier as it gives administrators more time to combat attacks in progress that are trying to probe for such information to progress the attack. Sure the attacker may not be able to break in at all due to the security layers in place, but why risk it? Why hand out a map to the treasure even though the treasure box is padlocked?
Again, we are using open software with industry standard well known and reviewed security implementations. We are not relying on security though some custom logic that is closed and has not been peer reviewed. We are not "secure" because we rely on the fact that people don't know the specifics of how things are configured.
If we are to apply your definition of "security by obscurity" one could argue that the authentication credentials used to access the servers are kept secret and as such is "security by obscurity". -
According to my dictionary the use of obscurity was spot on. Your desire to keep knowledge of the infrastructure private and not well known is pretty much the dictionary's definition.
Bullshit. By that definition, not keeping your password on a post-it note stuck to your monitor would be security-through-obscurity also.
In the industry, "security through obscurity" is used to describe the situation when knowing the technical and implementation details of the system is sufficient to break the security scheme. "Operational security" is used to describe the situation when the technical details of the system are kept private to limit the attack surface: knowing them does not break the security scheme, but it may help an attacker trying to break the security scheme by providing useful targeting information.
For example, if one accepts incoming SSH connections on a nonstandard port, and keeps that port number private (only telling those who do use those SSH connections), one applies operational security, not security through obscurity. Revealing the port does not compromise security per se, but it can help attackers. As security measures go, it is a very weak one, but if combined with diverting incoming connections from addresses that have recently tried to connect to other ports (especially the standard SSH port) to a designated honey pot, it can be a powerful tool for early automated attack detection. Essentially, it does not provide any added security per se, but it definitely can help in detecting intrusion attempts, and focusing human security monitoring to where it is most likely needed. Also, it does nothing at all against directed attacks.
Gnif uses the simplest operational security approach there is, one that is literal millenia old: minimize the attack surface by only providing information to those who actually need it.
The reason this is done, is that it makes attacks and intrusion attempts harder and more costly. Before anyone can attack, they need to know what they are attacking, where, and how. Even if an attacker uncovers this information, the security is not yet compromised; they still need to do the attack itself. If an attacker cannot uncover that information, they need to guess what kind of an attack might work, and usually try several ones to have any kind of chance at success, and failed attack attemps tend to be easy to detect.
What benefit would there be for gnif or Dave or other gnif's clients, if they revealed the system security scheme details publicly? Here, absolutely nothing; it would just make it easier for attackers to design and plan intrusion attempts to these systems.
There are certain forums where specific details and their application and usefulness are discussed, but to reduce the possibility of attackers exploiting the information discussed, they tend to be closed/insular, invitation-only, with real-world identity verification. Me, I prefer physical meetings and face-to-face discussions, because of body language. -
Sort of.
To be fair it doesn't necessarily increase the attack risk but that depends on who you are. Most of the attack IVs are drive by i.e. from network scans etc. No one is going to read something in this forum and go "yay lets tear eevblog a new asshole". No joking but in the scale of things no one really gives a fuck about EEVblog because the end game has no financial advantage. I mean you might have pissed off an adversary by banning them for being a prick when posting but that's about the only major risk vector. Realistically defending against automated network attacks is probably the only major worry, followed by casual idiots attacking problems with the forum software. Fundamentally the profile depends if you are an interesting target i.e. you have a lot of PII, financial data and/or information to leverage to generate money in some way. Not all endpoints are equal.
But at no point should you make anyone's life easier for them if they are an adversary and that means keeping schtum on configuration, location, versions, everything. A dumbass could work that out. Do you pin a sign on your door with "I have a nice TV and the window is over there?". Nope! -
No one is going to read something in this forum and go "yay lets tear eevblog a new asshole".
No, but there are many people that would love to cause Dave and gnif grief. The kind of people who might frequent the seedy side of the web, looking for nasty scripts and 0day exploits for free or for little money, to use against their many "enemies".
Not real security risks, but annoying assholes. And like you said, no reason to give them any more targeting information.Realistically defending against automated network attacks is probably the only major worry, followed by casual idiots attacking problems with the forum software.
Determining and deciding that is something gnif and Dave and gnif's other clients would have to discuss first. Like I said, gnif's operational security approach is the oldest, simplest one, and it makes sense. Any other choice or approach must be evaluated; and when you are a responsible sysadmin/provider, such evaluations take time and effort.
Note that I am not claiming you are wrong in your assessment. I am only saying that gnif's current approach requires no assessment, which makes it the easy, obvious choice. To choose a different option, someone has to make the assessment and take responsibility. It's only worth it if there is something weighty in the balance, and I personally only see curiosity on the other side of the balance, and no gain for Dave or gnif really.But at no point should you make anyone's life easier for them if they are an adversary and that means keeping schtum on configuration, location, versions, everything. A dumbass could work that out. Do you pin a sign on your door with "I have a nice TV and the window is over there?". Nope!
I'd change that slightly, replacing 'if they are an adversary' with 'if you are not absolutely sure they are your friend/partner and reliable'.
Then again, I myself am "paranoid" about giving even myself too much access. (Because if my account or private keys are compromised, I want to minimize the risk of harm to others. While I do jokingly call it "paranoia", it too is just sensible operational security.) -
Ahem! Gnif has previously said he doesn't want this thread being anything but server reports. The discussion of what constitutes security, etc, perhaps should go in the thread I started (to derail exactly that from here) over in:
https://www.eevblog.com/forum/security/kneejerks/ -
Please be aware that we are performing some changes that may disrupt this service for a short period. I will post here again when the work is complete.
-
Please be aware that we are performing some changes that may disrupt this service for a short period. I will post here again when the work is complete.
LOL, no kidding.
Please try again. If you come back to this error screen, report the error to an administrator. -
Please be aware that we are performing some changes that may disrupt this service for a short period. I will post here again when the work is complete.
LOL, no kidding.
Please try again. If you come back to this error screen, report the error to an administrator.
Lol same
Then it went to a 502 error for half an hour or so Wasn’t sure if it my was on my end or what
All good EEVBlog forum team! Thanks for the post
Fingers crossed for you all
-
Ok, sorry about the outages, we are making some changes to improve server security.
The forum should be fully working now with the exception of existing attachments while we wait for on disk permissions to change. If attachments are not yet loading please just wait for now before reporting a fault. I will post when this is completed.
Edit: Please note that there may be gateway timeouts while this is in progress as it's quite I/O intensive.