Server Error Reports

[magic]

The situation with attachments is a big mess and it looks like a forum bug.

It appears that posting an image occasionally overwrites the exact previous image posted by the same user, regardless of file format, thread or any posts in between made without attachments. No reliable trigger is known and it is unclear if the same could happen to non-image attachments.

A recent thread about it
https://www.eevblog.com/forum/chat/im-beginning-to-see-some-new-file-attachment-wrong-file-errors-again/

The inline images plugin got blamed last time and it has been half-disabled for over a year now. You can select the relevant option while posting, but the forum doesn't render resulting [ Attachment Invalid Or Does Not Exist ] tags. I seems to recall that Dave did an SMF upgrade shortly before the issue returned this year.

Sorry for not bringing this thread to your and Dave's attention earlier, I assumed that somebody (like maybe the OP) had already done so


edit
I'm not familiar with SMF internals and these plugins, but if you ask me, I would say that the location to write the attachment on disk gets written somewhere into the database, associated with the current user. Subsequently, a stupid race condition causes the file writing job to use the previous post's filename before the cached value is updated. Or maybe it doesn't get updated at all for some other stupid reason.

At any rate, the file name and path associated with the attachment doesn't seem to change. The contents are replaced. A few times we have seen XXX.JPG files with PNG contents in them or vice versa.

[gnif]

Sorry for not bringing this thread to your and Dave's attention earlier, I assumed that somebody (like maybe the OP) had already done so

No, I was aware of it but I was under the impression it was resolved. It was far worse at one stage.

I'm not familiar with SMF internals and these plugins, but if you ask me, I would say that the location to write the attachment on disk gets written somewhere into the database, associated with the current user.

The database just stores meta data and SMF writes each file to the disk with a name it generates, which is the unique primary key of the table combined with a hash of it's filename, the current time, and a random number.

Code: [Select]

$hash = sha1(md5($filename . time()) . mt_rand());
$filename = $id . '_' . $hash;


Subsequently, a stupid race condition causes the file writing job to use the previous post's filename before the cached value is updated. Or maybe it doesn't get updated at all for some other stupid reason.

Yeah, I am still not sure as to the cause as this should be impossible. PHP isn't multi-threaded per request per request, and since the filename/hash used is guaranteed by the database and the filename generation method that SMF uses to be unique, it should never happen.

What is even stranger is the post times of this latest incident was about 1/2 an hour apart, which rules out a race also. And finally, the new file that is written to disk incorrectly, is different from the file that it should be if it were just overwritten, it has also been altered/processed (this confirms it's a PHP/code bug, not fs or database)

Also what is further strange, SMF generates thumbnails when you try to view the thumbnail on the fly, it doesn't do it at image upload time. For this to have worked, the file on disk's contents needed to be correct to begin with.

I am on break for a few days, but as I can I will try to debug this futher.

[McBryce]

I had a similar problem on another forum. I found that it depended on what software was used to create or convert the image. I think we finally decided that it was a problem with the EXIF data containing wrong data.

McBryce.

[PlainName]

You probably already know this, but just in case.... I am getting database errors:

Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

In Buy&Sell and General Technical Chat. Obviously, it cleared up to post this.

[gnif]

Thanks, I will have a look.
I do have an alert for this instance but as it was fleeting I am unsure as to the cause at this time.

Edit: There was an issue with the database load balancer that has now been resolved.

[gnif]

Seems we had an outage during the night, one of the database servers had crashed out in a strange way. Cause is still yet unknown but being investigated.

[PlainName]

I came across that as it was going wrong, if it might help figure it out. I was trying to post and got:

The following error or errors occurred while posting this message:
Your session timed out while posting. Please try to re-submit your message.

Happened in the crowd funding section, and then when I tried to post here (of course). A the time I could read messages, just not post.

[gnif]

Yeah, thanks, I saw this also

Edit: Please be aware that I am still investigating this, we are seeing the database service get into a unrecoverable deadlock state and the cause is still unknown.

[gnif]

I have made some infrastructure changes that should not only resolve the crashing that has been occurring over the last few days, but also improve overall performance of the website quite considerably.
If you are noticing any improvement/degredation please provide feedback here:

https://www.eevblog.com/forum/chat/server-upgrade-feedback/

[tautech]

This after selecting to view new replies is a PITA:
Please try again. If you come back to this error screen, report the error to an administrator.

[gnif]

Already reported in the other thread, investigating

Edit: This is fixed

[m k]

I've recently had few hiccups.
First one was few minutes where page was finally loaded but content was database error.
Possibly just a database restart.

Now login and post were acting up, maybe timed out somehow but without updated Firefox window.
Like where post reply page is still present but new post is also live.
Maybe database machine has some issues.

I'm also quite sure I saw gnif's edit of a post about Steve Bannon and law where sub forum of deleted messages was mentioned.
Search can't find it anymore.

[gnif]

I've recently had few hiccups.
First one was few minutes where page was finally loaded but content was database error.
Possibly just a database restart.

We do not do database restarts, it was likely a peak load period, we get them from time to time.

Now login and post were acting up, maybe timed out somehow but without updated Firefox window.
Like where post reply page is still present but new post is also live.
Maybe database machine has some issues.

This is just latency in the server responding to you after posting, the new data was clearly inserted into the database and some kind of latency caused you to hang on the post page while waiting. There are no DB issues.

I'm also quite sure I saw gnif's edit of a post about Steve Bannon and law where sub forum of deleted messages was mentioned.
Search can't find it anymore.

When someone replies to a topic we can't just delete it, it must be manually moved to a "Deleted" area of the forums. We do not actually delete posts fully so that we have a record of why people were banned, etc. SMF by default has set to post a redirect link when you move a thread, I accidentally left this enabled. This area of the forum is not available to the public.

[RoGeorge]

Can log and browse but can not post from Ro, when using usual browsers (e.g. firefox/ubuntu).  This post was made through a VPN.  The error is session timeout, please try to logoff/login again, but whatever I do it does not fix the err.

Any advice how to fix this, please?

[gnif]

Right in the middle of some minor changes, that might be the cause. Please hold.

[gnif]

Work completed for tonight, still a few loose ends to sort out, but in no worse state for it

[gnif]

Just an update, one of the packages upgraded seems to be causing an issue with the database backup procedure which runs each night at 22:00 UTC. The database is being locked during the backup which should not occur, which is causing a brief outage while the backup happens.

I have implemented what I hope is a fix to this issue and running another backup to see if it's resolved.

[RoGeorge]

For me still not working.  Managed to post once from Ro, then timeout err again.  This one is from VPN.  I wonder if the cookies were tied to the IP, because I could post from Ro, then I couldn't edit it one minute later (with Modify button), then from there only session expired try to resubmit err.  Checking if it works from new IP only, and in the first minutes only.

Later edit
Worked from VPN (which implies new IP for me

Later later edit
Will stay here a little longer to test if I can only post in the very first minutes after a login from a new IP

[gnif]

What IP are you coming from when it times out (PM me)

[RoGeorge]

Will check now and PM you in a minute, from this login

[magic]

I noticed that my /unread/ list gets cleared randomly since maybe yesterday. I think it first happened after some outage (database connection error, bad gateway), but now it happened again without any outage that I noticed.

The attachment substitution bug is still present, like here.

[gnif]

I noticed that my /unread/ list gets cleared randomly since maybe yesterday. I think it first happened after some outage (database connection error, bad gateway), but now it happened again without any outage that I noticed.

This is not related to any server issues, perhaps a SMF bug, but there is nothing in the server deployment that can explain this.

The attachment substitution bug is still present, like here.

I am still tracing this, it's rare to catch it in action among all the noise of other posts going on so the cause is still under investigation.

[RoGeorge]

Trying now from Ro, with Brave browser.  Will try to attach a printscreen.

The following error or errors occurred while posting this message:
Your session timed out while posting. Please try to re-submit your message.

later edit
Strange that I can post now (once) from a new browser (Brave) I've never logged into eevblog before, but couldn't from LibreWolf or Firefox which I use usually, all same Ro IP.

[gnif]

Thanks, this is not a HTTP timeout as I was looking for, but rather a session tracking issue. Let me have a dig through the SMF source and see if I can spot anything obvious that we might be doing to trigger this.

[gnif]

I have made a change to the code to log additional information, can you please re-try the PM post. I need it to generate the error so I can see what is going on, feel free to spam me until it occurs again.