Author Topic: Server Error Reports  (Read 757569 times)

0 Members and 1 Guest are viewing this topic.

Online magic

  • Super Contributor
  • ***
  • Posts: 6921
  • Country: pl
Re: Server Error Reports
« Reply #600 on: April 20, 2023, 09:30:26 pm »
The situation with attachments is a big mess and it looks like a forum bug.

It appears that posting an image occasionally overwrites the exact previous image posted by the same user, regardless of file format, thread or any posts in between made without attachments. No reliable trigger is known and it is unclear if the same could happen to non-image attachments.

A recent thread about it
https://www.eevblog.com/forum/chat/im-beginning-to-see-some-new-file-attachment-wrong-file-errors-again/

The inline images plugin got blamed last time and it has been half-disabled for over a year now. You can select the relevant option while posting, but the forum doesn't render resulting [ Attachment Invalid Or Does Not Exist ] tags. I seems to recall that Dave did an SMF upgrade shortly before the issue returned this year.

Sorry for not bringing this thread to your and Dave's attention earlier, I assumed that somebody (like maybe the OP) had already done so :palm:


edit
I'm not familiar with SMF internals and these plugins, but if you ask me, I would say that the location to write the attachment on disk gets written somewhere into the database, associated with the current user. Subsequently, a stupid race condition causes the file writing job to use the previous post's filename before the cached value is updated. Or maybe it doesn't get updated at all for some other stupid reason.

At any rate, the file name and path associated with the attachment doesn't seem to change. The contents are replaced. A few times we have seen XXX.JPG files with PNG contents in them or vice versa.
« Last Edit: April 20, 2023, 09:47:44 pm by magic »
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #601 on: April 20, 2023, 11:39:02 pm »
Sorry for not bringing this thread to your and Dave's attention earlier, I assumed that somebody (like maybe the OP) had already done so :palm:

No, I was aware of it but I was under the impression it was resolved. It was far worse at one stage.

I'm not familiar with SMF internals and these plugins, but if you ask me, I would say that the location to write the attachment on disk gets written somewhere into the database, associated with the current user.

The database just stores meta data and SMF writes each file to the disk with a name it generates, which is the unique primary key of the table combined with a hash of it's filename, the current time, and a random number.

Code: [Select]
$hash = sha1(md5($filename . time()) . mt_rand());
$filename = $id . '_' . $hash;

Quote
Subsequently, a stupid race condition causes the file writing job to use the previous post's filename before the cached value is updated. Or maybe it doesn't get updated at all for some other stupid reason.

Yeah, I am still not sure as to the cause as this should be impossible. PHP isn't multi-threaded per request per request, and since the filename/hash used is guaranteed by the database and the filename generation method that SMF uses to be unique, it should never happen.

What is even stranger is the post times of this latest incident was about 1/2 an hour apart, which rules out a race also. And finally, the new file that is written to disk incorrectly, is different from the file that it should be if it were just overwritten, it has also been altered/processed (this confirms it's a PHP/code bug, not fs or database)

Also what is further strange, SMF generates thumbnails when you try to view the thumbnail on the fly, it doesn't do it at image upload time. For this to have worked, the file on disk's contents needed to be correct to begin with.

I am on break for a few days, but as I can I will try to debug this futher.
« Last Edit: April 20, 2023, 11:43:15 pm by gnif »
 

Offline McBryce

  • Super Contributor
  • ***
  • Posts: 2701
  • Country: de
Re: Server Error Reports
« Reply #602 on: April 21, 2023, 01:02:07 pm »
I had a similar problem on another forum. I found that it depended on what software was used to create or convert the image. I think we finally decided that it was a problem with the EXIF data containing wrong data.

McBryce.
30 Years making cars more difficult to repair.
 

Offline PlainName

  • Super Contributor
  • ***
  • Posts: 7038
  • Country: va
Re: Server Error Reports
« Reply #603 on: April 23, 2023, 09:54:35 pm »
You probably already know this, but just in case.... I am getting database errors:

Quote
Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

In Buy&Sell and General Technical Chat. Obviously, it cleared up to post this.
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #604 on: April 23, 2023, 10:10:47 pm »
Thanks, I will have a look.
I do have an alert for this instance but as it was fleeting I am unsure as to the cause at this time.

Edit: There was an issue with the database load balancer that has now been resolved.
« Last Edit: April 23, 2023, 10:51:43 pm by gnif »
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #605 on: April 24, 2023, 10:59:08 pm »
Seems we had an outage during the night, one of the database servers had crashed out in a strange way. Cause is still yet unknown but being investigated.
 
The following users thanked this post: EEVblog

Offline PlainName

  • Super Contributor
  • ***
  • Posts: 7038
  • Country: va
Re: Server Error Reports
« Reply #606 on: April 24, 2023, 11:13:15 pm »
I came across that as it was going wrong, if it might help figure it out. I was trying to post and got:

Quote
The following error or errors occurred while posting this message:
Your session timed out while posting. Please try to re-submit your message.

Happened in the crowd funding section, and then when I tried to post here (of course). A the time I could read messages, just not post.
 
The following users thanked this post: gnif

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #607 on: April 24, 2023, 11:23:42 pm »
Yeah, thanks, I saw this also

Edit: Please be aware that I am still investigating this, we are seeing the database service get into a unrecoverable deadlock state and the cause is still unknown.
« Last Edit: April 25, 2023, 05:28:14 am by gnif »
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #608 on: April 25, 2023, 11:36:20 am »
I have made some infrastructure changes that should not only resolve the crashing that has been occurring over the last few days, but also improve overall performance of the website quite considerably.
If you are noticing any improvement/degredation please provide feedback here:

https://www.eevblog.com/forum/chat/server-upgrade-feedback/
 

Offline tautech

  • Super Contributor
  • ***
  • Posts: 28925
  • Country: nz
  • Taupaki Technologies Ltd. Siglent Distributor NZ.
    • Taupaki Technologies Ltd.
Re: Server Error Reports
« Reply #609 on: April 25, 2023, 12:01:24 pm »
This after selecting to view new replies is a PITA:
Please try again. If you come back to this error screen, report the error to an administrator.
Avid Rabid Hobbyist.
Siglent Youtube channel: https://www.youtube.com/@SiglentVideo/videos
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #610 on: April 25, 2023, 12:09:14 pm »
Already reported in the other thread, investigating

Edit: This is fixed
« Last Edit: April 25, 2023, 12:58:29 pm by gnif »
 

Offline m k

  • Super Contributor
  • ***
  • Posts: 2243
  • Country: fi
Re: Server Error Reports
« Reply #611 on: June 02, 2023, 09:34:50 am »
I've recently had few hiccups.
First one was few minutes where page was finally loaded but content was database error.
Possibly just a database restart.

Now login and post were acting up, maybe timed out somehow but without updated Firefox window.
Like where post reply page is still present but new post is also live.
Maybe database machine has some issues.

I'm also quite sure I saw gnif's edit of a post about Steve Bannon and law where sub forum of deleted messages was mentioned.
Search can't find it anymore.
Advance-Aneng-Appa-AVO-Beckman-Danbridge-Data Tech-Fluke-General Radio-H. W. Sullivan-Heathkit-HP-Kaise-Kyoritsu-Leeds & Northrup-Mastech-REO-Simpson-Sinclair-Tektronix-Tokyo Rikosha-Topward-Triplett-YFE
(plus lesser brands from the work shop of the world)
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #612 on: June 02, 2023, 09:41:09 am »
I've recently had few hiccups.
First one was few minutes where page was finally loaded but content was database error.
Possibly just a database restart.

We do not do database restarts, it was likely a peak load period, we get them from time to time.

Now login and post were acting up, maybe timed out somehow but without updated Firefox window.
Like where post reply page is still present but new post is also live.
Maybe database machine has some issues.

This is just latency in the server responding to you after posting, the new data was clearly inserted into the database and some kind of latency caused you to hang on the post page while waiting. There are no DB issues.

I'm also quite sure I saw gnif's edit of a post about Steve Bannon and law where sub forum of deleted messages was mentioned.
Search can't find it anymore.

When someone replies to a topic we can't just delete it, it must be manually moved to a "Deleted" area of the forums. We do not actually delete posts fully so that we have a record of why people were banned, etc. SMF by default has set to post a redirect link when you move a thread, I accidentally left this enabled. This area of the forum is not available to the public.
« Last Edit: June 02, 2023, 09:58:34 am by gnif »
 
The following users thanked this post: m k

Online RoGeorge

  • Super Contributor
  • ***
  • Posts: 6475
  • Country: ro
Re: Server Error Reports
« Reply #613 on: June 16, 2023, 03:14:26 pm »
Can log and browse but can not post from Ro, when using usual browsers (e.g. firefox/ubuntu).  This post was made through a VPN.  The error is session timeout, please try to logoff/login again, but whatever I do it does not fix the err.

Any advice how to fix this, please?

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #614 on: June 16, 2023, 03:30:07 pm »
Right in the middle of some minor changes, that might be the cause. Please hold.
 
The following users thanked this post: RoGeorge

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #615 on: June 16, 2023, 05:06:40 pm »
Work completed for tonight, still a few loose ends to sort out, but in no worse state for it :)
 
The following users thanked this post: MK14, beanflying

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #616 on: June 17, 2023, 01:46:43 pm »
Just an update, one of the packages upgraded seems to be causing an issue with the database backup procedure which runs each night at 22:00 UTC. The database is being locked during the backup which should not occur, which is causing a brief outage while the backup happens.

I have implemented what I hope is a fix to this issue and running another backup to see if it's resolved.
 
The following users thanked this post: MK14

Online RoGeorge

  • Super Contributor
  • ***
  • Posts: 6475
  • Country: ro
Re: Server Error Reports
« Reply #617 on: June 17, 2023, 02:33:47 pm »
For me still not working.  Managed to post once from Ro, then timeout err again.  This one is from VPN.  I wonder if the cookies were tied to the IP, because I could post from Ro, then I couldn't edit it one minute later (with Modify button), then from there only session expired try to resubmit err.  Checking if it works from new IP only, and in the first minutes only.

Later edit
Worked from VPN (which implies new IP for me

Later later edit
Will stay here a little longer to test if I can only post in the very first minutes after a login from a new IP
« Last Edit: June 17, 2023, 02:38:03 pm by RoGeorge »
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #618 on: June 17, 2023, 02:36:39 pm »
What IP are you coming from when it times out (PM me)
 

Online RoGeorge

  • Super Contributor
  • ***
  • Posts: 6475
  • Country: ro
Re: Server Error Reports
« Reply #619 on: June 17, 2023, 02:39:32 pm »
Will check now and PM you in a minute, from this login
 
The following users thanked this post: gnif

Online magic

  • Super Contributor
  • ***
  • Posts: 6921
  • Country: pl
Re: Server Error Reports
« Reply #620 on: June 17, 2023, 03:33:27 pm »
I noticed that my /unread/ list gets cleared randomly since maybe yesterday. I think it first happened after some outage (database connection error, bad gateway), but now it happened again without any outage that I noticed.

The attachment substitution bug is still present, like here.
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #621 on: June 17, 2023, 03:41:23 pm »
I noticed that my /unread/ list gets cleared randomly since maybe yesterday. I think it first happened after some outage (database connection error, bad gateway), but now it happened again without any outage that I noticed.

This is not related to any server issues, perhaps a SMF bug, but there is nothing in the server deployment that can explain this.

The attachment substitution bug is still present, like here.

I am still tracing this, it's rare to catch it in action among all the noise of other posts going on so the cause is still under investigation.
 
The following users thanked this post: magic

Online RoGeorge

  • Super Contributor
  • ***
  • Posts: 6475
  • Country: ro
Re: Server Error Reports
« Reply #622 on: June 17, 2023, 03:47:55 pm »
Trying now from Ro, with Brave browser.  Will try to attach a printscreen.

Quote
The following error or errors occurred while posting this message:
Your session timed out while posting. Please try to re-submit your message.

later edit
Strange that I can post now (once) from a new browser (Brave) I've never logged into eevblog before, but couldn't from LibreWolf or Firefox which I use usually, all same Ro IP.
« Last Edit: June 17, 2023, 03:50:58 pm by RoGeorge »
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #623 on: June 17, 2023, 03:51:18 pm »
Thanks, this is not a HTTP timeout as I was looking for, but rather a session tracking issue. Let me have a dig through the SMF source and see if I can spot anything obvious that we might be doing to trigger this.
 

Offline gnifTopic starter

  • Administrator
  • *****
  • Posts: 1691
  • Country: au
Re: Server Error Reports
« Reply #624 on: June 17, 2023, 03:57:54 pm »
I have made a change to the code to log additional information, can you please re-try the PM post. I need it to generate the error so I can see what is going on, feel free to spam me until it occurs again.
 
The following users thanked this post: RoGeorge


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf