Author Topic: NEWS: Forum reverted to backup  (Read 10706 times)

0 Members and 2 Guests are viewing this topic.

Offline EEVblog

  • Administrator
  • *****
  • Posts: 33404
  • Country: au
    • EEVblog
NEWS: Forum reverted to backup
« on: May 02, 2021, 09:40:39 pm »
We had to revert to a backup of the forum last night.
I'm still need to evaluate the aftermath, this is just a thread to discuss what happened.
 
The following users thanked this post: SeanB, Whales, ogden, alm, I wanted a rude username, Hannibal_Hector

Online tautech

  • Super Contributor
  • ***
  • Posts: 22235
  • Country: nz
  • Taupaki Technologies Ltd. NZ Siglent Distributor
    • Taupaki Technologies Ltd.
Re: NEWS: Forum reverted to backup
« Reply #1 on: May 02, 2021, 09:58:47 pm »
Found this on Discord however even that thread has vanished:

Avid Rabid Hobbyist
 

Offline Whales

  • Super Contributor
  • ***
  • Posts: 1325
  • Country: au
    • Halestrom
Re: NEWS: Forum reverted to backup
« Reply #2 on: May 02, 2021, 10:11:00 pm »
Yes, that thread was made yesterday.  https://www.eevblog.com/forum/news/did-everyones-forum-post-counts-just-drop-by-between-1-and-5/msg3561390/#msg3561390

Hoping that the beast is tamed.  Thankyou Dave.

Offline I wanted a rude username

  • Frequent Contributor
  • **
  • Posts: 535
  • Country: au
  • ... but this username is also acceptable.
Re: NEWS: Forum reverted to backup
« Reply #3 on: May 02, 2021, 10:18:47 pm »
Thanks for the update. Glad I created that thread, if it's what tipped you off.

> TE thread is down to 18 pages.
> Our entire lives are gone


RIP all the comments we made between the backup time and restoration time. Feels like an alternate universe just got snuffed out.
 

Offline tom66

  • Super Contributor
  • ***
  • Posts: 4392
  • Country: gb
  • Electron Fiddler, FPGA Hacker, Embedded Systems EE
Re: NEWS: Forum reverted to backup
« Reply #4 on: May 02, 2021, 10:27:44 pm »
Does the SMF forum not support database transactioning?  Before doing a big operation, start a SQL transaction, see if it is bad, if it is, roll it back...

You don't want to transaction every operation (huge performance hit) but if you're ever unsure about what something will do, it should be done.

Not sure if you can tell SQL 'any operation in the future is to be put in the transaction log' and then turn that off.
 

Offline rsjsouza

  • Super Contributor
  • ***
  • Posts: 4919
  • Country: us
  • Eternally curious
    • Vbe - vídeo blog eletrônico
Re: NEWS: Forum reverted to backup
« Reply #5 on: May 02, 2021, 10:34:52 pm »
If it helps finding the extent of the data loss... I usually post once or twice a day and my post history shows a gap between a post in April 24th and the next on April 29th.
Vbe - vídeo blog eletrônico http://videos.vbeletronico.com

Oh, the "whys" of the datasheets... The information is there not to be an axiomatic truth, but instead each speck of data must be slowly inhaled while carefully performing a deep search inside oneself to find the true metaphysical sense...
 

Offline Hannibal_Hector

  • Newbie
  • Posts: 4
  • Country: us
Re: NEWS: Forum reverted to backup
« Reply #6 on: May 02, 2021, 10:35:54 pm »
This is what backups are for. You can put al kinds of levers in place to ensure nothing ever breaks, nothing is ever lost, nothing ever goes down... but there is always that one thing that gets you. And that's where the backups come in, and putting things in perspective, nobody got hurt! (I think?)
Rigol DS1054Z, Circuitmate DM25xl, Aneng 8008, Logic Probe
 

Offline andy3055

  • Frequent Contributor
  • **
  • Posts: 764
  • Country: us
Re: NEWS: Forum reverted to backup
« Reply #7 on: May 02, 2021, 10:40:19 pm »
Earlier today, it was saying that the site is down for maintenance. Is that also related?
 

Offline JohnnyMalaria

  • Super Contributor
  • ***
  • Posts: 1148
  • Country: us
    • Enlighten Scientific LLC
Re: NEWS: Forum reverted to backup
« Reply #8 on: May 02, 2021, 10:49:36 pm »
Earlier today, it was saying that the site is down for maintenance. Is that also related?

Yes, Dave shut the site to address his...um...nuclear attack on some threads.

I was reading the thread as it was all unfolding. Most surreal. Though for some this happened yesterday, it was earlier today for me. Either that, or I can't account for a day :)
 

Offline Whales

  • Super Contributor
  • ***
  • Posts: 1325
  • Country: au
    • Halestrom
Re: NEWS: Forum reverted to backup
« Reply #9 on: May 02, 2021, 10:50:24 pm »
Earlier today, it was saying that the site is down for maintenance. Is that also related?

Probably.  Loading a backup for a massive database takes some time and you don't want people interacting with an incomplete database, that can lead to all sorts of inconsistencies and craziness (assuming it even works at all).  Even viewing a topic before the full import is complete could lead to thousands of errors (eg posts and thread exists, but certain user's profiles do not).
« Last Edit: May 02, 2021, 10:52:03 pm by Whales »
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 33404
  • Country: au
    • EEVblog
Re: NEWS: Forum reverted to backup
« Reply #10 on: May 02, 2021, 11:17:57 pm »
Unfortunately I don't think the backup went back far enough, I'm still seeing huge numbers of deleted threads, about half as many as before, so progress, but still, I wouldn't rule out another reversion.
In this case we coincidentaly had a partial day backup and I believe that was used to recover. We might need to go back full day at least, or try an fix the threads manually from here.
 

Offline gnavigator1007

  • Supporter
  • ****
  • Posts: 337
  • Country: us
Re: NEWS: Forum reverted to backup
« Reply #11 on: May 02, 2021, 11:20:04 pm »
Completely forgot about the discord. I was wondering what was going on when I received 35 email notifications all at once for each topic I was subscribed to that had been removed. Received more some hours later for topics being moved. At least it was nothing malicious. Figure Dave is probably too busy to answer, but does anybody know how often the forum is backed up & how much storage is required to maintain?
 

Offline Whales

  • Super Contributor
  • ***
  • Posts: 1325
  • Country: au
    • Halestrom
Re: NEWS: Forum reverted to backup
« Reply #12 on: May 02, 2021, 11:54:18 pm »
[...] but still, I wouldn't rule out another reversion.

Can you temporarily freeze?  That gives time for TEA people to backup their new posts and time for you to scrounge through backups, all the while avoiding us making new stuff.

Offline EEVblog

  • Administrator
  • *****
  • Posts: 33404
  • Country: au
    • EEVblog
Re: NEWS: Forum reverted to backup
« Reply #13 on: May 03, 2021, 12:09:05 am »
Yep, will freeze posting until a decision is made on what direction to take.
 
The following users thanked this post: bitseeker

Offline EEVblog

  • Administrator
  • *****
  • Posts: 33404
  • Country: au
    • EEVblog
Re: NEWS: Forum reverted to backup
« Reply #14 on: May 03, 2021, 02:40:18 am »
I've re-enabled posting, and am currently going through restroing some accidently deleted theads.
As a result you may see your post count increase back toward normal level as threads are restored if you have posted in them.
 
The following users thanked this post: bitseeker, I wanted a rude username

Offline magic

  • Super Contributor
  • ***
  • Posts: 3685
  • Country: pl
Re: NEWS: Forum reverted to backup
« Reply #15 on: May 03, 2021, 05:22:28 am »
RIP all the comments we made between the backup time and restoration time. Feels like an alternate universe just got snuffed out.
Yep, a good chunk of Sunday (6AM~6PM UTC or so) is MIA.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 33404
  • Country: au
    • EEVblog
Re: NEWS: Forum reverted to backup
« Reply #16 on: May 03, 2021, 05:26:41 am »
I'm pretty sure I have now undeleted almost every thread that was accidently deleted.
I may have missed a couple, and if you are aware of any please let me know.
Everyone's post counts should be back to almost normal, although there will almost certainly be a few casulty posts, sorry about that.
The recovery from my goof was not 100%, but should be pretty darn close.
Thanks to gnif for a late night backup recovery and writing some script to fix some things.
Sorry to everyone for the goof up, it was entirely my fault  |O

Although having said that, I do believe it was ultimately caused by forum bug.
For those interested, I thought I'd be clever and search for keywords from a particular user which allowed me to mass delete posts 50 at a time (at request of the owner).
So although I was selecting individual posts from a user, the forum decided to delete the entire thread that contained that post. And I didn't realise this until someone posted that thread asking why their post count had dropped 5% or something.
 


Offline Ian.M

  • Super Contributor
  • ***
  • Posts: 10475
Re: NEWS: Forum reverted to backup
« Reply #18 on: May 03, 2021, 06:04:15 am »
For those interested, I thought I'd be clever and search for keywords from a particular user which allowed me to mass delete posts 50 at a time (at request of the owner).
So although I was selecting individual posts from a user, the forum decided to delete the entire thread that contained that post. And I didn't realise this until someone posted that thread asking why their post count had dropped 5% or something.
Obviously, if a user gets banned and all or most of their posts  have unacceptable content, you may need to mass delete them, but why bend over backwards to <expletive> up the forum at the request of a user?   Deleting posts that weren't unacceptable content, at best causes confusion, and if done in bulk for/by a single user smacks of "I'll take my ball and go home!".  Obviously, given the tools you choose to make available to us users, you cant easily prevent users stripping their content from the forum, but if they want to be immature, at least let them have to do all the grunt work to do so rather than wasting your and Gnif's time.

IMHO when you post a message on a public forum, you are granting the forum the right to display the content you have created in perpetuity*.  Once you hit [Post] or [Send] its out of your control. 

*  unless the forum offers users a more restrictive content creators license, granting the users more rights.
 
The following users thanked this post: amyk, thm_w

Offline EEVblog

  • Administrator
  • *****
  • Posts: 33404
  • Country: au
    • EEVblog
Re: NEWS: Forum reverted to backup
« Reply #19 on: May 03, 2021, 06:10:32 am »
For those interested, I thought I'd be clever and search for keywords from a particular user which allowed me to mass delete posts 50 at a time (at request of the owner).
So although I was selecting individual posts from a user, the forum decided to delete the entire thread that contained that post. And I didn't realise this until someone posted that thread asking why their post count had dropped 5% or something.
Obviously, if a user gets banned and all or most of their posts  have unacceptable content, you may need to mass delete them, but why bend over backwards to <expletive> up the forum at the request of a user?   Deleting posts that weren't unacceptable content, at best causes confusion, and if done in bulk for/by a single user smacks of "I'll take my ball and go home!".  Obviously, given the tools you choose to make available to us users, you cant easily prevent users stripping their content from the forum, but if they want to be immature, at least let them have to do all the grunt work to do so rather than wasting your and Gnif's time.

IMHO when you post a message on a public forum, you are granting the forum the right to display the content you have created in perpetuity*.  Once you hit [Post] or [Send] its out of your control. 

*  unless the forum offers users a more restrictive content creators license, granting the users more rights.

Sorry but I won't go into details about this.
This stuff up was entirely my doing, I tried to do something fancy and it backfired.
If it wasn't for the technical stuff up/bug, it would have been trivial work and no one would have noticed.
A similar thing could happen in theory during any routine admin cleanup task like spam account purging etc.
« Last Edit: May 03, 2021, 06:28:28 am by EEVblog »
 
The following users thanked this post: Ian.M

Online BradC

  • Super Contributor
  • ***
  • Posts: 1777
  • Country: au
Re: NEWS: Forum reverted to backup
« Reply #20 on: May 03, 2021, 06:14:25 am »
Thankfully nothing I post is worth reading!

A nice reminder in "whoopsie" recovery though.
 
The following users thanked this post: Ed.Kloonk

Online Ed.Kloonk

  • Super Contributor
  • ***
  • Posts: 2297
  • Country: au
Re: NEWS: Forum reverted to backup
« Reply #21 on: May 03, 2021, 06:22:59 am »
Thanks for all you do, Dave.

Thanks gnif.

 :-+
 
The following users thanked this post: SeanB


Offline Muttley Snickers

  • Supporter
  • ****
  • Posts: 2179
  • Country: au
Re: NEWS: Forum reverted to backup
« Reply #23 on: May 03, 2021, 06:33:53 am »
I'm still down around 100 posts but not that concerned about it. I do wonder though, what would happen if a member had reached 1000 posts and gained access to the supporters lounge but are now back under 1000 due to the glitch, do they still have access?
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 33404
  • Country: au
    • EEVblog
Re: NEWS: Forum reverted to backup
« Reply #24 on: May 03, 2021, 06:46:12 am »
I do wonder though, what would happen if a member had reached 1000 posts and gained access to the supporters lounge but are now back under 1000 due to the glitch, do they still have access?

I can fix that manually.
 
The following users thanked this post: Muttley Snickers


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf