good grief
I do hope you found a way to make MS bugger off
No, we need the indexing, blocking crawlers is generally not an option.
You want to reveal what you think the cause of the problem was? Or not.
Two things.
1) Microsoft (aka, Bing) hammering the search on the forum. Seems their crawler uses the search feature of the forum to look for keywords, which was causing quite high server load when the crawl started each time.
2) The podcast plugin for the forum
So it was essentially transient high CPU behind the problem?
Pretty much, assuming it is fixed now. The podcast plugin that Dave was using tracks every view/hit/listen and where they came from, but instead of keeping a count of the totals it was doing a 'SELECT COUNT(DISTINCT col)' query over 6.2 million records. Each time Dave logged into the wordpress admin interface it would re-count them, causing a huge spike in database load.
SMF also does some strange stuff to search the forums, it creates a temporary table (Hash) that it uses to store the result set into for further sorting. Since this table never actually exists, I had never seen the effect it was having. To resolve this I installed the Sphinx full text search engine onto the server and configured SMF to use that instead, now the search can be hammered as much as Bing want's and it wont cause spikes in DB load.
Here is a small sample of what Bing is feeding the search box, such a dirty method of crawling.
[Sun Jul 30 03:16:42.090 2017] 0.073 sec 0.073 sec [ext2/3/ext 0 (0,1000) @id_topic] [smf_index] valet tray leather desk or dresser organizer by tech swiss
[Sun Jul 30 03:16:45.231 2017] 0.001 sec 0.001 sec [ext2/3/ext 0 (0,1000) @id_topic] [smf_index] valet tray leather desk or dresser organizer by
[Sun Jul 30 03:16:51.472 2017] 0.001 sec 0.001 sec [ext2/3/ext 0 (0,1000) @id_topic] [smf_index] valet tray leather desk or
[Sun Jul 30 03:16:54.275 2017] 0.000 sec 0.000 sec [ext2/3/ext 0 (0,1000) @id_topic] [smf_index] valet tray leather desk
[Sun Jul 30 03:16:57.432 2017] 0.000 sec 0.000 sec [ext2/3/ext 0 (0,1000) @id_topic] [smf_index] valet tray leather
[Sun Jul 30 03:17:08.289 2017] 0.024 sec 0.024 sec [ext2/1/ext 466 (0,1000) @id_topic] [smf_index] evolution
[Sun Jul 30 03:17:40.449 2017] 0.021 sec 0.021 sec [ext2/1/ext 218 (0,1000) @id_topic] [smf_index] 121
[Sun Jul 30 03:17:51.521 2017] 0.041 sec 0.041 sec [ext2/3/ext 0 (0,1000) @id_topic] [smf_index] stash tea flavor variety pack gift set 80 count bags with
[Sun Jul 30 03:17:56.119 2017] 0.001 sec 0.001 sec [ext2/3/ext 0 (0,1000) @id_topic] [smf_index] stash tea flavor variety pack gift set 80 count
[Sun Jul 30 03:17:59.322 2017] 0.001 sec 0.001 sec [ext2/3/ext 0 (0,1000) @id_topic] [smf_index] stash tea flavor variety pack gift set 80
[Sun Jul 30 03:18:05.293 2017] 0.000 sec 0.000 sec [ext2/3/ext 0 (0,1000) @id_topic] [smf_index] stash tea flavor variety pack gift