-
I run a community site (techy hobby but not electronics related) and one big issue is with spammers. Currently around 50% of new signups are spammers, and each time one of these manages to get in, one wakes up in the morning to find the forum covered in adverts for fake passports, fake IELTS, fake all sorts of certificates... so I put some effort into not allowing them posting rights (which is a manual permission).
These are not bots. We are using a hidden google catpcha which is pretty good at blocking bots. They are humans and they put a lot of time into this. They sign up and then go away for a week (because most forums don't let you post for a few days) and then come back and dump their stuff.
Various things give away a spammer. One is setting their country to one which does not match their IP (this isn't 100%; some genuine people are on a SIM card from another country, on holiday, etc) but if the IP is in say India and they put their country as the USA then it will be a spammer. Another one is that the IP or their email address is a hit on one of the many "stop forum spam" lists. We also filter against a list of about 2000 throwaway email domains; this is important because forum notifications bounce back and then your whole site gets an elevated spam rating. Another is that the signup uses a female name or email address; in any "tech" hobby women are extremely rare so while this isn't 100% it is a strong indicator especially if you get one like mentioned below. Genuine female posters always get a hugely enthusiastic reception on a techy site and "windup merchants" ("genuine" posters with a grudge, not spammers) often use a female identity
Currently I am seeing a lot of people join up via VPNs; usually one called VYPR VPN. These are usually obvious spammers from other indicators so I don't let them post, but notably they are taking a lot of trouble to sign up, even uploading an avatar (the last one is a woman in a keep fit class, not wearing a whole lot ) and selecting the right country to match the signup IP.
I have two questions:
Do "normal people" use a VPN routinely for forum browsing? If so, what is the point, for non illegal activities? I see the latest Firefox comes with a VPN but it isn't free. If your browser supports a "non tracking mode" that chucks away cookies which does most of the job. A VPN changes the IP but doesn't alter any of the dozens of other client browser parameters which get potentially published to the server. This site will demonstrate the futility: https://amiunique.org/fp
Do people who use these "privacy VPNs" know which country that particular session terminates in? Historically a lot of spammers (and other "windup merchants") used the TOR browser and AFAIK on that one you don't know the terminating country (often it is in Africa, because the TOR browser is used for a lot of illegal stuff).
-
Yes, people use VPNs normally. If you are only using VPN for "illegal activities", then you are telegraphing to the world that you are currently doing something illegal when using VPN.
Also, people don't want to be spied on by their ISPs.
Also, some countries are overly zealous with their blocking of stuff (China, Russia). So using VPN that terminates in another country is almost a must.
One of the best captachas are the ones unique to the site. Something basic. Like this forum might ask you to solve Ohm's law or something like that. Lichess.org asks you to solve "mate in 1" puzzle before posting to their forum. Spam farms are bad at solving that stuff.
Pre-moderation on first post may help too. Especially if certain trigger words are present in the messaage. -
Captchas are a funny one.
I got someone to produce a captcha which asked to multiply two numbers. Both numbers were presented as graphics so a bot for this would need to do OCR. It took the guy a few days to code this (in Ruby but it could have been PHP). It then took another clever unix programmer about 10 mins to write a bot which did OCR and pasted in the answer I don't have the details to hand but there is a unix utility which does OCR on a specified region of the page, so you don't even have to write any "software". So if you want to create a custom bot which creates 100 new signups and posts 1000 adverts for fake IELTS or whatever, it's very easy.
And the standard google captcha ("identify all images containing a bus") is hated by everybody, and often is very hard to solve. Some research indicates that it drives away a large % of the audience.
The basic issue is that there are people dedicated to doing this and they aren't stupid like they always were in the past. So blocking them has to involve something which they are not aware you will be looking for.
Sure I see a VPN being handy in China etc but for most online communities in the Western world there is negligible participation from these countries.
Pre-modding of 1st post remains a very good method.
I have a lot of admiration for whoever runs EEVBLOG which does not exclude posters from China etc. In fact they can't, since they accept adverts from JLCPCB But at least having adverts means you can employ a paid person to do this job. I don't have that luxury.
-
That's why you need a captcha that involves basic knowledge of the forum's subject. You can't automate that.
Let's say you have a forum about gardening. Your captcha may ask what type of fertilizer is not compatible with tomatoes, or something like that. Even if they OCR the question, the answer still not solvable by the computer. And if the number of questions is large an they are updated from time to time, then bots may just give up.
I also don't believe EEvblog does anything special against spam, just the normal captcha and manual reporting by forum members when there is spam. -
Whist I do agree with asking questions, we need to bear in mind that this forum has a beginners section, so they have to be easy enough for someone with minimal knowledge to answer.
Severely limiting the number of accounts from the same IP and new posts per day/hour/miniute for those with under 100 posts would help a lot.
Moderation of the first few posts is a good idea, but labour intensive. It also needs to be smart. Quite often we get people signing up and creating pointless posts, such as "Thanks for the info.", "Nice site.", "I like electronics" etc. which aren't spam and don't violete the rules, only to change the signiture to spam, or start spamming later. Fortunately the staff here are smart enough to spot this pattern and ban such accouts, before they spam. Many users here have also seen this pattern, including myself and will report pointless posters. -
"And if the number of questions is large"
I agree, but somebody has to set that up.
"Severely limiting the number of accounts from the same IP and new posts per day/hour/miniute for those with under 100 posts would help a lot."
Indeed - that's a good tool. But VPN users just switch IPs for each session, and of course dump their cookies.
"Quite often we get people signing up and creating pointless posts, such as "Thanks for the info.", "Nice site.", "I like electronics" etc. which aren't spam and don't violete the rules, only to change the signiture to spam, or start spamming later. Fortunately the staff here are smart enough to spot this pattern and ban such accouts, before they spam. Many users here have also seen this pattern, including myself and will report pointless posters."
One clever tactic I have seen is to pick a random old thread and add a post to the end of it, by copying the first post in the thread (which is usually a question). It's really cunning, if you think about it. It produces a meaningful post without any effort. On a long thread, nobody will spot the duplication.
Also I find that forum users lose interest in reporting stuff. They might report outright obvious spam but the admins will find that anyway soon after they get out of bed
There may be a way of automatically checking email addresses and IPs against the "stop forum spam" sites. You probably have to pay for it... and every so often the link will break and somebody has to spend half a day of their life making it work again.
-
Yes. We also had spammers copying the first post in a thread. I suppose such users could be theoretically banned automatically, if the forum sofware were flexible enough to be controlled with a suitable script.
Can't you simply download lists of email addresses and IPs from stop forum spam sites and convert them to text format, then automatically block/ban said accounts?
I never tire of reporting spam, unless the forum has too much of it, in which case, after giving the forum staff a fair chance to deal with it, I leave.
It's important to have at least three moderators, in opposite time zones, for 24 hour coverage. -
Perhaps put posts from new members in an approval queue?
To really mess with the spammers, instead of outright banning them, blacklist them so that their posts are shown to them (making it look like it went through) and then silently discarded after a short time. -
That works only while their session is active. As soon as they log out or dump their cookies, they will see their post is missing
This feature has been implemented on some forums. It is sometimes called "annoying poster mode".
An approval queue is a tricky topic. You would think that any serious poster will happily wait some hours for their first post to be approved. But actually usability research shows this is not the case. Participation goes way down with any barrier you put up. One really needs to approve new signups within an hour or preferably much faster, otherwise most never come back. It's the way "modern people" are So I get an email on new signups and can approve them (clumsily) with just my phone, but can't do a whole lot of checking with the phone.
-
I have two questions:
Do "normal people" use a VPN routinely for forum browsing? If so, what is the point, for non illegal activities? I see the latest Firefox comes with a VPN but it isn't free. If your browser supports a "non tracking mode" that chucks away cookies which does most of the job. A VPN changes the IP but doesn't alter any of the dozens of other client browser parameters which get potentially published to the server. This site will demonstrate the futility: https://amiunique.org/fp
I'm going to say NO - most "normal" people don't even know what a VPN is - the ones who do either use it for security (as in to connect to a corporate LAN), or for nefarious purposes (as in hiding their location, or, accessing content that they would not or could not legally access).
The concept of using a VPN for privacy is ludicrous - first - think about this - how many "paranoid users" place their trust in an unknown VPN provider to provide them with privacy, what stops the VPN provider from snooping? Second - the encryption exists only between the VPN user and the VPN server - the data then travels unencrypted from the VPN server to it's final destination - the "relevant authorities" need only to "snoop" on the VPN provider's exit point to be able to access all the "paranoid users" unencrypted data - the data belonging to all the paranoid users.QuoteDo people who use these "privacy VPNs" know which country that particular session terminates in? Historically a lot of spammers (and other "windup merchants") used the TOR browser and AFAIK on that one you don't know the terminating country (often it is in Africa, because the TOR browser is used for a lot of illegal stuff).
It depends on the VPN - the VPN I use allows me to select a terminating country of my choice. -
what stops the VPN provider from snooping? Second - the encryption exists only between the VPN user and the VPN server - the data then travels unencrypted from the VPN server to it's final destination - the "relevant authorities" need only to "snoop" on the VPN provider's exit point to be able to access all the "paranoid users" unencrypted data - the data belonging to all the paranoid users.
If traffic goes over HTTPS, then none of those people could snoop without some serious certificate tampering.
One of the bigger things VPN protects against are non-securable parts of the process, like DNS lookups. And that is less of a problem with DoH becoming a thing.
The privacy part of the VPN comes from the fact that a company located in a remote place will probably have no interest in a person in another country. Unlike customer's ISP, which has all the incentive to grab as much data as possible.
If you are wanted to the point where it makes sense to track your behaviour across continents, then you are screwed no matter what. But this is not the case for majority of people. -
I use a VPN on my desktop machine, because I'm too lazy to set it up to only run the torrent traffic through the VPN.
-
These are not bots. We are using a hidden google catpcha which is pretty good at blocking bots. They are humans and they put a lot of time into this. They sign up and then go away for a week (because most forums don't let you post for a few days) and then come back and dump their stuff.
It may actually be bots, believe it or not. I remember reading about one way crooks defeat captchas so their bots can do their bidding: they forward the captcha onto a free porn site where horny humans have to solve the captcha. Then, armed with the right answer, the bot can solve the captcha on the target site. Dunno if this still works with modern captchas, it’s been a while. -
Disclaimer: I don't run a web forum anymore, and when I did it was for a small number of users. However, I did write a bot detector and auto-banner for a popular online game. The logic behind it was infallible, but eventually I came to realise I was wrong, and some obvious bots actually weren't. OK, on with the show...Quote
Do "normal people" use a VPN routinely for forum browsing? If so, what is the point, for non illegal activities?
Oh dear. Personally I don't use one but I am aware of people that do, for one or several of many reasons. Outstanding ones:
* Your ISP is not net-neutral and prioritizes certain traffic. The VPN hides that one's traffic is low-priority rubbish and it doesn't get clobbered.
* Your ISP monitors your data so they can spam you with 'relevant' adverts. And/or sell your stats onwards. Hey, being a Brit, don't you remember Phorm? The VPN prevents that.
Etc.QuoteThese are not bots. We are using a hidden google catpcha which is pretty good at blocking bots.
See my note above re logic. They could be bots, it's just that the bot offloads captcha decoding to wetware. Rooms of people sit in front of a PC and answer dumb questions, identify objects, etc. They have no idea where it comes from or what it's for, just that they will make a pittance if they do enough of them.
https://humancoder.com/
-
I use VPNs and Tor all the time, and I think this is becoming a new norm. As for eliminating spammers, you have few options:
* Gatekeeping strategies that recruits members only via invites. Forum is either read only or inaccessible until the user is invited somehow.
* Sandboxing new users into a "introduce yourself" sub-forum. Once the new users reached X amount of non-spam posts, they are granted full access to the rest of the forum.
* Organise a vigilant moderator team who are effective at cleaning up rubbish.
* Roll your own captcha that has Q&A about specific interests that spammers are unable to answer. For example electronics questions that only electronics geeks could answer.
* Combination of the above.
The truth is, once your forum is specifically targeted, you'll have a hard time eliminating all spam with one easy step. -
If you can't stop them attacking you by building better walls, then perhaps getting rid of the gold in your castle could work better? Eg: adding rel=nofollow to all links in posted content. This informs search engines to treat the links as untrustworthy, dramatically (completely?) reducing the value of link spamming your site. Many big sites do this, including Wikipedia.
* Gatekeeping strategies that recruits members only via invites. Forum is either read only or inaccessible until the user is invited somehow.
Example of this: https://lobste.rs/u#Hales . If a lot of spam users appear under someone they (and their whole tree) get banned.* Sandboxing new users into a "introduce yourself" sub-forum. Once the new users reached X amount of non-spam posts, they are granted full access to the rest of the forum.
A different variant of this is pre-moderating all content before it goes live. I wonder if there is much of an effort difference for mods?Quote* Roll your own captcha that has Q&A about specific interests that spammers are unable to answer. For example electronics questions that only electronics geeks could answer.
I highly recommend this, but you don't have to make the questions super hard. Any custom captcha will make your site only spammable by extra human effort. Ie you drag yourself out of the "lowest hanging fruit" pile where everyone using an off-the-shelf common system sits.
...which brings me onto...These are not bots. We are using a hidden google catpcha which is pretty good at blocking bots. They are humans and they put a lot of time into this. They sign up and then go away for a week (because most forums don't let you post for a few days) and then come back and dump their stuff.
That's really interesting. I don't know how successful Google's recaptcha is.
Maybe people run web-scrapers looking for specific forum software, then give these lists to the human work-force? Making the registration a bit weird or different could help defeat this?
I've written up on my own experiences doing anti-spam, but only for a very small website's comment section. Against automated attacks: even making your form fields slightly different to what is expected is very effective. Of course against human attacks this is mostly naught.
-
The google recaptcha is very good. It works in a complex way, probably linked to other stuff they run e.g. gmail's antispam measures (which are way over-zealous and make gmail unusable for any "international" business, but that's another topic). While we don't know what we don't know, if one was getting a load of failed signups then eventually some % of people will click on the Contact link and tell you, and this isn't happening.
Yes, there are people who search for specific forum software, usually PHP-BB, which being open source gets attacked and hacked mercilessly - unless administered with a lot of expertise. So e.g. the Magento online shops are getting hacked a lot. The forum in question here was custom written in Ruby on Rails, by an expert in security, and so far it has not been obviously hacked, but Ruby is practically impossible to maintain (was a big fashion ~10-15 years ago).
"Organise a vigilant moderator team who are effective at cleaning up rubbish."
That's a funny one. Experience shows that you either have to pay them the going rate (which requires a big site with lots of adverts and traffic) or you end up with a bunch of controlling-personality weirdoes who eventually tend to do a Colonel Kurtz in Apocalypse Now, piss off a load of people, and the site disintegrates.
-
"Organise a vigilant moderator team who are effective at cleaning up rubbish."
Depends on the community. If the forum is big enough and is respected, there will be plenty of volunteers who'd want to moderate. That said, moderation can be hard when it comes to social dynamics and community appeasement (i.e. you never win as a moderator, no matter what action you make). But cleaning up obvious spam bots, is really a no-brainer.
That's a funny one. Experience shows that you either have to pay them the going rate (which requires a big site with lots of adverts and traffic) or you end up with a bunch of controlling-personality weirdoes who eventually tend to do a Colonel Kurtz in Apocalypse Now, piss off a load of people, and the site disintegrates. -
Modding a forum is a very difficult job. 10x more so on a site which doesn't carry adverts or is too small to make money out of ads, and then the chances of finding a volunteer mod who does a good job in the long run are IME zero. I have never seen it work, since web forums started to be popular c. 2000. Mods just go weird, sooner or later. If the forum is basically unmoderated (which is a viable formula, for a while, until you get sued a few times) then it can run for quite a while. But a forum which is modded to e.g. not allow personal attacks needs very skilful modding, and it is almost impossible as a volunteer effort.
Coming back to spammers, on every forum an admin can see all activity (obviously) and bots are fairly obvious in the pattern and the timing. Of course they could be humans... but probably not because most of these are widespread attacks, not a vengeful hit on one specific site.
I'd say the use of a VPN, particularly some of them, is a strong indicator of a spammer. Most "real" people have a life and just can't be bothered about "privacy" issues. Google has already got everything on you, but it doesn't sell it in personal detail - unlike FB and such which is a really nasty company. -
Regarding moderation: just give most moderators the job of removing spam, pornography, copyrighted material etc. The personal stuff should be handled by a supermoderator, or the main site administrator.
I have moderated a site before and I just left the personal stuff alone. The trouble was, the administrator didn't give ordinary moderators the power to ban, so deleting spam was pointless, as it would just get reposted and is why I left, never to return. To be fair we did have a problem with a bad moderator before, which caused lots of people to leave, but that's only because the adminstrator too far too long to deal with it. Numberous people had complained about the bad moderator, over a long period, so there was no excuse!
If there was a need, which there isn't, I'd be happy to moderate here for free and I would stay out of the personal stuff, especialy if it involves me. If someone called me a cunt, I'd just report it and allow a supermoderator, or administrator to deal with it. -
I'd be wary of using AI to block nasty posts, because most of the time on this forum, they're about shoddy software, poor products, crappy companies etc. There are countless nasty posts about Microshaft & Winblow$ 10. Would your AI bot ban someone who said the scumbag who developed the Office 360 user interface should be stonned to death? It might be better at detecting pointless posts and obvious spam.
An obvious use of automation is to detect duplicate posts and copy, pasting. I don't know whether automatically banning duplicate posters should be done though. Often newcommers think it's okay, as they think they'll reach a wider audiance if they create the same thread in different sections and it has happened before acidentally. Perhaps such a ban could be temporary, with a message telling the user why they've been banned, for how long and how to appeal it, if they think it's a mistake. -
Many forums prevent duplicate posts if consecutive, not least because that is a common issue with submitting new posts if the connection is a bit dodgy.
But checking all the back through the thread for duplicate posts would be a new one
But I think, fundamentally, there is no solution to forum spam unless you manually approve all new signups. If you do that, it is pretty easy. One simple tip for admins: look at pre-signup activity and if there isn't any (i.e. negligible reading of the forum) then don't approve it. Almost no real person behaves like that, but 99% of spammers do. BUT you need to approve new signups quickly (not hours later) otherwise most will be lost for ever (people have a short attention span these days).
This wasn't a problem 10 years ago but today it is really big. Especially with fake passports / fake IELTS / etc being such a big thing. -
The attack-adjust dynamics of admin/user vs spammer is interesting to me and I appreciate the content in the thread. Of course, I immediately start thinking about where it is going.
Just some thoughts...
It seems like most of the processes in the thread seem to involve automated pre-moderation based on some pattern recognition. That is, excepting perhaps message #20, I am not seeing much on the automation of spam identification after the fact. Why is this? IOW, it seems like shuffling off identified spam into an invisible (to normal users) bin for later processing (a few seconds to days as per resources) where it would be returned to its posting status or deleted. Is it common and just has not been mentioned or is it too difficult or something else?
For example, many (most or all) email clients will sort out junk mail. Even to the extent of automated removal after XXX days. Of course you can see it and various places will remind you to 'look into your junk mail folder for our email and add us to your ok lists' or some such message.
The SME, or at least subject matter familiarity, captcha was mentioned, but wouldn't this simply be a matter of time before it no longer worked? That is, automated answers to such captchas would seem like they would only require some additional 'knowledge data bases'.
I am amazed at some of the bots I see on places, including here, and I am assuming, perhaps erroneously, that they are bots. Perhaps I simply lack vision but I don't see the justification for that much effort for such poor performance (maybe I just have not suspected the good ones).
I used to enjoy thwarting the phone spammers by blocking the numbers. I would check the effectiveness by not blocking some and see if they would call a second time. Now, they never call a second time. Apparently phone lines (and IDs) are cheap enough to be never-the-same-twice. Now I am much closer to having an allow list (for deciding if I will answer)...something that the forums already use with a minimum number of posts and something that the spammers strive to defeat.
VPNs are offered as part of packages of anti-virus software (two large vendors come to mind). People may use them because they paid for them and feel like they have to get their money's worth and believe the hype.
I went back and read my response to a thread we had a while ago about the future of online discussion https://www.eevblog.com/forum/chat/the-future-of-online-discussion-how-do-you-see-it/msg2899818/#msg2899818 I have not changed much as far as how I feel about it.
-
Web forums grew rapidly ~2000 onwards. Everybody who knew PHP and a database knocked one up. And people were desperate for online participations and flooded these sites.
Today, web forums struggle because their success needs people who can read and write and who have time to participate usefully.
These people are still around but many, especially the young, are being lost to "instant satisfaction" / "one liner" places like facebook, twatter, instagram, and such.
The fact that these sites are useless for archiving knowledge, and anyway almost nobody bothers to post anything informative there, doesn't concern most people.
Then there is a lot more nastiness around. The online world has become generally nastier. This is a problem largely because everybody who comes to a web forum today has been on FB etc where nastiness is pretty much the default, and they expect to be able to do the same on the web forum. They then have to be dealt with by the mod(s) and often they don't like that.
Usenet used to be brilliant but has mostly gone under the weight of spam, and few people having a decent client for reading it.
The forum I run is modded for zero personal attacks (which is rare, and could not be done if we had adverts because kicking and biting threads bring a lot more advert clicks) and ~98% of the regular group has always been good. But over the 10 years it's been going we have had some "moments" I have few regrets except that troublemakers should have been removed much faster. The big tech forums in the US remove them instantly and there is a lesson to learn there.
Quality online discussion will continue on quality well managed sites like this one, which have an educated audience. There it can survive. Once you get away from specialised subjects, to mundane stuff, it is a struggle and most of that will probably end up on FB where everybody has created a forum
Someone mentioned stopping search engines going in. That is absolutely not a good idea because good SEO is key to a forum collecting new people, to replace those who left, died, etc. Web forums still get the best SEO.
What is changing is the spammer landscape, because there are high value commodities they can sell. The next one will be fake vaccine passports, but I think the govts are onto that because everybody knows one of them will easily be worth 3 or 4 figures.
-
Do "normal people" use a VPN routinely for forum browsing? If so, what is the point, for non illegal activities?
I used a paid VPN by default.
Besides frustrating the data harvesting of my ISP, it allows the secure use of WiFi hotspots. Doing otherwise is an invitation to disaster.