|
|||||||||||||||
An FAQ for news.admin.net-abuse.email
Maintainer: James Farmer
|
||
TABLE OF CONTENTS
(Questions highlighted in red have been modified since the last release of this document.)
|
||
New section 1.4.1.6.1 about Bayesian filtering.
Removed or fixed lots of dead links.
I've added the following links:
|
||
The following document should, where not otherwise stated, be understood to represent the opinions and beliefs of the FAQ-maintainer only. I endeavour to ensure that these opinions and beliefs are as correct as possible, but take no responsibility for any problems caused by errors herein. This document should not be considered to represent the opinions of any individuals or organisations other than the FAQ-maintainer.
Please note that in this document, "we" is intended to collectively refer to all regular or semi-regular posters to the news.admin.net-abuse.email newsgroup, including those of all persuasions, and should not be read as indicating the existence of a "clique" comprising persons of similar viewpoints.
|
||
This is one of three documents I have compiled to comprise an FAQ for the news.admin.net-abuse.email newsgroup. Each document addresses points in a given area, specifically:
The SPAMFIGHTING OVERVIEW offers a taste of the many techniques people use to fight spam. The objective isn't to teach you how to fight spam (there are many far superior documents that do just this), but rather to introduce some of the techniques you can use and refer you to some more detailed works.
THE EVILS OF SPAM covers the more ethical, moral, and legal aspects of spam, including just what constitutes spam and the types of people who become spammers.
UNDERSTANDING NANAE aims to introduce all of the weird, wonderful, and sometimes impenetrable terminology that people use in news.admin.net-abuse.email (nanae). It covers both colloquialisms (e.g. "chickenboner") and technical terms (e.g. "direct-to-MX").
These three parts are designed to stand alone and don't have to be read in order; feel free to pick and choose just the bits you're interested in.
These documents shouldn't be considered to be "the" FAQ, as there are plenty of other FAQs that are superior in insight, detail, or depth of coverage. They are just an FAQ that I hope will answer some questions that have been troubling you.
These documents are currently maintained by James Farmer. If you have any suggestions for additions or corrections, then feel free to send an email to faqmaster@spamfaq.net.
The latest versions of all of these documents can always be found at http://www.lumbercartel.ca/archives/spamfaq.net/. There's also an index there, which is the easiest way to find the answer if you've got one question in particular - just find the word you're looking for and click on it!
These documents are somewhat extensive. For a quicker overview of the main things you'll need to know, have a look at George Crissman's excellent document "Your First Post to NANAE".
|
||
1.1.1 Whom is this document for?
This document is intended for anyone who feels confused about any of the spamfighting techniques discussed in the news.admin.net-abuse.email newsgroup. It aims to briefly summarise what each of the commonly used techniques is, and provide links to sites where you can find more detailed information.
This document is not a tutorial for spamfighters. While there is much in here that will be of interest to a newcomer, reading this document alone will teach you only what techniques you can employ to fight spam, not how to use them.
1.1.2 What is spam and why do we fight it?
These are issues that are discussed in great depth in the second part of this FAQ, "The Evils of Spam". However, to briefly summarise, spam is a type of email that endangers the very existence of the email system by threatening to overwhelm it with a massive and uncontrollable volume of messages. Spam usually takes the form of advertising or promotional material that arrives in your emailbox without you having requested it.
UBE (Unsolicited Bulk Email) and UCE (Unsolicited Commercial Email) are terms that are often used to describe different types of spam
More information on just what is spam and why it is bad can be found in the second part of this FAQ, The Evils of Spam.
|
||
1.2.1 I've received some spam... what can I do?
Most people ignore the spam they receive. They either don't have the time or the expertise to deal with it. Their decision is understandable, but in the end inaction only helps the spammers because they can point to statistics and say "I sent my spam to 7 million email addresses and only 190 people complained so the other 6,999,810 must have been happy to receive it".
Alternatively, spam-victims might try to use a spam's "remove address". The concept here is that by sending a message to a given email address you will tell the spammer to remove you from their mailing list. However, these things almost universally fail to work. In the rare cases where your "remove request" actually reaches the spammer, they'll just take it as an indication that email sent to your address is actually read by a human, and thus your address becomes _more_ valuable to them, and they send you _more_ spam.
The best thing to do is: complain, complain, complain! Most ISPs have Terms of Service (or Acceptable Use Policies) that forbid spamming, so if you can tell the spammer's ISP that their customer broke these rules, then you can get the spammer's account cancelled! As well as giving you personal satisfaction, this will serve as a deterrent to this and other spammers, and with any luck prevent him from profiting in any way from his spam.
(As an aside, an ISP will sometimes try to "educate" a spammer before terminating their account, as sometimes a company will send a spam without considering the issues involved. This topic is explored in the second part of this FAQ, "The Evils of Spam".)
1.2.2 How can I find a spammer's ISP?
The tricky bit is working out just who is the spammer's ISP. The address in the "From:" field is almost certainly forged in order to throw you off the scent (and may even belong to an innocent third-party), so you have to learn to read the "full message headers", which are a bit like a log of an email message's travels through the internet. The spammer will try to forge these too, but in most cases it's still pretty easy to work out which ISP the message came from.
Header-reading is beyond the scope of this document, but here are a few links where you can find out more:
BUT... when complaining, please remember that the people at the spammer's ISP are not the bad guys. They didn't know their customer would turn out to be a spammer. There is a great temptation to fire off a few pages of verbal abuse, but remember that you are angry with the spammer, not the abuse staff at his ISP. The spammer will have abused them too, probably breaking their Terms of Service. And there is nothing an ISP can do to prevent, completely, any chance of Internet abuse emanating from their machines. So be polite. Point out what has happened without dramatic or obscenity-clad embellishment. Hostile or infantile behaviour will do you no good at this stage.
If the abuse staff sends you a response that is blatantly offensive, then it may be time to revise your opinion of them (although always be aware of the potential for a misunderstanding), but you should start out from the assumption that these people are your friends.
Most abuse departments won't act against a spammer until a non-trivial number of complaints have been received. This is because people sometimes forget that they have signed up for legitimate mailing lists or requested other types of email, and complain about it as spam. If you are convinced that a message was spam but the spammer's ISP claims that it wasn't, then there are further steps you can take. We will discuss these in later sections of this document.
1.2.3 Can I do anything about a spammer's website?
Assuming that the ISP agrees to take action, the spammer's account with that ISP will often be cancelled. Unfortunately, the spammers have caught on that their accounts rarely last long after they send their spam, so they've taken to using cheap "throw-away" accounts, opened solely for the purpose of sending spam which advertises ("spamvertises") websites held on other providers. The spamming accounts will get cancelled soon after the spam-run is complete, but the website will remain intact and thus the spammer can safely benefit from their spam (in terms of sales over the web, or clicks on banner advertisements, or whatever). That's the idea, at any rate.
Largely, this doesn't work as most web-hosting companies have clauses in their Terms of Service forbidding the use of spam to advertise the websites they host. Sending a quick complaint to the hosting company will often result in the spammer's website being removed.
But how to find the web-hosting company? The spammers may try to conceal this, but there's one snag - they want potential customers to reach their website, which means that the website's URL is probably somewhere in the spam. Once you find it, you can use tools like "traceroute" and "whois" to work out who's hosting the site. Here are some useful online versions of these tools:
But if you'd prefer to run them from your desktop, rather than surfing over to a webpage every time you want to run a traceroute, then you can download versions of the tools from these links:
"traceroute" is a tool that gives you the list of machines on the Internet, where a message sent from the source machine to another machine would pass through. "Whois" is a tool for looking up the owner of a domain or IP address. A detailed look at either of these is beyond the scope of this document, but again here are some useful links:
NOTE: Make sure you know what you're doing before you start writing complaints based on the results of tools like "traceroute" or "whois", as it's very easy to make mistakes. In particular, don't automatically email every email address you see in a whois output - sometimes these are merely the writers of the whois servers! If in doubt, ask in the newsgroup for confirmation.
Spammers will often try to obscure the true address of their website by spamvertising the address of an intermediate site or giving the address in an obscure format, but in most cases it's pretty easy to work through their tricks. We'll look at this in more detail in section 1.3.1.
Using the result of a "whois" or "nslookup" tool, you can also find out whose providing nameservers or DNS services for a spammer's domain. These are just as vital to the website's operation as the web-hosting company - you may wish to complain about the spammer's activities to them as well.
1.2.4 What if the spam doesn't include a website?
Alternatively, the spam may not advertise a website and will instead be soliciting replies by email. You can use the techniques described above to work out who is hosting this email address ("drop-box") and complain to the provider, which will probably cancel the spammer's email account. Good, eh?
1.2.5 What if the spam doesn't even include an email address?
A few spammers - particularly chain-letter spammers - don't include any electronic ways of contacting them, giving only a postal address or a telephone number in their spams. In these cases, there tends to be less you can do.
Most postal addresses found in spams will actually be P.O. boxes (e.g. Mailboxes Etc). Some of these mailbox providers may have rules against business use or certain types of business uses (e.g. chain letters or MLM); if so and you complain, they may take action.
In fact, chain letters soliciting money are illegal pyramid schemes in many countries, so reporting them to the authorities may be a good idea. For example, in the United States you can forward such chain letters to your local postmaster or postal inspector, or the postmaster/postal inspector local to each address on the chain letter, or present them to the clerk at your local post office saying "I received this illegal chain letter asking for money". You can also send them by email to fraud@uspis.gov.
Incidentally, I do NOT recommend making personal visits to addresses advertised in spams. Nothing good can come of such episodes. If you desperately want to contact the spammer, send him a letter.
Many spams will include phone numbers you're supposed to call for more information. Sometimes these will play recorded messages giving the address of a website or an email address, in which case you can complain to the relevent ISP as usual. In other cases, it can be worthwhile checking the type of phone number it is - many spammers give premium-rate numbers and don't include legally required warnings, in which case you can complain the provider or the regulator or whatever is relevant to the locality. (On this note, _always_ check the call charges before calling a spamvertised phone number. If in doubt, don't call it.)
Note that in many countries, a freephone number can still detect your number even if you have call blocking enabled. Use a pay-phone if this worries you.
By the way, if you call a spammer's phone number and actually reach the spammer or his family, DON'T be abusive. It does no good and only makes the spammer feel like the victim.
(Well that's all I know. Can anyone think of anything more for this section?)
1.2.6 Who else can I complain to?
The key with most spamfighting is summed up by this simple motto: "Follow the Money". Have a look at the spam and the spammed website and see how the spammer's intending to earn off it. Is he using an external merchant to charge credit cards? If so, complain to them and often they'll stop dealing with the spammer. Does he have banner ads? If so, complain to the suppliers of the banner ads. If there's a form on the spammer's website that sends information to an email address, complain to the ISP of that email address. Most legitimate businesses on the Internet aren't keen to sully their reputations by working with spammers.
Remember: always be polite. The ISPs are not your enemies and a single polite word will get you a lot farther than a screenful of abuse.
As an aside, the U.S. Federal Trade Commission has a project for analysing and classifying spam, and have invited Internet users to forward their spam to uce@ftc.gov. This won't help you in the short-term but it could be of long-term benefit in the fight against spam. They also occaisionally take action against outright scams that are reported in this way.
1.2.7 What email address do I complain to?
At most ISPs, the address for sending complaints is "abuse@<isp-domain>, e.g. abuse@rcn.com or abuse@yahoo.com. However, a few ISPs have non-standard abuse department email addresses; in these cases it can be hard to know where to send your complaint. To the rescue comes abuse.net; a database of ISP abuse addresses. It can even forward complaints automatically to the relevant abuse addresses if you supply the complaint and the name of the Internet provider! Have a look at http://www.abuse.net/
1.2.8 Can't this all be automated?
All this reading headers, working out webhosting providers, and so forth is a pain. Spamcop is a service that aims to automate this process; you give it your spam and it writes and mails the complaint for you.
Spamcop has a reputation for sending complaints to a few incorrect places, so you have to keep an eye on what it's doing, but if you think you might find it useful, then have a look at http://www.spamcop.net/. (Note that www.spamcop.org has no relation to www.spamcop.net.)
A French-language service at http://www.spam-rbl.com seems to do something similar to SpamCop, but in French.
There's also downloadable anti-fighting tools, such as:
1.2.9 Should I hack into the spammer's computer?
No; hacking (or, to use the precise term, cracking) is very seriously frowned upon by most of the anti-spamming community. Apart from the fact that it's illegal, it allows the spammers to portray themselves as honest businessmen being assaulted by electronic terrorists. If we are to eliminate spam it is important that we retain the moral high ground.
|
||
1.3.1 Spammer Tricks
1.3.1.1 What are these weird URLs?
Some spammers try to "obfuscate" the address of their website in order to make it hard to see where to complain to. A number of common tactics include:
The Non-Dotted-Quad IP address
Most IP addresses have the "dotted-quad" form:
182.175.90.10
However, the IP address is also valid as one big decimal number, e.g.:
3064945162
The spammer hopes that by giving you the address in this form, you'll be confused. However, tools like traceroute and whois will quite happily work on either dotted-quads or big decimal numbers. If you're happier working with the dotted quads, there's a tool at http://combat.uxn.com/ that will convert back to them.
IP addresses can also be represented in Octal (prefixed '0') or hexadecimal (prefixed '0x'), or even as a mixture of these within a dotted quad, in which case the above IP address might become:
0266.0xaf.0x5a.012
The key thing to remember is that if it works in your web browser, it'll work in traceroute and whois too, so all this obfuscation by the spammer is really a wasted effort on their part. What a shame. :)
The Really Long Dotted-Quad IP address
The dotted-quad I.P. address is just a way of representing a 32-bit number using four 8-bit numbers. It's a bit like the way you might right "1153" as one thousand, one hundred, five tens and three units. Now, in a dotted-quad only the lowest eight bits of each number are significant - to continue the above analogy, if we had "one thousand, twenty-one hundreds, five tens and three units", we'd discard the "twenty" from the "hundreds" column (because that would mean an extra two thousands and if we really wanted them we'd have put them in the "thousands" column, so it must be an error, right?) and still be left with the number "1153".
Some spammers make use of this by setting the high-bits of the four numbers in the dotted quad to make the I.P. address rather long and confusing. For example:
http://10889035741470030830827987437816582766808.4153837
4868278621028243970633761010.913438523331814323877303020
44767688728495784090.54445178707350154154139937189082913
83522/
It looks daunting, but dealing with it is quite simple. Just take each of the four dotted quads and ignore all but the eight lowest bits (ie divide each by 256 and take the remainder). In the example above, you'll end up with:
http://216.242.154.226/
and from here you've got the I.P. address and can continue as normal.
Note that only the least-significant 32 bits have meaning in an I.P. address; any other bits are put there by the spammer to further confuse us.
Alternatively, the URL de-obfuscator at http://combat.uxn.com/ will happily decode this kind of really-long-dotted-quad URL for you.
The Username Trick
You can specify a username and password in a URL using the @ symbol. For example:
http://jjf:fred@www.myreallysecurewebsite.com/
will log me into www.myreallysecurewebsite.com using the username "jjf" and the password "fred". But if www.myreallysecurewebsite.com didn't need a username & password, the username & password are ignored. Spammers use this to conceal their website's location. For example, is the following website located on members.aol.com or www.twinlobber.org.uk?
http://members.aol.com@www.twinlobber.org.uk/ispammedyou/
If you know this trick, it's fairly easy to see through it, so the spammers have now taken to trying a double-bluff. The username has to come before the first slash after the "http://" bit, and so the spammers try things like this:
http://members.aol.com/@www.twinlobber.org.uk/ispammedyou/
This URL references the directory "@www.twinlobber.org.uk/ispammedyou" at members.aol.com, not a website at www.twinlobber.org.uk itself.
Many of the URL de-obfuscation tools given below for decoding Javascript-encoded URLs will also deal with this trick.
JavaScript
A _really_ nasty technique is to encode the URL in JavaScript; this can result in URLs that look to you and me like absolute gobbledegook!
Fortunately, help is at hand. Have a look at these resources:
1.3.1.2 Is the spammer's URL always the place to complain to?
Spammers know that no matter how hard they try to mangle their URL in the manner described above, some people will be able to decode them. Therefore, they sometimes try to hide their websites using other methods as well...
Page Redirections
Another tactic favoured by some spammers is to spamvertise one URL but have that URL "redirect" visitors to another. In this way, the spammer hopes to confuse us, to misdirect complaints, and if the site that's redirected to is taken down he can just change the redirection page to point to another, identical site and still profit from his spam run.
Fortunately, in most cases, page redirection can be followed simply by looking in your browser's history window. Once you recognise this, the thing to do is complain to the hosters of both the redirecting website _and_ the website it redirects to.
Frames
A variant on the Page Redirection trick is to have a webpage on one site that contains a frame around a webpage on a second site; this way "Location:" field of the browser will contain the URL of the first site (the one containing the frame) and not the URL of the second site (the one containing the actual content). In Netscape, you can get the URL of the second site by selecting "Page Info" from the "View" menu; in Internet Explorer, right-click on the webpage and select "Properties".
1.3.1.3 Why does the spammer's website's source code look so weird?
Many spammers have learned that anti-spammers get important information about their operations from the source code of their website. So they've taken to encoding their webpages in JavaScript; this is decoded into HTML by your web-browser in order to display the page, but when you try to look at the source you just see gobbledegook-like Javascript.
Fortunately, help is at hand. Have a look at these resources:
Alternatively, users of Internet Explorer 5.x can install the "Microsoft Web Developer Accessories" add-on from Microsoft. With this tool you can highlight a portion or all of a webpage, right-click (or shift+F10) and select "View Partial Source". You now see the plain HTML that the spammer's JavaScript sent to your browser.
Some spammers go to almost insane lengths to obfuscate their websites, but the key to remember is that they have to be decodable by your web-browser, so they're decodable by you too. John McGowan has written an excellent example of how he doggedly disected a spammer's website; this can be found at http://www.lumbercartel.ca/archives/spamfaq.net/examples/cyberdetective/.
1.3.1.4 How can I stop a spammers' website doing bad things to my computer?
Some spammers' websites can do some quite nasty tricks, such as switching Internet Explorer to full-screen mode and not letting you escape, or opening lots of pop-ups, or re-opening the site every time you try to leave it, and so forth. If you use IE, you can put the spammer's site in "Restricted Mode" which will disable all JavaScript, Java, ActiveX, cookies and anything else on the site the spammer will try to trick or trap you with. In other browsers you can disable JavaScript and Java from the configuration window.
You can also use the advert-removing program WebWasher to prevent abusive JavaScript code from executing. Look for it at http://www.webwasher.com/.
However, beware; some spammers know that many anti-spammers surf with JavaScript permanently disabled and have written websites that look as if they have been killed if JavaScript is disabled yet are still fully functional for surfers with JavaScript enabled. Some other spammers websites will immediately redirect you elsewhere if they detect you have disabled JavaScript.
1.3.1.5 What if a spammer's website has disabled right-click?
Spammers know that anti-spammers get a lot of information about their revenue chains by looking at the source code of their website. So they have taken to writing little bits of JavaScript that intercept right-mouse-clicks on their webpage to prevent the context-sensitive menu containing the "view source" option in Netscape and Internet Explorer from appearing.
In Internet Explorer, you can also type into the Address box "view-source:" followed by the URL in question to see the page source, for example "view-source:http://www.spamfaq.net".
This can, of course, be circumvented by deactivating JavaScript in your browser, but there is also a simpler solution, as the "view" menu on the menu bar allows you to bring up the page source in some versions IE and Netscape. Alternatively, Shift+F10 will simulate a right-click in some browsers. Some Windows keyboards also have a "context-sensitive menu key" which can be used to call up the menu you'd normally get by right-clicking. Note that some spammer's webpages will now intercept these keypresses as well as the right-click, but the "view" menu on the menu bar should still work. (If the website contains frames you'll only get the source of the frameset - type the URL of the frame itself into your browser. Sometimes it'll automatically stick itself back in the frame - if this happens, disable JavaScript. If the page requires JavaScript, try using the w3c.org validator.)
1.3.2 What can I do about Spam-Supporting ISPs?
Most ISPs hate spam. Sometimes, however, you'll come across an ISP that is either utterly clueless or refuses point-blank to act against its spamming customers. In these cases, there are a number of steps you can undertake.
1.3.2.1 Research
The first step is to check the archives to see whether anyone else is having a problem with this spammer or with this ISP. If you can contact others who are having the same problems as you, you can pool your resources to better achieve an affect.
1.3.2.1.1 news.admin.net-abuse.sightings & groups.google.com
news.admin.net-abuse.sightings is a newsgroup for reporting - not discussing - instances of Internet abuse. The idea is that anti-spammers post instances of the spam they see to this newsgroup, and then other anti-spammers can look in this newsgroup to see if other people are getting the same spam as they.
But it gets better. Google's newsgroup archiving service at http://groups.google.com/ archives most postings to news.admin.net-abuse.sightings (along with most postings to most newsgroups); you can use the advanced search feature to search these archives for instances of a particular spam! For example, if you've received a spam advertising the website "www.iamareallybadassspammer.com" you could search for "www.iamareallybadassspammer.com" in the forum (Google-speak for "newsgroup") "news.admin.net-abuse.sightings" and find some other people who have been spammed by that spammer.
Incidentally, the Google archives for news.admin.net-abuse.email are also a very useful resource for priming yourself on specific issues. There are few new ideas; most spam-related issues will have been discussed in this newsgroup at some point or another, and many spammers have too.
1.3.2.1.2 Halls of Shame
news.admin.net-abuse.sightings is a very useful resource but sometimes you need something a little more structured. Unlikely as it may seem, there are anti-spammers who dedicate whole websites to keeping track of the unrepentant spammers and those who run spam-support services. These can be very useful in discovering a spammer's M.O., or just why you're having trouble getting a spammer's account at a certain ISP killed. Here's just a handful of such sites...
The Spamhaus Project tracks spam support services and spam-friendly ISPs, and displays the results in a number of easy-to-navigate formats, with links to "whois" information, relevant abuse addresses, and the like. As well as currently-active spamhausen it lists deceased spamhausen, including how many times they have been terminated and by which ISPs, and when. There's even a "league" of leading spam-support services.
In a similar vein is Sapient Fridge's Spamware Sites Listing; a list of websites that are selling Spamware or supporting Spam in other material ways, each coming with various service providers (with cross-references), handy links to traceroute tools, and their status with the MAPS RBL.
The Spammer Quick Reference Guide has by no means as many technical whizz-bangs, but it looks like a quite useful list of who's spamming what.
ROKSO is a good reference of hard-core spam operations that get thrown off Internet providers time after time after time.
SenderBase seems to be a good way of checking the spam-reputation of a domain or I.P. address, including blacklists and other statistics:
1.3.2.1.3 Posting in news.admin.net-abuse.email
If this research turns up a blank, then don't forget that a great way to contact other spamfighters about a suspected spam-supporting ISP is to post in news.admin.net-abuse.email.
1.3.2.2 Education
Sometimes an ISP will support their spamming customer simply because the ISP themselves don't realise that spam is bad. In these cases, it may be worthwhile taking time to briefly explain (patiently and without expletives) the problems around spam and why the ISP should take action against their spamming customers.
If you try this, you'll soon be able to tell whether an ISP is genuinely ignorant and confused or is purposefully supporting spam.
1.3.2.2.1 What if the ISP doesn't speak English?
There are an increasing number of ISPs, most notably those in the Far East, but also some in Europe and other parts of the non-English-speaking majority of this planet, where the technical contacts don't speak English. This can obviously lead to a communication difficulty if you yourself aren't fluent in their native language.
One solution is to use the Babelfish automatic translation service, but this technology can be a little flakey at times. It's probably better to get a bilingual friend to translate for you if at all possible.
For persistent spammers from foreign countries, you may be able to seek help from some of the foreign-language email abuse newsgroups, such as:
it.news.net-abuse - Italian net abuse newsgroup
fr.usenet.abus.d - French net abuse newsgroup
de.admin.net-abuse.mail - German net-abuse newsgroup
hr.news.net-abuse - Croatian net-abuse newsgroup
nl.internet.misbruik - Dutch net-abuse newsgroup
pl.news.mordplik - Polish net-abuse newsgroup
As a last resort, there are some anti-spam documents written in non-English languages, to which you may be able to refer non-English-speaching providers.
(All suggestions for this section are greatly appreciated!)
1.3.2.3 Contact their Upstream
An ISP's "upstream" is a bit like an ISP's ISP. Apart from a few very large ISPs called "backbones", every ISP purchases its connectivity with the rest of the Internet from one or more other ISPs, which are called the "upstreams" of the first ISP. Many of these upstreams will have clauses in their contracts about spam, and if you can show them that their customer is allowing spam to come through their networks, they may well cut them off or pressure them to take action.
Occasionally, you'll find that a spammer has tricked you into thinking you're complaining to their ISP when really you're complaining to the spammer himself. In these cases, by going upstream you'll find the spammer's real ISP.
If an upstream provider refuses to act, you can try _their_ upstream provider, and so forth until you reach a backbone.
1.3.2.4 Publicise their Spam-Supporting
Spam is unpopular, so if you publicise the fact that a large organisation is supporting spam, then you may be able to force them to change their mind. A posting about them in news.admin.net-abuse.email is a good place to start. If the provider has their own newsgroups, then possibly one of them might be appropriate for a posting too. And then, if you're really determined, you can move on to online magazines, newspapers, and so forth.
1.3.2.5 Bitching
A very controversial tactic is that sponsored by http://www.bitch-list.net/. This is a service a little like abuse.net, except that it forwards email to _every_ known contact address for abusive and unresponsive ISPs. The idea is that by forwarding abuse reports to as many officials and unrelated departments as possible, the message will get through somehow.
However, this is considered by many (including the faq-maintainer) to be sending Unsolicited Bulk Email and thus wrong. And even if you can get over that moral hurdle, it is extremely impolite.
|
||
Spamfighting is very important for reducing the amount of spam we'll all receive in the future but it doesn't do much to affect your spam intake for today. This section looks at some popular methods that are used to reduce the amount of spam currently ending up in mailboxes.
1.4.1 How can an individual reduce the amount of spam they get?
1.4.1.1 How do spammers get our email addresses?
The obvious way to reduce the amount of spam you receive is to make sure that spammers don't have your email address! Before we can go further with this, however, we must learn how spammers get hold of email addresses in the first place. As it turns out, there are five main ways:
They pick them up when they're used publicly on the Internet, e.g. in a newsgroup posting or on a webpage. This is by far the most common way, and is known as "harvesting". Using your email address in a newsgroup or on a webpage is generally understood to solicit personal, topical replies from individuals, but is not a solicitation to receive broadcast advertising.
They buy a CD of addresses from another spammer. These addresses were probably harvested from newsgroups or webpages in the manner described above, and are often years out-of-date to boot. As the saying goes, there is no honour among thieves...
They guess them. For example, it's a fair bet that "joe@example.com" could be a valid email address, although there's no way of knowing to whom it leads. When spammers concentrate this technique on one domain it is sometimes called a "dictionary attack". (As it happens, joe@example.com isn't a valid email address, because "example.com" is a domain reserved for testing and examples.)
Our ISPs sell them our email addresses. This is extremely rare.
We give them to them. Always carefully read the privacy policy of any website before you give your email address to it, as sometimes email addresses are passed on or used for purposes other than those we intended when we gave them.
For a more detailed look at how spammers find email addresses, have a look at these documents:
1.4.1.2 Choose a non-obvious email address
Some spammers guess email addresses, so it may be a good idea to use something that spammers can't guess easily. For example, instead of joe@example.com, why not have joe34z@example.com?
1.4.1.3 Be careful with your email address
The only way to totally eliminate the chance of receiving spam is not to have an emailbox. Even if you have an emailbox and never ever show your email address to anyone else, there's still the chance that a spammer might guess your email address. However, there are a few less extreme steps you can take to at least reduce the amount of spam you receive...
Never, ever give your email address to a company you do not trust entirely. If in doubt, open a free email account with a web-based provider such as hotmail.com and use that address for communicating with the company; that way, if they do spam, you can close the account and you've only lost a free email account you weren't using for anything else.
Never, ever post to usenet using an unmunged email address you care about. Use a throw-away address from a free email provider or munge your email address as described in 1.4.1.4. (Some people have reported that you can reduce spam without impacting upon the ease of contacting you, by posting with a munged From: address or an unmunged Reply-To: address, but I can't believe the spammers won't catch on to this eventually.)
Never, ever allow your email address to appear on a website, including on a web-based discussion board.
Some people concerned about privacy enter made-up email addresses into online application forms and the like. This seems like a good idea, but it is important to make sure that the made-up domain you use doesn't actually belong to anyone, otherwise you'll just be sending spam to the innocent third-party who owns it. This can become a very serious problem for the owners of some domains popularly used in such forms.
BAD MADE-UP EMAIL ADDRESSES
walt@disney.com
go@away.com
GOOD MADE-UP EMAIL ADDRESSES
this@address.is.made.up.invalid
go@away.invalid
There are several free mail-forwarding services that can be used to reduce your spam-level. The idea is simple; you give a different mail forwarding email address to each company that asks for your email address, and the mail forwarder forwards all mail to these addresses to your usual mailbox. If a company ever starts to spam you, you just disable the forwarding address you gave them and you won't get their spam, without affecting your other incoming mail. Companies who provide this service include:
1.4.1.4 Address Munging
"Munging" is the act of mangling your email address so that it can still be read by a human but cannot be automatically harvested by spammers.
For example, my email address:
jjf@mungedeg.twinlobber.org.uk
Could be munged into any of the following:
jjf<at>mungedeg<dot>twinlobber<dot>org<dot>uk
jjf@mungedeg.twinlobber.org.uk.REMOVETHISTOSENDEMAIL
jjf@NOSPAM.mungedeg.twinlobber.org.uk.NOSPAM
fjj@ku.gro.rebbolniwt.gedegnum.REVERSE-TO-SEND-EMAIL
When munging, you have to be careful not to accidentally munge your own email address so that it's identical to someone else's, and should always munge the bits to the RIGHT of the @-sign and not just the bits to the LEFT (otherwise your ISP will still get your spam even if you don't yourself). Also, you should ensure that your munged domain name is NOT an existing domain (else the poor sod who owns it could get your spam).
Recent drafts of the Usenet message format RFC specifies that the From: line of a newsgroup posting must contain either a valid email address or an email address ending in ".invalid". Your munged email address should really comply with this forthcoming standard, e.g.:
jjf@REMOVE-CAPS-AND-INVALID.mungedeg.twinlobber.org.uk.invalid
Note that some spammers now have harvesting software that can remove widely-used munges like "NOSPAM".
1.4.1.5 Whitelisting
Some ISPs forbid their customers from using a munged email address. In these cases, whitelisting can be an alternative. In this, you set up your mail account such that some given word or string of characters must be in the subject line for any mail to be accepted, and then you explain this in any newsgroup postings and webpages containing your address. This way people can respond to you, but spam will be deleted from the server without you having to spend time downloading and reading it. This works especially well with webpages, e.g. use:
<A HREF="mailto:unmunged@example.com?Subject=FRIENDLYMAIL: Comments about my webpage">
Send me email!</A>
Then kill any mail that doesn't have FRIENDLYMAIL: in the subject line and have the rest forwarded to your real email address.
1.4.1.6 Filtering
There have always been people who have filtered spam using simple rules in their email client; for example, depending on your tastes, it may be a fair bet that any message with "FREE LIVE SEX" in the subject-line is spam, and can be deleted or filtered into a separate folder that the user will clean out by hand. However, this has always been a somewhat hit-and-miss approach, requiring hard work and made more difficult by the somewhat crude filtering capabilities of many popular mail programs.
More recently, personal spam-filters have started to appear. These sit between your mail program and your mailbox, using more advanced methods to filter or tag likely spam messages. The number of personal spamfilters has skyrocketed in recent months; I even wrote one myself (SpamPal). Most of them work in different ways, and will have differing strengths and weaknesses. Here's a few links to get you started:
Free spam-filters for Windows users:
Commercial/Shareware spam-filters for Windows users:
Spam-filters for Unix users:
Spam-filters for Macs users:
There are also various companies who will filter the spam from your mail without the use of additional software. These include:
1.4.1.6.1 What is Bayesian filtering?
Bayesian Probability Filtering is an increasingly popular spam-filtering technique which has been integrated into popular email programs such as Mozilla. The idea is that you "train" the filter to recognise spam from non-spam, by telling it whenever it makes a mistake. This can be quite succesful because everyone's spam is different and the types of legitimate mail everyone gets is different; for example, anything I get that mentions "Viagra" may be spam, but another person may have a bedroom issue and legitimately need to discuss Viagra with someone. (Or vice versa.) The down-side to Bayesian filters is that it takes an appreciable effort to train the them; pre-trained Bayesian filters aren't really practical.
1.4.1.6.2 Challenge-Response Tools
Challenge-response systems, also known as "Reverse Whitelisting" or "Permission-based" fitering, take a different approach to traditional spam-filters. Whereas traditional filters start from a stand-point that all mail is good then try to detect the spam, Challenge-Response systems start by assuming all mail is spam then only letting through people on a "whitelist". If the user receives mail from someone not on a whitelist, the system "holds up" the mail and sends a "challenge" message to the sender. If sender replies ("responds") to the "challenge" message, the original message is "released" and allowed into the user's mailbox, and the sender is "whitelisted" so any future emails will be allowed through without this rigmarole. The theory here is that the spammers won't bother to reply to the "challenge" - most of them are using forged email addresses so they won't even receive the "challenge".
Put like that, it sounds like quite a good idea. But the simplicity of the solution doesn't reflect the complexity of the real world, and challenge-response has a number of problems:
Mailing lists, especially discussion lists. If I send a message to a mailing list with 1000 subscribers, would I receive - and have to respond to - 1000 challenge messages? Many Challenge-Response systems allow the user to whitelist a mailing list automatically, but this can be unreliable (and judging by experience plenty of people forget).
Automated mailings - generated by a computer with no human intervention - have no human sender who can respond to the challenge message. This immediately breaks things like password reminder messages, confirmed opt-in mailing lists, Cron job notifications and so forth. Again, these things could be whitelisted manually - but you have to remember, and anyway guessing the email addresses most of them will be sent from would be difficult.
Forged sender addresses. Spammers often forge the addresses of enemies or just random individuals as the senders of their spam - if a spammer forges me as the sender of a 1,000,000-recipient spam-run, the last thing I want to receive is a "challenge" message from each and every victim!
And of course, simple challenge-response systems can be fooled if the spammer stops using forged email addresses and sets up a simple bot to reply to the challenges. It has been suggested that challenge-messages could include a graphic image containing a number that has to be typed into the subject of the response, in order to prevent automatic responding, but this breaks the system for blind users and adds an extra hoop for senders to jump through. While it's tolerable if you only communicate with one or two new people every day, if you're (like me) exchanging emails with many new people every day (if you work in support, for example) then going through a prolonged challenge-response procedure for everyone - or even a fair proportion of senders - would be an enormous pain at best.
1.4.1.7 If I use a tool to send "bounce messages" for any spam I get, will I get less spam in the future?
When you send an email message to an address that doesn't exist, you receive a "bounce message" back. (If you've never seen a bounce message, try sending an email to "joe@example.invalid" and you'll get one back within minutes.) There's a school of thought that says that if you could somehow send fake "bounce messages" in response to the spam you receive, spammers will remove you from their mailing lists and you'll get less spam in the future. To this end, there are various tools - the most well-known being MailWasher - that will generate such "fake" bounce messages.
The general consensus on news.admin.net-abuse.email is that this is a bad idea. Here's a few reasons why:
There is lots of anecdotal evidence that suggests spammers as a rule are not interested in removing dead email addresses from their lists - for example, The Story of Nadine.
The return address in almost all spam messages these days is forged, probably because the spammer knows his mailing list has lots of bad addresses and he doesn't want the bounce messages to fill up his own mailbox. So any "fake bounce" you generate probably won't reach the spammer anyway.
So at best, your "fake bounce" will hop around between mailservers consuming computing resources before being quietly dropped. However, a lot of spammers forge their spam to look like it came from the email address of a real person - either someone who's annoyed them (e.g. an anti-spammer) or just some poor soul picked at random. So your fake bounce message - together with those of everyone else who uses such a tool - would end up in the mailbox of this entirely innocent third-party. (My own email address has been forged in this way and let me tell you it isn't a pleasant experience - I have no idea how many of the thousands of bounce messages I received were real and how many were fake, but the last thing I'd have needed to receive were even more.)
By examination of the headers and included information in a bounce message, it's possible to make a reasonable inference as to whether it is real or fake. So even if your bounce message did somehow reach the spammer, his systems may well be able to figure out that it's fake and ignore it appropriately.
1.4.2 How can an ISP reduce the amount of spam their customers get?
1.4.2.1 Stop Accepting All Email
This will immediately reduce the spam intake of their customers to zero. Unfortunately, it also destroys email as a usable communication medium. In order to prevent this becoming necessary whilst still taking action to reduce their customers' spam levels, many ISPs adopt policies that are midway between blocking everything and doing nothing...
1.4.2.2 Filtering
One tactic used by some ISPs to cut down on spam is filtering. The ISP scans incoming mail and any messages that match the pattern of a known piece of spam are discarded. The big danger with filtering is that of false positives; users are unlikely to be very pleased if some non-spam mails are mistaken for spam by the filter and never arrive.
Some of the filtering techniques discussed in 1.4.1.6 can also be applied across an entire I.S.P., although there may be additional risks due to questions of scale.
1.4.2.2.1 DCC
DCC (Distributed Checksum Clearinghouse) is based upon a very simple idea - if only we knew what email everyone was getting, we could detect what was bulk and what was personal. DCC works by collecting "checksums" of incoming messages (and not the email messages themselves) in distributed databases, and counting the frequency with which each checksum occurs. Using this information, spam can be filtered out. The down-side is that solicited bulk email must be whitelisted or it too will be filtered out.
The DCC code is currently available for a variety of Unix-like systems, and is intended to work best when installed close to the mail server.
1.4.2.3 DNSBL lists/Blackholing/Blocklisting
Blackholing (or Blacklisting) is a variation on filtering whereby an ISP refuses to accept any email from machines that have a reputation for producing a disproportionate amount of spam. Many administrators have had some success with this tactic, although there are two main problems with it: firstly, someone will have to add more spam-sending machines to their list as more emerge if the effectiveness of the list is to be maintained, and secondly it is hard for the ISP to know when a machine on the list has reformed and is no longer emitting spam.
Of course, with any type of blackholing, any legitimate email from machines on the blackhole list will be lost along with the spam emails.
The main tool for blackholing are so-called DNSBL Lists. These are publically available lists of IP addresses that can be queried using a DNS lookup. There are a wide variety of DNSBL lists listing IP addresses according to various criteria; an individual site will have to choose the services to use based upon their own requirements. It isn't possible for me to discuss or link to every single DNSBL service, but I will cover a few that are most frequently discussed in the newsgroup.
But first, a word of warning. If you configure your server to use an external listing service you are turning over part of the control of your server to that service. You should exercise caution when you do this, and keep an eye on how the list is being used. If you have no means of your own to verify the integrity of the service you should pay some attention to a newsgroup such as news.admin.net-abuse.email or news.admin.net-abuse.blocklisting and be alert for any reports that the service you have chosen has started to slip in quality.
You should also bare in mind that most of these DNSBLs are provided as a public service, but if you don't have a contract with the maintainer they may be withdrawn at any moment. On occaision, withdrawn DNSBLs have been set to reject everything in order to get people to stop using them quickly. Carefully monitor your mailserver and any third-party DNSBLs you choose to utilise.
A few specific DNSBLs are mentioned below, because these are the DNSBLs that are most frequently discussed in news.admin.net-abuse.email. However, there are a LOT of DNSBL services out there, and you would do well to evaluate more than the handful listed in this document before choosing which ones to implement.
There is a sister group to news.admin.net-abuse.email dedicated to discussion of blocklists; news.admin.net-abuse.blocklisting.
1.4.2.3.1 MAPS
Mail Abuse Prevention Systems LLC is a Californian company who were one of the pioneers of DNSBL lists. They offer a number of different services, including the famous RBL (Realtime Blackhole List), DUL (Dialup Users List), RSS (Relay Spam Stopper), and NML (Nonconfirmed Mailing List).
MAPS have fallen out of favour with many regulars of news.admin.net-abuse.email since they stopped making their services freely available. Users now require a static IP address, and need to sign a contract (although there is no monetary fee for individual and hobbiest sites). However, they are still used by many thousands of Internet sites, and have a reputation for causing a minimum of collateral damage.
1.4.2.3.2 Spamhaus SBL
The Spamhaus SBL (Spamhaus Block List) lists all I.P. addresses belonging to known spammers, spam operations and spam support services. It draws on data from the Spamhaus Project and ROKSO as well as other sources..
1.4.2.3.3 SPEWS
The Spam Prevention Early Warning System, or SPEWS, is one of the most controversial DNSBL lists. For one thing the people behind it have chosen to remain anonymous and silent. For another, its policies are surrounded by mystery. It is believed that SPEWS lists spammers and hosts connected with them, presumably based upon some kind of evidence, but the exact criteria they use is uncertain. Mind you, it certainly seems to catch a lot of spam.
SPEWS' website has in the past suggested that SPEWS listings are discussed in news.admin.net-abuse.email, which is why you see so many SPEWS-related threads in the newsgroup. These days, such listings are more properly discussed in news.admin.net-abuse.blocklisting
1.4.2.3.4 SpamBag.org
SpamBag.org publish a list of the parts of the Internet controlled by "anti-social elements" (such as those who send large amounts of junk email), as defined by some very detailed criteria layed out on their website. By blocking traffic from machines on this list, providers can protect their customers from such anti-social elements.
1.4.2.3.5 SpamCop BL
The SpamCop Blocking List DNSBL service is based upon an analysis of the complaints sent through the SpamCop service - the sites that generate the most complaints get listed. While this is a very effective method of stopping lots of spam, it can also result in some alarming mistakes and false-positives, and so this experimental DNSBL list should only be used in caution.
1.4.2.3.6 Collateral Damage
Most blackhole lists try to be as specific as possible with the exact parts of the Internet that they list. However, sometimes an upstream ISP will move a spamming customer around in their I.P. space, in order to avoid such lists, and it will become necessary to list the entire ISP. However, that ISP will have other, non-spamming customers, who will also suffer the ill-effects of being in the list; these innocents have become collateral damage in the spam wars.
Collateral damage is sadly inescapable, and is directly the fault of those companies who support spammers in this way. Organisations and individuals so affected are advised to find themselves a different, more responsible ISP to escape the collateral damage blackhole.
The analogy of living in a slum neighbourhood is often invoked for those innocent people who become collateral damage, and I find it very appropriate. If you live in a bad part of town, you may find that pizzas won't be delivered after dark, taxis won't hang around, and so forth. Similarly, if you live in a spam-supporting ISP then many other organisations simply won't want anything to do with you. Just like living in a slum, you have two options: either help clean up the neighbourhood (persuade the ISP to stop supporting spam) or move somewhere nicer (find another ISP).
1.4.2.3.7 I'm not a spammer but I'm being blackholed! How do I fix it?
What has almost certainly happened to you is that your internet provider, or their upstream, has been facilitating spam or spammers in one way or another. Therefore large parts of the Internet have taken the decision to protect themselves from spam by accepting no email from these providers and all their customers.
You are probably an innocent caught in the middle; you're not a spammer but your email is bouncing and you can't contact your friends or your family or your customers. You're entirely justified in feeling very angry about this.
But the many Internet Providers who are shunning your provider are not the right targets for your anger, and neither are the organisations that recommended that your provider be blocked. Instead, you should direct your anger towards your own provider (or their upstream). After all, its their policies, freely decided upon, that have lead to you being cut off from parts of the Internet. If you have a Service Level Agreement with them then you should study it; if your provider is not providing the promised level of service then you may be able to claim compensation or take legal action against them.
If you can persuade your provider to mend their ways, then you will be on the road to becoming free of the blackholings. Alternatively, your only real option is to move to another, less spam-friendly Internet Provider.
You may wonder why the blackholing can't be made specific to the active spammers of the providers, or why just your own I.P. address cannot be removed from the blackhole. Unfortunately, this is not practical, as too many I.S.P.'s have in the past moved their spammers to new I.P. addresses to help them to evade blackholing. To guard against this, the entire I.S.P. in question is generally blackholed.
Your situation is regrettable, and we all wish this wasn't necessary. We feel much sympathy for you, but ultimately we feel more sympathy for the millions of victims of your I.S.P.'s pet spammers.
Occaisionally, you may encounter some problems because your I.S.P. has assigned you an I.P. address that once belonged to a particularly notorious spammer; such addresses often persist in providers' local blocking lists for months or even years after the spammer in question has departed. Since your address is probably present in hundreds or even thousands of such lists, getting it removed from them all will be a next-to-impossible task, so your best course of action in this case would be to ask your I.S.P. for a new I.P. address (and maybe take them to task for selling you damaged goods).
(You may also want to read the answer to question 1.4.2.3.6, which covers this issue from the other direction.)
1.4.3 How can an ISP reduce the amount of spam their customers send?
With difficulty. However, experience has shown that there are a few things that can make a difference...
If an ISP has a reputation for dealing with spammers quickly and decisively, many spammers will avoid them. If spammers are dealt with very rapidly indeed, the ISP may be able to shut down a spam-run before it has completed.
An ISP can have a clause in their terms of service that allows them to charge "clean-up fees" to any customers that send spam. Unfortunately, many spammers sign up using stolen credit-card numbers, and in these cases clean-up fees aren't much of a deterrent. It can be messy to collect clean-up fees, too.
An ISP can implement "port 25 filtering" (see 3.5.3 in "Understanding NANAE") to prevent their customers from spamming via open relays. Note that this, however, will prevent their customers from using external mailservers for legitimate reasons too.
An ISP can regularly "port-scan" their users, to check that they aren't running any open proxies or open relays that could be abused by spammers. This is particularly important for so-called "24/7" ISPs, such as ADSL or cable providers.
An ISP can monitor the email traffic generated by a customer. If a customer who hadn't previously sent more than three or four emails a day suddenly sends a hundred thousand messages, for example, it's a fair bet that he's a spammer and it would be nice if there were systems that would inform the ISP and let them take a closer look.
|
||
1.5.1 Why do anti-spammers fight spam?
There's no collective answer to this - different people will have different motivations. However, three of the most common ones are:
1.5.2 Aren't anti-spammers just a load of anti-business communists?
No. Some anti-spammers own businesses, and most of the rest work for businesses. Anti-spammers are generally NOT anti-business. In fact, many anti-spammers happen to believe that businesses that cannot survive without stealing the computing resources of others (i.e. spamming) should go the way of the dodo. It's called "capitalism".
1.5.3 Aren't anti-spammers just a load of anti-commerce net-nazis?
See 1.5.2 above.
1.5.4 Don't anti-spammers just want to control email on the Internet?
No. Controlling all email on the Internet, apart from being a practical impossibility due to the distributed nature of the system, would be an extremely big job to undertake purely to satiate a few egos.
1.5.5 Why don't anti-spammers spend their time stamping out porn instead?
Porn isn't what gets anti-spammers hot-under-the-collar; spam is. Anti-spammers are drawn from a surprising cross-section of society and you'll find that they hold wildly divergent views about the contentious issues of the day, pornography included. However, they are drawn together by the simple opinion that spam endangers the email system, which they really rather like.
1.5.6 Why don't you anti-spammers just get a life?
We have lives. Part of our lives involve sending and receiving email and so we want to protect this when it is endangered.
1.5.7 Are anti-spammers all Systems Administrators?
Sometimes, when reading news.admin.net-abuse.email, you can get the impression that in order to be an anti-spammer you have to be a technical wizard and run your own mailserver. This isn't the case at all, and the point to remember here is that the only people who contribute to highly-technical discussions will be those with highly-technical knowledge, but this doesn't mean that there's not less-technically-minded people reading.
Anti-spammers tend to be drawn from many sectors of life with many different types of knowledge. Some do run their own networks and their own mailservers, but many do not. This FAQ-maintainer, for example, is a Java programmer. Many anti-spammers don't even work in the computer industry; they can be florists or brick-layers, brain surgeons or secretaries. It doesn't matter. The skills needed for most spamfighting are fairly easy to learn and the more voices that are heard on this issue, the better.
1.5.8 If you anti-spammers are so smart, why am I still getting spam?
So who said we were smart? ;-)
As a problem, spam has not been solved. We will probably never be able to completely eliminate spam from this world, any more than we can expect to eliminate robbery, assault, or bad music. Realistically, our aim must be to reduce the spam levels as much as possible, to a level where it doesn't greatly impinge on the usability of electronic mail.
That's an achievable goal. We aren't there yet, and we have a long way to go, but we've come a long way too. Someday, someway, we _will_ get there.
|
||
No document of this magnitude can be the work of only one man. I would like to thank everyone who offered ideas and suggestions, everyone who pointed out grammatical errors and gaps in my logic, and places where I was just plain getting things wrong. This wouldn't have been possible without you, people.
|
||
You may copy and redistribute this FAQ in unmodified form by any means or media you see fit.
You may modify the presentation of this FAQ as you see fit, so long as the content remains unaltered.
You may modify the content of this FAQ so long as you appropriately credit both your changes and the original authors of this FAQ. At a minimum, the link to the FAQ's website _must_ remain in place.
|
|||||||||||||||