Spam belt tightening done badly rejects legitimate mail
Posted 21 Jan 2004 at 20:04 UTC by lkcl
Wanadoo, the French ISP, recently volunteered its entire dial-up IP range to the SMTP blacklists: BTOpenworld, a leading UK ISP, followed suit last week. Yahoo is touting their own proprietary extensions to SMTP; BT recently managed to mess up the DNS configuration of post.btinternet.com, resulting in them getting RBL'd. If several Small - and major - ISPs cannot configure their servers correctly such that mail can get through, what hope is there?
This is just insane.
All we want to do is to send email, like we could ten years ago. I recently got sufficiently fed up with SPAM to install marc.merlin.org's sa-exim package with spamassassin, razor, pyzor and exim4: it's FANTASTIC. i dropped the spamassassin threshold to 1.5, added four friends to the /etc/spamassassin/local.cf file as whitelisted entries, and i now receive ONE spam message per two to three weeks.
everything is logged: i grep the whole lot every couple of weeks to see if i missed anyone.
Last month, i got phone calls from a couple of friends saying that their messages had bounced: they had received messages saying "firstname.lastname@example.org doesn't exist". i had _no_ clue what they were talking about, so decided to check it out.
what i found was that the ISP that they had used had so badly configured their SMTP server (typically by their host not having an MX record for their _own_ SMTP server or sometimes not even having an IP address in their own DNS records _at all_) that my SMTP server was flatly refusing to accept mail from them.
it turns out that the RFCs on SMTP specify what is needed to be done in these circumstances: you must inform postmaster@thesender'sdomain.co.uk - so in that case it would be email@example.com - NOT firstname.lastname@example.org.
_however_, most people who have email redirected, and handled by their ISP, don't _have_ a postmaster email@example.com - most people don't even know what a postmaster _is_.
so, what happens?
the message to inform them of the problem gets bounced - fortunately to them!
do they care?
do they xxxx!
all they want is their email to be delivered.
am i going to change my SMTP configuration?
am i xxxx!
i'm not a nanny: i'm not going to hold their ISP's hand to get them to fix their broken DNS configurations, especially after one experience where the ISP responded with "oh, it must be because you don't have any web pages with us. if you create some web pages the problem will go away."
... by magic, i presume?
ironically, my SMTP server hadn't even had to work up a sweat running spamassassin to detect these problems: they were _basic_ MX record and DNS record cock-ups.
next, we come on to BT. by the way, if anyone works for BT and is reading this PLEASE tell the admins to fix their damn server post.btinternet.com!
recent changes to BT's SMTP configuration have left them with two IP addresses: 220.127.116.11 and 18.104.22.168. remember, these are the guys dealing with the _entire_ btopenworld ADSL and 56k customer base!!!
one of these is post.btinternet.com, the other is carbon.btinternet.com. if you telnet to port 25 to carbon.btinternet.com, you get no response. if you telnet to port 25 to post.btinternet.com, you get an SMTP header greeting you from carbon.btinternet.com!
yet, if you look up an MX record for btinternet.com, it's listed as hosts moongate and stargate.
this is _so_ frustrating.
because of these mess-ups, anyone sending mail from BTopenworld dialup networks is likely to get blocked by _basic_ SPAM-checking rules let alone something like spamassassin.
if the big boys can't get it right, what hope is there for any of _us_?
now we come to yahoo, who recently proposed that they intend to extend SMTP, a la hashing, which is quite likely to be easily forged.
pointless and fruitless.
we have gone from being able to communicate by email to being attacked by viruses that install hundreds of thousands of noddy SMTP servers on compromised hosts, world-wide, notwithstanding SPAM advertisers who think it's their God-Given Right To Send Anything To Anyone They Please, It's In The Constitution And Our Rights Apply To The Entire World Not Just Our Country, Dammit.
and people are forced to think of ways to stop these idiots from overloading our ability to communicate, and they frequently get it wrong.
what's the answer?
I'll plug my own project, CAKE. Seriously, I think there are two major thrusts to the answer. The first is a strong system of public-key backed pseudonyms, which is what CAKE's main purpose is. The second is a built in challenge response protocol that requires the person the challenge is issued to perform some sort of task or give you money.
The task could be some sort of CAPTCHA thing designed to identify humans. Or it could be some sort of expensive computation that is feasible to do to communicate with one person, but isn't feasible to do for each person when you want to send mail to millions.
Damn, how about this, posted 22 Jan 2004 at 06:49 UTC by tk »
Maybe we should reedirect all spams to the moronic postmasters.
fixes or fittings?, posted 22 Jan 2004 at 12:06 UTC by lkcl »
the issue as i see it is that retro-fixes to an existing system just ain't gonna cut it.
it's more a social engineering thing than anything else.
you have to convince people to implement such a scheme on windows, linux, solaris, mac, everything, and in postfix, exim, sendmail, qmail - the works.
maybe that's possible, maybe it isn't.
personally i think that an approach that stands a much better chance of success would be to design an _alternate_ service to SMTP, backed by reputable ISPs and the IETF, designed from the ground up with security, authentication, identification and filtering in mind.
at least then you would stand a chance of being able to make a global announcement, "better SMTP than SMTP! no more spam!"
hey, i'd be delighted if it were possible to retro-fit MTA filters into SMTP that were easy enough for any programmer to write a plug-in module on any system, and delighted if it were taken up and actually installed - and used - correctly.
...maybe we're all looking at this the wrong way: maybe it's possible to mandate the use of IPsec or IPv6 and to use that as a chance to cut unauthorised spammers off at the knees...
lkcl: no, posted 22 Jan 2004 at 13:19 UTC by tk »
As pointed out before, "If foolish coders
ignore the old standards, why would they magically pay attention to the new
thanks ..., posted 22 Jan 2004 at 22:15 UTC by lkcl »
... for the reference. ... much as i am confused to say it, and mad as it is, the present situation is about it!
other than... how about creating VPNs (tinc, openvpn, ipsec).
e.g. how about this:
ISPs create, amongst themselves, VPNs that they pass mail between themselves over that VPN (store and forward?).
ISPs only allow email into their network from trusted - registered - users. it should be easy for users to register with another ISP's SMTP service if an ISP misbehaves.
so, a bit like DNS where there are TLDs and it goes down from there, mail could be routed "up" the tree to the SMTP-TLDs and then "down".
the rules are that you do NOT allow SPAM from one level to the next,
and if you do, you can be cut off at the knees, by your peers, not by some arbitrarily managed blacklist.
the smaller and more involved the group of peers is, the more likely they are to help each other out, and the more likely they are to take swift and _correct_ action.
ultimately, users (us) would not "see" any of this: you would have mail delivered a la SMTP.
the significant difference is that, rather than at the moment where email can get trashed by some company running brain-dead SMTP server that doesn't check the RBLs properly, they can trust the top-level SMTP servers delivering to them _and_ they can block the firewall rules in the surefire confident knowledge that no legitimate SMTP traffic will ever come to them from anywhere but their upstream SMTP provider.
i'm writing this as i think it, out loud, i'll try to clarify it a bit, later, as it pulls in several ideas and existing practices that i have heard of, but the hierarchy bit and VPN bit is a new idea (hey to me, at least).
MX Records, posted 23 Jan 2004 at 21:38 UTC by johnnyb »
The poster seemed to indicate that the SMTP servers need MX records. I don't think that's the case, and could be the cause of your difficulties. MX records are only needed for receiving mail, not sending it.
apologies, posted 24 Jan 2004 at 15:04 UTC by lkcl »
thank you for correcting my misconceptions. i check the exim logs and see a lot of messages rejected by the exim4 + sa-exim combination, because there is no MX record possible to be retrieved for the name: turning this around, i misconcluded that a sender would have to have an MX record.
this report in the times newspaper, today, quotes 70% of all email traffic, 3 billion per day handled by the lovely aol.com, as being spam, and likely to be 90% by the end of the year.
indication from spamhaus interview on how serious this is getting
if spamhaus' assessement is correct, in that 90% of all email by the end of the year is going to be spam, and that there are 400,000 "granny-boxes" out there with windows-virus-installed SMTP servers, we could indeed be looking soon at the total breakdown of email communication.
based on this assessement, it is almost worthwhile having in place a strategy to replace email.
hasn't it occurred to anyone yet that microsoft's product is causing such a serious problem?
The VPN idea, posted 24 Jan 2004 at 19:08 UTC by Omnifarious »
I don't like the VPN idea at all. It smacks of a centralization that will be even more harmful than the spam problem. I prefer adding things to email clients, and changing the nature of the messages that are transmitted so that it's much harder for spammers to send you messages that you'll actually read.
hierarchy and peer cooperation. in the UK there are 25 major ISPs: i would expect those 25 ISPs to be responsible for... say... store-and-forwarding all *.uk domain name based email. for example.
not everyone can have stuff added to email clients, that's the whole point.
for example, if using GPRS, you _certainly_ don't want your 14,400 (allegedly 33,600 baud) bandwidth - for which you pay £1.50 per megabyte - loaded 90% with total shit: that makes it ten times more expensive - approximately £0.20 - per actual real email received (est.)
the VPN part isn't strictly necessary: it kinda "codifies" however the relationship between ISP peers and their communication paths up and down the domain hierarchy.
store-and-forwarding is becoming common: firewalling to only accept incoming SMTP from the forwarding SMTP server is perhaps not so common but is something i would like to see people take up as "the norm".
at least it will cut out the 400,000 or so compromised NT systems.
in the end it should even be possible for ISPs to simply block SMTP port 25 even between their dial-up clients!
If people stop responding to spam, it'll stop being sent. And if that's not what happens, the Internet infrastructure is in big trouble no matter what. At the current exponential growth rate, it won't be long before 90% of the backbone bandwidth is used by spam.
Any kind of exclusive arrangement at all seems the wrong thing to do to me.
There's new stuff that gets added to email clients all the time. 10 years ago, no email client had MIME. Now, only Unix email clients widely support PGP/MIME. New stuff can get added to email clients. If it solves a pressing problem that a lot of people have, it will be added to email clients extremely quickly.
Solution, posted 25 Jan 2004 at 14:29 UTC by Malx »
There are people able to sent SPAM.
There are people paying for SPAM to be sent.
There are people reading and using SPAM for their benefit.
So ... Up until all of them exist SPAM will be. No blocking will help. :)
Is there any solution? Yes - if there are people who need it, then we should do it! But in the manner which is not abusing ISP or users.
1. We should create Spam Delivery Protocol (ICD - Internet Comercials Delivery).
2. We should convert ICD to e-mails for the end-users, but with addition of "unsubscribe" mechanism.
3. We should popularize the idea among business, ISPs, users, admins and SPAM-senders
If you could do system, which will cost less and be close-targeted then SPAM, then there will not be any reason to pay SPAMers. You just pay ICD representative for delivering your info! :)
It is essential that first ICD message is delivered and only then you could "unsubscribe" from all like this (so it must be split in categories).
It is essential to implement special protocol so you could deliver only one copy of message to ISP and then it should be sent to every interested customer. So the "unsubscribe" messages whould be handled be the local ISP!
Why ISP will ever receive those messages?! Of course because there are 1) interest from end users in receiving those messages, 2) ICD-sender company would set contract with that ISP. And it will save them money to recive only one copy and not to hire special admins for SPAM-fighting
Why company pay for it? Because it now are able to select region to deliver message to and it will not be punished for doing so. And they know how to pay for delivery!
Why end users will be happy with that? Because he will recive information he really needs! It could receive list of today TV-casts or plan of holidays or shoping discounts or (for geeks) new projects relesed.
And the last - how to beat your own complex against SPAM? :)
Just think of ICD as new system like NNTP(global delivery, regional settings, clear names) + IRC(tree delivery mechanizm, one message to every, ban/ignore server side) + multicast MBONE(subscribe/unsubscribe) + RSS feeds(news from companies) + E-mail(familiar user interface).
Are you voting for ICD? ;)
Let's convert SPAM! :)
a contract with the providers: no spam, no money.
it would be quite easy to set up companies based on guaranteed or minimised spam.
marketing strategy: save time and money not having any spam delivered to your company.
lkcl: wrong-headed, posted 26 Jan 2004 at 06:23 UTC by tk »
Paying providers to send you extra junk? This is a**-backwards. The trend
these days is that web companies are asking users to pay in order
not to receive advertisements. It's obvious which way the
preferences are going.
Show me any free web hosting service / free e-mail service / free whatever
which asks people to pay for more advertisements.
When the ad becomes the thing you really need now, it is no more spam, it is "usefull info".
Example - you need a computer and recive an "ad" about computers from 10 companies. Then you are able to select best for you.
Or you are reciving notifications about new posts of Advogato - again it is the same "ad" but YOU need this information and you CAN unsubscribe it any time. Could you list all notification maillist you subscribed to now? (security, annonce etc) They are the same! :)
It is the best thing when people whould receive information they need.
I do not know about other countries, but in Russian (and russian speaking region) there is http://subscribe.ru/. There are 16809 mailing lists (that means one way list - you can't post there, only the owner of list could post messages to it). And 2148549 people are subscribers.
Even 10 years ago there were commercial newsgroups delivered via UUCP. You could subscribe them to recive advertizements! And they were popular, because it was really usefull info. It is way better then to search through bunch of web-sites. :)
There were people who need nice colorfull WEB - and it is created now.
There are people who need SPAM - and we should provide nice way of delivering it. Arn't you agree? :)
Already paid for, posted 28 Jan 2004 at 11:16 UTC by tk »
The people who want to send these ads will already pay the web hosts.
tk, posted 29 Jan 2004 at 12:33 UTC by Malx »
But the E-mail is easie to use. And it consumes less time to find anything you need :)
You have not answerned about number of mail-lists you are on ;)
Some thoughts that chip at the realities in spam
ICD is a wonderful idea, but why would any spam-advertiser want to use anything but spam? It's cheap, and it's blazingly effective. There were, mid-90's or so, opt-in mailing lists, they failed miserably.
if spam was about advertising as information, they why do I never receive any spam these days other than for Viagra, Nigerian officials or some other obvious scam? I don't remember the last time I saw UCE where the C meant "commercial" ... most commercial companies instead send me Dear Info emails addressed to my website, but addressed and with a valid reply-to, and those messages are in the 1% of the load.
filtering with fancy AI filters that log the false-positives works to a point, but it's getting so that spam attachments now seriously eat into the stupid bit-caps that the telco/cableco ISPs are inflicting. There may be places in the world where bandwidth is unlimited, but not in Ontario.
Now let's ponder some further realities and ask ...
- Daddy, where does Spam come from?
Way back in the old days, spam came from improperly configured Sendmail servers; although this was fixed nearly 10 years ago (and as much as 5 years ago in the third world) there is a popular urban myth that persists on the compromised university-lab mailhost.
Spammers don't use compromised servers anymore, they use hundreds of thousands of compromised Windows boxes. Long live supercomputing! And why Windows? Because
- Windows users tend to have DSL or better connections but lack any sort of monitor ability on their bandwidth use
- Windows itself has only one security level, root
- Windows has notoriously bad track record for thinking in a security conscious way about rolling out new features. they deploy first, ask questions later
- What can Microsoft do?
Microsoft doesn't want to fix spam, spam is Microsoft's best friend. They could fix spam in the next release with two reasonably doable but non-trivial changes to their O/S strategy:
That's it, that's all. Spam would be stopped dead because it would now find the desktop as impervious to relaying as the current mail hosts. They'd then make max-protection the default, and maybe throw in their own hack-act version of iptables in each box while there's mucking about in the code ...
- implement a multi-tier permissions system so virii cannot infect the O/S.
- double check the permission system so they know they actually did (1) right.
But Microsoft won't do that. Not a chance. They've known of this solution since at least 1995 and they haven't made a single step in that direction. It could be because organized crime buys an awful lot of computers, but I don't think so. I think the real reason may be far more sinister:
- Email continues to be the killer-app online with far more people using Email than any other single protocol. Unfortunately, Email is an open standard, meaning any vendor can implement SMTP and create an email client --- Email cannot be owned.
- Spam attacks email, threatening the one network application that 90% of net users cannot live without. It's not that they'd go back to telephone and snail-mail, they must have electronic messaging, and spam is killing email...
- Microsoft has never made any move to fix the permission system. NT has permissions, but not the desktop, leaving the average cable/dsl connected desktop as wide open as a 1992-vintage university server ... only there are millions of them. What, instead, is Microsoft's spam strategy? Glad you asked ...
- Black Penny, which Gates has announced will roll out "within two years" is proprietary electronic messaging that will be the defact standard messaging system on 90+% of the world's desktops by 2006. Just as today you can't exchange docs with a lot of MsOffice shops (OOo works but not that well and not bilaterally, and it will be worse when MS patents DOC's XML), in that near future you will not be able to exchange messages with any of your corporate clients ... not unless you have Black Penny.
QED: If Spam did not exist, it would have been necessary for Microsoft to invent it. Spam is their universe unfolding exactly as they want it.
I personally like the Remington 870 approach to fixing spam -- find the sending unit(s) and fill them full of 1 oz. 12 guage 00 shot at 1400 FPS.
Don't most modern nations have anti-spam laws on the books? Can't we, as taxpayers, get some of our legislative bodies to arise from their phat asses and kindle some enforcement of these laws? Why keep pushing technical solutions to what are basically social problems?
Does anybody here remember the mid-'80s copy protection craze? It was a daily race between the creators of copy protection (for diskettes, no less!) and the crackers of copy protection. I had one gig where there was a club of guys whose sole enjoyment in computers was breaking copy protection. Once guy was telling me he had over a thousand titles for three different platforms (MS-DOS, Apple II, Mac) but had never really used any of this stuff. He didn't want to do productive work on his box, just break copy protection schemes.
Let's examine this problem from a larger perspective. Compare junk email to junk snail mail, telemarketers, and other forms of advertising. What can be done to limit the intrusion of spammers into our daily communications? How about following up on the trail of spammers as is done on the senders of faxes? Remember, it costs money to receive a fax (paper, toner, blah blah blah). In the US it is not legal to send a fax without a return phone number enclosed. (Somebody back me up on this -- is that just commercial faxes?)
What happens if the whole civilized world starts insisting that all emails have a valid, living and breathing, non-machine-generated, return address before it is conveyed? What happens if the civilized world stops forwarding email without such return addresses and at the same time starts prosecuting the people who send such crapola? D'ya think all this might make a difference?
We need to fix spam at its source, which is the profit motive that makes scummy people willing to polute an entire communication medium just to make a couple of sales. If it costs to spam then the spam will stop. Nothing short of this will have any impact that I can see.