SPF is a very broken anti-forgery technique which you should not implement. You should neither publish SPF records, nor check SPF on incoming email.
If you use SPF, you will be causing genuine email to be rejected. There are much better alternatives which address the forgery problem without throwing the baby out with the bathwater.
SPF is Sender Policy Framework, formerly known as Sender Permitted From. It makes the assumption that mail from a given email address will only ever be seen when sent directly from certain IP addresses which are associated with that address, and it gives the domain owner a way to specify which IP addresses are 'acceptable'.
So, for example, I might state that mail from the infradead.org domain will only be seen from hosts with hostnames ending in 'infradead.org', from the IPv4 subnets 81.187.226.96/29 or 81.2.98.173/30, or from the IPv6 subnet 2001:b80:10b::/48. The idea would be that if you ever see a mail claiming to be from an address at infradead.org which is coming from a different host, then you should know that it's a forgery.
(That's about as far as the pretty graphical but mostly content-free official explanation goes, but if you take a moment to look closer or think about it, there's a little bit more to it than that...)
Unfortunately, this assumption is false. You do see perfectly genuine mail from my domain, from machines other than mine. This happens due to mail forwarding. People tend to change their ISP quite often, but don't want to have to tell everyone that they've changed their email address. So they have an account elsewhere, at a vanity domain or on another computer, and they forward mail from that address to whichever is their current ISP, or their employer.
SRS
There is a proposed solution to this -- the SPF advocates say that the whole world should 'upgrade' to make the initial flawed assumptions on which SPF is based come true.
Their idea is that when a server forwards a mail, it shouldn't just use the original sender's address as has been done for the last couple of decades. Instead, it should munge the sender's address as it resends the mail, faking an address at the domain which is actually responsible for the forwarding. Then, if a bounce is generated, that faked address receives the bounce and the bounce needs to be forwarded on to the original sender of the original mail.
This forwarding of bounces could be easily abused if done naïvely, so the idea is that we also include a timestamp to time-limit it, and a cryptographic hash which can only be generated locally. And of course you have to include the original sender's address; both the domain and the username. So when Bill Gates sends me an email to my vanity domain, for example, and my computers forward it to my current ISP, they'd actually end up rewriting the sender address to: SRS0=hh=tt=microsoft.com=bgates@infradead.org instead of leaving it intact.
This rewriting scheme they call 'SRS', and they need it to be implemented everywhere before they can use SPF without throwing away genuine mail.
------------------------------------------
Problems with SPF
You may have managed to spot one or two potential problems with this scheme all by yourself, but some people do seem to have trouble so let's spell them out explicitly:
SRS is not common. If you publish SPF records, you are going to be asking people to throw away genuine email which you did actually send.
If you check SPF records strictly for anything larger than a toy domain with a handful of users, then you're likely to be rejecting genuine email which is sent to you or your users.
There is little incentive for people to deploy SRS. It's a cumbersome and convuluted workaround for a broken assumtion.
Those who believe SPF to be an insanely stupid idea will strongly resist, and those who are simply uninterested in SPF have little reason to want to use it. Yet SPF require these these uninterested third parties to upgrade too -- it's not just limited to those who are participating.
People are very slow to deploy new ideas on the Internet, and especially with respect to email, even when those ideas are good. Many people haven't even managed to deploy Extended SMTP (ESMTP) yet, even.
There will be a porcine implementation of RFC1149 before SRS is ubiquitous.
Some people claim that SPF directly combats spam. It doesn't. SPF attempts to address forgery. In fact, a large amount of spam rates an SPF 'pass' result, because spammers have rapidly adopted SPF for themselves.
You still need a blacklist or other kind of trust database, to tell you which domains are trustworthy and which are not. But we already have lots of blacklists; it's just that we list the IP address instead of the domain name, to tell you which hosts are trustworthy and which are not. We don't need to break email and force everyone to upgrade to some bizarre new scheme just for that.
Although the uptake of SRS on forwarding sites will be low, it can still be a common spammer trick. Any spamming host can do the SRS trick to 'take responsibility' for forwarded mail. If your ISP receives mail from the address I used in the above example -- 'SRS0=hh=tt=microsoft.com=bgates@infradead.org' -- then they have no real way of telling whether it really did come from Bill Gates via my servers, or whether it's a fake. SPF would just accept the mail, depending on my domain's reputation in the trust database. It's all about how much you trust the one server which is offering the mail -- it's not an end-to-end authentication.
So SPF is really not any better than blacklisting by IP address or HELO name. Again, the breakage it imposes is not worth it.
The original sender address is useful information, and can be lost if an intermediate host mangles the mail by using SRS.
A sender address may be present in a blacklist, or may fail SMTP callouts. Yet an intermediate host 'takes responsibility' for the mail in question, causing it to be accepted anyway.
Sender addresses are also used for mail filtering. The sender address is the most reliable method of filtering mailing list traffic into its own folder; using Cc: and/or To: headers has both false positives and false negatives, and using the List-Id: header has false positives too. Again, SRS mangles this information.
How good are your spam filters? By rewriting the address of mail you're forwarding so that it appears to come from your own domain, you put your own reputation on the line. You could
be blacklisted for mail which you claimed even though you didn't send it and you have no real knowledge of the original sender.
In the simple case without forwarding, SPF does manage to give a clear indication that a mail is valid. But that's not really very useful in practice. What we need is a blacklist but SPF gives us only a whitelist. We can't safely use that to reject mail -- we can only reject mail if it's clearly invalid, but SPF can only honestly say 'valid' or 'unknown'.
------------------------------------------Alternatives
In summary, SPF doesn't really achieve what it sets out to achieve, and it breaks far more than it's worth. The costs far outweigh the benefits.
Some SPF advocates would claim that you can't make an omelette without breaking eggs; that the spam problem has got so bad that we have to break things and switch to a new email system.
That's silly for two reasons. Firstly, if you're going to break the world and make people upgrade, you might as well just switch wholesale to something like Internet Mail 2000 rather than pretending it's still compatible with SMTP. SMTP has enough other problems (like the way bounces are done, the difficulty of implementing variable preferences for different users, etc.) that we might as well fix it all at once, if we find it necessary to make such an incompatible change.
Secondly, and more relevantly, there are alternative schemes which give a measure of protection against address forgery but without the flawed assumptions, the loss of genuine mail, and the need for the rest of the world to 'upgrade'. You would have to be mad to choose all the problems of SPF, when other schemes are far saner can and offer the same benefits:
These schemes (and some others) are all fairly similar in nature. They offer a cryptographic solution to the forgery problem, where a sender adds headers to the mail with a signature, and that's verified by the recipient by checking against a published key for the domain or user in question.
Google, Yahoo and Earthlink, amongst others, are already using DomainKeys on outgoing email. AOL are reportedly going to be doing so in the early part of this year (2005).
These solutions don't suffer from the forwarding problem; they're compatible with email as we know it today, and offer the same protection as SPF does but without the false rejections.
There is an effort under way to merge these into a single proposal for approval by the IETF. See the MASS web page for more details.
These proposals are based on the same idea, originally inspired by the concept of using SRS on your own outgoing email instead of on forwarded mail.
For example, I never send email with MAIL FROM:<dwmw2@infradead.org<; it's always rewritten to appear to be from a time-limited SRS-encoded address.
This means that I can instantly start rejecting bounces targeted at the raw email address, because I know they're bounces to mail I did not actually send. And this, in turn, means that anyone doing SMTP sender verification callouts will reject faked mail claiming to be from that address, because a bounce cannot be delivered.
Since SMTP callouts are unpopular in some quarters, SES takes the concept further and provides alternative methods for validation of reverse-paths, and also allows for a message digest to be included in the reverse-path, to prevent replay attacks in the event of an encoded reverse-path being discovered and abused by a spammer.
By implementing SES or BATV, you can instantly stop accepting bounces to mail which you didn't send.
CSV is a very simple method for identifying servers by their HELO greeting instead of just by their IP address. It uses SRV records in DNS to specify which IP addresses may use a given hostname in their HELO greeting. It also offers a method of storing the reputation database in DNS.
Since SPF can only really be used for evaluating the trust level of the host which is actually sending the mail, CSV is sufficient to use as a viable alternative.
------------------------------------------Further reading
"When banks start DomainKeys or S/MIME signing all outbound mail, I promise to give up SPF and Sender ID."
-- Meng Weng Wong, inventor of SPF.
How about just a .45 to the back of the head for all spammers?
[ahem]
But seriously, folks...what happened to the promise of PGP encoding for all SMTP traffic that we heard about in years gone by? Aren't any of the simple solutions viable? Do we need proposals for changes that will impact the entire Internet and won't be backward compatible? Please.
Where are all the simple solutions? Like ISP enforcement of bulk email quantities? Or the ICANN's monitoring and DNS-blacklisting of all those crappy Taiwanese SMTP servers? Or any of the sixty-seven jillion other proposed solutions to crushing spam? Oy.
But seriously, folks...what happened to the promise of PGP encoding for all SMTP traffic that we heard about in years gone by? Aren't any of the simple solutions viable?
PGP isn't really the answer in general because it requires MIME and doesn't reliably survive mailing lists. You'll get broken signatures and hence false rejections.
However, the crypto solutions described in the 'Alternatives' section above improve on that idea by hiding the signature in the header and being applicable only to the most recent sender address -- which would be the mailing list itself in the case of a mailing list, so you don't have to worry about breaking the original sender's signature.
Do we need proposals for changes that will impact the entire Internet and won't be backward compatible? Please.
No, we don't need such proposals; that's my point. The alternatives given above are much more compatible with the real world of today than SPF is. Take your pick of those instead.
The article makes a lot of presumptions over what exactly SPF does and is.
Yes, SPF allows you to specify rules stating that a certain server can send mail from your domain. Yes, that can break forwarding. Yes, SRS sucks. No, that doesn't mean SPF is useless or broken.
First, you should *not* be publishing -all SPF records. A -all command in an SPF record means "deny all" and would usually be the last item in the list of SPF commands. The problem is, of course, that doesn't handle forwarding. However, you *can* publish a ?all record. That means that "any host that doesn't match a previous rule might be valid, or maybe it's not, I don't know."
What's the use of this? It eases policy. For example, many organizations have started blocking mail from free mail hosts like Yahoo and Hotmail because the domains are spoofed so very often in viruses and spam. (Note: that *also* breaks forwarding, along with all other mail from those domains.) Using SPF, Yahoo and Hotmail can let you know what the real servers are. You can then add a rule to your domain that will throw away Yahoo and Hotmail messages only if you can't guarantee they're valid. In this case you'd still dump forwarded messages, but the alternatives for these organizations is even worse.
Some mail systems use *very* resource-intensive anti-spam measures. Or use cumbersome sytems like white-lists with reply verifications. Using SPF, you can let mail bypass these systems if the SPF record verifies as guaranteed valid.
The above scenario does have a hole: spammers can use SPF records, too. The answer is something that *ALL* long-term anti-spam solutions need: domain ratings database. Namely, a way to check if a given domain is a spammer domain, with checks for things like the age of the domain and whether it has a web site and so on, in addition to more concrete ratings. So then you have mail in four classes:
* Explicitly denied. (Yahoo knows that 1.2.3.4 spoofs their mail and isn't a forwarding service/user, so they deny it explicitly. If you get mail from 1.2.3.4 with a yahoo domain, you should drop it.) * Unknown. (Not explicitly granted or denied; the default for most hosts.) * Verified, possible spammer. (It is definitely from the domain owner, but the domain *might* belong to a spammer. The mail should be scanned with your stronger, and likely slowest, content scanners.) * Verified, not a spammer. (It is definitely from the domain owner, and the domain owner isn't a spammer.)
So in the first and last case your mail system's life just got a let less over-loaded. ;-) In the second and third case your system's load didn't really change at all, save for the SPF lookup itself, which is pretty lite-weight and will be cached if you run a caching domain server.
Now, that's not all. SPF can do far, far more than just check the sending machine's IP address and the envelop domain. SPF can tie into other authentication systems as well.
Let's say you use DomainKeys. (*gasp* it also breaks things - that's what happens when you fix a flawed and insecure protocol, you have to change it.) You can use SPF to describe your DomainKeys setup, specify rules for when it should be used, and so on. SPF becomes your globally-readable configuration file for your DomainKeys setup.
Let's say you use PGP keys, in which each user of your company has their own key. (Unlikely setup, but possible.) You could define a method for specifying a key server and use SPF to tell mail servers about that key server. Mail servers that support this system would get a mail, check the domain, see that the SPF record tells it to check a PGP key against the server, and does so. If the mail is not signed or the key isn't valid according to the specified key server then the receiving mail server knows the mail was forged. No, this PGP mechanism doesn't exist yet (that I know of), but you could help form the specification for it, and simply utilize SPF in that specification to greatly reduce the amount of code that implementations need to implement (as they will likely already have SPF reading code).
SPF is the Sender Policy Framework. It's a framework for defining policy. While the most common and simplest policy the framework is capable of using does indeed break forwarding, and while some people may think that SMTP can never be evolved to handle that problem, it is simply untrue that this weakness will actually affect SPF's over-all usefulness. There are other policy mechanisms that SPF supports and new policy mechanisms will likely be added with time. In the future we are likely to see different organizations using one of several different popular mechanisms based on whichever best fits its needs.
Yes, you could publish a record ending in '?all'. That's what I meant by my reference to whitelists. And yes, you can use whitelists to avoid time-consuming spam checking in some cases. But that's not what the SPF advocates are getting people to do -- in fact some of the SPF advocates are even treating '?all' as if it were '-all' in some cases. Without the ability to safely publish records ending in '-all', it's basically useless for its intended purpose, although there are still ways you can abuse it to achieve interesting results.
One possibility you didn't mention is that a record which says only '-all' for a domain which never sends mail is actually sane. But that again is beside the point. Since people can't safely reject mail for a 'fail' SPF result, I don't see the point in doing that even for those domains. And you don't need SPF for that anyway.
The example of an explicit 'fail' for a known spoofer is tenuous at best. Yahoo doesn't keep track of these -- other people do though. And the offenders don't spoof only one domain either. This is what IP-based blacklists are for -- again you don't need SPF for that.
Your example of a "globally-readable configuration file for your DomainKeys setup" doesn't seem to make much sense to me, and again it doesn't bear that much relation to SPF as it is advertised and advocated. Even if there's a practical way to do as you suggest, it's a private thing. I happen to use DNS TXT records as a globally-readable alias database for my virtual domains -- but the globally readable nature isn't really a feature; the fact that DNS allows easy replication across all my MX hosts and DDNS allows domain-owners to modify their own aliases is the feature. If you want to use SPF-like syntax as some kind of private configuration method, you gain nothing by using actual SPF records, surely?
Yes, you're also right that SPF can be theoretically be extended with new mechanisms which aren't based on IP addresses, and aren't entirely broken. But that hasn't happened yet, although the 'exists' mechanism can already query for the existence of a fairly arbitrary host name, and hence be used in conjunction with a stunt DNS server to achieve some interesting results.
But again that's not really the point. Since a recipient can't trust that an SPF 'fail' result is sufficient to reject the mail, the flexibilty of potential extensions to SPF grammar is in vain. Unless we're going to see a new 'reallyfail' result code, perhaps? :)
You can abuse SPF records in interesting ways to achieve some vaguely useful things, I agree. But SPF as it is presented, as an anti-forgery mechanism based on IP addresses, is not workable. Furthermore, because so many people are publishing records ending in '-all' when -- as you correctly point out -- they shouldn't, you can't trust a 'fail' result. So even the interesting abuses like stunt DNS servers aren't really a useful way to publish information to third parties in a way that allows them to safely reject faked mail.
When I first heard of SPF, I thought it was useless and misguided. Their misleading-by-omission marketing doesn't exactly help the situation. However, I am now fully in favour of SPF, with the following stipulations:
1) It is nothing but a whitelist. Anybody who thinks otherwise is crazy. However, it allows the sender (that's me) to define the rules the receiver will use to tell if the mail is really from me or not. In other words, if my rules are sufficiently strict, you can guarantee that the message you're getting either is, or is not, from me. If my rules aren't sufficiently strict, then you can't (and I get my flexibility to use mail forwarders and things). But if you know for sure that the message is from me, and you know for sure you want messages from me (eg. you've sent messages to me in the past) then you can have a 100% reliable whitelist for messages from me; I can completely bypass your normal spam scanner, and messages from me will never be lost. In short: I, the sender, control whether you'll incorrectly reject messages from me. If I claim I'll never use a mail forwarder, and then I use a mail forwarder, then I'm an idiot.
2) It's unarguably better than the current system. That's a very strong statement. Why is it unarguably better? Because it will work fine (no worse than the old way) if nobody publishes SPF records, if you publish "infinitely permissive" SPF records, if you check SPF records or if you don't. It's a general solution that includes the old system as one of its allowed implementations, so if any variation of SPF deployment is potentially better than the old system, you can't lose.
3) Reliable whitelists permit strengthening your blacklists. If you know for sure when a message is from me, and you know that you want messages from me, then you'll be assured that messages from me are never, ever blocked, until the end of time. If you send a message to me, your anti-spam system can auto-whitelist my email address. (Without SPF, mindless address-based whitelists are pointless because spammers fake messages from people you've communicated with in the past.) And if everyone you've ever corresponded with is auto-whitelisted, then you can turn up the insanity level on your spam scanner: only messages from people you don't know could ever be blacklisted by accident (false positives). If that happens, they can always send you a less-suspicious-looking email next time.
4) SRS sucks but is orthogonal so we can ignore it. Well, yes. Enough said.
In the meantime, I don't agree that it's better than the current system. If it could only be used for whitelisting, or if there was even a way to publish records which could stipulate that they are only to be used for whitelisting, then that might be true. But people are using it for other things, and some are even rejecting mail if they hit '?all' at the end of a record.
Leaving your From: address as one which you forwards to you but you can no longer even access the mail server for is broken behavior, and trivially fixed. Reasonable accomodations can be made, but that one isn't reasonable.
Your premise is up their with the old arguments that newgroup commands should be issued only after a long beurocratic process on usenet because the awfulness of some nntp servers combined with the awfulness of some file systems caused them to melt if there were too many newsgroups. Please fix your own personal setup before demanding that the entire internet be architected so you don't have to bother.
You speak of "Leaving your From: address as one which forwards to you but you can no longer even access the mail server for", but I don't see how that is relevant to the discussion. SPF isn't about the From: address, and you don't specify what type of access you mean -- do you mean access to send mail by SMTP AUTH, access to read mail from a local mailbox, or access to change your .forward file on the host in question?
Like many who support SPF, you seem very confused. Please could you try to be more specific about what you mean and why you think it's broken, and then we can help you to understand.
dwmw, I was referring directly to your original post.
Oh, I'm 'confused'? If you had attempted a rebuttal I'd respond, but your complete lack of one strongly indicates an attempt to get the conversation bogged down in technical detail, a highly disingenuous tactic, although one I find unsurprising given how overtly destructive the behavior you're recommending is.
The fact remains, you only have one example of functionality which SPF damages, and that functionality is highly dubious at best and clearly worth sacrificing. In fact, many admins of machines whose former users are abusing forwarding by not re-subscribing to mailing lists at their new address view breaking that functionality as very much a feature.
Yes, I do think you're confused. Many people who support SPF are confused, and your responses in particular have certainly lacked coherency.
My original post didn't refer to the From: address at all in the context of SPF, and neither did it mention anything about not being able to access the mail server. You seem to have gone off into the weeds -- I can't respond because I can't actually work out what you're trying to say or how it relates to SPF.
I'm sorry that you feel my request for clarification is a 'highly disingenuous tactic' and that you don't feel able to explain your meaning more coherently. I'm only vaguely surprised that you seem to consider a technical discussion to be a bad thing -- many SPF advocates seem to take that point of view, in my experience.
If you change your mind in future, please let me know. But I am only really interested in technical discussion, I'm afraid -- the purely emotional knee-jerk response of "spam is bad, and we must force the world to change and throw away valid mail to help protect against it", and adopting the first hare-brained scheme which comes along instead of the saner alternatives, isn't really something which can be fruitfully discussed in a forum such as this.
Btw, if forwarding sites don't want to continue forwarding for old users, they have no obligation to do so. It's bizarre for you to say that the admins of such sites may view SPF's breakage as a 'feature'. Partly because they can stop forwarding for themselves if they really want, and partly because they can't actually publish SPF records which prevent forwarding through their site -- SPF would be published by the original sender, and checked by the ultimate recipient. Again, what you say just doesn't really make sense.
New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.
Keep up with the latest Advogato features by reading the Advogato status blog.
If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!
del.icio.us
Digg
Google bookmark
reddit
Simpy
StumbleUpon
Furl
Newsvine
Technorati
Tailrank