Older blog entries for dwmw2 (starting at number 203)

Why are people so bloody clueless about email? I received this in snail mail from my bank today:

Account Number xxxxxxxx Sort Code xx-xx-xx
Your statement
Your statement for the above account, is ready to view by logging in to online banking at www.natwest.com.

Unfortunately, we have been unable to deliver this alert to you by email. This may be because the email address we hold for you (DAVID@WOODHOU.SE) is incorrect.

That has to be almost the most clueless bug report I've ever seen. It should have included at least some of:

  • Precise date and time of the latest delivery attempt
  • Sender's email address
  • Sending server IP address
  • Which MX host was being delivered to
  • The rejection message from the MX host

If I hadn't been running my own mail server, I'd have had no way to work out what happened — no ISP is going to go trawling through their logs looking for a needle in a haystack based on virtually nothing.

Since I do run my own, I was able to log into all the MX hosts for that domain, look through the historical mail logs on each of them and I happened to find their failed message among all the lots of other people trying to fake mail from NatWest:

2009-04-21 00:38:20 +0000 1Lw40C-0002sE-3D H=mailhost7a.rbs.com [155.136.80.121] F=<OnlineBanking@Information.natwest.com> rejected after DATA: Your message lacks a Date: header, which RFC5322 says it MUST have.

Upon calling them to tell them of their problem, I was asked "who says our mails lack a Date: header?" and "who says that they should?".

After dealing with that, I left the first-line support person with three items to pass on to Nat West's technical team:

  1. The lack of Date: header on their outbound mail
  2. The uselessness of the letter they send when they can't deliver email
  3. The fact that they are converting email addresses to upper case, when localparts may well be case-sensitive
I wonder what the odds are of any of them actually getting fixed?

Maybe I should have added "you're sending outbound mail without GPG-signing it" as a fourth item? :)

26 Mar 2009 (updated 26 Mar 2009 at 08:55 UTC) »

Today is the third birthday of GNOME bug #336076, which I filed to report a particularly idiotic regression in Evolution's IMAP code. (Update: It looks like I also posted about it on Advogato, too.)

Instead of just issuing a simple STATUS command to check the status of each folder for new mail, Evolution started to actually open the folder, fetch the headers for all new mail in it, re-fetch the flags for all mail in it.... and it does this for every folder that it's checking (which, with bug #336074 still unfixed, is all folders — not just the active folders. So in my case it was continuously re-fetching the flags for years of archived mail in folders which are marked on the server as being inactive.)

This meant that it took Evolution two HOURS to start up that first time, when connected across the Internet. Even when I ran it on a local machine which was connected to the server by Gigabit Ethernet, it still took 23 minutes to start up; downloading half a gigabyte of mail before it was usable.

I don't know what's scarier — the fact that this utterly moronic regression got into the code base in the first place (what in fuck's name were they thinking?), or the fact that GNOME 2.26 went out last week with it still not fixed, three years later.

I've actually moved my older archived mail folders off to a separate server to work around bug #336074, and I've stopped checking for new mail in folders other than the INBOX to work around bug #336076, which is a PITA but is the only way to keep Evolution even vaguely usable — and it's still extremely bad over a slow connection, such as GPRS (or connecting home from China).

It's not just at startup, either. It goes off into the weeds frequently, doing this stuff in the "background" while I'm waiting for it to fetch the mail I just clicked on. Sometimes, I end up using pine to read my email while I'm waiting for Evolution to do whatever weird crack-inspired stuff it's doing with the IMAP server and start responding again.

I think it's about time that the choice of default mail client for GNOME was re-evaluated. Evolution seems to be mostly stagnant, and the changes that are being made seem to be entirely dubious. Version 2.24 was a significant regression in many ways. Evolution is definitely letting the side down.

This kind of post inevitably leads to people mailing suggestions for an alternative MUA. Changing MUAs is a painful process, but I think after the 2.24 release I've reached the point where I'm going to have to give up on Evolution. Things I really need from the MUA are:

  • Graphical folder 'tree' showing the number of new mails in each folder (currently broken/disabled in Evolution as described above).
  • Ability to reach mail server over ssh: ssh $MAILSERVER exec imapd
  • No mangling of outgoing or incoming patches
As far as I'm aware, the latter two requirements rule out Thunderbird. I think I'm going to try Sylpheed. Last time I did that, it would SEGV at startup, which quickly put me off — but I'm sure that's fixed now, and I've heard good things about it. Next alternative if I can't get on with that is probably kmail.

Whatever I use, it would also be nice if it handled the calendar stuff that the Outlook/Exchange weenies use — preferably with the calendar on the Exchange server, but just using its own calendar (as I do in Evolution) would be fine.

(Of course, Evolution being the steaming pile of crap that it is, it fucks up the calendaring too. It has its own idea of what the timezone is, perhaps because it thinks it might be in a different timezone to the rest of the system? So for someone who travels a lot and uses the calendar infrequently, it's fairly much guaranteed that a meeting will be displayed in some arbitrary, wrong, timezone. And just for fun, it stupidly displays the meeting times without any hint about the time zone. )

I finally got round to writing up some documentation on the greylisting setup that I use, and that we've been shipping in an exim-greylist package in Fedora for some time.

This setup avoids some of the common mistakes that greylisting implementations make, and tries hard to avoid delaying mail except where it's actually likely to be a benefit to you. Mostly, that means:

  • Remember which hosts actually do retry, and never delay mail from those hosts in future.
  • Only delay mails which actually look suspicious in some way; don't just delay everything blindly.
  • Avoid greylisting for hosts on the DNS Whitelist database.
It's amazing how many greylisting implementations miss all three of these fairly obvious points. I often see my outgoing mails being delayed due to greylisting, by hosts which I deliver mail to all the time. That's just stupid. They know it's going to be retried, so all they achieve is a delay on mail that they're going to accept later anyway.

I also see a lot of greylisting which happens at RCPT time, without even looking at the mail. I appreciate that some people claim that they don't want to use the extra bandwidth to actually look at the mail, or the extra CPU time. I think that's a very poor decision, if it means you're delaying mail that has absolutely nothing wrong with it. Bandwidth and CPU time on a mail host really shouldn't be an issue these days. Some people even do it at RCPT time when the sender is empty (a bounce), which means that sender verification also fails (temporarily) and they end up delaying their own outgoing mail.

Using dnswl.org is something I added quite recently, and also makes a lot of sense — if the host is registered as a known mail server, it's almost certain to retry the mail and therefore you gain nothing by greylisting except for a delay.

This greylisting is done purely in Exim's ACL configuration, which is quite versatile enough to handle it — there's no need to call out to external software at all. For storage, it uses an sqlite database, again using Exim's built-in capabilities rather than calling out to an external database server. (Thanks to Jeff Garzik for that bit; I used to use simple text files with a fairly evil hack to append to them, but he converted it to sqlite for me after I added sqlite support to Exim.)

It shouldn't shock me, I know, but...


Date: Tue, 13 Jan 2009 08:24:17 +0000
To: David Woodhouse <dwmw2@infradead.org>
Subject: Re: UK - United Kingdom (KMM81905425V35010L0KM)
From: Yahoo! UK & Ireland Login Support <uk-account@cc.yahoo-inc.com>
X-Bad-Reply: 'Re:' in Subject but no References or In-Reply-To headers

Hello David,

Thanks for writing to Yahoo! UKIE Customer Care.

In order for us to proceed, please reply to this email and provide us
with your desired new alternate email address. Please note that this
cannot be a Yahoo! Mail address.

We will then be able to update your account information.

We look forward to your reply.

Regards,

Joseph

Customer Care - Yahoo! UK & Ireland

CONFIDENTIALITY NOTICE: This email and any attachment is confidential
and may be legally privileged. It is intended for the named recipient
only.




Original Message Follows:
-------------------------

On Sun, 2009-01-11 at 23:35 +0000, Yahoo! UK & Ireland Login Support wrote:
> Currently, we are unclear on exactly what it is you want done to your
> account. Please reply to this message with a more detailed explanation
> of what you need done.

Have you managed to find a competent adult to help you read my message
of January 8th?

Perhaps you could just refer them to
http://advogato.org/person/dwmw2/diary/197.html which has all the
required information.

To ensure that I'm talking to a real person rather than an automated
script, please include, but _paraphrase_, the following text in your
response:
    "I understand that Yahoo's automatic reminder emails are broken,
     and we need to get our own technical people to fix the system."

--
dwmw2

WTF?


Today 14:11:31

Cleared CLEARED 00 - GPMS case has been cleared
Today 14:09:03

BT 15 - Previous notes confirm the fault is due to overhead cable damage. So please book appointment via usual interaction with BTW for the further investigation in the customer premises

OK, we believe it's overhead cable damage. So... why are you clearing the fault? Just another fraudulent attempt to stop the timer from counting up and avoid having to pay the correct amount of compensation, presumably?

And why in hell do you want us to book an appointment so you can investigate in my premises? There are no BT overhead cables running through my house, I assure you!

I've seen the cable which is probably at fault. It's here. A fracture in the cable could probably explain the crackling I hear when I make an analogue call, and the other strange effects like the fact that the DSL often drops as soon as I pick up the handset.

What confuses me, though, is that BT closed this section of road a month ago, in order to put new copper in to provide a second line to my house — a second line which I've been waiting for since October. But they didn't notice and fix this problem, and they're still claiming that there are NO SPARE PAIRS in this section of the route! How in hell does that work? What were they doing when they closed the road, in that case?

Fucking Useless Telco.

My DSL line has been unusably slow and unreliable since around mid-November, and BT's responses in the fault ticket are, as usual, varying between the stonkingly incompetent, the dishonest, and the outright fraudulent.

They claim to have sent an 'external engineer' on January 3rd, who claims to have tested the line and reported it OK... despite the fact that we were in France skiing on the 3rd, and the line was connected at a speed below the Fault Threshold Rate all of that day, until the evening.

They seem to be referring to this alleged visit as "Chargeable" despite the fact that we clearly did not ask for their "Special Faults Investigation" service, so it would be a violation of the Unsolicited Goods and Services Act 1971. And despite the fact that the charge is for "repair activity beyond the NTE", and they would have to have broken into my house to do any of that in my absence.

They keep clearing the ticket with idiotic claims such as "EU Equipment/settings/drivers believed incorrect" despite the fact that they've been here (on a previous occasion when I was actually present) and checked it all.

They seem to have learned another trick recently, to fraudulently avoid the 'Service Level Guarantee' clock counting up and having to pay compensation for the extended failure. They've taken to reassigning the ticket back to us, telling us to wait for the BRAS profile to change. But the BRAS profile is just a reflection of the state of the line, limiting IP traffic so that we don't saturate the ATM link. So effectively they were saying "we won't bother with this until the problem magically fixes itself... and we're not letting the fault timer count up while we wait, either".

They also tried arbitrary delays — clearing the ticket with "GPMS notes state that the line is not stable so case needs to be slept for 8 hours. So setting a sleeper for remaining time.". As if another 8 hours is going to make a difference when the line has been buggered since mid-November.

They closed the previous ticket outright, even though the line was still dropping, so this particular ticket has only been open since December 30th — but BT's blatant fraud means that they've only just admitted to exceeding the 40-hour Service Level Guarantee, almost two weeks later.

As usual, it's quite scary reading the fault ticket. The responses from the BT side seem completely nonsensical at times, as if they haven't even bothered looking at the fault history. They've even cleared the ticket reporting that the line is in sync, while it's clearly connected at a rate below the fault threshold. They repeat themselves and almost never seem to respond directly and coherently to anything — neither the direct questions and statements in the ticket, nor the facts of the case. It's difficult to understand how anyone could be so massively incompetent.

I sincerely believe that British Telecom is too incompetent and dishonest to be permitted to hold the monopoly position it does. A huge number of people have no choice but to deal with them, and the watchdog is notoriously toothless.

I fantasise that the government will realise this and issue a compulsory purchase order to re-nationalise the infrastructure which connects British homes, then allow the existing British Telecom to go bankrupt. They seem to be completely beyond help.

I don't know what would happen to the infrastructure after this — I normally wake up at this point. But whether it remains nationalised or whether it's sold to someone else, it couldn't possibly be as bad as British Telecom.

Argh. How is it that people can be so fucking stupid?

Are Yahoo actually employing people to do their customer service, or just dragging in crack-smoking monkeys off the street to do it for a laugh?
This reminds me of the exchange with Vodafone, where they never bothered to read anything I wrote, but just kept asking the same identity questions over and over again... even when I answered them.


Message-Id: <200901112335.n0BNZjsU033833@mail-relay1.yahoo.com>
Date: Sun, 11 Jan 2009 23:35:53 +0000
To: David Woodhouse <dwmw2@infradead.org>
Subject: Re: UK - United Kingdom (KMM81869195V16943L0KM)
From: Yahoo! UK & Ireland Login Support <uk-account@cc.yahoo-inc.com>
X-Bad-Reply: 'Re:' in Subject but no References or In-Reply-To headers

Hello David,

Thanks for writing to Yahoo! UKIE Customer Care.

Currently, we are unclear on exactly what it is you want done to your
account. Please reply to this message with a more detailed explanation
of what you need done.

Also, in your reply, please include the:

- Yahoo! ID
- alternate email address
- date of birth
- postal code
- secret question and answer

that you supplied when you created your account. We will then be able to
verify your account.

Regards,

Customer Care - Yahoo! UK & Ireland

CONFIDENTIALITY NOTICE: This email and any attachment is confidential
and may be legally privileged. It is intended for the named recipient
only.




Original Message Follows:
-------------------------

On Sat, 2009-01-10 at 23:48 +0000, Yahoo! UK & Ireland Login Support wrote:
> Currently, we are unclear on exactly what it is you want done to your
> account.

This was fully stated in my mail of January 8th.

I'm sorry; I don't mean to be rude, but I don't think I could explain it
in words of fewer syllables. You'll have to find a competent adult to
read it to you.

I am attaching that email for your convenience, in case you can't find
it. Please forward it to someone who is capable of dealing with it... as
I requested on the 8th.

--
dwmw2

[ Attachment 1.2 Type: message/rfc822][ Forwarded message displayed below ]

From: David Woodhouse <dwmw2@infradead.org>
Date: Thu, 08 Jan 2009 15:36:07 +0000

On Thu, 2009-01-08 at 13:44 +0000, Yahoo! UK & Ireland Login Support wrote:
> Hello David,
>
> Thanks for writing to Yahoo! UKIE Customer Care.
>
> We have received the account information you have provided and
> understand that you are unable to login to your Yahoo! account.

I have found the reason why your reminder message wasn't getting though.
Please forward this to whoever is responsible:

Your reminder messages are being sent out without a Message-Id: header.
This header is supposed to contain a unique identifier for each message,
and the email RFC (RFC5322) says that every message SHOULD have such a
header.

Since most messages lacking such a header, in violation of the standard,
are spam or virus traffic, many mail servers reject them. Including
mine.

I have implemented a temporary workaround, and I was able to receive the
reminder -- so I do have my username and password now; thanks. But you
should fix the problem anyway, of course.

--
dwmw2

[ End of Forwarded Message 1 ]

Wheee! Opened my first Christmas present a couple of days ago. A PCI ADSL2+ card, fully supported by Linux.

Many thanks to the folks at Traverse Technologies and Xrio for the effort they've put into developing this hardware — and for sending me a board.

At last I'll be able to have a properly supported Linux box, with legal drivers, as the endpoint for my ADSL lines.

There are a few things to improve; adding DMA support is a priority right now. But we do have the capability to update the FPGA from software now, so they're starting to ship real hardware to customers.

Following my post on Friday, we're now at almost 19,000 bogus 'clears' from the BT side, as they dishonestly keep clearing the fault in order to avoid being held accountable to their own Service Level Guarantee. All of the clears have been rejected by the ISP, of course — but BT have arranged for their systems to fall over when the ISP responds too quickly, while they can re-clear the fault from their side within a second or so of it being passed back to them.

So it looks like BT are winning the game with the 'chess clock' — although the line has been fairly much unusable for a week now, and the fault was reported 6 days ago, the clock has yet to reach a single day, let alone the SLG of 40 hours. Purely because they keep kicking the fault back to the ISP's side while we're waiting for BT to take action.

There's no way we can attribute this to their normal incompetence — this is blatant fraud on the part of British Telecom.

28 Nov 2008 (updated 28 Nov 2008 at 21:12 UTC) »
"Any sufficiently advanced incompetence is indistinguishable from malice"
Grey's Law

The above rule (inspired, no doubt, by Clark's Law and Hanlon's Razor) is often applicable to the actions of British Telecom.

Regular readers of my drivelings may be surprised to find that I actually think I've been kind to BT in reporting their actions as incompetence, rather than anything more sinister.

Sometimes, however, it's virtually impossible to give them the benefit of the doubt. One example of this is their repeated insistence on a 'Special Faults Investigation' (SFI) engineer visit, when we're asking them to fix faults. SFI is an optional service which we don't usually want, and they repeatedly try to charge for it after explicitly being told that it was not required — a criminal offence under the Unsolicited Goods and Services Act 1971. In fact, although it's supposed to be a charge for services beyond the limit of their responsibility, in the customer's own equipment or wiring, they've also been known to apply charges after themselves clearly stating that no such work has occurred.

(Update: The ISP reports that today they've had someone in BT refusing point blank to fix a fault unless an SFI is booked. This is a clear-cut fault where the end user has tried two separate modems, RJ11 cables and filters and the problem persists — it's definitely a BT fault.)

I have a similar problem with my line, although thankfully BT haven't refused to send an engineer to investigate. Since about 2:30AM on Monday morning, my DSL line has turned to crap — it seems to be caused by RF interference from something outside my house. I did similar tests to the above, with different cables, different filters and different modems. The usage graphs look like this, with ugly purple stalactites eating my connectivity:

Even when it's connected, it's getting lots of errors and running at half the speed it should be; below the 'Fault Threshold Rate'.

When the engineer arrived, I'd disconnected everything else on the line and connected the DSL modem directly to the master socket, at the ISP's request. With no filter, since there was no analogue equipment present.

The engineer held up the little adaptor which lets you plug an RJ11 into the BT socket, and told me that it was "not a filter"... strangely. He then went away and reported in the fault ticket that "customer also had filter incorectly [sic] installed which I believe to be a genuine mistake as he had already had a nte2000 master plate installed". A doubly confusing statement, since I didn't have a filter installed — the filters I use are the NTE2000 faceplate type, and were quite clearly sitting on the floor near the socket. I'd also told him why I'd removed it. So his report in the ticket really does look like it's an excuse to charge for the SFI, pretending that there was some work done on my side of the line.

It gets better than that, though — they also have a Service Level Guarantee which says that faults should be fixed within 40 hours, and they're required to pay compensation if they fail to meet it. There's a kind of 'chess clock' which counts the time during which the ticket is with BT for action, and BT keep 'clearing' tickets with totally bogus responses, as shown in the Cisco-IPv6 tickets I posted here and here.

They do this so much that the ISP found it necessary to implement an autoresponder, rejecting the bogus 'clear' and flipping the clock back to BT so that the SLG clock kept counting. Obviously, they apply this with care, only when they're sure that BT aren't clearing the ticket for genuine reasons, but only their fraudulent attempts to avoid the SLG.

Today, we found that BT have implemented their own autoresponder, flipping the ticket back to the ISP even while we're waiting for BT to take action. Again, it's really hard to put this down to BT's normal institutional incompetence — it cannot really be interpreted as anything but a deliberate attempt to defraud their customers by circumventing the Service Level Guarantee.

I've posted a copy of the fault ticket, although I truncated it at about the 50th clear/reject cycle. We're up to about 1000 cycles now, and counting... :)

Fucking Useless, Criminal, Telco.

194 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!