Open Projects provides interactive facilities for coordination and support to groups and projects involved with open source. We run between 1,500 and 2,000 clients and are home to such projects as Debian GNU/Linux and Enlightenment. We've had our share of difficulties recently, but we're continuing on.
The past few weeks have been quite an experience. Last week one of our hubs on Open Projects started going up and down like a yoyo. I'd seen that behavior in this normally very reliable server in recent weeks and not thought much of it, since the company in question was in the process of moving its facilities and reliability issues do sometimes creep in during such moves. But we soon obtained a little bit more insight into the problem. After watching the server perform a loop-de-loop, I received a /MSG from a rather peremptory and anonymous skript kiddie informing me that if I didn't permanently remove the sponsor's server from the network, he would kill my home ADSL line and take down Open Projects until he got his way. It seems he feels the sponsor owes him money. I'm afraid I wasn't very polite in my response. Feeling that one can hardly allow psychotic delinquents to dictate network policy, I explained to him that while he might very well be able to take down our network, he was not going to set policy, and specifically I would not entertain the notion of removing our sponsor's machine.
The last week has been interesting. Apparently this petulant child has something over 45Mbps to play with, and he's moderately competent with SYN attacks and so on. In various incidents throughout the week he packeted ISP's and universities and small companies to death to demonstrate his, uh, prowess with borrowed equipment. Currently he has proclaimed that he'll be taking down our network once a day for an hour until his wishes are granted. All I can say is that he's going to be doing it for a long time if that's the case; the heat death of the universe isn't due to arrive for some time.
Throughout this experience I have noticed it's very difficult to coordinate much of a response from ISPs and backbone providers. An unofficial contact at uu.net explained that we must notify his security people while an attack was taking place for them to have any chance of thwarting it. They thoughtfully provided him with an email address rather than a telephone number to give to us, explaining that this is a matter of policy. Perhaps they don't understand that packeting can affect services like email. Or perhaps they are simply extremely comfortable, their owners having cornered much of the backbone market after the last round of industry mergers. My employer's ISP was targeted, and so far the people at the ISP seem a little bewildered, though they're game to fight the good fight. Some folks with very nice bandwidth contributed a server today to see if we couldn't keep our hubbing working through an attack, and the skript kiddie seems to have gone after their routers, leaving very little in the way of evidence behind him as to his point of origin.
As a first, one of our admins contacted the FBI at our request. I'm not sure this will accomplish anything useful, but it's certainly worth a try. It is worth noting that, as a philosophical anarchist, I'm usually not inclined to bring in the muscle of a law enforcement agency to resolve such disputes, preferring to reason with the party or parties involved. But in cases where the problem user has learned his manners from repeated viewing of Robocop, well, there's not much one can do but consider the business to be a declaration of war.
At any rate, it seems to me that this otherwise very mundane set of attacks points to a long-standing problem with the Internet: Denial-of-service attackers have location indirection, but content services and users are left in plain sight as targets for their efforts. I'm hoping Corridors will helpful in dealing with this problem, though it's a fairly long-term project (and constantly in search of additional expertise to finish the design and begin the actual implementation). Meanwhile, we go on, attempting to devise kludges to improve the robustness of ircd in the face of all-out attack.
Any assistance from the readership in combatting problems which we have never experienced in quite this magnitude would be greatly appreciated.
Thanks to the Magenet people and Diane Bruce and F. John Rowan of the hybrid ircd project for their assistance. Thanks to the many users and admins of OPN, whose patience and support have been impressive. And thanks especially to VA Linux for their help and support; they've been real heroes and deserve a great deal of praise. And no, we're not going to delink their server, however many or few seconds we have to comply. ;)
There are, already available or coming onto the market, a variety of tools that can limit the impact of such denial of service attacks.
They are based on Diferentiated Services and QoS (Quality of Service). What makes these tools nice is that they allow you to limit, at
the
router, the amount of bandwidth that can be eaten by any single connection or group of connections, although that is not the only option
available. Assuming you have, or can get, policy enabled routers of some type (Cisco and Bay/Nortel are the 2 that spring to mind) you
can start limiting the impact of your friendly neighbourhood script kiddy fairly quickly.
There is, of course, a bad news side to this story. DiffServ and network traffic shaping in general are fairly new to the world,
and
in all
honesty, the standards are somewhat of a moving target these days. There is very little agreement between the vendors and IETF on
even simple things like definitions, so a large dose of caveat emptor is required.
The other part of this you will need to look at is the management side of DiffServ. There are at least a couple of front-ends for
managing router policy and network policy, some better than others. Since my day to day job revolves around these things, I'll leave it
to
others less directly involved to discuss the specific vendor options here. My answer would be unashamedly biased in favour of a
specific
vendor, and this is not the place for such things :)
I realize there isn't a lot of detail here, and I apologize for that. DiffServ and router/network policy is a very large and complex
area.
Rather than occupy a large amount of space explaining options and uses, I think you and your employer would be better served by a
healthy dose of research. I would be more than happy to point you at my employers options in this area, outside this forum, if you're
interested and I could provide more detailed information if that is appropriate to your needs. Please note that this is potentially a very
expensive option, and there is not (to the best of my knowledge) an open source solution in this area. The benefit is that not only can
you
limit the potential damage from DoS', but it opens up a vast array of options in controlling how your available network resources are
utilized.
Thanks for your comments, rasputin. One of the problems is that Open Projects is a relatively-distributed network, and that we depend on our sponsors for server account space. I believe that, in the long term, such problems are best resolved by adding client-server and client-client indirection which, in the current design of the Internet, would have to be implemented in an additional protocol layer.
In the meantime, we struggle along as best we can. Did I mention to anybody that their cryptographic and steganographic experience would be very useful to the Corridors project? :) :) Anyone who is interested can send email to me at the feedback address found on my diary entries.
Being involved with IRC networks on and off for quite some time, it still surprises me that such a well understood problem like DoS can still exist. It is laziness in the network administrators, pure and simple.
This exact subject has been discussed to death on lists such as NANOG, and it is well-known that QoS and related tools will not help a DoS type attack. Your network will still get the traffic, because the larger networks are .. well, large.
BUT, I don't think its up to the larger networks to filter traffic. The current "internet design" and its very heavy asymmetry in routing paths makes filtering in the core very difficult. The problem lies in filtering customer links. Customers shouldn't be able to send spoofed packets unless they have a good reason (eg half-duplex satellite feeds). And even then, you're talking about 1% of the online population who needs this.
*sigh* I think you're stuck Lilo. Track the script kiddie down and squash him. He'll either get caught, or he'll stop.
On a humerous note, I find the two versions of "DOS" both equally disturbing. :)
*sigh* I think you're stuck Lilo. Track the script kiddie down and squash him. He'll either get caught, or he'll stop.k,
That pretty much nails it. Spoofed IP packets need to be squashed at origin, which essentially requires the cooperation of all of the providers, since skript kiddies will tend to find providers which don't comply. The answer has ultimately got to be location indirection for content nodes (client and server), <tireless-recruiting>which is what the Corridors project is all about.</tireless-recruiting>. 8)
Another comment I wanted to make here: To clarify, it seems very unlikely that the skript kiddie has a real grudge against VA. Regardless of the rationalizations he just seems to enjoy watching people ping out....
Pretty much no other instant messaging system seems to have these kinds of problems with mass DOS attacks. I think the problem is more social than technical - IRC attracts the wrong kind of crowd.
Pretty much no other instant messaging system seems to have these kinds of problems with mass DOS attacks. I think the problem is more social than technical - IRC attracts the wrong kind of crowd.I would argue that it's a combination of problems. IRC networks handle the largest user base of the interactive messaging systems available. Handling a large user base is potentially a very powerful thing....if you can connect people well to each other, and encourage random conversation. But IRC was never meant to scale to the size it has, so it lacks good indexing of interaction areas and some other things.
But it's just a fact that if you connect a lot of people together, some of them are going to want the feeling of power you get from a DoS attack. That ability to deny service needs a technical solution, and location indirection seems to be essential.
It's obvious that with the widespread popularity of IRC, surely someone else (MANY else) would have thought of and refined new IRC-server designs long ago, right? So in what way would your Corridors be related to the design work of these visionaries, trying to come up with irc3, the next generation of distributed IRC service? I think you're much more likely to get developers if you can coordinate with the old irc3 designers and be able to call your new project the official irc3 service.
I think you're much more likely to get developers if you can coordinate with the old irc3 designers and be able to call your new project the official irc3 service.I'm not sure that would be the right approach. The transport architecture of Corridors is intended to have a completely different emphasis than the transport structure of irc3; location indirection is key and it's considerably more decentralized. And the interactive service is just one service laid on top of the transport layer, and it's really not a whole lot like irc, in emphasis or implementation.
There is nothing wrong with having a number of projects to do overlapping things. With a relatively large number of projects, one or more is more likely to take off and serve a need. In my opinion, we do no one a favor to try to merge approaches and reduce the diversity of potential solutions.
Maybe not, but since irc3 is still so much well-designed vaporware itself, I thought your project would at least benefit a lot from looking at their design proposals, and the irc3 team would probably give you their blessing immediately because they don't have their own implementation anyway. Something to consider, I thought?
On SorceryNet we have a number of approaches to denial of service attacks, although massive packet flooding is one that's just not easily solved. We try and promote a culture in which people are erspected as individuals, and yet remain intolerant of nuking, port scans, "hacker talk" and so forth. On the technical side, we don't allow connections from open wingate servers, which helps a little. But the IRC infrastructure is fragile, and easily upset.
Like you with corridors, I've been working on irc++, intended to be a more robust and scalable service (the paper there is a very incomplete draft of a conference paper).
One problem we had at SorceryNet was a skript kiddie who worked at a major ISP; he happily packet flooded us, and complaining to the ISP was not likely to help. Luckily for us, he went away again; my next recourse was going to be to contact the board of directors for the ISP involved.
The latest problem seems to be massively distributed attacks using sub7 bots. These bots can usually be disinfected remotely, but it's a real pain to have a few hundred bots all join and start flooding. Again I think the real problem is cultural, but there is also the problem of (effectively) anonymous internet use destroying accountability. I do0n't have an answer to that.
Again I think the real problem is cultural, but there is also the problem of (effectively) anonymous internet use destroying accountability. I do0n't have an answer to that.The right approach to accountability is to harness trust models. I'm not averse to anonymity. The problem in the typical distributed denial-of-service attack is that the attacker has location anonymity, but the service he's attacking does not. The latter needs to be changed, to give the attacked party the tools to deal effectively with such problems.
New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.
Keep up with the latest Advogato features by reading the Advogato status blog.
If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!
del.icio.us
Digg
Google bookmark
reddit
Simpy
StumbleUpon
Furl
Newsvine
Technorati
Tailrank