Older blog entries for LaForge (starting at number 232)

Problems with OpenVPN on high-latency satellite links

So far I never had a need to look in detail how the OpenVPN protocol actually looks on the wire. It seems like not many people had that much of a close look, as the wireshark plugin is fairly recent (from 2012 I think) while OpenVPN is around for ten more years than that. If I was an OpenVPN developer, the wireshark plugin would be the first thing I'd write to help debugging and development. At least that's what I've been doing from OpenPCD to SIMtrace and through the various GSM and other protocols I encounter...

The reason for my current investigation is some quite strange and yet-unexplained problems when running OpenVPN on high-latency satellite links. I'm not talking about high-bandwidth VSAT or systems with dedicated / guaranteed bandwidth. The links I'm seeing often have RTT (as seen by ICMP echo) of 2 seconds, sometimes even 5. This is of course not only the satellite link, but includes queuing on the ground, possibly the space segment and of course the terminal, including (possibly) access arbitration.

What struck me _very_ odd is that OpenVPN is sending tons of UDP messages with ridiculously small size during the TLS handshake when bringing up the tunnel. Further investigation shows that they actually internally configure a MTU of '0' for the link, which seems to be capped at 100 bytes control payload, plus HMAC and OpenVPN header resulting in 124 to 138 bytes UDP payload.

Now you have to consider that the server certificate (possibly including even a CA certificate) can be quite large, plus all the gazillions of TLS handshaking options in ServerHello, the first message from server to client. This means that OpenVPN transmits that ServerHello in something like 40 to 60 fragments of 100 bytes each! And each of the fragments will have to be acknowledged by the remote end, leading 80 to 120 UDP/IP packets _only_ for the delivery of the TLS ServerHello.

Then you start reviewing the hundreds of OpenVPN configuration options, many of them related to MTU, MSS, fragmentation, etc. There is none for that insanely small default of 100 bytes for control packets during hand-shake. I even read through the related source code, only to find that indeed this behavior seems hard-coded. Some time later I had written a patch to add this option, thanks to Free Software. It seems to work on client and server and brings the ClientHello down to much smaller 4-6 messages.

The fun continues when you see that the timeout for re-transmitting fragments that have not been ACKed yet is 2 seconds. At my satellite RTT times this of course leads to lots of unneeded re-transmissions, simply because the ACK hasn't made its way back to the sender of the original message yet. Luckily there's a configuration option for that.

After the patch and changing that option, the protocol trace looks much more sane. However, I still have problems establishing a tunnel in a number of cases. For some odd reason, the last fragment of the ServerHello is not acknowledged by the client, no matter whether patched or unpatched OpenVPN is being used. I get acknowledgements always only up to fragment N-1 after having transmitted N. That last fragment is then re-transmitted by the server with exponential back-off, and finally some 60 seconds later the server gives up as the TLS handshake didn't finish within that time. Extending the TLS handshake timeout to 120 seconds also doesn't help.

I'm not quite sure why something like 39 out of 39 fragments all get delivered reliably and acknowledged, but always the last fragment (40) doesn't make it to the remote side. That's certainly not random packet loss, but a very deterministic one. Let's see if I can still manage to find out what that might be...

Syndicated 2013-09-05 02:00:00 from Harald Welte's blog

Attending HITCON and COSCUP in Taipei

It is my pleasure to attend the HITCON 2013 and COSCUP 2013 conferences in July/August this year. They are both in Taipei. HITCON is a hacker/security event, while COSCUP is a pure Free/Open Source Software conference.

At both events I will be speaking at the growing list of GSM related tools that are available these days, like OpenBSC, OsmcoomBB, SIMtrace, OsmoSGSN, OsmoBTS, OsmoSDR, etc. As they are both FOSS projects and useful in a security context, this fits well within the scope of both events.

Given that I'm going to be back to Taiwan, I'm looking very much forward to meeting old friends and former colleagues from my Openmoko days in Taipei. God, do I miss those days. While terribly stressful, they still are the most exciting days of my career so far.

And yes, I'm also going to use the opportunity for a continuation of my motorbike riding in this beautiful country.

Syndicated 2013-06-05 02:00:00 from Harald Welte's blog

Rest In Peace, Atul Chitnis

Today, very sad news has reached me: Atul Chitnis has passed away. Most people outside of India will most likely not recognize the name: He has been instrumental in pineering the BBS community in India, and the founder and leader of the Linux Bangalore and later FOSS.in conferences, held annually in Bangalore.

I myself first met Atul about ten years ago, and had the honor of being invited to speak at many of the conferences he was involved in. Besides that professional connection, we became friends. The warmth and affection with which I was accepted by him and his family during my many trips to Bangalore is without comparison. I was treated and accepted like a family member, despite just being this random free software hacker from Germany who is always way too busy to return the amount of kindness.

Despite the 17 year age difference, there was a connection between the two of us. Not just the mutual respect for each others' work, but something else. It might have been partially due to his German roots. It might have been the similarities in our journey through technology. We both started out in the BBS community with analog modems, we both started to write DOS software in the past, before turning to Linux. We both became heavily involved in mobile technology around the same time: He during his work at Geodesic, I working for Openmoko. Only in recent years his indulgence in Apple products was slightly irritating ;)

Only five weeks ago I had visited Atul. Given the state of his health, it was clear that this might very well be the last time that we meet each other. I'm sad that this now actually turned out to become the thruth. It would have been great to meet again at the end of the year (the typical FOSS.in schedule).

My heartfelt condolences to his family. Particularly to his wonderful wife Shubha, his daughther Anjali, his mother and brother. [who I'm only not calling by their name in this post as they deserve some privacy and their Identities is not listed on Atuls wikipedia page].

Atul was 51 years old. Way too young to die. Yet, he has managed to created a legacy that will extend long beyond his life. He profoundly influenced generations of technology enthusiasts in India and beyond.

Syndicated 2013-06-03 02:00:00 from Harald Welte's blog

OsmoDevCon 2013 preparation update

OsmoDevCon 2013 is getting closer every day, and I'm very much looking forward to meet the fellow developers of the various Osmcoom sub-projects. Organization-wise, the catering has now been sorted out, and Holger has managed to get a test license for two ARFCN from the regulatory body without any trouble.

This means that we're more or less all set. The key needs to be picked up from IN-Berlin, and we need to bring some extra extension cords, ethernet switch, power cords and other gear, but that's really only very minor tasks.

There's not as much formal schedule as we used to have last year, which is good as I hope it means we can focus on getting actual work done, as opposed to spending most of the time updating one another about our respective work and progress.

Syndicated 2013-03-29 01:00:00 from Harald Welte's blog

Hardware outage affectiong osmocom.org, deDECTed.org, gpl-violations.org

As usual, murphy's law dictates that problems will occur at the worst possible moment. One of my servers in the data center died on March 20, and it was the machine which hosts the majority of the free software projects that I've created or am involved in. From people.netfilter.org to OpenPCD and OpenEZX to gpl-violations.org and virtually all osmocom.org sites and services.

Recovery was slow as there is no hot spare and none of my other machines in the data center have backplanes for the old SCA-80 hard disks that are in use by that particular machine. So we had to send the disks to Berlin, wait until I'm back there, and then manually rsync everything over to a different box in the data center.

To my big surprise, not many complaints reached me (and yes, my personal and/or business e-mail was not affected in any way)

Recovery is complete now, and I'm looking forward to things getting back to normal soon.

Syndicated 2013-03-29 01:00:00 from Harald Welte's blog

Update on what I've been doing

For the better part of a year, this blog has failed to provide you with a lot of updates what I've been doing. This is somewhat relate to a shift from doing freelance work on mainline / FOSS projects like the Linux kernel.

In April 2011, Holger and I started a new company here in Berlin (sysmocom - systems for mobile communications GmbH). This company, among other things, attempts to provide products and services surrounding the various mobile communications related FOSS projects, particularly OpenBSC, OsmoSGSN, OpenGGSN, but also OsmocomBB, and now also OsmoBTS + OsmoPCU, two integral components of our own BTS product called sysmoBTS.

Aside from the usual software development, this entails a variety of other tasks, technical and non-technical. First of all, I did more electrical engineering than I did in the years since Openmoko. And even there, I was only leading the hardware architecture, and didn't actually have to capture schematics or route PCBs myself. So now there are some general-purpose and some customer-specific circuits that had to be done. I really enjoy that work, sometimes even more than software development. Particularly the early/initial design phase can be quite exciting. Selecting components, figuring out how to interconnect them, whether you can fit all of them together in the given amount of GPIOs and other resource of your main CPU, etc. But then even the hand-soldering the first couple of boards is fun, too.

Of all the things I so far had least exposure to is casing and mechanical issues. Luckily we have a contractor working on that for us, but still there are all kinds of issues that can go wrong, where unpopulated PCB footprints can suddenly make contact with a case, or all kinds of issues related to manufacturing tolerances. Another topic is packaging. After all, you want the products to end up in the hands of the customer in a neat, proper and form-fitting package.

On the other hand, there is a lot of administrative work. Sourcing components can sometimes be a PITA, particularly if even distributors like Digikey conspire against you and don't even carry those low quantities of a component that we need for our 100-board low quantity runs. EMC and other measurements for CE approval are a fun topic, too. I've never been involved personally in those, and it has been an interesting venture. Luckily, at least for sysmoBTS, things are looking quite promising now. Customs paperwork, Import/Export related buerocracy (both in Germany as well as other countries) always have new surprises, despite me having experience in dealing with customs for more than 10 years now.

Also significant amount of time is spent on evaluating suppliers and their products, e.g. items like SIM/USIM cards, cavity duplexers, antennas, cables, adapters, power amplifiers and other RF related accessories for our products.

The thing that really caught me off-guard are the German laws on inventory accounting. Basically there is no threshold for low-quantity goods, so as a company on capital (GmbH/AG) you have to account for each and every fscking SMD resistor or capacitor. And then you don't only have to count all those parts, but also put a value at them. Depending on the type of item, you have to use either the purchasing price, or the current market price if you were to buy it again, or the price you expect to sell the item for. Furthermore, the trade law requirements on inventory accounting are different than the tax laws, not often with contradictory aims ;)

In the end it seems the best possible strategy is to put a lot of the low-value inventory into the garbage bin before the end of the financial year, as the value of the product (e.g. 130 SMD resistors in 0402 worth fractions of cents) is so much lower than the cost of counting it. Now that's of course an environmental sin, especially if you consider lots and lots of small and medium-sized companies ending up at that conclusion :(

So all in all, this should give you somewhat of an explanation why there might have been less activity on this blog about exciting technical things. On the one hand, they might relate to customer related projects which are of confidential nature. On the other hand, they might simply be boring things like dealing with transport damage of cavity duplexers from china, or with FedEx billing customs/import fees to the wrong address...

Overall I still have the feeling that I was writing a decent amount of code in 2012 - although there can never be enough :) Most of it was probably either related to OsmoBTS, OpenBSC/OsmoNITB or the various Erlang SS7/TCAP/MAP related projects. The list of more community-oriented projects with long TODO lists is growing, though. I'd like to work on SIMtrace MITM / card emulation support, the CC32RS512 based smartcard OS, libosmosim (there's a first branch in libosmocore.git). Let's hope I can find a bit more time for that kind of stuff this year. You should never give up hope, they say ;)

Syndicated 2013-02-08 01:00:00 from Harald Welte's blog

Back from FOSDEM 2013

As (almost) every year, I attended the annual incarnation of FOSDEM. It is undoubtedly (one of?) the most remarkable events about Free Software in existence. No registration, no fees, 24 tracks in parallel, an estimated 5000 number of attendees. I also like that it brings together people from so many different communities, not _just_ the Linux or Gnome or KDE or Telephony or Legal people, but a good mixture of everything.

I have to congratulate the organizers, who manage to pull this off, year after year again. And as opposed to many other events, they do so quietly and without much recognition, I feel. I'd also like to thank the many volunteers working tirelessly before, at and after the event. Last, but not least, I'd like to thank the local university (ULB Solbosch) hosting the event.

What made me truly sad though, is the amount of littering that surprisingly many of the attendees did. This was particularly visible in the Cafeteria. Imagine an event run by volunteers, who put in a lot of time and effort. Imagine an event where food and drinks are sold by volunteers at such low prices that there can barely be any profit at all. And then imagine people eating there and leaving all their rubbish around, as if they were in some kind of restaurant where they are being served and where somebody is cleaning up after them. It really makes me feel very bitter to see this. Don't people realize that those very volunteers who are creating the event will then have to put in _their_ spare time just because those who just enjoyed their coffee or lunch didn't have the extra 30 seconds of bringing their trash to the trashcan? I feel ashamed for members of our community who behave this way. Please think next time before acting and show your respect to the people behind FOSDEM.

Syndicated 2013-02-04 01:00:00 from Harald Welte's blog

Why I hate phone calls so much

The fact that I have more than 20 missed phone calls on my land line telephone after only half a day has passed triggers me to write this blog post.

It is simply impossible to get any productive work done if there are synchronous interruptions. If I'm doing any even remotely complex task such as analyzing code, designing electronics or whatever else, then the interruption of the flow of thoughts, and the context switch to whatever the phone call might be about is costing me an insurmountable amount of my productive efficiency. I doubt that I am the only one having that feeling / experience.

So why on earth does everybody think they are entitled to interrupt my work at any given point in time they desire? Why do they think whatever issue they have rectifies an immediate interruption in what I am doing? To me, an unscheduled phone call almost always feels like an insult. It is a severe intrusion into my work-flow, and has a very high cost to me in terms of loss of productivity.

Sure, there are exceptional absolute emergencies (like, a medical emergency of a family member). But just about anything else can be put in an e-mail, which I can respond to at a time of my choosing, i.e. at a time I am not deeply buried into some other task that requires expensive context switching and the associated loss of productivity. And yes, a response might be the same day, some days later, or even a week or more later. There are literally hundreds of mails of dozens of people that need to be responded to. I can never even remotely answer all of them in a timely manner, even if I'm working 12-14 hours a day up to 7 days a week.

Right now I'm doing the only reasonable thing that is left: Switch off all phones. And to anyone out there intending to contact me: Please think twice before calling me on the phone. Almost anything can be put in an e-mail. And if you really want to have a phone call, please request a scheduled phone call in an e-mail containing a very detailed agenda and explanation of the topic.

Syndicated 2013-01-16 01:00:00 from Harald Welte's blog

Strain of bad luck

From roughly September to December 2012 I seem to have had a quite unusual strain of bad luck and set-backs. I don't want to go into the details here, as most of the issues are of quite private nature.

This has kept me quite distracted from a lot of my other activity. Projects like the various Osmocom sub-projects, gpl-violations.org are in desperate need of attention, and I have severely neglected my responsibilities in the Chaos Computer Club Berlin e.V. :(

I don't even want to talk about actual paid work, where customers also had to put up with repeated schedule slips and lack of availability.

I let down friends and colleagues at a number of occasions, as I was unable to keep up with anything that remotely resembles my typical work schedule.

Last but not least, I regrettably have also not felt much of an urge to write many blog posts here.

My sincere hope and expectation is that things are going to improve quickly in 2013. At least most of issues from the last half year have been resolved. Now I need to work through a considerable back-log of work and find more time for my volunteer projects in the FOSS and hacker worlds. However, this will need some time and I would like to ask for some patience. I do intend to be up to speed with things just like before.

In this spirit, I am looking forward to a productive and exciting 2013. Happy hacking und Viel Spass am Gerät

Syndicated 2013-01-02 01:00:00 from Harald Welte's blog

29C3. The end of an era?

When I first heard that the annual CCC congress was moved to Hamburg, my immediate reaction was: Fine, but I wouldn't want to be involved in it. For the last 15 years I've been attending the CCC congress every year, in most years as a speaker, and in many years in some (small) contributing role, first in the team doing the video recordings, and in the last couple of years setting up a GSM network. Contributing to an event is easy if your home/lab is within 20minutes, so if you need another strange cable/adapter/tool/whatever, you can just go and grab it. Doing that at an event that's multiple hours of driving away, in a new/unknown venue is an entirely different story. I have more than enough stress already with (paid) work and the various FOSS projects that I'm leading or involved in.

I have no interest in "just" attending the event. That never was a primary reason for me. In all those years, I've probably attended an average of one talk each year. The event for me was about being able to contribute something actively.

Now, months after those thoughts and my decision not to attend, there is a schedule for the 29C3 available. And to say the least, I am shocked. The entire event seems to have turned into a SIGINT, rather than an xxC3. Lots of talks on politics and society, and lots of German talks.

The debate on implications of technology on society, culture, politics, etc. is an important debate, there is no doubt. And so far I always had the feeling that the xxC3 had a pretty good balance between hard-core technical talks and those non-technical talks. But if I look at the schedule this year, it really looks like an incarnation of the SIGINT conference. With too many German talks you are scaring off the international community. And with focussing on non technical topics, you scare away the die-hard technical hackers. So why move to a larger venue, if you at the same time seem to limit the scope of the event?

Meanwhile I have heard of a number of friends and colleagues who seem to share this view. A number of people who have attended in previous years are not interested in attending this year due to the issues mentioned above.

It's sad to see, but I somehow have the feeling that 29C3 might be the end of an era. The end of a highly successful series of events with exceptionally strong technical talks. To me, xxC3 has always been unique and special. No other event would ever compare to it. Who will fill the gap for the die-hard technical topics? I am feeling quite sad, up to the point that I want to start mourning about "the good old times".

I'm not writing this to put blame on anyone. It just reflects my personal and highly subjective view. Let's see what people will say after 29C3 has actually happened. Let's see how successful it is in terms of number of attendees, and in terms of feedback from participants.

Syndicated 2012-12-18 01:00:00 from Harald Welte's blog

223 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!