Presentation: what is http2
We had the 14th meetup with foss-sthlm yesterday and I talked about http2 for the almost 100 attendees in the audience. See my slides at slideshare:
Presentation: what is http2
We had the 14th meetup with foss-sthlm yesterday and I talked about http2 for the almost 100 attendees in the audience. See my slides at slideshare:
curl and the road to IPv6
I’d like to comment Paul Saab’s presentation from the other day at the World IPv6 Congress titled “The Road To IPv6 – Bumpy“. Paul works for Facebook and in his talk he apparently mentioned curl (slide 24 of the PDF set).
Lots of my friends have since directed my attention to those slides and asked for my comment. I haven’t seen Paul’s actual presentation, only read the slides, but I have had a shorter twitter conversation with him about what he meant with his words.
The slide in question says exactly this:
- Very hostile to the format of the IPv6 address
- Wants everything bracket enclosed
- Many IPv6 bugs that only recently were fixed
Let’s see what those mean. Very hostile to the format of the IPv6 address and Wants everything bracket enclosed are basically the same thing.
Paul makes a big point about the fact that if you want to write a URL with an IP address instead of a host name, you have to put that IP address within [brackets] when the IP address is an IPv6 one, which you don’t do if it is an IPv4 one.
Right. Sure. You do. That’s certainly an obstacle when converting slightly naive applications from IPv4 to IPv6 environments. This syntax is mandated by RFCs and standards (RFC3986 to be exact). curl follows the standards and you’ll do it the same way in other tools and clients that use URLs. The problem manifests itself if you use curl for your task, but if you’d use something else instead that something else would have the same issue if it follows the standards. The reason for the brackets requirements is of course that IPv6 numerical addresses contain colons and colons already have a reserved meaning in the host part of URLs so they had to come up with some way to handle that.
Then finally, Many IPv6 bugs that only recently were fixed he said.
I’m the main developer and maintainer of the curl project. This is news to me. Sure we always fix bugs and we always find stupid things we fix so there’s no doubt about that we’ve had IPv6 related bugs that we’ve fixed – and that we still have IPv6 related bugs we haven’t yet found – but saying that we fixed many such bugs recently? That isn’t something I’m aware of. My guess is that he’s talking about hiccups we’ve had after introducing happy eyeballs, a change we introduced in release 7.34.0 in December 2013.
curl has had IPv6 support since January 2001. We’re on that bumpy road to IPv6!
groups.google.com hates greylisting
Here’s a Wikipedia article for you: Greylisting.
After you’ve read that, then consider the error message I always get for my groups.google.com account when you disable mail sending to me due to “bouncing”:
Bounce status Your email address is currently flagged as bouncing. For additional information or to correct this, view your email status here [link].
“Google tried to deliver your message, but it was rejected by the server for the recipient domain haxx.se by [mailserver]. The error that the other server returned was: 451 4.7.1 Greylisting in action, please come back later”
See, even the error message spells out what it is all about!
Thanks to this feature of Google groups, I cannot participate in any such lists/groups for as long as I keep my greylisting activated since it’ll keep disabling mail delivery to me.
Enabling greylisting decreased my spam flood to roughly a third of the previous volume (and now I’m at 500-1000 spam emails/day) so I’m not ready to disable it yet. I just have to not use google groups.
Reducing the Public Suffix pain?
Let me introduce you to what I consider one of the worst hacks we have in current and modern internet protocols: the Public Suffix List (PSL). This is a list (maintained by Mozilla) with domains that have some kind administrative setup or arrangement that makes sub-domains independent. For example, you can’t be allowed to set cookies for “*.com” because .com is a TLD that has independent domains. But the same thing goes for “*.co.uk” and there’s no hint anywhere about this – except for the Public Suffix List. Then, take that simple little example and extrapolate to a domain system that grows with several new TLDs every month and more. The PSL is now several thousands of entries long.
And cookies isn’t the only thing this is used for. Another really common and perhaps even more important use case is for wildcard matches in TLS server certificates. You should not be allowed to buy and use a cert for “*.co.uk” but you can for “*.yourcompany.co.uk”…
If you read the cookie RFC or the spec for how to do TLS wildcard certificate matching you won’t read any line putting it crystal clear that the Suffix List is what you must use and I’m sure different browser solve this slightly differently but in practice and most unfortunately (if you ask me) you must either use the list or make your own to be fully compliant with how the web works 2014.
In curl and libcurl, we have so far not taken the PSL into account which is by choice since I’ve not had any decent way to handle it and there are lots of embedded and use cases that simply won’t be able to cope with that large PSL chunk.
Wget hasn’t had any PSL awareness either, but the recent weeks this has been brought up on the wget list and more attention has been given to this. Work has been initiated to do something about it, which has lead to…
I’ve mostly cheered the effort so far and said that I wouldn’t mind building on this to enhance curl in the future if it just gets a suitable (liberal enough) license and it seems to go in that direction. For curl’s sake, I would like to get a conditional dependency on this so that people without particular size restrictions can use this, and people on more embedded and special-purpose situations can continue to build without PSL support.
If you’re interested in helping out in curl and libcurl in this area, feel most welcome!
Meanwhile, the IETF has set up a new mailing list called dbound for discussions around PSL and similar issues and it seems very timely!
what’s –next for curl
curl is finally getting support for doing multiple independent requests specified in the same command line, which allows users to make even better use of curl’s excellent persistent connection handling and more. I don’t know when I first got the question of how to do a GET and a POST in a single command line with curl, but I do know that we’ve had the TODO item about adding such a feature mentioned since 2004 – and I know it wasn’t added there right away…
Starting in curl 7.36.0, we can respond with a better answer: use the –next option!
curl has been able to work with multiple URLs on the command line virtually since day 1, but all the command line options would then mostly apply and be used for all specified URLs.
This new –next option introduces a “boundary”, or a wall if you like, between options on the command line. The options set before –next will be handled as one request and the options set on the right side of –next will start adding up to another request. You of course then need to specify at least one URL per individual such section and you can add any number of –next on the command line. If the command line then gets too long, we also support the same logic and sequence in the “config files” which is the way you can specify command line arguments into a text file and have curl read them from there using -K or –config.
Here’s a somewhat silly example to illustrate. This fist makes a POST and then a HEAD to two different pages on the same host:
curl -d FOO example.com/input.cgi --next --head example.com/robots.txt
Thanks to Steve Holme for his hard work on implementing this!
http2 in curl
While the first traces of http2 support in curl was added already back in September 2013 it hasn’t been until recently it actually was made useful. There’s been a lot of http2 related activities in the curl team recently and in the late January 2014 we could run our first command line inter-op tests against public http2 (draft-09) servers on the Internet.
There’s a lot to be said about http2 for those not into its nitty gritty details, but I’ll focus on the curl side of this universe in this blog post. I’ll do separate posts and presentations on http2 “internals” later.
http2 (without the minor version, as per what the IETF work group has decided on) is a binary protocol that allows many logical streams multiplexed over the same physical TCP connection, it features compressed headers in both directions and it has stream priorities and more. It is being designed to maintain the user concepts and paradigms from HTTP 1.1 so web sites don’t have to change contents and web authors won’t need to relearn a lot. The web will not break because of http2, it will just magically work a little better, a little smoother and a little faster.
In libcurl we build http2 support with the help of the excellent library called nghttp2, which takes care of all the binary protocol details for us. You’ll also have to build it with a new enough version of the SSL library of your choice, as http2 over TLS will require use of some fairly recent TLS extensions that not many older releases have and several TLS libraries still completely lack!
The need for an extension is because with speaking TLS over port 443 which HTTPS implies, the current and former web infrastructure assumes that we will speak HTTP 1.1 over that, while we now want to be able to instead say we want to talk http2. When Google introduced SPDY then pushed for a new extension called NPN to do this, which when taken through the standardization in IETF has been forked, changed and renamed to ALPN with roughly the same characteristics (I don’t know the specific internals so I’ll stick to how they appear from the outside).
So, NPN and especially ALPN are fairly recent TLS extensions so you need a modern enough SSL library to get that support. OpenSSL and NSS both support NPN and ALPN with a recent enough version, while GnuTLS only supports ALPN. You can build libcurl to use any of these threes libraries to get it to talk http2 over TLS.
(This still describes what’s in curl’s git repository, the first release to have this level of http2 support is the upcoming 7.36.0 release.)
Users of libcurl who want to enable http2 support will only have to set CURLOPT_HTTP_VERSION to CURL_HTTP_VERSION_2_0 and that’s it. It will make libcurl try to use http2 for the HTTP requests you do with that handle.
For HTTP URLs, this will make libcurl send a normal HTTP 1.1 request with an offer to the server to upgrade the connection to version 2 instead. If it does, libcurl will continue using http2 in the clear on the connection and if it doesn’t, it’ll continue using HTTP 1.1 on it. This mode is what Firefox and Chrome will not support.
For HTTPS URLs, libcurl will use NPN and ALPN as explained above and offer to speak http2 and if the server supports it. there will be http2 sweetness from than point onwards. Or it selects HTTP 1.1 and then that’s what will be used. The latter is also what will be picked if the server doesn’t support ALPN and NPN.
Alt-Svc and ALTSVC are new things planned to show up in time for http2 draft-11 so we haven’t really thought through how to best support them and provide their features in the libcurl API. Suggestions (and patches!) are of course welcome!
Hardly surprising, the curl command line tool also has this power. You use the –http2 command line option to switch on the libcurl behavior as described above.
To reduce transition pains and problems and to work with the rest of the world to the highest possible degree, libcurl will (decompress and) translate received http2 headers into http 1.1 style headers so that applications and users will get a stream of headers that look very much the way you’re used to and it will produce an initial response line that says HTTP 2.0 blabla.
See the README.http2 file in the lib/ directory.
I just want to make this perfectly clear: http2 is not out “for real” yet. We have tried our http2 support somewhat at the draft-09 level and Tatsuhiro has worked on the draft-10 support in nghttp2. I expect there to be at least one more draft, but perhaps even more, before http2 becomes an official RFC. We hope to be able to stay on the frontier of http2 and deliver support for the most recent draft going forward.
PS. If you try any of this and experience any sort of problems, please speak to us on the curl-library mailing list and help us smoothen out whatever problem you got!
HTTPbis design team meeting London
I’m writing this just hours after the HTTPbis design team meeting in London 2014 has ended.
Around 30 people attended the meeting i Mozilla’s central London office. The fridge was filled up with drinks, the shelves were full of snacks and goodies. The day could begin. This is the Saturday after the IETF89 week so most people attending had already spent the whole or parts of the week before here in London doing other HTTP and network related work. The HTTPbis sessions at the IETF itself were productive and had already pushed us forward.
We started at 9:30 and we quickly got to work. Mark Nottingham guided us through the day with usual efficiency.
We all basically hang out in a huge room, some in chairs, some in sofas and a bunch of people on the floor or just standing up. We had mikes passed around and the http2 discussions were flowing back and forth depending on the topics and what people felt about them. Some of the issues that were nailed down this time and will end up detailed in the upcoming draft-11 are (strictly speaking, we only discussed the things and formed opinions, as by IETF guidelines we can’t decide things on an offline meeting like this):
There were some other things too handled, but I believe those are the main changes. When the afternoon started to turn long, beers and other beverages were brought out and we did enjoy a relaxing social finale of the day before we split up in smaller groups and headed out in the busy London night to get dinner…
Thanks everyone for a great day. I also appreciated meeting several people in real-life I never met before, only discussed with and read emails from online and of course some old friends I hadn’t seen in a long time!
Oh, there’s also a new rough time frame for http2 going forward. Nearest in time would be the draft-11 at the end of March and another interim in the beginning of June (Boston?).
Out of all people present today, I believe Mozilla was the company with the largest team (8 attendees) – funnily enough none of us Mozillians there actually work in this office or even in this country.
A payment on a flight story
“Three beers and three chicken sandwiches, thanks” – I said, graciously handing over my VISA card to the flight stewardess to pay for everything for me and my two Haxx friends Björn and Linus. 20 something Euros. Neither of us were carrying any cash.
We were happy that we had seats in the 7th row on the way back to Stockholm since when we flew down to FOSDEM Brussels just two days earlier we were in the 18th row and by then they had ran out of sandwiches. Brussels Airlines on a direct flight.
The lady inserted the card into her handheld card-reader, messed around a while since it wouldn’t “take” at the first attempt and then she handed it to me to enter my PIN. So I did.
Ms stewardess pressed some buttons or something and then she said, “no it doesn’t work” and continued to try inserting my card in basically every variation you can (especially having the chip side turned out has to be a very clever way) while repeating to me that it doesn’t work. This is the same card I’ve used numerous times during the weekend and I used it several times at the airport less than an hour ago. I know it works.
“OK let’s say you’re right”, I sighed as I really didn’t think I nor my card were to blame but I also didn’t feel like just causing trouble. I handed her my second VISA card. “Here, try this instead then”.
“No it doesn’t work either”. This certainly wasn’t right.
Linus reached over and offered his MasterCard. Perhaps there was a VISA problem with the device but “Sorry sir, it doesn’t work” came back. We’re now at farce level. Björn joins the fun and offers a 4th card, another MasterCard.
By this time we’ve eaten most of the sandwiches and were enjoying the beers.
The lady continued to struggle and it still didn’t work. She was starting to act a bit troubled about this and I asked her if she really insisted that all our cards are broken and she admitted that she suspected the machine wasn’t working quite as it should. Then she leaned towards the male steward who was serving drinks a couple of steps away. They fiddled a bit more and then she came back to us.
“That worked, I swiped it” she said without much further explanation and returned with Björn’s card and the receipt for the purchase. I felt quite done with this by now so I didn’t ask nor pushed her why she didn’t do that earlier. I was glad it finally was fixed. So much for me paying, now Björn did it instead. Oh well, truly fascinating that they would do like this on an airline where people have to do purchases with credit cards all the time.
Then Björn looked at the receipt he got back:
Pringles and a bottle of water, 5 Euro.
It made all three of us burst out in uncontrolled laughter. It was then followed by some contemplation what it actually meant. What exactly did Björn pay for? Did he only get the wrong receipt or did he pay for it? If so, who paid for our food and drinks?
It is 2014 and we haven’t come further than this.
(Update: Björn reminded me that we did verify the last 4 digits of the card number on the receipt and it didn’t match any of our cards…)
My FOSDEM 2014
I’m back home after FOSDEM 2014. A big THANK YOU from me to the organizers of this fine and totally free happening.
Europe’s (the World’s?) biggest open source conference felt even bigger and more crowded this year. There seemed to be more talks that got full, longer lines for food and a worse parking situation.
Nothing of that caused any major concern for me though. I had a great weekend and I met up with a whole busload of friends from all over. Many of them I only meet at FOSDEM. This year I had some additional bonuses by for example meeting up with long-term committers Steve and Dan from the curl project whom I had never met before IRL. Old buddies from Haxx and Rockbox are kind of default! :-)
Talk-wise this year was also extra good. I’ve always had a soft spot for the Embedded room but this year there was fierce competition for my attention so I spread my time among many rooms and got to see stuff about: clang the compiler, lots of really cool stuff on GDB, valgrind and helgrind, power efficient software, using the GPU to accelerate libreoffice, car automation and open source, how to run Android on low-memory devices, Firefox on Android and more.
I missed out the kdbus talks since it took place in one of them smaller devrooms even though it was “celebrity warning” all over it with Lennart Poettering. In general there’s sometimes this problem at FOSDEM that devrooms have very varying degrees of popularity on the different talks so the size of the room may be too large or too small depending on the separate topics and speakers. But yeah, I understand it is a very hard problem to improve for the organizers.
As a newbie Firefox developer at Mozilla I find it fun to first hear the Firefox on Android talk for an overview on how things run on that platform now and then I also got references to Firefox both in the helgrind talk and the low-memory Android talk. In both negative and positive senses.
As always on FOSDEM some talks are not super good and we get unprepared speakers who talks silently, monotone and uninspired but then there’s the awesome people that in spite of accents and the problem of speaking in English as your non-native language can deliver inspiring and enticing talks that make me just want to immediately run home and try out new things.
The picture on the right is a small tribute to the drinks we could consume to get our spirits up during a talk we perhaps didn’t find the most interesting…
This year I found the helgrind and the gdb-valgrind talks to be especially good together with Meeks’ talk on using the GPU for libreoffice. We generally found that the wifi setup was better than ever before and worked basically all the time.
Accordingly, there were 8333 unique MAC addresses used on the network through the two days, which we then can use to guesstimate the number of attendees. Quite possibly upwards 6000…
See you at FOSDEM 2015. I think I’ll set myself up to talk about something then. I didn’t do any this year.
HTTP2 – the next step
The HTTPbis working group of the IETF had an interim meeting in Zurich January 22nd to 24th. I participated from remote and I listened in on the discussions over webex and followed the jabber room while the meetings were going on, addressing HTTP2 protocol issues one by one ironing out quirks and progressing forward.
I won’t bore you with details why I wasn’t present in Zurich.
Here’s a couple of quick and brief reflections from my perspective:
Listening in from remote like this is not at all adequately compensating for not being there. A room full of people discussing something is really hard to follow from remote and completely impossible to interact with. It is better than not being able to listen in at all, but it is certainly not a replacement for being there.
It is amazing how much faster people can come to conclusions and fix issues when being in the same room. Issues that have been lingering in the tracker for a very long time could be dealt with and closed within minutes. Things like what to call the protocol in ALPN or to remove the ability to switched off flow control. Not all issues of course…
HTTP2 draft-09 that soon will become draft-10 to reflect the updates from this meeting and more, is from my perspective quite far in its process. It is clearly at a point that seems to be OK with most people and the discussions are now just about details. Of course the devil is in the details and I’m not saying it can’t take a long time to settle on them, but the structure and main concepts of the protocol are probably now established.
There were not very many proxy or server people at the interim. Most of the people seemed to be client-side oriented and some service oriented. I’m personally client-side biased myself but I hope this doesn’t lead to us deciding on things that the “other side” will have problems with down the line.
Firefox nightly supports HTTP2 draft-09 (for https:// URLs) and twitter supports it server side. Enable it in the browser by entering “about:config” in the URL bar and change the config entry called “network.http.spdy.enabled.http2draft” to true. Done.
Some of the biggest HTTP2 changes brought up compared to what draft-09 says include:
There was also a whole lot of discussions about TLS for http:// urls, proxys, MITM for SSL, opportunistic encryption and more but I believe most of those issues remained at the same position as before – I missed out on parts of the last afternoon so I may have missed some details. It’ll all be revealed in draft-10 and the mailing list I’m sure!
New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.
Keep up with the latest Advogato features by reading the Advogato status blog.
If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!