Older blog entries for dtucker (starting at number 52)

12 Dec 2004 (updated 13 Dec 2004 at 08:45 UTC) »

First, in the interests of disclosure: I'm one of the OpenSSH and OpenBSD developers, and the person responsible for the Portable branch of OpenNTPD. That said, the following is purely my own opinion.

I've decided to address some of the issues raised by Brad Knowles in an article entitled OpenNTPd Considered Harmful. While I think some of his points are valid (and some of those have been addressed since it was written), I disagree with others and some are quite misleading.

stratum and refid (ntptrace)
The comments about stratum and refid (the comment about ntptrace): these are valid points. They have been have been fixed for a while in the development tree and now in the openntpd 3.6.1 release.

I don't think the criticism about portability is warranted. At the time that article was written Portable OpenNTPD already supported Solaris (it was the 2nd target I did after Linux) and HP-UX support has since been added. I don't think it's valid to criticise a project that's only existed for a couple of months for "only" running on Linuxes, *BSD's, Mac OS X, and Solaris (which covers the 3 main *nix families in use today, Linux, BSD, System V). I suspect it's probably portable to any platform with a POSIX interface and an adjtime() syscall (or something that behaves like it) [update: and a good entropy source (such as /dev/random or a self-seeding OpenSSL)]; it already runs on every system that I have access that meets those criteria with the exception of one (and that one is for non-technical reasons).

Brad also says that the OpenNTPD developers "seem to be violently opposed to using certain methods which are known to be more portable, in favour of using techniques which are specific to OpenBSD". I'm not aware of what he's referring to here. There are several instances of changes in OpenBSD for the purposes of improving portability (eg 1, 2, 3) and if there are ways of doing things in -Portable to improve portability then I would be interested to hear them.

Finally, he describes the automated test procedures they use on the "flock of test/development machines we have at our disposal". Since the article was written I have added build and basic functionality tests for OpenNTPD to tinderbox automated tester that we also use for OpenSSH (the machines which appear will vary depending on which machines I have running at the time). BTW the key phrase here is at our disposal: we can't test on systems we don't have access to.

Split development
About the split OpenBSD/Portable development model, Brad says that it "pretty much guarantees that the two code bases will quickly diverge quite dramatically". This is not necessarily so: as a counterexample, OpenSSH has operated with this model for five years quite successfully. It requires discipline, but it can work. It is not a coincidence that OpenNTPD's portability layer looks like OpenSSH's.

In the case of OpenNTPD, the split between OpenBSD and Portable is quite clean and the differences in the common code are small, a total of approximately ~70 lines for 3.6.1p1 including man page diffs. (The diff is in the Portable tarball).

Clock disciplining
The comment about clock disciplining (compensation for systematic skew or drift) is a fair point, within limits. Right now OpenBSD doesn't permit changing of tickadj at the default securelevel so another mechanism is needed in the kernel. When this happens, ntpd will probably learn how to do it. (Note: this does not necessarily prevent it from being done on other platforms, for example see my experimental patch that adds crude skew compensation via adjtimex to Linux, which is implemented with zero changes to the common code.)

I would like to point out, though, that the charge of "jumping the clock forward or backward" is very misleading: at the time the article was written, OpenNTPD would never cause the clock to jump either forward or backward: the offsets are filtered (although the filters could be improved) and the adjustments were always done via adjtime (a "slew") which means the time is always monotonically increasing. Even now, the only time it will use settimeofday (a "step") for the first adjustment, and only if this behaviour is enabled with the -s option. In contrast, the reference implementation in its default configuration will step the clock any time the offset exceeds 128ms (see the ntpd man page, specifically the "-x" option).

Alternative server modes
Brad criticizes OpenNTPD for not imlementing broadcast, multicast and anycast modes because "the NTP protocol can put a heavy load on the server, and trying to handle thousands of clients, each in direct one-to-one communications, just isn't feasible".

Dr David Mills describes a Stratum-1 server running on a SPARC IPC (a 25MHz machine) serving 734 clients while using 1.54% of the CPU and concludes "that substantial numbers of clients with no significant degradation on other network services". Arithmetic suggests that even this long-obsolete machine could support thousands of clients.

While OpenNTPD doesn't support these modes now, it may in future, however it's not clear how much of a gain these would be in many environments. Broadcast mode is limited to the local network, thus it would only be necessary in environments where there are more clients in a single subnet or broadcast domain than can be handled by a single server. Multicast mode requires the network to support multicast routing (eg IGMP, DVMRP, MOSPF), although in an environment that supports it, it could result in a reduction of WAN traffic. Anycast mode is described in RFC 2030 as "provisional", and doesn't help with load, it only helps in locating a server, after which it operates in unicast mode.

Authentication and crypto
Regarding the comment about authentication and crypto: it depends on what your threat profile is. Brad correctly points out that it's not a zero-sum game. It is, however, a tradeoff: you're trading a reduction in risk that someone will be able to perform an active attack against your clock for an increased risk of a security issue in the additional code required to implement it. There's no way to prove the relative weights in this tradeoff, it's a matter of opinion.

As Brad points out, OpenNTPD's "cookies" always provides some protection against blind spoofing of responses. They're by no means perfect, but they're a simple measure that provides cheap insurance and requires no configuration or additional support in the server.

The symmetric-key authentication in NTP v3 seems to have limited deployment (probably due to key distribution issues and administrative overhead, and it appears that most public servers don't offer it) and is listed as optional in RFC 1305.

I can't comment much on the public-key modes in NTP v4 as at the time of writing the NTP v4 spec is not available, although from the description of the Autokey protocol it appears that it is based on X.509v3 certificates. I will only point out that OpenSSL's ASN.1 parser and X.509v3 code runs to around 15,000 lines (probably due to the general wackiness of X.509 [1]), and this is a significant amount of additional code to be running in a daemon that accepts input from the network (via a transport that, as Brad points out, can be trivially spoofed).

Lack of Features
(also known as "simplicity"). Guilty as charged.

If you need a feature the OpenNTPD doesn't have (and, indeed there are many), or need greater accuracy than it can provide, or if you just prefer it, then by all means use ntp.org's software (or any other software for that matter). If you want a small daemon that will do a decent job of keeping your clock in sync while running mostly unprivileged, then OpenNTPD may suit. If it does, great. If not, and you decide to use something else, that's fine too. You have another option, which is why I started the Portable branch in the first place.

[1] For an overview of said wackiness, see Peter Gutmann's PKI Tutorial, subtitled "Everything you Never Wanted to Know about PKI but were Forced to Find Out", and for a comprehensive treatment see his X.509 Style Guide).

OpenSSH celebrated its fifth birthday this week.

Busy working, training and getting ready for the next OpenSSH release. Users of OpenSSH with PAM will hopefully be happier. Test a snap, please :-)

11 Jul 2004 (updated 11 Jul 2004 at 09:08 UTC) »
OpenBSD -current now has a small, simple NTP daemon as an alternative for the large, featureful reference implementation. It's small and should be secure (it's privilege separated) and should serve the needs of most people (who don't need the power or features of the reference implementation.)

I've taken this, added autoconf and a portability layer much like (indeed, based heavily on) OpenSSH and produced a portable version. Currently it works on Linux and FreeBSD in addition to OpenBSD. Depending on how well it works out, support for other platforms might be added.

While it's small, it's fully functional for basic sync-to or sync-from requirements, but it doesn't (yet) have the capability to act as a stratum-1 time source. On the other hand, it's an order of magnitude smaller than the reference implementation :-)

$ du -ks /usr/sbin/ntpd /usr/local/sbin/ntpd
328     /usr/sbin/ntpd
36      /usr/local/sbin/ntpd
(both compiled normally, dynamically linked and stripped).

Correction: The latter is compiled without optimization, with -O2 the size is 28KB. I also forgot to mention the funny part: the "configure" script is bigger than rest of the source code combined.

OpenSSH 3.8.1p1 will be released soon (it's a bugfix release), so I sent a call-for-testing.

We run regular tests of the OpenSSH tree (the results go to the tinderbox) and I've been looking at improving the test coverage by building with non-gcc compilers on my Linux box. So far I have found:

  • TenDRA, but the last time I tried it it would not build on my box.
  • tcc, a small Linux/x86-only compiler.
  • icc, which I just found has a non-commercial non-expiring license option.

I played with tcc which seems interesting (and fast), but ran into problems with its library-search behaviour. It would search -L paths last, and it would always use a dynamic library, even if a static library was before it in the library path.

Its code turned out to be quite easy to work with, so I modified it to handle -L/-l more like gcc does and submitted the patch back to the author. I also found what is probably a portability bug in OpenSSH. Unfortunately, some of the OpenSSH binaries produced by tcc segfault for reasons I've been unable to determine, so it's currently not suitable for regular build-and-test use.

Sorted some of the pending OpenSSH bugs into fix for 3.8.1p1 and aim for the next major release.

For the past couple of releases, we've opened a "Release Engineering" bug to tie all of the pending fixes shortly before the release, to make sure stuff didn't get missed. I found myself sorting bugs into "fix now" and "fix later", so I decided to start tracking the "fix later" ones too. There's no guarantee that the listed ones will be done, but it should give folks a rough idea what the plan is.

clarkbw, regarding your self-assessment proposal: I have a book (Maverick! by Ricardo Semler) that describes a much higher stakes version of what you're proposing: at the Brazillian company Semco, most of their senior staff set their own salaries!

Apart from modesty and peer pressure, they have a very real incentive to keep it realistic: the easiest way to solve a budget problem is to get rid of someone drawing an excessive salary! They found that most people set realistic salaries, and of those that didn't all but one set a salary lower than expected.

Anyway, the book is a fascinating read even if you're not into management theories. (I'm not: I picked it up on a whim in a bookshop one day, then realised I had been standing there reading it for over 20 minutes, so I bought it). The book is about 10 years old, it would also be interesting to know what has changed since then.

OpenSSH 3.8 is out. Coming soon to a mirror near you...

There is going to be a new OpenSSH release soon. If you want the release to work on your system, you ought to try out a snapshot.

Particularly, you should try it if you're interested in PAM fixes or Kerberos/GSSAPI support. If you have access to an unusual system or configuration then tests on those are particularly helpful since we can't test all of them.

You can see the bugs that ought to be fixed before 3.8

Hmm, this asking-for-testing thing usually gets only limited responses, since most people prefer to complain after the event. Maybe I should try a little reverse psychology...

There will be a new OpenSSH release soon. Under no circumstances should you try a snapshot. You especially should not test it if you're interested in PAM fixes or Kerberos/GSSAPI support. This goes double if you have an unusual system or configuration, because since we're psychic we always know what's going to break on system's we've never seen or heard of, and magically know what to do to avoid problems.

Yeah, that ought to do it :-)

More OpenSSH hacking. I'm trying to figure out why the new(ish) PAM chauthtok via keyboard-interactive code displays the PAM messages properly on some platforms (Redhat, AIX) but doesn't on others (Solaris, HP-UX). I added some instrumentation to the PAM code to help debug it and spent some time poking sshd with a debugger, but without definitive results. It seems to be related to to message "style" set by PAM when responding to the request, but needs more investigation.

I also noticed a problem whereby the if the PAM authentication thread dies without completing the authentication (either success or failure), sshd will block indefinitely for that session. I have a fix for that, which will go in once a few style nits are sorted out.

Spent a little time looking at dropbear, another SSH2 server implementation. While fiddling with it I got it running on HP-UX, fixed what I suspect was a bug and sent the patch for both back to the author.

43 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!