Upgrade SSH & Apache

Posted 25 Jun 2002 at 18:19 UTC by todd Share This

Please be advised that if you run sshd (which is still a good idea, encrypting passwords and administrative commands over the network is a wise choice) you should update to the new privelege separated sshd and verify that it is operating in this capacity.

Here are the details of why.

Just wanted to make sure everyone was aware of this. Sorry if it's out of place or anything, I'm sure I'll hear about it if it is. But hopefully it will be of some service to everyone.

Another vulnerability that you should be aware of is the Apache 'Chunk Handling Vulnerability'.

If you have not updated for the purposes of security your apache web server in the last month, you should be seeking to find the updated version.

A few vendors you might wish to visit are:

  • OpenBSD http://www.openbsd.org/errata.html
  • RedHat http://www.redhat.com/apps/support/errata/
  • SuSE http://www.suse.com/us/support/security/index.html

Other vendors should have links from their web page detailing procedures for updating as well.

You have time before the the end of this week to safely update sshd to the PrivelegeSeparation functionality. Very early next week, the details of the vulnerability will be announced. Of course, when a vulnerability exists, whether the details are widely announced or not, if there is an available fix, applying the fix in a timely manner is the wisest course of action!

problems with the ssh vulnerability announcement, posted 25 Jun 2002 at 19:49 UTC by jbuck » (Master)

I'm disappointed at the way the openssh vulnerability is being handled. If it enables a remote root exploit, it's appropriate to be cautious, but not enough information is being provided to allow sysadmins to make a reasonable assessment. Evidently the "fix" is partially broken, but the announcement is vague about the details. It also appears that some of the confidential lines of communication are not being used, or else that Theo has not convinced some parties as to the reality or severity of the bug. Also, it appears from the wording of the message that even after the "fix" it is possible for someone to break in as user sshd but be trapped in a chroot jail; presumably this would enable a DoS attack. On the other hand, perhaps Theo just means that if a new exploit were found the cracker could at most get into the chroot jail as an unprivileged user, it is unclear.

I'm not asking for complete disclosure, but I'd like to see more disclosure than has occurred so far. And what I write above may be a complete misinterpretation, but without more information I can't come to a correct interpretation. Conclusion: I don't know at this point what the best course of action is.

no "fix" available, posted 25 Jun 2002 at 21:05 UTC by jschauma » (Observer)

Actually, Theo is quite clear about this ("3.3 does not contain a fix for this upcoming bug."): As of today, there is no fix available, the upgrade to 3.3 with UsePrivilegeSeparation simply assures that when the vulnerability is exploited, the cracker finds herself in an empty chroot. Theo explicitly states that they are still working on the fix itself, which, supposedly, will be released early next week or so.

Re more information, check http://docs.freebsd.org/mail/current/freebsd-security.html.

In a nutshell, posted 25 Jun 2002 at 21:09 UTC by Pizza » (Master)

Here's the rundown for all those who are criticizing Theo and the OpenSSH team for how they're handing this:

  • All versions of OpenSSH, including 3.3 are vulnerable to a remote exploit. This will be fixed in a release on Monday, along with full disclosure.
  • On OpenSSH older then 3.3, this exploit becomes a remote root exploit and there's nothing you can do about it other than disable ssh entirely.
  • If you use OpenSSH 3.3 with the new privlidge seperation, the exploit still exists but instead the attacker ends up in a non-privlidged chroot'd jail, in effect unable to do anything.
  • Privlidge seperation is not completely functional on all platforms. In particular, some PAM systems have problems that the OpenSSH team are requesting vendor help to solve.
  • The privlidge seperation code also breaks compression on Linux 2.2.x systems, thanks to incomplete mmap() functionality. You can disable compression until this is properly fixed.

Which would you rather have, a remotely exploitable root hole into your machines, or the ability to use compression on ssh connections?

Upgrading to OpenSSH 3.3 will not fix this exploit, but it decreases its severity considerably. Once the bug is fully disclosed, you can bet your ass that lots of hax0rs and script kiddies will be banging on every ssh server they can find.

Yes, there's going to be a new version of OpenSSH in a week. But until it's released and you've upgraded to it, you're still vulnerable. So the best you can do until then is to upgrade to 3.3 and be considerably less vulnerable.

In an nutshell..., posted 25 Jun 2002 at 22:03 UTC by thom » (Master)

We haven't been told a thing about what the problem is, how high the risk is, or any other details that sys admins and vendors actually need.
Saying "Upgrade OpenSSH to 3.3 now! I'm not going to tell you why!" is not really acceptable or fair communication with anyone.
It seems that both the ISS and Theo De Raadt feel that the current system for vendor disclosure and basic politeness are things they can freely ignore.
It also seems that they both feel that releasing reasonable (ie, working under all situations) fixes is something they don't have to bother with. (see, for example, ISS' broken fix for apache, and the fact that PrivilegeSeperation doesn't work with linux kernels <2.4)

Oops, posted 25 Jun 2002 at 22:05 UTC by thom » (Master)

uh, s/PrivilegeSeperation/PrivilegeSeperation and compression/ in my last.

reason for the critique, posted 25 Jun 2002 at 22:40 UTC by jbuck » (Master)

My critique of Theo is that he did not include the information that was contained in Pizza's post above, and he chose instead to waste bandwidth attacking Alan Cox and HP. A cynic might suppose that he wanted to avoid uttering the sentence "OpenBSD contains a remote root exploit". Believers in the "evil eye" might conclude that the OpenBSD and Apache folks did a bit too much bragging and brought down a curse. :-)

Now that more information is available, it's clear that 3.3 is an attempt at a stopgap solution and the real fix will be in 3.4. But the problem is in the bad communication and in continuing relationship damage with developers from other projects.

I don't know the context for Alan Cox's remark about Theo possibly slipping us a trojan, but suspect that it was a joke, or at least an implicit request for more info (which could have been communicated confidentially). If not, Alan deserves as much criticism as Theo. These things are too important to let petty personal disputes impede the ability to work together on fixes.

OK, here's what I object to, posted 25 Jun 2002 at 22:47 UTC by jbuck » (Master)

Vendors have confidential mechanisms set up for discussing security bugs, but Theo refuses to play. See this message. To quote:

I am not nearly naive enough to believe that we can release a patch for this issue to any vendor, and have it not leak immediately.

Theos will be Theos, posted 26 Jun 2002 at 01:03 UTC by splork » (Master)

Yes, Theo is going on a power trip assuming he knows best how to protect the world from evil (defined as all users != him+friends or anything which might break OpenBSD's running "no security holes in the base system without any useful software installed in NNN days" record). We are afterall talking about one of the the ego's that helped cause a major round of project forking in BSD land. ;)

Quit whining, posted 26 Jun 2002 at 02:26 UTC by deekayen » (Master)

My gosh people. Theo isn't evil. Humans are flawed, therefore so is software. Even software made famous by the "secure by default" motto is found vulnerable now and then. Just upgrade and be happier your boxen are more secure than before. Otherwise, try a few weeks in my job (installing MS critical updates on my school's computer labs). I've been doing it since February, and never run out of things to do.

re: reason for the critique, posted 26 Jun 2002 at 02:53 UTC by jschauma » (Observer)

Now that more information is available, it's clear that 3.3 is an attempt at a stopgap solution and the real fix will be in 3.4.
I still insist that this is not clear only "now that more information is available", but that is exactly what Theo's been saying from the very beginning in this email. Anybody who says otherwise has not read that email.

ssh alternatives, posted 26 Jun 2002 at 04:26 UTC by yakk » (Master)

OpenSSH is a scary pile of code. I've looked through it quite a bit while writing my own ssh implementation. I'm not sure how comfortable I would be to run OpenSSH on a machine that had to actually be secure. We've had several remote root exploits so far and theres no end in sight.

Remember theres always ssl telnet :)

apt-get install telnet-ssl telnetd-ssl

Re: OK, here's what I object to, posted 26 Jun 2002 at 07:54 UTC by djm » (Master)

Vendors have confidential mechanisms set up for discussing security bugs, but Theo refuses to play.

We tried to do that last time, but the bug was leaked and we had to rush the release. The vendors have proven themselves not to be trustworthy.

IMO this is the best possible way to deal with this. We are giving everyone a chance to immunize themselves by making a workaround available before we release the real fix (which will be used to build an exploit).

This isn't about personalities.

Re: ssh alternatives, posted 26 Jun 2002 at 07:54 UTC by djm » (Master)

OpenSSH is a scary pile of code... Remember theres always ssl telnet :)

If you think OpenSSH is scary you obviously haven't looked through OpenSSL and telnetd.

You have been publically complaining about OpenSSH for a while. If you think you can do better then please show us some code. Either send us bug reports (the only bug report you filed got fixed within 12 hours), patches (haven't seen any from you ever) or go and write your own sshd.

*sigh*, posted 26 Jun 2002 at 13:37 UTC by todd » (Master)

My June 25th diary entry says just about all there is to say about the above discussion.

More fuel for the fire, posted 26 Jun 2002 at 15:32 UTC by AlanShutko » (Journeyer)

The ISS advisory came out today.

So much for withholding information from vendors to stop it from being leaked.

So much for waiting until OpenSSH 3.4 is available.

It turns out that you are not vulnerable if you add "ChallengeResponseAuthentication no" to your config file.

It's possible that this isn't the vulnerability everyone was talking about, and we'll be seeing another advisory on Monday, but the Security Focus database seems to indicate that they are the same.

now that we know the details ..., posted 26 Jun 2002 at 17:18 UTC by jbuck » (Master)

I'm really disappointed in how this was handled. Theo claimed that he did not trust the established security community not to prematurely leak information on what the bug was, and promised that 3.4 would be out Monday with a fix. I'm sure that a number of people made plans based on his word. Then he went ahead with a release five days early, meaning that it appears that Theo himself cannot be trusted not to leak information before agreed-apon dates. I am convinced that there are many sysadmins out there who had not yet done anything about disabling sshd because they thought they had a bit more time to get around to installing 3.3. Fortunately they aren't too hosed because it turns out that there is a one-line config file temp-fix.

Yes, it turns out that there was an easy fix all along that did not require any software updates, meaning that the Debian folks didn't have to do that emergency release of 3.3 that broke compression (since potato still uses a 2.2.x Linux kernel, and the privsep code in 3.3 causes compression to break, evidently because of some issue with mmap).

Perhaps Theo feared that if he provided the simple fix, the bad guys would immediately go and read the challenge-response code to check for flaws. Perhaps Theo believes so strongly in the privilege separation concept that he thought it justified to stampede the entire OpenSSH user community into using it immediately (now, I like the idea; it is good to minimize the amount of software that runs as root). But, given that a one-line fix in the config file sufficed to fix the bug, and given that there are still some portability issues with the privilege separation code, the CERT process would have worked in this case.

Todd, your analysis in your diary is wrong, because you argue against straw men. No one was calling for full disclosure to the public. There are confidential forums where bugs can be discussed; if Theo doesn't trust such forums, he could have picked out selected developers he trusts from other OS development communities. The Debian, Red Hat, and FreeBSD folks have a very good record of keeping security bugs confidential until the agreed-apon date for release, and with proprietary OS vendors the record is even better (there we have the opposite problem, their process is too damned slow).

Mind you, I'm grateful for all Theo's good work in OpenBSD and OpenSSH. But that doesn't immunize him from criticism when he screws up.

hrm, posted 26 Jun 2002 at 17:58 UTC by todd » (Master)

I understood iss was driving the timeline not Theo.

what are we doing wrong here?, posted 26 Jun 2002 at 19:28 UTC by cpw » (Apprentice)

This announcement fills me with sadness.
OpenSSH on OpenBSD should be the security flagship product of the most paranoid distribution. Remote root holes in openssh suggests that this community is doing something fundamentally wrong - if Theo and co. can't get it right despite devoting all their efforts to it, what hope does anyone else have?

re: alleged wrong doing, posted 26 Jun 2002 at 19:51 UTC by todd » (Master)

Try realizing that the only guarantee is that an attempt at fixing everything known is in progress.

There is no guarantee that OpenBSD is 100% free from any flaws or bugs, nor will there ever be.

Read the goals and you'll note that one line says:

Reading all of security.html is recommended. You'll find that in no way does it suggest that there are no bugs. Infact it states that new security related bugs are continually found and fixed.

IMHO, it is the high standard wrt trivial code fixes (that sometimes are found to be security related later on), security *FIXES*, and auditing for anything (bad practice, security issue) found in the rest of the tree that sets OpenBSD apart.

re: alleged wrong doing, posted 26 Jun 2002 at 20:40 UTC by todd » (Master)

Also worth noting, I am not aware of anyone who was not effected.

on code scariness and network servers, posted 26 Jun 2002 at 22:40 UTC by splork » (Master)

yakk: the original code that OpenSSH was based from (the original ssh, 1.2.13 and earlier) was extreemly disturbing code. stop whining and write your own if you don't like it.

Anyone that wants to run a secure network server should really not be running one written in any language allowing pointers.

Look for a client written in an interpreted language such as java or python (see the mindterm ssh client, i don't know if anyone has written such a server).

Languages like that eliminate a significant number of potential points of failure due to traditional buffer overflow or off by N counter problems. Sure those languages themselves do have to interface with the C library, system calls and potentially other external libraries but that limits the points where potential problems can occur to a set of interfaces used by lots of programs, not just yours.

wrong doing, posted 1 Jul 2002 at 22:47 UTC by cpw » (Apprentice)

(I wasn't meaning to diss OpenBSD, nor do I expect it to be bug free. I think I was just disappointed that code audits aren't a silver bullet, which was not terribly realistic of me.)

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

Share this page