Upgrade SSH & Apache
Posted 25 Jun 2002 at 18:19 UTC by todd
Please be advised that if you run sshd (which is still a good idea, encrypting
passwords and administrative commands over the network is a wise choice) you
should update to the new privelege separated sshd and verify that it is
operating in this capacity.
Here are the details of
Just wanted to make sure everyone was aware of this. Sorry if it's out of place or anything, I'm sure I'll hear about it if it is. But hopefully it will be of some service to everyone.
Another vulnerability that you should be aware of is the Apache 'Chunk
If you have not updated for the purposes of security your apache web server
in the last month, you should be seeking to find the updated version.
A few vendors you might wish to visit are:
- OpenBSD http://www.openbsd.org/errata.html
- RedHat http://www.redhat.com/apps/support/errata/
Other vendors should have links from their web page detailing procedures
for updating as well.
You have time before the the end of this week to safely update sshd to the
PrivelegeSeparation functionality. Very early next week, the details of the
vulnerability will be announced. Of course, when a vulnerability exists,
whether the details are widely announced or not, if there is an available fix,
applying the fix in a timely manner is the wisest course of action!
I'm disappointed at the way the openssh vulnerability is being handled.
If it enables a remote root exploit, it's appropriate to be cautious,
but not enough information is being provided to allow sysadmins to make
a reasonable assessment. Evidently the "fix" is partially broken, but
the announcement is vague about the details. It also appears that some
of the confidential lines of communication are not being used, or else
that Theo has not convinced some parties as to the reality or severity
of the bug. Also, it appears from the wording of the message that even
after the "fix" it is possible for someone to break in as user sshd but
be trapped in a chroot jail; presumably this would enable a DoS attack.
On the other hand, perhaps Theo just means that if a new exploit were found the cracker could at most get into the chroot jail as an unprivileged user, it is unclear.
I'm not asking for complete disclosure, but I'd like to see more disclosure than has occurred so far. And what I write above may be a complete misinterpretation, but without more information I can't come to a correct interpretation. Conclusion: I don't know at this point what the best course of action is.
Actually, Theo is quite clear about this ("3.3 does not contain a fix for this upcoming bug."): As of today, there is no fix available, the upgrade to 3.3 with UsePrivilegeSeparation simply assures that when the vulnerability is exploited, the cracker finds herself in an empty chroot. Theo explicitly states that they are still working on the fix itself, which, supposedly, will be released early next week or so.
Re more information, check http://docs.freebsd.org/mail/current/freebsd-security.html.
In a nutshell, posted 25 Jun 2002 at 21:09 UTC by Pizza »
Here's the rundown for all those who are criticizing Theo and the OpenSSH team for how they're handing this:
- All versions of OpenSSH, including 3.3 are vulnerable to a remote exploit. This will be fixed in a release on Monday, along with full disclosure.
- On OpenSSH older then 3.3, this exploit becomes a remote root exploit and there's nothing you can do about it other than disable ssh entirely.
- If you use OpenSSH 3.3 with the new privlidge seperation, the exploit still exists but instead the attacker ends up in a non-privlidged chroot'd jail, in effect unable to do anything.
- Privlidge seperation is not completely functional on all platforms. In particular, some PAM systems have problems that the OpenSSH team are requesting vendor help to solve.
- The privlidge seperation code also breaks compression on Linux 2.2.x systems, thanks to incomplete mmap() functionality. You can disable compression until this is properly fixed.
Which would you rather have, a remotely exploitable root hole into your machines, or the ability to use compression on ssh connections?
Upgrading to OpenSSH 3.3 will not fix this exploit, but it decreases its severity considerably. Once the bug is fully disclosed, you can bet your ass that lots of hax0rs and script kiddies will be banging on every ssh server they can find.
Yes, there's going to be a new version of OpenSSH in a week. But until it's released and you've upgraded to it, you're still vulnerable. So the best you can do until then is to upgrade to 3.3 and be considerably less vulnerable.
In an nutshell..., posted 25 Jun 2002 at 22:03 UTC by thom »
We haven't been told a thing about what the problem is, how high the risk is, or any other details that sys admins and vendors actually need.
Saying "Upgrade OpenSSH to 3.3 now! I'm not going to tell you why!" is not really acceptable or fair communication with anyone.
It seems that both the ISS and Theo De Raadt feel that the current system for vendor disclosure and basic politeness are things they can freely ignore.
It also seems that they both feel that releasing reasonable (ie, working under all situations) fixes is something they don't have to bother with. (see, for example, ISS' broken fix for apache, and the fact that PrivilegeSeperation doesn't work with linux kernels <2.4)
Oops, posted 25 Jun 2002 at 22:05 UTC by thom »
uh, s/PrivilegeSeperation/PrivilegeSeperation and compression/ in my last.
My critique of Theo is that he did not include the information that
was contained in Pizza's post above, and he chose instead to waste
bandwidth attacking Alan Cox and HP. A cynic might suppose that he
wanted to avoid uttering the sentence "OpenBSD contains a remote root
exploit". Believers in the "evil eye" might conclude that the OpenBSD
and Apache folks did a bit too much bragging and brought down a curse. :-)
Now that more information is available, it's clear that 3.3 is an attempt at a stopgap solution and the real fix will be in 3.4. But the problem
is in the bad communication and in continuing relationship damage with
developers from other projects.
I don't know the context for Alan Cox's remark about Theo possibly slipping us a trojan, but suspect that it was a joke, or at least
an implicit request for more info (which could have been communicated
confidentially). If not, Alan deserves as
much criticism as Theo. These things are too important to let petty
personal disputes impede the ability to work together on fixes.
Vendors have confidential mechanisms set up for discussing security bugs,
but Theo refuses to play. See
I am not nearly naive enough to believe that we can release a patch
for this issue to any vendor, and have it not leak immediately.
Yes, Theo is going on a power trip assuming he knows best how to protect the world from evil (defined as all users != him+friends or anything which might break OpenBSD's running "no security holes in the base system without any useful software installed in NNN days" record). We are afterall talking about one of the the ego's that helped cause a major round of project forking in BSD land. ;)
Quit whining, posted 26 Jun 2002 at 02:26 UTC by deekayen »
My gosh people. Theo isn't evil. Humans are flawed, therefore so is software. Even software made famous by the "secure by default" motto is found vulnerable now and then. Just upgrade and be happier your boxen are more secure than before. Otherwise, try a few weeks in my job (installing MS critical updates on my school's computer labs). I've been doing it since February, and never run out of things to do.
Now that more information is available, it's clear that 3.3 is an attempt at a stopgap solution and the real fix will be in 3.4.
I still insist that this is not clear only "now that more information is available", but that is exactly
what Theo's been saying from the very beginning in this
email. Anybody who says otherwise has not read that email.
ssh alternatives, posted 26 Jun 2002 at 04:26 UTC by yakk »
OpenSSH is a scary pile of code. I've looked through it quite a bit while writing my own ssh implementation. I'm not sure how comfortable I would be to run OpenSSH on a machine that had to actually be secure. We've had several remote root exploits so far and theres no end in sight.
Remember theres always ssl telnet :)
apt-get install telnet-ssl telnetd-ssl
Vendors have confidential mechanisms set up for discussing security bugs, but Theo refuses to play.
We tried to do that last time, but the bug was leaked and we had to rush the release. The vendors have proven themselves not to be trustworthy.
IMO this is the best possible way to deal with this. We are giving everyone a chance to immunize themselves by making a workaround available before we release the real fix (which will be used to build an exploit).
This isn't about personalities.
OpenSSH is a scary pile of code... Remember theres always ssl telnet :)
If you think OpenSSH is scary you obviously haven't looked through OpenSSL and telnetd.
You have been publically complaining about OpenSSH for a while. If you think you can do better then please show us some code. Either send us bug reports (the only bug report you filed got fixed within 12 hours), patches (haven't seen any from you ever) or go and write your own sshd.
*sigh*, posted 26 Jun 2002 at 13:37 UTC by todd »
My June 25th diary entry says just about all there is to say about the above discussion.
The ISS advisory came out today.
So much for withholding information from vendors to stop it from being leaked.
So much for waiting until OpenSSH 3.4 is available.
It turns out that you are not vulnerable if you add "ChallengeResponseAuthentication no" to your config file.
It's possible that this isn't the vulnerability everyone was talking about, and we'll be seeing another advisory on Monday, but the Security Focus database seems to indicate that they are the same.
I'm really disappointed in how this was handled. Theo claimed that he did not trust
the established security community not to prematurely leak information
on what the bug was, and promised that 3.4 would be out Monday with a
fix. I'm sure that a number of people made plans based on his word.
Then he went ahead with a release five days early, meaning that it appears
that Theo himself cannot be trusted not to leak information before
agreed-apon dates. I am convinced that there are many sysadmins out
there who had not yet done anything about disabling sshd because they
thought they had a bit more time to get around to installing 3.3.
Fortunately they aren't too hosed because it turns out that there is a
one-line config file temp-fix.
Yes, it turns out that there was an easy fix all along that did not
require any software updates, meaning that the Debian folks didn't have
to do that emergency release of 3.3 that broke compression (since potato
still uses a 2.2.x Linux kernel, and the privsep code in 3.3 causes
compression to break, evidently because of some issue with mmap).
Perhaps Theo feared that if he provided the simple fix, the bad guys would
immediately go and read the challenge-response code to check for flaws.
Perhaps Theo believes so strongly in the privilege separation concept
that he thought it justified to stampede the entire OpenSSH user community into using it immediately (now, I like the idea;
it is good to minimize the amount of software that runs as root). But, given that a one-line
fix in the config file sufficed to fix the bug, and given that there
are still some portability issues with the privilege separation
the CERT process would
have worked in this case.
Todd, your analysis in your diary is wrong, because you argue against
straw men. No one was calling for full disclosure to the public.
There are confidential forums where bugs can be discussed; if Theo
doesn't trust such forums, he could have picked out selected developers he
trusts from other OS development communities. The Debian, Red Hat, and
FreeBSD folks have a very good record of keeping security bugs confidential until the agreed-apon date for release, and with proprietary
OS vendors the record is even better (there we have the opposite problem,
their process is too damned slow).
Mind you, I'm grateful for all Theo's good work in OpenBSD and OpenSSH.
But that doesn't immunize him from criticism when he screws up.
hrm, posted 26 Jun 2002 at 17:58 UTC by todd »
I understood iss was driving the timeline not Theo.
This announcement fills me with sadness.
OpenSSH on OpenBSD should be the security flagship product of the most paranoid distribution. Remote root holes in openssh suggests that this community is doing something fundamentally wrong - if Theo and co. can't get it right despite devoting all their efforts to it, what hope does anyone else have?
Try realizing that the only guarantee is that an attempt at fixing everything known is
There is no guarantee that OpenBSD is 100% free from any flaws or bugs, nor will there ever be.
Read the goals and you'll note that one line says:
Reading all of security.html is recommended. You'll find that in no way does it suggest that there
are no bugs. Infact it states that new security related bugs are continually found and
IMHO, it is the high standard wrt trivial code fixes (that sometimes
are found to be security related later on), security *FIXES*, and auditing for
anything (bad practice, security issue) found in the rest of the tree
that sets OpenBSD apart.
Also worth noting, I am not aware of anyone who was not effected.
: the original code that OpenSSH was based from (the original ssh, 1.2.13 and earlier) was extreemly disturbing code. stop whining and write your own if you don't like it.
Anyone that wants to run a secure network server should really not be running one written in any language allowing pointers.
Look for a client written in an interpreted language such as java or python (see the mindterm ssh client, i don't know if anyone has written such a server).
Languages like that eliminate a significant number of potential points of failure due to traditional buffer overflow or off by N counter problems. Sure those languages themselves do have to interface with the C library, system calls and potentially other external libraries but that limits the points where potential problems can occur to a set of interfaces used by lots of programs, not just yours.
wrong doing, posted 1 Jul 2002 at 22:47 UTC by cpw »
(I wasn't meaning to diss OpenBSD, nor do I expect it to be bug free.
I think I was just disappointed that code audits aren't a silver bullet,
which was not terribly realistic of me.)