Serious Debian/Ubuntu openssl/openssh bug found

via Reddit, this Debian Security announcement:

‘Luciano Bello discovered that the random number generator in Debian’s openssl package is predictable. This is caused by an incorrect Debian-specific change to the openssl package (CVE-2008-0166). As a result, cryptographic key material may be guessable.

It is strongly recommended that all cryptographic key material which has been generated by OpenSSL versions starting with 0.9.8c-1 on Debian systems (ie since 2006! –jm) is recreated from scratch. Furthermore, all DSA keys ever used on affected Debian systems for signing or authentication purposes should be considered compromised; the Digital Signature Algorithm relies on a secret random value used during signature generation.’

and, of course, here’s the Ubuntu Security Notice for the hole:

Who is affected

Systems which are running any of the following releases:

  • Ubuntu 7.04 (Feisty)
  • Ubuntu 7.10 (Gutsy)
  • Ubuntu 8.04 LTS (Hardy)
  • Ubuntu “Intrepid Ibex” (development): libssl <= 0.9.8g-8
  • Debian 4.0 (etch) (see corresponding Debian security advisory)

and have openssh-server installed or have been used to create an OpenSSH key or X.509 (SSL) certificate. All OpenSSH and X.509 keys generated on such systems must be considered untrustworthy, regardless of the system on which they are used, even after the update has been applied. This includes the automatically generated host keys used by OpenSSH, which are the basis for its server spoofing and man-in-the-middle protection.

It was apparently caused by this incorrect “fix” applied by the Debian maintainers to their package. One wonders why that fix never made it upstream.

Bad news….

Update: Ben Laurie tears into Debian for this:

What can we learn from this? Firstly, vendors should not be fixing problems (or, really, anything) in open source packages by patching them locally - they should contribute their patches upstream to the package maintainers. Had Debian done this in this case, we (the OpenSSL Team) would have fallen about laughing, and once we had got our breath back, told them what a terrible idea this was. But no, it seems that every vendor wants to “add value” by getting in between the user of the software and its author.

+1!

For what it’s worth, we in Apache SpamAssassin work closely with our Debian packaging team, tracking the debbugs traffic for the spamassassin package, and one of the Debian packagers is even on the SpamAssassin PMC. So that’s one way to reduce the risk of upstream-vs-package fork bugs like this, since we’d have spotted that change going in, and nixed it before it caused this failure.

Here’s a question: should the OpenSSL dev team have monitored the bug traffic for Debian and the other packagers? Do upstream developers have a duty to monitor downstream changes too?

This comment puts it a little strongly, but is generally on the money in this regard:

the important part for OpenSSL is to find a way to escape the blame for their fuck-up. They failed to publish the correct contact address for such important questions regarding OpenSSL. Branden (another commenter –jm) noted that the mail address mentioned by Ben is not documented anywhere. It is OpenSSL’s responsibility that they allowed the misuse of openssl-dev for offtopic questions and then silently moving the dev stuff to a secret other list nobody outside OpenSSL knew about.

I’m sure Debian is willing to take their fair share of the blame if OpenSSL finally admits that their mistake played a major role here as well. After all the Debian maintainer might have misrepresented the nature of his plans, but he gave warning signs and said he was unsure. But as it appears now all the people who might have noticed secretly left openssl-dev, the documented place for that kind of questions. This is hardly the fault of the maintainer.

Update 2: this Reddit comment explains the hole in good detail:

Valgrind was warning about unitialized data in the buffer passed into ssleay_rand_bytes, which was causing all kinds of problems using Valgrind. Now, instead of just fixing that one use, for some reason, the Debian maintainers decided to also comment out the entropy mixed in from the buffer passed into ssleay_rand_add. This is the very data that is supposed to be used to see the random number generator; this is the actual data that is being used to provide real randomness as a seed for the pseudo-random number generator. This means that pretty much all data generated by the random number generator from that point forward is trivially predictable. I have no idea why this line was commented out; perhaps someone, somewhere, was calling it with uninitialized data, though all of the uses I’ve found were with initialized data taken from an appropriate entropy pool.

So, any data generated by the pseudo-random number generator since this patch should be considered suspect. This includes any private keys generated using OpenSSH on affected Debian systems. It also includes the symmetric keys that are actually used for the bulk of the encryption.

A pretty major fuck-up, all told.

Update 3: Here’s a how-to page on wiki.debian.org put together by the folks from the #debian IRC channel. It has how-to information on testing your keys for vulnerability using a script called ‘dowkd.pl’, details of exactly what packages and keys are vulnerable, and instructions on how to regenerate keys in each of the (many) affected apps.

It notes this about Apache2 SSL keys:

According to folks in #debian-security, if you have generated an SSL key (normally the step just prior to generating the CSR, and then sending it off to your SSL certificate provider), then the certificate should be considered vulnerable.

So, bad news — SSL keys will need to be regenerated. Add ‘costly’ to the list of downsides. (Yet another update: this hasn’t turned out quite that badly after all — many CAs are now offering free reissuance of affected certs.)

Looking at ‘dowkd.pl’, it gets even worse for ssh users. It appears the OpenSSH packages on affected Debian systems could only generate 1 of only 262148 distinct keypairs. Obviously, this is trivial to brute-force. With a little precomputation (which would only take 14 hours on a single desktop!), an attacker can generate all of those keypairs, and write a pretty competent SSH worm. :(

Update: voila, precomputed keypairs, and figures on the viability of remote brute-forcing the keyspace in 11 minutes.

This entry was posted in Uncategorized and tagged , , , , , , . Bookmark the permalink. Both comments and trackbacks are currently closed.

22 Comments

  1. Brian
    Posted May 13, 2008 at 16:16 | Permalink

    It was never submitted upstream – an OpenSSL developer says they ‘would have fallen about laughing, and once we had got our breath back, told them what a terrible idea this was’: http://www.links.org/?p=327

  2. Posted May 13, 2008 at 16:26 | Permalink

    thanks for the link, Brian — just read that myself.

  3. Posted May 13, 2008 at 17:18 | Permalink

    I’m concerned about two things here: obviously this has raised the whole upstream/distributor-relations businesss, which is fine; it also appears there’s a bit of heat involved as well, which could be better channelled.

    This is a great opportunity for testing paranoia levels: those who cry all is breached should write a debian-screwup-detector utility that cracks keys to prove the point.

  4. Posted May 13, 2008 at 17:19 | Permalink

    I don’t think the openssl devs should be so smug. Personally I care about the software I release and monitor bug reports at debian, ubuntu & fedora at least. I’m not too sure do the openssl devs care much about their users TBH: http://www.pixelbeat.org/programming/lib_crypto.html

  5. Michael Alan Dorman
    Posted May 13, 2008 at 20:15 | Permalink

    Except, of course, that the Debian developer in question did ask the OpenSSL people about his proposed change, and they didn’t fall about laughing:

    http://marc.info/?l=openssl-dev&m=114651085826293&w=2

    Though no one came forward and said, “Go for it”, no one said, “Hey, are you sure you’re not about to neuter your pool of entropy” even though Kurt actually went into some detail about his concerns.

    So the Debian maintainer’s solution may have been wrong, but I think it’s a stretch to suggest he was negligent, and Ben Laurie is just dead wrong.

    Mike.

  6. Posted May 13, 2008 at 21:46 | Permalink

    I’d have to sort-of-disagree with Padraig here. While I think it’s ideal if you can keep an eye on how other people use your software and amend it, it’s not easy to keep on top of all the potential changes as well as maintaining your own original version.

    Personally I believe that if you’re going to amend any software you didn’t write yourself, then make sure you at least ask the original authors if they would add the patch in to the source.

    this has a number of purposes. 1. by pushing the change into the source, your work will be vetted by people that are more familiar with the package than you. 2. your work will eventually be improved and extended by other people. 3. by having your work in the source, you will not need to amend your own derivatives every time a new release comes out.

  7. Posted May 13, 2008 at 22:06 | Permalink

    Fedora has “send stuff upstream” as an explicit policy:

    http://fedoraproject.org/wiki/PackageMaintainers/WhyUpstream

  8. Paul Hedderly
    Posted May 14, 2008 at 08:26 | Permalink

    Debian has a similar policy: http://www.debian.org/doc/manuals/developers-reference/ch-developer-duties.en.html#s-upstream-coordination

    and the link in the earlier comment pointing to the openssl-dev mailing list where the Debian Dev asks about his problem shows Debian do take relationships with upstream very seriously.

  9. Posted May 14, 2008 at 09:45 | Permalink

    +1 for what Michael Alan Dorman said. I think there are some procedures in need of review on Debian’s side (patches not accepted upstream should be periodically re-reviewed, imho) but this went under OpenSSL’s radar too.

  10. Posted May 14, 2008 at 10:01 | Permalink

    Be sure to read the comments at Ben Laurie’s site — an important point is that openssl-dev is not actually the list used by the OpenSSL dev team, but a list for third-party developers using OpenSSL. So that’s a partial reason why it slipped under the OpenSSL guys’ radar.

    However, the Debian package could be considered on-topic for that list, arguably — and someone with an @openssl.org address replied to that mail, saying “I’m in favour of removing them”. If I was in the Debian dev’s shoes I’d probably take that as a “go ahead” too….

    I still think upstream developers have a duty to monitor downstream changes and bug traffic. At the very least, downstream packages are the main way that Linux users will encounter your code — it pays to ensure they work well, for the same reasons it pays to produce regular releases, have public mailing lists, etc.

    As I said in the comments at Ben’s, it’s not hard to do, and that’s one thing in Debian’s favour — they are very transparent in this respect.

    Next: here’s the $64000 question: if SSL Certificate Signing Requests were generated using a vulnerable version of openssl, in order to purchase third-party CA certs, are they in turn vulnerable? I’d assume the answer is yes, which makes things even messier :(

  11. Paul Hedderly
    Posted May 14, 2008 at 10:10 | Permalink

    Sadly the £32,938.67 answer is ‘yes’ – since the CA’s just add data to the public part of the cert you create…

    And yes it is very messy. And potentialy costly.

  12. Posted May 14, 2008 at 10:16 | Permalink

    Justin, Ben’s description of openssl-dev is the first time I have heard of it being for that purpose, and that directly contradicts what is written on openssl.org itself.

  13. Posted May 14, 2008 at 10:30 | Permalink

    well spotted Jon!

    sure enough http://openssl.org/support/ describes that list’s purpose as:

    ‘Discussions on development of the OpenSSL library. Not for application development questions!’

    sounds pretty spot-on-topic to me….

  14. Nix
    Posted May 14, 2008 at 21:58 | Permalink

    Branden Robinson and others have pointed out that before this snafu erupted, the only reference to the openssl-team@openssl.org address was in a comment on the bug tracker. The source distro and web pages all said that openssl-dev@openssl.org was the place to go.

    But of course anyone intending to write a patch to some package should read every single comment to every bug in that package’s bug tracker first… ;}

  15. niklas
    Posted May 15, 2008 at 19:24 | Permalink

    “however, the Debian package could be considered on-topic for that list, arguably — and someone with an @openssl.org address replied to that mail, saying “I’m in favour of removing them”. If I was in the Debian dev’s shoes I’d probably take that as a “go ahead” too….”

    .Why did you remove the part about “if it helps you with debugging”? Trying to spin it? That’s really dishonest citing.

    I’m frankly scared by the horde of Debian apologists, do this reflect the Debian community? Will they sweep this under the rug and go on as before? I don’t like this, I have Ubuntu just about everywhere.

  16. Posted May 15, 2008 at 19:46 | Permalink

    niklas: let’s not quabble over issues of citation like so many micro-managers when there’s real work to be doing fixing your keys.

    It is neither debian’s nor openssl’s fault specifically, but rather a failure in communication between the two parties. Sith happens, scripts to detect weak keys exist, the world keeps spinning.

  17. Posted May 16, 2008 at 12:07 | Permalink

    Just to reiterate how important it is to update your keys/certs, the SANS Internet Storm Center (http://isc.sans.org/) have raised their Infocon status to yellow.

    While over at Metasploit HD Moore has released the “Debian OpenSSL Predictable PRNG Toys” http://metasploit.com/users/hdm/tools/debian-openssl/ to exploit this vulnerability.

    Regards

    Brian

  18. David
    Posted May 19, 2008 at 09:55 | Permalink

    Found this link to 2003 correspondence on someone else’s blog:

    http://rt.openssl.org/Ticket/Display.html?id=521&user=guest&pass=guest

    Here is a snippet, exhibiting the ‘if valgrind complains, it must be wrong’ mentality.

    At lines 467-469 in crypto/rand/md_rand.c is an interesting thing:

    ifndef PURIFY

    MD_Update(&m,buf,j); /* purify complains */

    endif

    That is the code that causes the problem (I just verified it with Valgrind). Does it have any bad side affects to always skip that code? Since both Purify and Valgrind is unhappy with that function call, something must be wrong with it.

    @Nix One might not expect the Debian maintainer to read every single comment in the bug tracker, but one might expect him to read the FAQ for the software he is maintaining.

  19. Posted May 19, 2008 at 17:18 | Permalink

    Looks like if you’ve ever used your DSA key on one of these affected debian systems, you’ll need to regenerate also :( http://blog.sesse.net/blog/tech/2008-05-14-17-21_some_maths.html

    So that means I need to revoke my DSA public key which I generated years ago in a million difference places, as I think I signed stuff on my Feisty desktop :(

  20. just me
    Posted May 19, 2008 at 19:01 | Permalink

    lol good points but the story isn’t represented fully… laurie’s got his head up his ass – the proposed change was posted to openssl-dev for comment: http://marc.info/?t=114651088900003&r=1&w=2 which they blessed. lol.

    if anyone is going to use archaic code and rely on admitted atypical use, quoting laure “usually it is bad to have any kind of dependency on uninitialised [sic] memory”, then the author must document it. plain and simple.

    as referenced by the openssl-dev link even the openssl developers didn’t even have a clue. if they don’t know how the code works, how is anyone else supposed to?

    this is software dev 101: document your shit.

  21. David
    Posted May 20, 2008 at 14:14 | Permalink

    Thought I might check up on when the relevant item concerning valgrind was added to the OpenSSL faq. The pages on the Wayback Machine are here:

    http://web.archive.org/web/*/http://www.openssl.org/support/faq.html

    (Link above apparently not clickable due to asterisk.)

    The last archived page is August 21, 2007. The entry regarding valgrind was not included in any of them, so far as I can see, so it must be a recent addition to the FAQ. So, although one OpenSSL team member was complaining, back in 2003, that ‘This has come up many times before’, they were not prompted to add an entry about it to the FAQ.

  22. Martin Cochran
    Posted May 24, 2008 at 19:11 | Permalink

    Clearly there is some blame to be shared with both parties, but there are a few points worth mentioning/reiterating:

    • Given Debian’s upstream policy, if a patch was submitted (thus, conforming to that policy), the openssl team would have caught this error.

    • Why isn’t it common knowledge that changing crypto code is a bad idea for everyone but vetted crypto experts? Especially for the maintainer of security-related packages? (????)

    • Another common sense argument: The functions that were changed had buf as a parameter. The changes commented out the lines where buf was used to update the hash… After looking at this code for about 10 minutes, it’s apparent this is clearly not a good idea.