By Dan Goodin in San Francisco
21st May 2008
It's been more than a week since Debian patched a massive security hole
in the library the operating system uses to create cryptographic keys
for securing email, websites and administrative servers. Now the hard
work begins, as legions of admins are saddled with the odious task of
regenerating keys too numerous for anyone to estimate.
The flaw in Debian's random number generator means that OpenSSL keys
generated over the past 20 months are so predictable that an attacker
can correctly guess them in a matter of hours. Not exactly a comforting
thought when considering the keys in many cases are the only thing
guarding an organization's most precious assets. Obtain the key and you
gain instant access to trusted administrative accounts and the ability
to spoof or spy on sensitive email and web servers.
Security pros have rightfully reacted swiftly to word of Debian debacle.
But if you think last week's patch is like most other security fixes,
you're dead wrong. Installing it is probably the easiest part of mopping
up the resulting mess. Once it's installed, admins will be forced to
search sometimes sprawling systems for every key that's ever interacted
with the buggy version of Debian and a host of other OSes and
applications that relied on it.
Certificates for defective keys will have to be revoked, new keys will
have to be generated and, in the case of SSL certificates, registered
with VeriSign or another certificate authority. No one knows how many
keys need to be replaced, but it could number in the hundreds of
thousands or millions. The keys are used for Secure Sockets Layer (SSL)
transactions, which authenticate servers handling trusted websites and
email, and to authenticate Secure Shell (SSH), which provides encrypted
channels between sensitive computers.
The heft and tedium of tracking down, testing and regenerating so many
keys, and the cost of paying certificate authorities to register them,
has left some people feeling pessimistic about the prospects the problem
will be fixed anytime soon.
"There's the pain-in-the-ass factor and then there's the cost factor,"
says Jacob Appelbaum, an independent security researcher, as he ticks
off the reasons he believes organizations will be slow to tackle the
problem. Sure, some will make an earnest effort, but "even those people
are going to be overwhelmed and patch a lot of their systems but not all
of them," he adds.
Weakened White House
Among the weak SSL certificates at time of publication is this one
belonging to Whitehouse.gov. It's of little consequence, since the site
doesn't conduct secure transactions, but it does show the ubiquity of
the problem. The key is owned by content delivery provider Akamai
Technologies and is used by about 20,000 websites. Akamai is in the
process of replacing it.
Akamai has escaped relatively unscathed. All its keys involved in
sensitive transactions are generated using a highly customized Debian
derivative that didn't include the buggy random number generator. The
single key used by Whitehouse.gov and the other Akamai customers, which
was generated using a separate system running on standard Debian is the
only one affected, says Andy Ellis, Akamai's senior director of
"I can't imagine how painful this will be for people who are using large
data centers with hundreds of certificates," Ellis said.
The unwieldy cleanup effort is akin to the aftermath of a serious Flash
vulnerability found in December to be plaguing tens of thousands of
websites. Three months after a patch was released, the sites - many
carrying out banks financial and other sensitive transactions - remained
vulnerable because they had yet to remove and regenerate an estimated
500,000 buggy flash applets. Both the Debian and Flash vulnerabilities
are unusual, because applying the patch represents only the beginning of
the healing process.
The Debian bug was introduced in September 2006. It vastly reduces the
amount of entropy used when programs like the Apache webserver,
Sendmail, Exim and some implementations of Kerberos use OpenSSL to
perform basic cryptographic functions. As a result, attackers can crack
SSL keys, x.509 certificate keys, SSH keys, and digital signatures in
fewer than 33,000 guesses, rather than the seemingly-infinite number of
tries that would normally be required.
Tools available from Ubuntu and Metasploit author HD Moore are designed
to aid in the process of detecting weak keys, but Appelbaum, the
independent researcher, says certain conditions will prevent even
diligent searches from finding everything. For example, keys with
nonstandard sizes may not be flagged even though they're vulnerable.
"What that means is you have tools that may cover large swaths of the
key space, but they won't cover all of the key space," he says.
So if your organization hasn't begun a thorough audit of all the keys in
its portfolio, now is the time to get to it. Like an outbreak of lice at
the children's grade school, its an unpleasant task eradicating the
pests, but it's got to be done.
"This is a bit of a nightmare for anybody who used Debian" or programs
that relied on its OpenSSL library, says Vincent Danen, the security
team manager for Mandriva, a Linux distribution that was not affected by
the bug. "If you're running a Debian shop and you have 100 certificates,
depending on who you've got as a certificate authority, you could be
looking at big bucks to regenerate your keys and get them re-signed. It
could take months or even years for all the keys to get weeded out."
Attend Black Hat USA, August 2-7 in Las Vegas,
the world's premier technical event for ICT security experts.
Featuring 40 hands-on training courses and 80 Briefings
presentations with lots of new content and new tools.
Network with 4,000 delegates from 50 nations.
Visit product displays by 30 top sponsors in
a relaxed setting. http://www.blackhat.com