By Ellen Messmer and Phil Hochmuth
Heavy fallout continues on several fronts from a security researcher's
recent disclosure that unpatched Cisco routers can be subverted by
buffer-overflow attacks and shell-code exploits.
Among the developments last week: Cisco continually revised its
security bulletin, adding details as to how versions of unpatched IOS
software could be undermined by a "specifically crafted IPv6 packet."
Sources at Cisco say testing will continue indefinitely and could
include findings related to more than simply IPv6-related exploits.
The researcher who touched off the uproar, Michael Lynn, says he is
now the subject of inquiries by FBI agents, and he continues to defend
the propriety of his actions.
The episode rekindled debate about "responsible disclosure," the
notion that information about major security problems should be made
public in a way that brings minimal risk to customers.
According to Lynn and other experts, what Lynn described and
demonstrated at the Black Hat Conference on July 27 could potentially
lead to manipulation of Cisco router tables, denial-of-service attacks
and access to confidential data.
Through a security advisory, Cisco has indicated that the way some
unpatched IOS routers handle IPv6, which has seen little adoption in
North America outside of research labs, is a conduit for the type of
buffer-overflow exploit revealed by Lynn. But last week, a Cisco
spokesman acknowledged the exploit may be possible in other ways.
"There's ongoing information gathering and more testing," says Cisco
spokesman John Noh.
Cisco last week also released a new patch for Cisco IOS-XR, its new
carrier-focused router operating system, which was introduced last
year for its CRS-1 Internet core router, and ported to the 12000
series of carrier routers this year.
Experts and users say the hole in IOS appears not to be an immediate
concern based on what is public knowledge at the moment, since patches
are available. But what concerns some is that Lynn's exploit
techniques take router hacking to a new level, which eventually could
have security implications for Cisco customers.
"Strategically, this is a very serious issue for Cisco," says David
Lawson, vice president and director of global security practice at
Greenwich Technology Partners, a New York integration and consulting
firm that specializes in Cisco technology. "It proves something we've
been saying in the security field for a long time, that a router is
Many IOS exploits in the past would simply cause a router to crash or
reload itself, he adds.
"The big key to what [Lynn] did was to demonstrate a way to fool [the
router] into thinking it was already crashing, so that it didn't
initiate the shutdown sequence. If you can do that, that opens up the
ability to open up other exploits. Now you can actually get code
running that does god-only-knows what."
As for the question of responsible disclosure and whether Lynn
represented that ideal or not, opinions continue to differ.
"I personally wouldn't have done it the way he did it," says Justin
Bingham, CTO at security vendor Intrusic, referring to Lynn's action
in defying Cisco and Internet Security Systems (ISS) - his employer
until he quit just hours before giving his demonstration. "I like my
career being a security researcher and a lot of that is based on trust
with your customers and other companies."
Lynn, who has acknowledged breaking non-disclosure agreements in
speaking out about the router exploit, says he took the step out of
concern that withholding the knowledge would help would-be attackers
and even posed a national security concern.
"The vulnerability which I demonstrated-but didn't give any
information about-was properly disclosed to Cisco months in advance,"
Lynn says. "They had patches publicly available for months before I
went on stage.
"That said, the disclosure debate is one that needs to happen. The
idea of full disclosure is just about as dangerous as no disclosure at
all. As with most things, we have to find the proper balance."
While Lynn has settled one lawsuit with Cisco and ISS, agreeing not to
disclose anything he knows about the exploit, his problems don't seem
to be over. The FBI is investigating him and interviewing friends and
roommates, he says.
ISS, which declined to discuss the Lynn matter last week, has sought
to stop the spread of the electronic version of the presentation
slides that Lynn showed at Black Hat-many of which are labeled with
the ISS logo-by threatening legal action against Web sites posting
ISS has benefited from its research by including preemptive
protections for the vulnerabilities in its Proventia IPS product line
and Internet Scanner products. ISS had been planning to make a big
splash at Black Hat by unveiling the Cisco router flaw, but backed
down when Cisco balked. But Lynn, after quitting his job at ISS, spoke
Customers want more info.
Cisco customers say they would like to know about these types of
security problems as soon as possible.
"I'd like to be the first one to find out," says Bob Lescaleet, MIS
department manager at Pace Suburban Bus Service, a government agency
in Arlington Heights, Ill., serving a six-county region. "I'm not sure
Cisco should have kept this quiet as long as they have."
John Monaghan, vice president of IT for Marnell Corrao Associates, a
Las Vegas construction and architectural firm that uses Cisco routers
and firewalls in its corporate and field offices, says he was troubled
that Cisco was working with ISS on how to present the shell-code
exploit at a hacker conference, but not telling customers about the
"We are concerned that a vulnerability has existed, and that Cisco
didn't come clean and let us know about it," Monaghan says. "As far as
getting information from Cisco, it's more of a pull from our end than
a push from their end. You had to dig through an awful lot of rhetoric
to find out that this vulnerability only has to do with IPv6."
"As a user, you worry if there's stuff out there already in the wild,"
says Dennis Schwind, network specialist at Miami University in Oxford,
Ohio. "Cisco is not telling us anything about" the shell-code exploit,
he says. "You're just left saying, I sure as hell hope this isn't big.
That's really what you're left [with], because there isn't any real
detail on what the real impact would be if this is exploited other
than the 'execution of arbitrary code,'" he says, referring to
language used in Cisco's security notice issued last week.
Microsoft weighs in
Microsoft last week offered its view on responsible disclosure, saying
it entails seeking to ensure there's a fix in place before publicly
identifying a flaw-but that there should a time frame for this, says
Stephen Toulouse, Microsoft's security program manager in the
Microsoft security response center.
In general, Microsoft supports the Guidelines for Security
Vulnerability Reporting and Response published under the aegis of the
Organization for Internet Safety.
These guidelines, while declaring there's "no single universally
appropriate time frame for investigating and remedying security
vulnerabilities," does state that 30 days is a "good starting point."
The guidelines also suggest a 30-day "grace period" during which the
remedy and information about the security problem is shared only with
people and organizations "that play a critical role in advancing the
security of users, critical infrastructures and the Internet."
However, Toulouse says if a security vulnerability is highly critical,
he would consider releasing information within a day.
Symantec, which has IPS products but doesn't do the type of security
research ISS does, didn't have the advance knowledge about the exploit
that ISS did, says Alfred Huger, senior director of engineering at
Symantec Security Response. Nonetheless, he noted that sometimes
researchers do share information about exploits across vendor
boundaries, usually based on personal relationships.
Huger says Symantec would probably have treated the situation
differently than ISS and Cisco did based on its own corporate
guidelines for responsible disclosure, which give an IT vendor 30 days
to correct an identified problem before going public.
McAfee President Gene Hodges said his company's policy is "to share as
much information as you need to share and nothing more." The Cisco
router flaw is "a very important vulnerability, probably one that's
had the biggest impact of anything we've seen all year."
Among the questions surrounding the Cisco router exploit is whether a
researcher's attempt to use reverse engineering and disassemble code
to discover flaws is illegal - a charge raised against Lynn by Cisco
and ISS in legal filings.
"In the anti-virus business, that's exactly what we do," Hodges says.
"You put it in the de-compiler and try to figure out how it operates."
Mark Rasch, chief security counsel at security firm Solutionary in
Omaha, Neb., says, "Reverse engineering is not clearly illegal."
Lynn maintains that he was simply following orders from his
"It seems to me there is a license agreement dispute over that now,
but the license was with ISS, not me," Lynn says.
Sept 16-18th, 2005
San Diego, California