By Kim Zetter
Aug. 01, 2005
LAS VEGAS -- Security researcher Mike Lynn roiled the Black Hat
conference Wednesday when he resigned from his job at Internet
Security Systems to deliver a talk about a serious vulnerability in
Cisco IOS, the operating system powering its routers, defying efforts
by the router manufacturer and his former employer to block the
In the aftermath, Lynn reached a legal settlement with Cisco and ISS
in which he agreed to erase his research material on the
vulnerability, to keep secret the details of the attack, and to
refrain from distributing copies of his presentation, among other
Now facing an FBI investigation -- and sudden celebrity status in the
tech world -- Lynn discusses the events leading up to this week's
disclosure, and what he thinks it means for the security of the
internet in an exclusive interview with Wired News.
Wired News: Can you tell me how all of this started? You were asked by
your employer, ISS, to reverse-engineer the Cisco operating system,
Michael Lynn: I was very specifically told.... It was January 26th and
Cisco had just announced a totally different vulnerability than the
one I demonstrated. They'd announced a vulnerability for something
called "Multiple Crafted IPv6 Packets Cause Router Reload" (as they
worded it in their patch message). But that's a very vague term. It
just says, "Hey, something is wrong in IP6 with the router reload" ...
but it didn't say you could be in control of it.
ISS wanted to get protection in their products (against this problem)
so that their customers wouldn't be as affected by it. So they called
up Cisco to try to get some more details for it ... and Cisco wouldn't
give (the information) to them. So (ISS managers) came to me and said,
"Can you reverse-engineer ... can you disassemble IOS ... to find out
what their vulnerability is?"
WN: So this was a different vulnerability from the one you
demonstrated at the conference this week?
Lynn: Yes, but (Cisco) had (also) found the vulnerability that I
demonstrated on stage about two weeks before I (found it).
WN: Then what happened?
Lynn: So on January 27th, ISS comes out with their response to this
vulnerability -- the advice to their customers based on my
analysis.... I stayed up all night basically (to research it).
I realized in looking at this (that the program) is actually way worse
than Cisco said.... So (our guy) calls up ... Cisco and says, "OK, we
aren't 100 percent sure that we found the same bug that you're talking
about, but it's important we find out because the one we found has
much, much greater impact. You said there's (the possibility) of a
denial-of-service attack. But the one we found is fully exploitable."
Cisco said, "You guys are lying. It is impossible to execute shell
code on Cisco IOS." At that point (ISS) management was annoyed....
They were like, "Mike, your new research project is Cisco IOS. Go find
out how to exploit bugs on Cisco IOS so we can prove these people
WN: In your speech you said you worked on the reverse engineering with
cooperation from Cisco.
Lynn: We did, in fact. The cooperation came later. They didn't start
that way, and they were not happy to begin with.... They didn't
cooperate in the actual reverse engineering itself. They cooperated in
the research effort, I would say, in finding vulnerabilities and
WN: They didn't stop you.
Lynn: They didn't stop us, and at this point there was some
back-and-forth communication. (Lynn spent the next month researching
WN: After you came to them with the serious flaw and said, "This is
the bug we found...."
Lynn: They said, "We don't believe you." And (ISS managers) said ...
"come down to Atlanta and we'll show you." And that's never happened,
by the way, at ISS. They've never brought somebody, let alone a
competitor, into the office just to show them (something).... Mike
Caudill, (Cisco's) customer advocate, came out. And they also sent out
an engineer ... who described himself as an IOS architect.... I was
told he helped design parts of the source code.... And his jaw hit the
ground. He was very impressed, he was just (saying), "Wow, that's
cool." That was June 14th.
WN: Cisco saw your Black Hat presentation long before they decided to
pull it. When did they see it?
Lynn: Probably June 14th, the day that they came out (to Atlanta). We
told them about the vulnerabilities well before (that).
WN: So at what point did they get nervous about the talk?
Lynn: When they saw the listing of the presentation on the Black Hat
site is when they actually called us back and said, "Wait, you guys
were serious?" And we said, "Yes, we were serious." Incidentally, it
was ISS who submitted (the talk) for Black Hat. I was told (by ISS),
"Hey, you want to go to Black Hat? We'd like you to do it."
WN: So ISS knew the seriousness of the bug.
Lynn: Yes, they did. In fact, at one point ... they apparently didn't
get it, and they actually wanted to distribute the full working
exploit very widely inside the company.... I was told ... "Give this
to all the sales engineers and to all the pen testers."
WN: Why would they want you to do that?
Lynn: Well, because it bruises Cisco, remember? Mind you, this was
something that Cisco hadn't gone public with yet and that's not useful
to pen testers because what do they advise their customers to do (to
protect themselves if no information about the vulnerability has been
I told them, "You do realize if you do that, it's going to leak?" And
(one of the ISS guys) says, "That's Cisco's problem." And then
(another ISS guy) turns to me and says that they need to understand
this could be their Witty worm. I was like, Whoa, what meeting did I
(The Witty worm was a particularly aggressive and destructive code
released by someone last year that targeted computer systems running a
security program made by Internet Security Systems and even more
specifically targeted military bases using the software. It infected
more than 12,000 servers and computer systems in about an hour.
Because of the worm's speed in spreading and its creators' apparent
knowledge of who ISS' customers were, some security experts speculated
that someone working for or connected to ISS might have been
responsible for writing and releasing it.)
At that point, I told them all no, and they fought it and I resigned
right there on the spot. And this was about a month ago.
I thought they were handling this in a non-ethical manner. Because it
was just way too fast and loose with who can see this.... I mean, I
don't even want people to see it now. (ISS talked him out of the
resignation by agreeing to give him control over who could see or have
So we start moving forward with the talk and we're working with Cisco,
and Cisco seems OK with it.
WN: They had already released information about what you found before
your speech, right?
Lynn: Yes, and the fix. The fix was about six months before the
WN: So they already knew how serious the problem was.
Lynn: If they didn't know, they should have.
WN: But they didn't indicate to their customers how serious it was.
Lynn: No, they did not.
WN: And Cisco saw your Black Hat presentation long before they decided
to pull it, right?
Lynn: Probably June 14, the day that they came out (to Atlanta).
(Then) it was two weeks ago, I was first told that Cisco might want to
come onto (the) stage with me and say a couple words. And I said,
provided the words aren't something to the effect that "he's a liar,"
I'm OK with it.... It didn't really matter. It lent credence to my
talk. And it's good because I felt my talk really needed to be taken
(However, the plan changed even more and Lynn was told to remove any
mention of reverse engineering from his talk or cancel the
presentation. If he did neither, he would be fired.)
Mind you this is a complete reversal. Like a week or so prior, the
night of the close of the fiscal quarter, and they were all
celebrating that they hit the numbers, the CEO invited me out for a
beer, and he just couldn't say enough awesome things about this talk.
WN: Was Cisco threatening them?
Lynn: I asked point-blank, "Are you being threatened by Cisco?" They
said no.... To be perfectly honest, I don't think there was any legal
threat. I think that it was more of a "scratch our back and we'll
(Cisco asked him to wait a year until it could release a new version
of its operating system. When he didn't back down, Cisco threatened a
lawsuit against Lynn and Black Hat. Then with Black Hat's cooperation,
Cisco arranged to tear out pages with images of Lynn's slides from the
WN: You met with the feds after your talk, and someone gave you a
challenge coin (a special coin created for members of the military to
commemorate challenging missions)?
Lynn: Yes, they did, actually. And I didn't know what it was, so I
didn't thank him properly.... This was a really funny story. (Right
after my talk, this) guy walks up with a very, very impressive badge
... and says, "I need to speak with you. Now."
WN: What agency was it?
Lynn: Air Force (Office of Special Investigations). NSA, is what I'm
told, but he wouldn't show me his credentials. There were a lot of
flashy badges around from lots of three-letter agencies. So they take
me to a maintenance area and I'm surrounded by people ... and one of
them says (to another guy), "You've got the van ready?" I'm going, "Oh
my god." And they go, "Just kidding!... Oh, man, you rock! We can't
thank you enough." And I'm just sitting there, like still pale white.
They all shook my hand.
I get the feeling that they were in the audience because they were
told that there was a good chance that I was about to do something
that would cause a serious problem. And when they realized that I was
actually there to pretty much clue them in on ... the storm that's
coming ... they just couldn't say enough nice things about me....
Also, US-CERT (Computer Emergency Response Team) asked me if I would
come up to D.C. in a week or two and help them formulate the nation's
strategy for cybersecurity.
WN: So this new version of the operating system that they're coming
out with, that's in beta testing.
Lynn: It's actually a better architecture ... but it will be less
secure.... That's why I felt it was important to make the point now
rather than sweep it under the rug. I think it's something that we can
The problem now ... is that if you want to attack something ... you're
going to (have to) hack one machine (at a time) and take control of
the part of the network (it's on). If you had (the exploit) up running
against the new version that's in beta now, you can take everything.
That's the difference between something you can make a worm out of and
something you can't make a worm out of.
(Right now) nobody patches Cisco routers because there's been this
culture (that) there's just never anything that can go wrong (with
them). So, unless there's some really critical thing that's making it
crash, people don't install the patches.... We have to change the
public perception about patching now, and that cause is not best
served by pretending that there's not a problem and saying maybe you
can talk about this next year.... The time to talk about this is
before the critical problem comes around.
WN: Cisco has said this is not a critical flaw that you found.
Lynn: I would agree with them in part and disagree with them. In a way
I would say, yes, it's actually not all that exceptional in that all
it proved is it's just like any other computer -- they're all
hackable. Because in any complicated system, people make mistakes.
It's our very nature.
But in the sense that the potential impact of something like a router
worm (attacking the routers) is no big deal, I would strongly
disagree. Unlike most other vulnerabilities or exploits, when you ...
take control of another machine, it's very difficult, if at all
possible, for you to ... destroy the hardware.... But on a router?
This is (a scenario in which) the network is down, and it's down in a
way that it's not getting up again. How do you ship the patch when the
network won't (be up so you can distribute it)? Are you going to mail
out a CD? But there's no CD drive.
The real point is there's a ticking clock but we still have plenty of
time. I wanted people to be afraid a little bit ... because I needed
people to act. But at the same time, now that I think they already
are, I will say it's not as bad as you probably think it is. Not yet
... because the version that makes this an unstoppable critical
problem is not out yet.
Sept 16-18th, 2005
San Diego, California