By Kim Zetter
Nov. 07, 2005
On Wednesday, Cisco Systems released a patch for what has become known
as the Black Hat Bug: a serious vulnerability in the operating system
running Cisco routers, which drive traffic through much of the
internet and control critical infrastructure systems.
Cisco's move closes the book on a controversy that began last July,
when Mike Lynn, a computer security researcher speaking at the Black
Hat security conference in Las Vegas, demonstrated that an attacker
could use the bug to crash Cisco routers or control them remotely.
Before Lynn's talk concluded, the dark conference room was already lit
with the glow of cell phones from audience members urging their IT
departments to immediately patch their Cisco routers.
Lynn was lauded by much of the security community for disclosing the
problem. But for his troubles, he and Black Hat organizers were
slapped with legal injunctions. Lynn had been asked by his employer,
Internet Security Systems, to reverse-engineer the Cisco router to
find the flaw, and both Cisco and ISS initially sanctioned his Black
Hat presentation. But two days before the talk, Cisco demanded that
slides of the presentation be removed from the conference book and
CD-ROM. And after the talk, the FBI began investigating Lynn for
allegedly stealing trade secrets.
The legal wrangling finally ended this week, and the FBI case against
Lynn has closed. Lynn spoke with Wired News in July to tell his side
of the story. Now Black Hat founder Jeff Moss talks about what
happened from his perspective and why companies continue to repeat the
mistakes of their predecessors in trying to suppress the full
disclosure of security bugs and punish security researchers.
Wired News: Describe how events unfolded at Black Hat.
Jeff Moss: We realized something bad was happening on ... Monday
morning (July 25). One of the Cisco representatives, Mike Caudill,
came by and said, "Hey, can I see the printed material (for the
conference)?" I said, "Well, we don't give our books out until Tuesday
at 4 p.m." (before the conference opens). "I'll let you look, but
we'll need the book back."
So he flips to Mike Lynn's presentation and basically says "Holy crap!
This isn't supposed to be in here. ISS told us only an abstract was
going to be printed in the book." I said, "How can we accept a speaker
with only an abstract? Of course there's going to be slides." Now it's
about 20 hours until we started handing out the bags with the books
and the CDs in it, and Cisco gets on the phone to their legal
department and gets everybody all spun up....
(Cisco claims that Lynn is revealing proprietary source code in some
of the slides and wants them removed. After Moss agrees, Cisco's
people spend hours ripping out Lynn's presentation from thousands of
conference books and reburning CD-ROMs.)
If Cisco is saying there's proprietary Cisco source code in there,
it's hard for me to evaluate that (just hours before the show). If
it's true and it is really proprietary and really would be breaking
the law ... I would want to remove it. Mike Lynn said don't worry
about it. If they want to remove it, remove it. The printed materials
in the book had more details than what Mike had on his PowerPoint
slides. He was thinking that with those details removed, he'd be able
to give his talk, because he wouldn't be revealing any of the stuff
that Cisco was concerned about. And then it became clear that it
really wasn't specifically that source code, it was pretty much the
whole talk in general that Cisco was really nervous about.
WN: But they agreed that he would speak anyway, right?
Moss: (By) Tuesday around 2 p.m., Cisco had pulled all of the material
out of the books. The (revised) CDs were starting to show up, and it
looked like everything was fine. Cisco was happy, ISS was happy, and
it looked like we dodged that bullet.
As soon as the show was over, and we're cleaning up the show and
everything looks like it's done, all of a sudden FBI agents call me on
the phone and want to talk to me. It turns out that while Black Hat
and Mike Lynn were negotiating with Cisco and ISS, somebody at ISS in
Atlanta calls the local FBI field office in Atlanta and claims theft
of trade secrets. So while we're negotiating in good faith and trying
to resolve this, behind the scenes ISS has fired up the FBI on Mike
WN: Debates about full disclosure have been going on for years, and a
number of companies have created firestorms from trying to suppress
information about flaws or punishing researchers, such as Dmitri
Sklyarov, who got into trouble with Adobe. Why haven't companies
learned the lessons about trying to suppress information?
Moss: There must be something that's fundamental in human nature. Or
people are coming into the business too quickly and don't have any
sense of history. It doesn't portray a positive image that these are
talented professionals pursuing security research, and it doesn't do
any of us service.
WN: You've said that you felt Mike Lynn followed all of the proper
procedures that a researcher should follow for responsible disclosure
of vulnerabilities. And yet Cisco and his own company turned on him.
Moss: It's disturbing because you can play in your mind how this can
happen to any person working for any company. And if that starts
happening, it's just going to be a big stifling of innovation, and
it's going to drive researchers underground. Or they're just going to
only post on full-disclosure lists under fake handles.
WN: Some companies purchase vulnerability information about their
products from independent researchers and have them sign
non-disclosure agreements preventing them from telling anyone outside
the company about the flaws. What do you make of bartering crucial
information like that? I'm reminded of the federal agents who thanked
Lynn after his Black Hat presentation for giving them information
about their systems that Cisco didn't give them.
Moss: Yes, that was what was really frustrating. If Cisco is not even
telling the feds, then where does the greater good end and the profits
Mike Lynn, under the full-disclosure model that I subscribe to,
informed Cisco, and Cisco had plenty of time (before his presentation)
and released the patch.... Free research was done on Cisco's products.
It was a third party that invested time and money, and Cisco got a
benefit out of it. Well, everybody got a benefit out of it because it
made a better product and they fixed the problem in its current form.
And all everybody (else) gets out of it is a lot of misery and legal
bills. In my ideal world the vendor, Cisco, would be thanking Mike for
improving their product and apologizing to the community for not
finding the problem themselves.
WN: There has long been debate in the security community about making
companies legally responsible for releasing products with security
flaws. Should software companies be held responsible for failing to
disclose or act on information they discover about vulnerabilities in
products after releasing them?
Moss: I'm opposed to creating more laws. We have so many of them, and
they're so poorly enforced. But I think what we need is some sort of
guidance ... not necessarily a law forcing the companies to disclose a
bug, but ... some sort of protection for the bug finder. Is (bug
research and disclosure) considered protected speech, sort of like the
First Amendment? (Should there be) an exception under Digital
Millennium Copyright Act for reverse-engineering for security
purposes? It would be really nice to have some kind of uniformity. (So
that) people know, if you're doing security research in the United
States, this is how the game is played legally. There's not that kind
of clarity yet. And nobody wants to be the DMCA test case.
WN: Researchers often hold onto really big disclosures so they can
present them at conferences and make a splash. Should conferences
serve this function for revealing information like that?
Moss: I think the function of conferences is a very important one.
Researchers want to get a chance to be face to face with their peers
and share information and then to show off and push other people. It
advances the state of the art a bit.
I was asked by somebody in some three-letter (government) agency if I
planned to change anything about the show (after the problems this
year). Because they were concerned that if I had to neuter the content
or had to fundamentally change the way the show ran to try to avoid
these problems in the future, it would impact the quality of the
content. And they didn't want that to happen. They viewed the content
as valuable, and they were frightened that the Cisco-ISS deal would
have somehow affected what researchers do. I said no, that I can't see
changing anything. I think what we offer the public is valuable. I
think people in the government realize it's valuable, otherwise the
show wouldn't be so successful.
One of my concerns is that if you start punishing these researchers or
publicly threaten them with lawsuits, they'll just go underground, and
that really then doesn't offer the company any chance to communicate
with them or learn from them. Why risk getting sued by telling a
company about a bug?
Some researchers now just think that it's too much effort. They have
to play politician now (with the companies) when all they want to do
is play researcher.... There are some vulnerability-assessment tools
that have come out ... that (uncover) five or six vulnerabilities (in
software) that have never been announced. The (product) vendors don't
know about them. The people who write the tools are just busy writing
them, and they don't want to spend time holding the hand of all these
manufacturers. That's kind of interesting, because the first chance
that these vendors have of knowing there's a problem with their
product is when somebody calls them up and says, "Hey, I just
downloaded this tool and found five problems (in your product)."
WN: What benefits have come from the Ciscogate incident?
Moss: There were so many people sitting in that session who
immediately picked up the phone to call their IT departments and told
them to immediately patch all of their gear right now. That was kind
of funny because nobody ever messes with their Cisco gear. It sort of
works and nobody ever touches it. In one fell swoop, it forced
everybody to update their gear and not only fixed the Mike Lynn (bug),
but it fixed all of the previous Cisco bugs that nobody had bothered
to patch. So by Mike demonstrating (the problem), I think it made
everyone wake up ... and realize, hey, we've got to treat routers just
like we treat computers, and we've got to start patching and staying
on top of these patches.
Earn your Master's degree in Information Security ONLINE
Study IA management practices and the latest infosec issues.
Norwich University is an NSA Center of Excellence.