By Sarah D. Scalet
June 05, 2007
Roger Johnston knows about security vulnerabilities, and not only
because he works for the Los Alamos National Laboratory, which has
experienced more than its share of security problems of late (including
the loss of classified materials last autumn). As leader of the
laboratory's Vulnerability Assessment Team, a research group devoted to
improving physical security, Johnston is the guy who gets brought in to
find security problems, not only at his own agency, but also at other
agencies and at private companies. His team has been hired to conduct
vulnerability assessments at government agencies with such high security
stakes as the International Atomic Energy Agency, the Department of
State and the Department of Defense, as well as at private companies
that are developing or considering the use of high-tech security
Senior Editor Sarah D. Scalet recently spoke with Johnston about
strategies for running an effective vulnerability assessment and then
communicating the results without also putting your job on the line. To
help security leaders identify specific areas that need improvement,
Johnston also developed a quiz that identifies the 28 attributes of a
flawed security system. "We see the same things over and over," he says.
"These are the common unifying themes." Find out how you rate. (Note:
Johnston emphasized that his statements here are his own opinions and do
not necessarily reflect the official position of the Los Alamos National
Laboratory or the U.S. Department of Energy, its parent organization.)
CSO: You basically spend your days finding problems with things. Are
people afraid to cook for you?
ROGER JOHNSTON: Yeah, well, we always try to have an upbeat message.
There are often very simple fixes to problems. Say you're using a
tamper-indicating seal for cargo security. When you inspect the seal,
maybe you simply spend an extra second or two looking for a little
scratch in the upper right-hand corner to discover an attack.
CSO: So training is a key to that upbeat message?
JOHNSTON: Right. We're very strong believers in showing security
personnel a lot of vulnerability information. Often, low-level security
people aren't given the information they need to do a good job. If they
know what they're supposed to be looking for, instead of just turned
loose and told to report "anomalous incidents," they generally will do a
lot better. You really haven't spent a lot of extra money, and it
doesn't necessarily take a great deal of time.
CSO: When you're doing a vulnerability assessment, what's the best way
to get into the mind-set of the adversary?
JOHNSTON: That's the real trick. The problem with a lot of vulnerability
assessments is that they're done by very sincere security people who
have devoted their lives and careers to being good guys. They really
don't want security to have any problems. It's not a matter of
dishonesty; it's just human nature. Also, in many cases security
personnel come from military or police backgrounds. That kind of
training and discipline can be very useful, but those backgrounds don't
typically tend to attract people who are wildly creative.
You want to look around your organization and find people who are
outside-the-box thinkers. They don't have to be in the field of
security. You're looking for people who would normally be your worst
security nightmare--people who are loophole finders, smart alecks, kind
of skeptical. They're people who have to prove things for themselves and
aren't sure they buy everything they hear from authority.
CSO: So you're looking for people who've been in trouble for violating
some security policy?
JOHNSTON: I don't want to push it too far. If they're wanted in 35
states for felonies, maybe that's not exactly who you want looking at
your critical security vulnerabilities. It's more about finding the
people who won't automatically toe the party line. These are people in
your organization who are already thinking about how they could beat
your security. They're probably not going to do it, but that's just the
way they think. They may be graphic artist types; they may be the smart
aleck on the loading dock who's always questioning the boss.
CSO: There's more of that ethos in the information security culture than
in the physical security culture.
JOHNSTON: Absolutely. There's a huge cultural gap, of course, between IT
security and physical security, and that's much of the problem of
convergence, trying to bring the two together. I think IT is better off
in this regard. A lot of the people who work on computers automatically
think that way.
CSO: What's the risk of conducting a vulnerability assessment from the
point of a good guy?
JOHNSTON: When vulnerability assessments are done by good guys thinking
like good guys, number one, they let the existing security
infrastructure and hardware and strategies define the vulnerability
issues. For example, if there's a fence, they'll think about ways the
bad guys might get over the fence. But of course that's all backwards.
We need to think about what the bad guys want to accomplish and then
decide if we even need a fence. Number two, there's that tendency not to
want to try to find problems.
CSO: Not only are they possibly making themselves look bad if they find
a problem, they're also creating more work for themselves, right?
JOHNSTON: Absolutely. In many cases when the fix is very simple,
organizations are very reluctant to do it, because that is sometimes
thought of as saying, "We've been screwing up all these years." So you
don't want to go with people who have a history of doing a vulnerability
assessment and then telling you everything is swell. There are always
vulnerabilities, and they are always present in very large numbers. Any
vulnerabilities assessment that finds zero vulnerabilities is completely
CSO: When you actually do the assessment, are there warm-ups you can do
to get yourself in the mind-set of a bad guy, or are there ways you
should set up the room?
JOHNSTON: A lot of vulnerability assessment needs to be very similar to
classic brainstorming. A lot of the tools that are applied to creative
thinking in other fields can be applied directly to vulnerability
assessments. This is kind of a radical position. A lot of people in the
security business are not comfortable with this 1960s hippy,
touchy-feely, "let's all get together" approach.
CSO: I'm imagining a bunch of beanbag chairs.
JOHNSTON: Yeah. A lot of people would much rather have a rigorous,
quantitative approach, and I would claim that's largely a sham. I don't
think it's a mistake to use analytical tools like a security survey, but
we would like to combine those more closed-ended, straightforward tools
with creative thinking. The fact is that creativity has been studied
extensively over the last 50 years, and there's a lot of understanding
of how you create an environment where people come up with good ideas.
It's not quite the seat-of-the-pants, wacky kind of thing that it might
look like from the outside.
CSO: Should the CSO even be there?
JOHNSTON: You don't want the boss in the room, because it constrains
people. What you need are really nutty ideas, so we strongly encourage
thinking about attacks that involve Elvis impersonators and flying
monkeys and the use of space aliens. Early on, it's very important not
to editorialize. Later on, we're going to prioritize them and think
about the practicality of them. In many cases, we have people say,
"Well, if I had the space aliens come down with a ray beam, they could
do the following." Later on, it turns into a very viable attack, once we
get rid of the space aliens and the laser beams.
CSO: Does this take hours? Days? Weeks?
JOHNSTON: It depends. If you're looking at a very complex security
program, you may want to spend two or three weeks just kind of
freewheeling. But you don't just sit around and do ideas. You generate
nutty ideas, and then you go back to the program or the hardware and
play around a little bit to see if those nutty ideas might have some
merit. Then you get back together again, and you think of more nutty
ideas based on what you learned. We're very much in favor of hands-on
work, and not just thinking in abstractions. Toss the device around.
Chat up the security guards. Kick the fence. Play with the system and
try to understand how it behaves.
CSO: When the CSO tells his or her company about a vulnerability, we've
seen that there can be a kind of "shoot the messenger" effect. (See
"Don't Shoot the Messenger" from the August 2006 CSO.) What are ways
they can avoid that or at least mitigate the effect?
JOHNSTON: We try to encourage people think about a vulnerability not as
bad news. It's great news. When you find a vulnerability, you can do
something about it.
CSO: But you still have to take people down the path of, something
terrible could happen.
JOHNSTON: All our vulnerability assessment reports start out by pointing
to the good things. There are always good things. Sometimes
they're an accident, but by pointing them out, you get them recognized.
Also, at the very beginning we always point out that we're going to find
more vulnerabilities than they can possibly mitigate. We're going to
make more suggestions for changes than you can possibly implement.
That's OK. The bottom line is, vulnerability assessors are not here to
tell you what changes to make. We're here to point out what we think are
problems and what we think may be solutions. It's up to you to decide
what you do with the findings.
This binary thinking about security--that something is either secure or
not secure, or that we have to have all the vulnerabilities covered or
we're not doing our job--is really nonsense. Security is a continuum,
and there are always going to be vulnerabilities you can't do anything
about. That doesn't mean anybody is screwing up. That's just the way
CSO: In coming up with this laundry list of problems and possible
solutions, is there oftentimes an 80/20 thing at play, where you can
solve 80 percent of the problems with 20 percent of the solutions?
JOHNSTON: It does work that way. People say, "Gee, you're telling me I
need to make this one little change, and this attack and this attack and
this attack and this other attack basically go away?" It's really quite
surprising. Sometimes the vulnerabilities are extraordinarily complex,
and the solutions, while they may not be 100 percent perfect, are often
really painless. We don't always have the most realistic view--we work
for the government--about what's economically viable to implement.
Sometimes what we think is simple isn't really simple in the real world.
But that's OK too. Sometimes our suggestions get the end users thinking,
and then maybe they come up with their own solution.
CSO: You've brought a couple of industrial-organizational psychologists
onto your team. Why?
JOHNSTON: Industrial-organizational psychology has been applied across a
wide range of fields, but for some weird reason, not security. When we
first got these psychologists to work with us, they just couldn't
believe that no one had applied all these powerful tools in industrial
psychology towards security problems. Increasingly, we're using them to
understand the human factors associated with security. In the end,
security is really about how people interact with technology, how people
use and think about technology, and how the technology was designed to
enhance what people are already doing.
CSO: What kinds of things have the industrial-organizational
JOHNSTON: The main one early on was the recognition that the security
guard turnover problem is a huge problem. The numbers typically run
between 40 percent and 400 percent per year. McDonald's has a turnover
rate of about 35 to 40 percent, so McDonald's does a better job than
security of finding the right people and hanging on to them. There are
plenty of organizations that do very fine with turnover rates that don't
pay people very well and don't necessarily represent fabulous careers.
There are ways that IO psychologists have developed over the last couple
decades that help these companies, but the tools never have been applied
to security. The first things that our guys did was publish some papers
basically saying, "Hey, wake up, we don't need to do any new R&D, there
are all these tools already proven out there." They involve things like
understanding who you hire and creating a realistic picture in their
mind of what the job is like. If you simply do that, turnover rate
We're just beginning to look more specifically at how IO psychology
applies to vulnerability assessments. It's a totally open field. One
problem we want to look at is the tamper-indicating seals that are used
for cargo security. We know from experience that some people are really
good at finding seals that have been tampered with, and some people
aren't. But we don't know why. One of the things we want to do is study
the people who are good at it and try to understand what it is that
they're doing or what characteristics they have that make them good. One
of the studies we want to do, and we haven't found anybody to fund it,
is an eye-tracking study. We want to look at what seal inspectors are
looking at. You give them this little eyeglass thing, and it tells what
their eyes are looking at. It's used all the time to judge
advertisements for TV; advertisers stick audiences in front of the
proposed commercial to see if they're really looking at the product or
they're looking at the pretty girl in the background. We want to apply
this technology to understanding what the people who are effective at
finding seals that have been tampered with are looking at. Maybe we can
train people better, or maybe we can do a screening exercise to find the
people who are really good at it, for whatever reason.
CSO: I know you've done a lot of work around what to do once you
actually find a vulnerability. Can you tell me about the Vulnerability
Disclosure Index that you and your group have created?
JOHNSTON: One of the problems with finding a vulnerability is, exactly
who do you tell? We have found vulnerabilities that were specific to the
sponsor of the vulnerability assessment, and of course if they pay for
the work, they get the findings. No issue there. But we'll find things
that have more general applicability. Now the question is, what do you
do? A classic example is spoofing a global positioning system.
Everyone's focused on jamming GPS devices, but that's not an interesting
attack, because the GPS receiver knows it's not getting satellite
signals from space. Spoofing, however, turns out to be surprisingly
easy. You can feed fake coordinate information to a GPS receiver.
CSO: How could the bad guys use that to their advantage?
JOHNSTON: A lot of national networks, like for financial transactions,
get their critical time synchronization from the GPS satellite signals.
If someone fed the GPS fake information, the networks could crash within
milliseconds. It could potentially be very serious. There's some
recognition that jamming might be an issue, but in our view spoofing is
the far more serious issue and is not widely recognized. Now, do we
discuss this? Do we write papers about this problem? Or do we just keep
our mouths shut?
This kind of problem crops up all the time, but there are some fairly
straightforward, simple signs you're looking for. If there are a whole
lot of good guys who don't seem to be very sophisticated in
understanding the vulnerabilities, and there are only a small number of
bad guys, you probably ought to just publicize it to the world. If the
attack is pretty obvious--and I think GPS spoofing is--the bad guys are
going to figure it out anyway. So again, you probably ought to just tell
the whole world. On the other hand, if it's kind of a specialized
security device not being used by very many people, but a whole bunch of
potential bad guys might want to exploit it, then maybe you don't need
to be publicizing that vulnerability. Instead, you want to seek out the
specific end user and point out the potential problem. The Vulnerability
Disclosure Index is a sort of semiquantitative attempt to try to provide
some guidance as to whether you should disclose this vulnerability, how
publicly, and in how much detail you should go.
CSO: Vulnerability disclosure has been especially contentious in the
field of IT security. (See "The Chilling Effect" from the January 2007
CSO.) Does this Vulnerability Disclosure Index apply to IT
vulnerabilities as well?
JOHNSTON: It's really meant for physical security. IT lives in a very
different world. Let's say you're playing around on your home computer,
and you find a very serious software vulnerability. There's some
controversy, but most people agree you should do the following: You
should contact the software company and say, "I think there's a problem
here." You give them a chance to fix that. If after a while they're just
stonewalling and not doing anything, then maybe you go public. Once they
fix the problem, it's no big deal. Everybody who bought the product
typically does checks on whether there are upgrades.
Physical security is not like that. In many cases the physical security
systems are from a bunch of different vendors and may be put together by
a third-party vendor. Often there's no one company to go to complain
about a potential vulnerability. Moreover, the fix isn't just some
software download. The fix may require servicepeople going out and
changing parts, and it could be very expensive, very disruptive. Before
you get everybody all wound up about a physical security vulnerability,
you may want to think about, is it even going to be practical to fix it?
CSO: You've written that when the vulnerability assessment is chartered,
the sponsor owns the findings, but that that doesn't necessarily
"relieve the vulnerability assessors of their responsibility to warn
others of a clear and present danger." This might strike fear into the
hearts of CSOs who think they're going to hire someone to do a
vulnerability assessment and the contract will ensure that the findings
JOHNSTON: A typical example would be if a company is considering a
commercial security device. Let's say we do a vulnerability assessment
on that device and oh my gosh, if you poke it with a paperclip it will
quit working. And we know that commercial device is being used for a
wide variety of applications, including corporate security, U.S.
national security and nuclear safeguards. We believe we have some moral
responsibility to tell people there might be a problem. Most companies
we've done that for have had no problems and in some cases encourage us
to do exactly that.
Senior Editor Sarah D. Scalet can be reached at sscalet (at) cxo.com. If
you would like to see a copy of a paper Roger Johnston wrote about
vulnerability disclosure, contact him at rogerj (at) lanl.gov.
Copyright 2002-2007 CXO Media Inc. All rights reserved.
Attend Black Hat USA, July 28-August 2 in Las Vegas,
the world's premier technical event for ICT security
experts. Featuring 30 hands-on training courses and
90 Briefings presentations with lots of new content
and new tools. Network with 4,000 delegates from
70 nations. Visit product displays by 30 top
sponsors in a relaxed setting. Rates increase on
June 1 so register today. http://www.blackhat.com