By Kevin Poulsen
Apr, 12, 2006
A computer failure that hobbled border-screening systems at airports
across the country last August occurred after Homeland Security
officials deliberately held back a security patch that would have
protected the sensitive computers from a virus then sweeping the
internet, according to documents obtained by Wired News.
The documents raise new questions about the $400 million US-VISIT
program, a 2-year-old system aimed at securing the border from
terrorists by gathering biometric information from visiting foreign
nationals and comparing it against government watch lists.
The Aug. 18 computer failure led to long lines at international
airports in Los Angeles, San Francisco, Miami and elsewhere, while
U.S. Customs and Border Protection, or CBP, officials processed
foreign visitors by hand, or in some cases used backup computers,
according to contemporaneous press reports.
Publicly, officials initially attributed the failure to a virus, but
later reversed themselves and claimed the incident was a routine
But two CBP reports obtained under the Freedom of Information Act show
that the virulent Zotob internet worm infiltrated agency computers the
day of the outage, prompting a hurried effort to patch hundreds of
Windows-based US-VISIT workstations installed at nearly 300 airports,
seaports and land border crossings around the country.
"When the virus problems appeared on (CBP) workstations Thursday
evening, the decision was made to push the patch, immediately, to the
... US-VISIT workstations. Most workstations had received the patch by
midnight and US-VISIT was back in operation at all locations," reads a
CBP summary of the incident.
The Department of Homeland Security's US-VISIT program office declined
to comment on the documents.
Former White House cybersecurity adviser Howard Schmidt says the
incident is typical of a large agency struggling with complex networks
and evolving threats. "We've got catching-up to do in all areas,
particularly areas having to do with national security and public
safety," says Schmidt. "I hope you and I, 10 years from now, look back
and say, 'Wow, I'm glad we survived that.'"
Launched in January 2004, and expanded since then, US-VISIT is a
hodgepodge of older databases maintained by various government
agencies, tied to a national CBP-run network of Windows 2000
Professional workstations installed at U.S. points of entry. The
system has processed more than 52 million visitors, and allowed border
officials to intercept more than 1,000 wanted criminals and
immigration violators, according to DHS. Some US-VISIT locations are
now testing gear to read new RFID-equipped passports.
While the idea of US-VISIT is universally lauded within government,
the program's implementation has faced a steady barrage of criticism
from congressional auditors concerned over management issues and
cybersecurity problems. Last December, the DHS inspector general
reported that the program might be vulnerable to hackers.
The nearly 6-year-old Windows 2000 operating system was a particularly
burdensome choice on Aug. 9, when Microsoft announced a vulnerability
in the software's plug-and-play feature that allowed attackers to take
complete control of a computer over a network. In an unusually quick
mating of vulnerability with attack, it took only four days for a
virus writer to launch an internet worm, called Zotob, that spread
through the security hole.
Operating somewhat more slowly, it took CBP officials until Aug. 16 --
a full week after Microsoft released a patch for the hole -- to start
pushing the fix to CBP's Windows 2000 computers. But because of the
array of peripherals hanging off of the US-VISIT workstations --
fingerprint readers, digital cameras and passport scanners -- they
held off longer on fixing those machines, for fear that the patch
itself might cause a disruption.
"The push was not made to the US-VISIT workstations during the initial
install due to concerns with the possible impact of the patch on the
unique workstation configurations," reads one of the CBP reports.
Officials -- not unreasonably, say security experts -- wanted to test
the patch before installing it. But as a consequence, hundreds of
computers networked to sensitive law enforcement and intelligence
databases were left with a known vulnerability -- a security hole
rated "critical" by Microsoft because it allows attackers to take
control of a machine remotely.
It wasn't until Zotob made itself at home on the CBP network Aug. 18
that the agency launched a fevered effort to secure the US-VISIT
terminals, which sit on local area networks that are in turn connected
to CBP's wide area network.
Even as officials raced to install the patches, the US-VISIT computers
were failing at major U.S. entry points around the country, including
airports in Dallas, Houston, Los Angeles, Miami, New York, San
Francisco and Laredo, Texas, according to press reports at the time.
A DHS spokesman told the Associated Press the next day that a virus
caused the outages. But in December, a different DHS spokesman told
CNET News.com that there was no evidence that a virus was responsible,
and that it was merely one of the routine "computer glitches" one
expects in any complex system.
The newly released documents call that claim into question.
The government did not part with the pages lightly. After an initial
FOIA request was rebuffed, Wired News filed a federal lawsuit,
represented by Megan Adams, a law student at the Stanford Law School
Cyberlaw Clinic. Only then did CBP release six pages of heavily
redacted documents, including one page that is completely blacked out.
(The lawsuit is ongoing.)
The redactions leave it unclear whether the virus itself shuttered the
system, or whether the patch, or the process of installing it,
contributed to the outage. For example, one sentence reads, "Initial
reports confirmed that the US-VISIT workstations were (redacted)
impacted" by the virus. The blacked-out portion might as easily read
"severely" as "not."
Other redactions appear less tactical: A public Microsoft security
bulletin is included, but with the bulletin number (MS05-039) blacked
Perhaps most significantly, the pages do not reveal how the Zotob
virus made its way onto the private CBP network -- an ominous
migration that demonstrates that computers used in protecting U.S.
borders are accessible, via some path, from the public internet, and
could be subject to tampering.
"That machine was reachable from some network, that was connected to
some other network, that was connected to the internet," says Tim
Mullen, a Windows security expert and CIO of security firm AnchorIS.
"There was some series of connections that manifested itself in those
machines getting compromised."
A September report by the DHS inspector general found computer
security at CBP wanting. In a scan of 368 devices on CBP networks,
investigators identified 906 security vulnerabilities rated as medium
or high risk. They criticized CBP for failing to implement a
comprehensive security testing program, among other issues.
"Our vulnerability assessments identified security concerns resulting
from inadequate password controls, missing critical patches,
vulnerable network devices and weaknesses in configuration
management," the report concludes. "These security concerns provide
increased potential for unauthorized access to CBP resources and
In a second report in December focused on US-VISIT, the inspector
general concluded that the mainframe databases at the backend of the
system were generally secure. But investigators found vulnerabilities
elsewhere in the system's architecture that "could compromise the
confidentiality, integrity and availability of sensitive US-VISIT
In particular, the report found system vulnerabilities at the U.S.
points of entry where the US-VISIT workstations are operating. It
blames the weaknesses on poor communications between administrators in
the field and those at US-VISIT's Virginia data center. In February,
the Government Accountability Office -- Congress' investigative arm --
followed up with its own investigation of the program, faulting
US-VISIT for not having an overall security plan.
Besides management issues, the system has been criticized as a
slapdash effort at stringing older technology together into a modern
security screening system. "Biometrics have been introduced into an
antiquated computer environment," the 9/11 Commission noted of the
program. "Replacement of these systems and improved biometric systems
will be required."
Schmidt agrees, though he says the problem is hardly limited to
US-VISIT. "We have to start moving at industry speed, not government
speed, when it comes to the deployment of new technologies," says
Schmidt. Instead of running Windows 2000, "I'd be racing to run the
beta of the next generation of operating system ... and not worry
about legacy stuff that we know isn't going to be supported too much
longer and has had issues."
Prior to infecting CBP, the Zotob virus reportedly caused disruptions
at The New York Times, ABC and CNN's headquarters in Atlanta, as well
as some offices on Capitol Hill. In late August, the FBI announced the
arrest of two men in connection with the worm: 18-year-old Farid
"Diabl0" Essebar in Morroco, and a 21-year-old Turkish man named
Atilla Ekici, known online as "Coder."
LayerOne 2006 : Pasadena Hilton : Pasadena, CA
Infomation Security and Technology Conference