By Kevin Poulsen
Nov, 02, 2006
A Morocco-born computer virus that crashed the Department of Homeland
Security's US-VISIT border screening system last year first passed
though the backbone network of the Immigrations and Customs Enforcement
bureau, according to newly released documents on the incident.
The documents were released by court order, following a yearlong battle
by Wired News to obtain the pages under the Freedom of Information Act.
They provide the first official acknowledgement that DHS erred by
deliberately leaving more than 1,300 sensitive US-VISIT workstations
vulnerable to attack, even as it mounted an all-out effort to patch
routine desktop computers against the virulent Zotob worm.
US-VISIT is a hodgepodge of older databases maintained by various
government agencies, tied to a national network of workstations with
biometric readers installed at airports and other U.S. points of entry.
The $400 million program was launched in January 2004 in an effort to
secure the border from terrorists by thoroughly screening visiting
foreign nationals against scores of government watch lists.
While the idea of US-VISIT is universally lauded within government, the
program's implementation has faced a steady barrage of criticism from
congressional auditors concerned over management issues and
cybersecurity problems. When Zotob began to spread last year, DHS'
inspector general had just finished a six-month audit of US-VISIT's
security; the resulting 42-page report, released in December, would
conclude that the system suffered "security related issues (that) could
compromise the confidentiality, integrity and availability of sensitive
US-VISIT data if they are not remediated."
Zotob was destined to make those theoretical issues real.
The worm had its roots in a critical vulnerability in Windows 2000's
plug-and-play feature that allowed attackers to take complete control of
a computer over a network. Microsoft announced the hole Aug. 9, and it
took only four days for a teenage virus writer in Morocco to launch
Zotob, which spread through the security hole.
The workstations at the front end of US-VISIT run Windows 2000
Professional, so they were vulnerable to attack. Those computers are
administered by the DHS' Bureau of Customs and Border Protection, which
learned of the plug-and-play vulnerability Aug. 11, according to the new
documents. The agency's security team began testing Microsoft's patch
Aug. 12, with an eye to installing it on more than 40,000 desktop
computers in use in the agency.
But as CBP started pushing the patch to its internal desktop machines
Aug. 17, it made the fateful decision not to patch the 1,313 US-VISIT
Because of the array of peripherals hanging off the US-VISIT computers
-- fingerprint readers, digital cameras and passport scanners --
officials believed additional testing was needed to ensure the patch
wouldn't cause more problems than it cured. The agency was testing the
patch at a US-VISIT station at a border crossing with Mexico in Nogales,
By that time, Zotob was already flooding DHS compartments like water
filling a sinking battleship. Four CBP Border Patrol stations in Texas
were "experiencing issues related to this worm," reads one report. More
ominously, the virus had made itself at home on the network of an
interconnected DHS agency -- the Immigrations and Customs Enforcement
bureau, or ICE. The ICE network serves as the hub for traffic between
the US-VISIT workstations and sensitive law enforcement and intelligence
databases, and US-VISIT visibly slowed as traffic slogged over ICE's
On Aug. 18, Zotob finally hit the US-VISIT workstations, rapidly
spreading from one to another. Phone logs offer a glimpse of the mayhem
that ensued. Calls flooded the CBP help desk, with callers complaining
that their workstations were rebooting every five minutes. Most are
explained in the "status" line of the log with the single word "zotob."
Though accounting for only 3 percent of its Windows 2000 machines, the
US-VISIT computers quickly became "the largest impacted population
within (the CBP) environment," reads a summary of the incident.
At international airports in Los Angeles, San Francisco, Miami and
elsewhere, long lines formed while CBP screeners processed foreign
visitors by hand, or in some cases used backup computers, according to
press reports at the time. At CBP's data center in Newington, Virginia,
officials scrambled overnight to distribute the tardy patch. By 8:30
p.m. EST on Aug. 18, a third of the workstations had been fixed. By 1
a.m., Aug. 19, 72 percent were patched. At 5 a.m., 220 US-VISIT machines
were still vulnerable.
"In retrospect," reads an executive summary of the incident, "CBP should
have proceeded with deploying the patch to the US-VISIT workstations
during the initial push."
A spokeswoman for DHS' US-VISIT program office refused to comment on the
incident this week. ICE declined to speak to the virus' infiltration of
its backbone network, referring inquiries back to DHS.
While DHS and its agencies are taciturn about discussing security
issues, they couldn't hide the travelers stranded on the wrong side of
Customs at airports across the country. The day after the infection, DHS
publicly acknowledged a worm was responsible. But by December, a
different story emerged; a department spokesman speaking to CNET
News.com claimed there was no evidence that a virus caused the August
incident. Instead, the problem was merely one of the routine "computer
glitches" one expects in any complex system, he said.
By then, Wired News had already filed a Freedom of Information Act
request with CBP seeking documents about the incident. The request
received a cool response. An agency representative phoned us and asked
that we withdraw it, while refusing to answer any questions about the
outage. When we declined, CBP misplaced the FOIA request. We refiled it,
and it was officially denied, in total, a month later. After an
administrative appeal went unanswered, we filed a federal lawsuit in
U.S. District Court in San Francisco, represented by the Stanford Law
School Cyberlaw Clinic.
After we sued, CBP released three internal documents, totaling five
pages, and a copy of Microsoft's security bulletin on the plug-and-play
vulnerability. Though heavily redacted, the documents were enough to
establish that Zotob had infiltrated US-VISIT after CBP made the
strategic decision to leave the workstations unpatched. Virtually every
other detail was blacked out. In the ensuing court proceedings, CBP
claimed the redactions were necessary to protect the security of its
computers, and acknowledged it had an additional 12 documents, totaling
hundreds of pages, which it withheld entirely on the same grounds.
U.S. District Judge Susan Illston reviewed all the documents in
chambers, and ordered an additional four documents to be released last
month. The court also directed DHS to reveal much of what it had
previously hidden beneath thick black pen strokes in the original five
"Although defendant repeatedly asserts that this information would
render the CBP computer system vulnerable, defendant has not articulated
how this general information would do so," Illston wrote in her ruling
(emphasis is lllston's).
A before-and-after comparison of those documents offers little to
support CBP's security claims. Most of the now-revealed redactions
document errors officials made handling the vulnerability, and the
severity of the consequences, with no technical information about CBP's
systems. (Decide for yourself with our interactive un-redaction tool.)
That comes as no surprise to Steven Aftergood, who directs the
Federation of American Scientists' Project on Government Secrecy. In the
wake of Sept. 11, the Bush administration has been keen to expand its
ability to withhold information from the public under the FOIA, and most
commonly offers security concerns as the explanation.
"The Justice Department more or less explicitly told agencies to do so,"
says Aftergood. "Many requests yield greater disclosure on appeal, and
time and again FOIA lawsuits succeed in shaking loose records that an
agency wanted to withhold."
Despite the outward silence, it's clear Zotob left a lasting mark on
An inspector general report released a month after the US-VISIT outage
recommended CBP reform its patch-management procedures; a scan found
systems still vulnerable to security holes dating from 2003. And in the
aftermath of the attack, CBP resolved to "(i)nitiate timely
distributions of software and application elements for testing and
pre-staging events," according to one of the internal documents.
Phone logs released under the court order show that Zotob lurked on
CBP's networks as late as Oct. 6, 2005 -- nearly two months after
Microsoft released its patch.
The call logs also show a lingering presence of Zotob in the agency's
On Oct. 12, 2005, a user phoned the help desk to advise it of a new
critical Microsoft vulnerability that had not been patched on the
caller's machine. "The workarounds require administrator access," the
caller is reported as saying. "I do not have admin rights."
"Please open a ticket to update my CBP laptop with the latest security
patches from Microsoft," the caller says. "It is vulnerable, just like
it was during the Zotob outbreak."
Visit the InfoSec News store!