By Sharon Fisher
October 09, 2006
Continental Airlines Inc. last week ran its first full-blown test of a
new disaster recovery facility that it opened earlier this year in an
former bomb shelter built by an eccentric Chinese oil baron.
The Houston-based airline decided to improve its business continuity and
disaster recovery facilities last October after hurricanes Katrina and
Rita, said John Stelly, managing director of technology. At that time,
the company's off-site facility was located just 20 miles north of
Houston and ran on the same power and telecommunications grid as the
city, he said.
In March, Continental chose a bomb shelter run by Westlin Corp. in
Montgomery, Texas. The shelter, which about 50 miles north of Houston,
was originally designed in the 1980s to support up to 700 people for
months. The facility extends 50 feet below ground and is not only
farther from downtown Houston but is also served by a different power
company and is on a different power grid, Stelly said.
Continental signed the lease by May 15 and had completed some
retrofitting by July 1. While the airline had done some testing since
July, the first full-scale operational test, which featured Continental
running its entire organization for one shift from the facility, took
place on October 3, Stelly said.
The facility takes up about 2,000 square feet of the 40,000-sq.-ft.
underground bunker, which is where servers and other equipment are
housed, Stelly said. The facility's primary shortfall -- literally -- is
that its ceilings are only about 10 feet high, so Continental dealt with
that by using shorter racks and more square footage, he said.
Stelly is not sure to what extent Continental will make use of some of
the bunker's more eccentric facilities, such as 1,000-pound
concrete-filled blast doors, jail cells and gun ports.
In addition, Continental leases 12,500 square feet in a four-story
building next door to the shelter that was built to the same standards
and at the same time as the underground facility by the same designer.
The building is made of reinforced concrete with bulletproof glass. Both
facilities feature two generators with enough diesel fuel on-site to run
for a couple of months, Stelly said.
Inside, the building is set up like any other Continental office, right
down to the Continental logo and carpeting so employees will feel at
home, Stelly said. There are 275 Intel-based Hewlett-Packard Co. DC7600
workstations with Philips Electronics NV 17-in. and 19-in. monitors on
the desks, including some with dual monitors, he said.
Inside the bunker, an Avaya Inc. S8700 IP private branch exchange is set
up to run the voice capability with a local survivable processor, which
provides all the features if the connection to the main system is lost;
a Hewlett-Packard storage-area network; and Intel-based HP servers that
run the applications, which are primarily custom software that keep
Continental's 375 planes flying, Stelly said.
Between the bunker and the building is dual-fiber connectivity, and
connecting the building to Continental's WAN are two 45Mbit/sec.
circuits, Stelly said.
Continental is not revealing the price of the project, but it was "a few
million dollars," Stelly said. Despite the new equipment, it was
cost-effective compared with services such as those from SunGard Data
Systems Inc. because the company did a lot of the work itself, he said.
Once a month, technical support people visit the site to certify
everything and make sure updates are performed, and twice a year -- once
in the spring before hurricane season, and once in the fall after
hurricane season -- the organization will run a full-scale operational
test, Stelly said.
But there's more to disaster recovery than just having the equipment in
place. It's also important to have the procedures for how to implement
it as well. By consulting with city, county and state emergency
management personnel in the Gulf area, and through consulting with the
Texas Engineering Extension Service, part of the Texas A&M University
System in College Station, Continental has developed a more
comprehensive business continuity plan.
For example, because the regional emergency management authority plans
to call for evacuation two days before a hurricane makes landfall,
Continental will make its evacuation call three days before landfall,
Stelly said. Continental's chief operating officer can make the decision
to activate the off-site location, based on a Category 3 or greater
hurricane that is projected to make landfall within a 100-mile radius of
Houston's central business district.
Continental has also contracted with a local hotel in the area to put up
its staff, as well as with a bus company to provide transportation
between the hotel and the disaster recovery site. It is the employees'
responsibility to get up to the hotel, but once they're there, they will
be taken care of, Stelly said. In addition, because organizations have
learned that employees will take care of their families first,
Continental will fly employees' families to other cities where they have
friends and family, he said.
Visit the InfoSec News store!