Computer Co-location Facility Vulnerabilities

How terrorists could smuggle and detonate explosives

Jonathan A. Zdziarski jonathan@networkdweebs.com

August 5, 2003

Other Security-Related Articles: [ Chrooting daemons and system processes | Creating root-kit proof saferooms ]

"These facilities are one of only a few places where an individual is capable of introducing heavy, unchecked equipment, leaving it in or near a large public concentration of business, and is able to communicate remotely with the equipment from virtually anywhere in the world. "

I’ve spent a significant portion of the past ten years of my professional career working for corporations with large co-location facilities. Co-location facilities provide a cost effective data center solution for many companies, both small and large, enabling remote hosting of equipment in a climate controlled environment usually with several redundant high-speed connections to the Internet. These facilities are responsible for a significant percentage of electronic business performed in the United States and other countries.

In having the privilege of working with a number of these facilities, I’ve also had the opportunity to witness the vulnerabilities that could give themselves over to terrorist activities. Prior to September 11 2001, I was able to dismiss these fears with the thought that “nobody would ever want to blow up the city block”. Unfortunately today these vulnerabilities are both a valid and justifiable concern.

Many co-location facilities are strategically placed in areas where a significant amount of business is occurring…major peering points, large corporate concentrations, and many general terrorist targets. Some facilities are within immediate proximity to targets such as the New York Stock Exchange and the public and private networks that are responsible for the Internet as well as military and public service networks. A single target among many, if taken out, could seriously cripple the Internet let alone the number of critical private networks sharing the same fiber. Due to the placement of such facilities, they are unfortunately an ideal target for terrorists to take advantage of. These facilities are one of only a few places where an individual is capable of introducing heavy, unchecked equipment, leaving it in or near a large public concentration of business, and is able to communicate remotely with the equipment from virtually anywhere in the world.

UPDATE:
As suspected, terrorists are beginning to use more high-tech methods of attack. The government recently found a large warehouse full of several everyday electronic devices (such as cameras and laptops) that have been modified to contain tazers, explosives, and other types of equipment. The airline industry has been tasked to closely inspect such equipment during a security screening. To my dismay, a majority if not all co-location providers still fail to perform any such inspections on incoming hardware.

This short article addresses some of the key vulnerabilities in these facilities, why they are important issues to address, and offers some potential solutions. While this article is by no means a complete study, I hope it presents enough information to convince some individuals in the field to start taking these issues seriously, while at the same time I have removed specific areas of interest (specific peering exchange locations, financial district facility locations, and primarily a list of facilities and their vulnerabilities) to prevent this document from ever becoming a national security risk.

Basic Overview of a Co-Location Facility

Before delving into the different vulnerabilities these facilities are faced with, it is necessary to get a general overview of what the average facility looks like.

Components of a facility

A co-location facility is for the most part a large data center located either in its own building or sharing an office building where slices of space (usually anywhere from 1/5 of a rack to multiple racks) are carved out and leased to an individual or corporation. The leasing party (e.g. the customer) is responsible for moving their equipment into the facility, setting it up on the racks, and connecting it to whatever network is provided by the facility (if any). Most facilities are comprised of the following components:

Telecommunications equipment for network connectivity
Racks to mount the customer’s equipment to with overhead conduits for cables
Cages or cabinets to physically isolate systems
A raised floor for air conditioning, power, conduits, flood prevention, etc.
A fire control system (usually a Halon, Carbon Dioxide, or Potassium-based system) along with crash bars on all doors
An authentication system for entrance/exit. This could be a card reader, biometric device, or a security guard.
Security cameras to view an overall state of the facility

Most facilities are unmanned, and remotely managed from a NOC, or network operations center. This NOC could be in the room next door, on a different floor, or in a nearby building. The NOC is usually responsible for the electronic state of the entire company including their network, external customers (leased lines), internal customers (co-located customers), peering sessions, and etcetera. There is generally no group focused specifically on the co-location facility’s process and procedures, although there are occasionally one or two overworked individuals responsible for the cabling and such.

Access Procedures

There are generally two groups of individuals clear to access these facilities at any given time: customers who have procured space, and telco/fire/electircal maintenance technicians from vendors, the city, and other such individuals.

Customers
From a customer perspective, the standard procedure to procure rack space in a co-location facility is to sign and pay on a contract through the company’s sales representative. The customer’s network access will then be provisioned and rack space assigned. Access control for customers is generally more formal than maintenance: the customer will most likely be given access cards, a key, or other means to access the facility at which point it will be their responsibility to install their equipment and bring it online.

Maintenance
From a maintenance perspective, the individual wishing to access the facility will generally be asked to provide an identification card and/or information about the service call. Since the maintenance may be for a specific customer, there is generally no maintenance authorization log. An individual performing maintenance on the facility may be from one of many local carriers with hardware in the facility, an electrician configuring a rack for a customer, fire marshall (who frequently inspect facilities), or even a "tech" from a customer sent out to unplug a system. Guest cards are usually given to the technician upon arrival, or the technician is escorted into the facility and left to work.

Since a majority of these co-location facilities are unmanned (perhaps with the exception of a security guard), it is the customer or maintenance technician’s responsibility to conduct themselves in a professional manner by not stealing or sabotaging the equipment of another customer or vendor. Should an individual visit the facility during a lull time (the evening, for example), they will be virtually alone. The individual will usually bring the necessary equipment in on a cart, swipe their card, possibly sign in, and be cleared for access to the facility.

While there are usually some deterrents in place, such as cages for larger customers, security cameras, and possibly an on-site security guard, these facilities are generally not monitored with much attention or scrutiny. Generally offenses of this nature are only detected after the fact. The responsibility of the security guard is to insure that unauthorized individuals are not allowed into the facility, that nobody walks in with a baseball bat and starts smashing equipment, etcetera. They are not by any means equipped with the ability to determine if a chassis is armed with explosives – or even if the equipment a customer is working on belongs to them. Even to engineers who work in a co-location facility on a daily basis, most of the work customers in the facility perform are considered to be “their technical stuff” and rarely ever noticed let alone challenged.

Common Equipment

Authorized guests of the facility will frequently have several large pieces of heavy equipment stored in the facility. These can include enterprise-class servers or mainframes, network equipment, telco gear, large batteries, and possibly even their own hardware closets. It is not uncommon to see computer systems five or six feet high with large locking doors for disk storage or processor blades nor is it uncommon to see a group of deep-cycle batteries with bare terminals connected to telco equipment. Other equipment involved usually includes copper and fiber optic cabling, miscellaneous small devices, and possibly even the customer’s own remote access cameras.

The basic vulnerabilities

Now that we’ve taken a basic look at the design and procedure of co-location facilities, it is time to discuss the vulnerabilities of these facilities. As I said before, these facilities are one of only a few places where an individual, with some money in hand, is able to bring heavy equipment in virtually unchecked, be able to leave it in a place where it will remain untouched, and remotely communicate with that equipment from anywhere in the world. This makes co-location facilities an ideal target for terrorists to use as a means of introducing explosive devices (even possibly a small nuclear device) into a large business or residential area, and detonate the explosives remotely having enough time to leave the country if so desired. Since co-location facilities are frequently located in buildings where there are concentrations of telecommunications networks tied together, even a small explosion capable of taking out only the building could seriously cripple commerce.

Maintenance Practices: A feat of social engineering

Telco...Fire...Electrical...Security...Building Management...there are so many different maintenance practices occuring on a co-location facility and such a lack of a centralized process that just about anyone with a common knowledge of social engineering can gain access to a co-location facility. Since most companies do not implement tight security for such visits (which can frequently occur unannounced), there are some significant vulnerabilities that could easily allow an individual to gain access. The typical social engineering attack begins with introduction, followed by a trust-building excercise (such as establishing a relationship with the target, or staging one or more "safe" visits), ending with the covert attack itself which usually remains undetected. Some examples of scenarios include:

A telco engineer from the local phone company calls your office to inform you that they will need to perform maintenance on one of the smartjacks in your facility on [Some Number] Wall St. The next day, an engineer shows up with his toolbelt and informs the NOC personnell that this is "next on his list", pretending to be mostly clueless about the specifics, just that "this is where he was told to go and fix a smartjack". The telco engineer is allowed into the colo with his toolbox and some small equipment with an explosive device inside, which he is able to inconspicuously leave bolted to a rack in plain view - dismissed as telco equipment.

A gentleman dressed professionally in a dark blazer with a walkie-talkie pays you a visit introducing himself as building security (this obviously applies only to facilities located in managed corporate buildings). He says a smoke detector went off in your co-lo, and wants someone to escort him into the facility. While down there, he pretends he can't find the problem, and asks if he can bring his technician back with him. The NOC technician either props the door open or gives the gentleman a guest access card to come back later. 20 minutes later, the gentleman comes back with some equipment and pretends to be looking for a faulty electrical connection. Once the NOC tech gets bored and leaves, he installs a small but powerful explosive device underneath the floor, or up above in a conduit.

Unfortunately these feats of social engineering have occured all too often in many types of businesses when an individual wants to gain unauthorized access to a building. Fortunately, they have thus far been unrelated to terrorism, however one thing is certain: it is far easier to take the avenue of misdirection and advantage of the staff's unawareness than it is to create a conflicted, forceful entry into the facility.

The Customer: Sales Pitch Procedures Without Familiarity

Co-location facilities being a corporate business, it is the goal of a vast majority of facilities to give business to any individual or corporation that can afford the services. There are generally no requirements to purchase rack space other than meeting the financial obligation and agreeing to the contract. The corporate strategy behind these facilities also compensates for the fact that many new small businesses will need these facilities’ services.

For this and many other reasons, a vast majority of co-location facilities are not very familiar with all of their customers. Certainly no background checks are performed on individuals or businesses seeking to lease rack space in the facility. In many cases a business license is also not necessary as an individual is just as valuable a customer as a business.

The first step a terrorist group would likely perform in a scenario such as this would be to find a company with a co-location facility either:

Strategically placed near an area where there is either a large concentration of people or business.
Hosting a significant business or businesses where financial transactions or other targeted commerce could be attacked

The sales representatives for the facility are usually more than happy to give an individual a guided tour through their facility and will generally like to brag about big name customers in the facility as well. This walk through the facility, while helping to close the deal, also gives a potential attacker the following information:

Location of most or all security cameras (not even necessary to this discussion)
Understanding of the general security in the building
Knowledge of any businesses in the facility that would make good targets
Knowledge of the part of the city they are located in and what targets are nearby
How much space the attacker will have to place their equipment, and where

Once the attacker has this information, they can make an even better judgment about whether or not the facility is a suitable point of attack. Should the attacker desire to become a customer, a contract will be filled out. The customer will then agree to a set of terms and conditions, provide billing information, and hand over a check. Once this is complete, the provisioning process begins and the rack space is assigned. In many cases, this entire process can take less than 24 hours.

Introduction of Equipment

Once the attacker/customer has been given access to the co-location facility, equipment may be introduced into the facility and bolted to the racks. In all my experience I have never found a single provider that allows their employees to inspect the customers’ hardware or require a copy of the keys to such hardware remain on the premises. The attacker can easily smuggle in whatever equipment they desire without detection. This can include explosive devices, small nuclear devices, electromagnetic devices, etcetera.

Note:
I recently received an email informing me that the gamma radiation from a nuclear device will cause fiber to turn opaque. Although this can be avoided with the proper shielding, should you ever have a problem with opaque fiber in a facility, be diligent to insure it is not a result of such radiation. Small radiation detection devices can be purchased to detect radiation in a room.

Due to the large and heavy nature of most enterprise class computer equipment, plenty of free space is already available in any large computer chassis to accommodate such devices as well as keep them cool and stable. The Sun E450 is an ideal example of such a chassis, as a significant amount of free space is already available and would require little or no modification to hide such a device. We used to call them the beer coolers for their size and cooling capabilities. There is also the Cisco 12000 series routers, large "stackable" type computer systems, and plenty of other options as well.

Using the serial cable or other interfaces, these devices can be connected to a computer which is then connected to the Internet via the network connection supplied by the facility. Some of the possible ways such weapons could be smuggled in include:

Placing them inside a large computer chassis
Removing the hard disks from a RAID unit and using just the bay doors as cover
Bringing in a large metal box that “looks technical” enough not to be suspicious

Once the attacker has brought in their equipment, it can sit in plain daylight inside a computer chassis, or even underneath the raised floor where it will be out of sight. The employees of most facilities are not expected to nor do they “meddle” in their customers’ business. I’ve found in many facilities, one could easily steal another company’s hardware while being watched through a camera by the NOC dismissed with the thought, “They must be with Company ABC. I’m not going to challenge them and get in trouble”.

Detonation of Equipment

Aside from timers and RF detonators, there is IP. As the co-location business involves Internet connectivity, very high speed connections are usually provided to the customers. Connecting this to the computer that has been brought in provides a remote means of detonation from anywhere in the world over any layer of encryption the attacker finds necessary. Should a small nuclear device have been smuggled in, detonating the equipment from inside the building could easily take out the city block.

A more complex approach might include scouring a news site for mentions of particular key words such as “Bin Laden Captured”. Once the system has been connected to the Internet, it can be trained to detonate on any event ranging from the system date to the score at the last Yankee's game.

Damage to in-house Equipment

Should the attacker be more interested in damaging equipment from another business located in the facility, smaller explosives will most likely be used. Something as simple as water itself can damage and bring down a large computer network. Unfortunately, cages do not keep water out, and many of the different designs of overhead conduits make it even easier to irrigate every machine in the room.

Possible Solutions

Now that some of the very basic vulnerabilities have been discussed, let’s take a look at some of the ways security can be improved in these co-location facilities. These are just a few suggestions out of many that would both serve as a deterrent and help detect such violations.

Unshakeable maintenance procedure

This method of gaining access to a facility hinges completely on the lack of proper procedures and authorization checks at a company. The "ignorance factor" is taken advantage of to authenticate phone calls, uniforms, and paperwork. The steps to fixing the hole in the maintenance arena include:

A centralized maintenance manager. Require all maintenance appointments, including emergencies, to go through a central maintance manager (or group). All appointments should be logged and confirmed with the vendor (by calling _them_ at the phone numbers already listed for them). Emergencies should be handled in cooperation with the NOC (to confirm if a particular vendor is having an issue, there should be an open ticket - albeit 5 minutes old) and building management.

Authentication of all telephone calls and documentation. All telephone calls into the company for maintenance should be logged, and the numbers called back before the conversation can even start. On top of this, the vendor should identify themselves and be reachable at a phone number already on file. NO maintenance requests should be accepted from a cellular phone. All appointments should be confirmed twice prior to allowing access. All documentation should be checked and compared with the formal documentation already on file for the particular vendor.

Supervised maintenace. A small group of individuals should be assigned to oversee any and all maintenance performed. While the individuals do not necessarily need to understand the complete technical nature of the maintenance, they must be able to identify what actions are appropriate for the maintenance being performed. For example, is a telco vendor installing a smartjack supposed to bolt some large equipment to a rack?

Equipment and toolbox inspections. All equipment and large toolboxes, etc. entering the facility should be briefly inspected. While it is unlikely that the words 'ACME NUCLEAR BOMB' will appear on the side, the individuals inspecting the hardware should have adequate knowledge to identify suspicious looking devices. There are many portable bomb detection devices on the market today which can also be used to scan all incoming equipment.

Know your customers

An undetected attack in this arena hinges on the ability for the individual to become a customer. The biggest problem with the co-location business is that there is not a very solid relationship between the provider and the customer. When an individual rents an apartment, their credit and references are checked, the individual is interviewed, and based on the character of the individual, it is determined whether or not they are allowed to rent. Unfortunately due to the technology industry failing and in the name of privacy and professionalism, renting rack space in a co-location facility requires no such check in most cases. The only reason most sales representatives even know what their customers do is from casual conversation and not background checks. Once the baton passes to provisioning or network operations, nobody knows anybody. The customer is now a number, and so no questions are asked.

Facilities located in areas with a high concentration of people or businesses should make every attempt to know who their customers are and where they’ve come from. Having background checks part of the standard contract, visiting their place of business, and learning as much about the customer are all ways to insure that you’re not doing business with the wrong group. This will force attackers to build long term relationships with their targets making it all the more possible for them to expose a suspicion.

New customers should provide more specific information about their business. Are they a web hosting company? An online trading company? What do they intend to use the hardware in the rack space for? Sure they won’t say “to blow up Wall Street” but if they don’t have an answer, that’s suspicious. Does the customer have a website? How did they pay…cash, personal check, company check, credit card or wire transfer?

Some information can be discerned from common interaction with the customer. If the company has been around for several years, has offices, and a staff then it is most likely a legitimate business. On the other hand, if the company is a new virtual company with a residential address, there is significant reason to raise an eyebrow. Co-location facilities already have the liberty to discriminate based on many other criteria such as whether or not the company spams, broadcasts pornography, etcetera. Insuring that the customer has a justifiable business is certainly a responsible approach to new customers.

Non-discriminating Hardware Inspection Policies

Some basic policies regarding the hardware that is stored in the facility can help give the provider some additional means of inspecting equipment and insuring new customers don’t attempt to smuggle explosive devices. Having a clause in a contract that requires a copy of all system keys to be stored on the premises and a requirement stating the provider “reserves the right to inspect any hardware brought into the facility for malfunction or malicious use” gives the provider the ability to perform periodic checks of all hardware for not only this purpose, but to insure there are no fire risks, loose power cables, or any other facility hazard. Such a policy is easy to justify.

Explosive detection tools are available as well to scan new hardware. Even K-9 officers can be trained to detect up to 11 distinct odors of explosives. An occasional pass-through of the facility by a trained K-9 officer can provide a non-intrusive way to check out new customers without the need to even touch their hardware. Other actions such as checking underneath the raised floor for any devices, closely monitoring a customer’s actions and behavior, and etcetera are all less intrusive ways to keep an eye out for suspicious activity..

Paying attention to the kind of hardware the customer brings in is another good way to identify suspicious activity. Is the user a dialup Internet provider? Why do they not have any dialup access equipment such as modem banks? Are they a web hosting company? Why do they have only one large server instead of several small ones? Are they a Linux shop? Why do all of their terminals have little window icons on them? As small as they are, these inconsistencies can pile up and help to identify a customer who is not really who they say they are (even if you're just looking for spammers).

Finally, implementing a network provisioning period that allows the customer to install their hardware before their network connectivity or POTS lines become available will give the provider plenty of time to perform any such inspections before they are given the ability to communicate with their hardware from anywhere in the world. Granted this would not stop a bomb on a timer, however an individual who would seek to launch this type of attack would most likely do so for the purpose of doing it remotely.

Restricting building and rooftop access

If the facility is located in the same building as a targeted corporation, or if there is a possibility of a different style of attack (for example, a releae of biological toxin from the roof top), the individual may attempt to use this opportunity to attack in other ways. By having 24 hour access to the co-location facility, the attacker is also given 24 hour access to the entire building (in many cases). Having a building security guard monitor the status of individuals in the building will help detect if a customer who is supposed to be on the 10^th floor is snooping around on the 4^th floor. Policies for introducing new equipment outside of normal business hours can be implemented requiring permission from the facility provider. Additional steps similar to these can and should be taken to secure the building outside of business hours.

Many facilities providers allow their customers to network to the roof of the building where antennas or satellite gear are mounted. Restricting this type of access to where the customer must be accompanied by staff during normal business hours will help prevent any “soft target” attacks on the building.

External Self-Assessment

Hiring an outside firm to test the effectiveness of the policies a company has in place for knowing/choosing their customers and managing the facility will help expose any loose individuals in the company. Hire a firm to create a fake company with some suspicious ties and attempt to become a customer. Do they succeed? What about equipment policy violations? Can they effectively pose as a maintenance technician and gain access? A terrorist isn't likely to be walking in with guns blazing and take over the building, but rather play a game of invisibility and misdirection, taking advantage of the very nature of of "good-willed" individuals...these external tests can help identify these weak areas.

Conclusion

In summary, there may be no easy cookie cutter plan for improving the security of co-location facilities; however it is of the utmost importance to take the necessary steps to protect these facilities from a terrorist attack. Detecting the vulnerabilities in your individual corporation is the first step. Once they are exposed, finding an effective way to fix them will help make the facility a secure place of commerce rather than the next target of attack

The bottom line is: how difficult is it for anybody out there to sneak their way into your facility as a tech or a customer, and sneak in a dangerous device? If the answer is “not very difficult” then you have some vulnerability in your policies that could potentially expose you to an attack.