Customs acts fast to shore up systems

Scrambling to avoid a repeat of a systems meltdown last month that snarled travel for tens of thousands of international passengers at LAX, U.S. customs officials have fast-tracked an overhaul of their operations here and around the nation.

Los Angeles International Airport officials say they are encouraged by the response of customs officials, who were put in a hot seat after their widely publicized system failure Aug. 11.

About $15.3 million has been allocated to refurbish technical systems at the country’s 104 major airports and international border crossings within the next six to nine months, bringing the upgrade project to a close about nine months ahead of schedule, said Ken Ritchhart, assistant commissioner in the Office of Information and Technology with U.S. Customs and Border Protection.

“Everything has been moved up,” Ritchhart said. “We recognize the fact that having folks sit on airplanes for hours is not acceptable, so we have to look at new procedures . . . to make sure it doesn’t happen again.”

The outage at LAX that led to the speedup of the nationwide overhaul was the result of a single malfunctioning computer that prevented customs from screening international passengers for security risks.

Officials had no suitable backup system, and more than 17,000 arriving passengers were stranded in planes for hours while about 16,000 departing passengers waited at their gates for updates from carriers with little information.

By Thanksgiving, or Christmas at the latest, the entire customs system at LAX will be redone, with not only new workstations, network switches, routers and cables, but also a snazzy new satellite backup system that will allow screeners to access network databases should local routers fail, Ritchhart said.

Airport officials said they have seen signs that the outage -- a huge embarrassment for customs officials and a source of outrage from passengers -- has touched a nerve at the agency’s highest levels.

“The tangible steps that have been taken, they’re encouraging, and, frankly, they’re impressive,” said Paul Haney, deputy executive director for airports and security for Los Angeles World Airports, the agency that operates LAX. “We actually have tangible evidence that this is being taken very seriously and there’s a commitment to ensure that there is never a repeat of this nature.”

Customs officials traced the source of the LAX system outage to a malfunctioning network interface card on a desktop computer in the Tom Bradley International Terminal.

The card, which allows computers to connect to a local area network, experienced a partial failure that started about 12:50 p.m., slowing down the system, said Jennifer Connors, an official in the office of field operations for the customs agency.

As data overloaded the system, a domino effect occurred with other computer network cards, eventually causing a total system failure a little after 2 p.m., Connors said. The system was restored about nine hours later, only to give out again late the next day, a Sunday, for about 80 minutes, until about 1:15 a.m. that Monday. The second outage was caused by a power supply failure, Connors said.

During the Saturday outage, customs officials implemented their slower and more limited backup system, which included setting up laptops at workstations and using the laptops to search through copies of files from the downed network on a CD-ROM and scan lists of names.

Last week, airport officials received 100 laptops, weeks earlier than promised. In addition, they are being allocated 30 more customs officials to help process passengers more quickly.

Customs officials said that, instead of just one information technology supervisor and one technician on duty as was the case Aug. 11, the agency now has seven such personnel on staff at LAX. They also plan to hire at least three more systems technicians and will explore other ideas, such as having another airport handle some of LAX’s screening load.

“Maybe Atlanta’s not too busy and can handle these five flights, and then phone it in, to redistribute the workload,” Ritchhart said.

In addition to the new satellite site, Ritchhart said customs is also looking at a more long-term idea of hiring additional Internet service providers to ensure more redundancy in the backup system.

Ritchhart said the LAX workstations were about 4 years old, with networking switches and routers 6 years old and cables about 20 years old. Nearly all had reached the end of their life expectancy, he said.

Local airport agency officials also have been dealing with the results of the outage that left them fielding dozens of angry phone calls and e-mails from passengers seeking reimbursement for missed cruises or hotel rooms.

Officials recently held a three-hour meeting to brush up on emergency procedures, including notifying people via the airport agency’s website (which was not done during the Aug. 11 incident).

The entire public relations staff underwent training last week on how to post urgent information on the site without having to contact information technology staff.

“It was a shortcoming on the part of my public information officer and the command post to implement that, and then . . . there was a shortcoming on the information technology people to get it posted, so it never got posted,” airport agency spokeswoman Nancy Castles said of last month’s incident.

Also last week, airport officials approved adding 21 more visitor service representatives at the Bradley terminal; officials said their presence could help in communicating with travelers in the event of another incident.

Along with new flight displays that will arrive with the planned renovation of the Bradley terminal, airport officials also plan to set out portable signboards during any future outage that will allow them to update those coming to pick up passengers and advise them on how long it may take to get through customs and baggage claim.

Just days after the outage, Los Angeles City Council members Bill Rosendahl and Janice Hahn called for an immediate report from the airports agency on actions taken by customs to permanently correct the computer malfunction problem.

They also called for the agency to report on contingency plans for working with customs and other officials to properly deal with passengers during another breakdown.

--

tami.abdollah@latimes.com