The Single Router In Southern Arkansas That Ground Air travel To A Halt

The Aviation world hit the panic switch Friday. A nationwide ground stop stranded tens of thousands of people after aging air traffic control technology buckled under new equipment, according to officials. The more than four-hour halt grounded thousands of flights and resulted in more than 25,000 delays, as of 9:30 p.m. ET on Nov. 13.

The Federal Aviation Administration first noticed communication glitches at its Indianapolis center, which oversees high-altitude flights over a swath of middle America.

Soon, they spilled into communications between pilots and air traffic control. There was static and interruptions. Pilots had questions.

Initially, FAA employees tasked with troubleshooting the problem—one that eventually grounded much of U.S. travel—began their hunt in Indianapolis, according to people familiar with the situation.

But then, they zeroed in on two locations: an FAA radio site in El Dorado, Arkansas; and a radar site in Amarillo, Texas, the people said.

Network gear at the Arkansas site, and possibly the one in Texas, overloaded the communications system that thousands of air-traffic controllers use to guide pilots through the nation’s airspace.

“We saw two places that were actually pinging our network at a very high rate, a thousand connections over every four to five seconds,” Rebecca Guy, the FAA’s technology chief, said at a Saturday press conference.

These pings traversed a tangle of copper wires and fiber-optic cables that feed into the FAA Telecommunications Infrastructure, or FTI. The decades-old system is outdated and undergoing a multibillion-dollar upgrade to move into the fiber-optic age.

Defense contractor L3Harris Technologies oversees the system, which includes key connections for radar, communications, and flight data. “L3Harris takes the nationwide ground stop and its impact on travelers, airlines, and air traffic operations seriously,” the company said in a statement.

The system has suffered repeated failures in recent years as officials struggle to replace it with a modernized network and newer routing hardware. But efforts to maintain and replace the mission-critical network have taken years and become more expensive over time.

The risks FTI failures pose to the FAA’s air-traffic system are well known. A 2023 outside review warned that “system risk could quickly reach unacceptable levels” due to increased outages and inadequate funding for overhauls.

FAA Administrator Bryan Bedford said the agency spends hundreds of millions of dollars annually maintaining its antiquated telecom systems.

Defense contractor L3Harris Technologies oversees the system, which includes key connections for radar, communications, and flight data. “L3Harris takes the nationwide ground stop and its impact on travelers, airlines, and air traffic operations seriously,” the company said in a statement.

The system has suffered repeated failures in recent years as officials struggle to replace it with a modernized network and newer routing hardware. But efforts to maintain and replace the mission-critical network have taken years and become more expensive over time.

The risks FTI failures pose to the FAA’s air-traffic system are well known. A 2023 outside review warned that “system risk could quickly reach unacceptable levels” due to increased outages and inadequate funding for overhauls.

FAA Administrator Bryan Bedford said the agency spends hundreds of millions of dollars annually maintaining its antiquated telecom systems.

Generally, changes to the system—like switching out copper wires for fiber—are carried out overnight in busy areas. Officials thought daytime updates in more remote locations shouldn’t affect many flights. Transportation Secretary Sean Duffy said officials would re-evaluate when they do such work.

Not long after L3Harris installed a new router made by HPE’s Juniper Networks at the Arkansas location, the machine began sending out a high volume of data Friday afternoon.

Data from that router and another at the Texas location overwhelmed the system, government and industry officials said. It disrupted communications at 17 of the FAA’s 22 centers that oversee high-altitude flying, Bedford said, prompting the FAA to order the ground stop.

About 60 Juniper routers are active and operating normally in the FAA’s air-traffic network, a government official familiar with the system said. It wasn’t clear whether Friday’s problems resulted from a bad router or from L3Harris not configuring it properly.

“Was it a faulty installation? Was it a faulty configuration? Or was it just a bad piece of router technology that we put into this system that propagated these errors?” Bedford said Saturday, noting the FAA is still investigating.

An HPE spokesman said the Juniper devices appeared to have been working as intended. “They did not fail,” he said.

FAA officials identified a router feature that appeared to be causing the data overload, according to government and industry officials. A government official described it as a broadcasting feature that sent data more broadly than necessary, and said the agency has since disabled it on all the Juniper routers.

As passengers rushed to rebook flights, book rental cars, and reserve hotel rooms, the FAA tried to diagnose the problem.

When the FAA cut off the routers in Arkansas and Texas from the broader network, the communication problems stopped. “We disabled both of those right away and were able to stabilize our network,” said Guy, the FAA’s technology chief.

The FAA still doesn’t know why the router went rogue. Duffy said they have found no evidence of nefarious causes and are instead zeroing in on a mechanical problem with the router itself or “incompetence” by L3Harris.

An L3Harris spokeswoman said the contractor is supporting the agency’s probe into what went wrong.

Duffy said that until all the copper wires are replaced at the end of 2028, more disruptions could come.

“This was an FAA system that didn’t meet the moment. Our contractor didn’t meet the moment. Or the equipment didn’t meet the moment,” he said.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.