
‘Unprecedented incident’ took down ‘brain’ of network, Rogers execs tell MPs
Rogers told MPs its widespread service outage was caused by a “truly unprecedented incident” that took down the “brain” of its network.
The company’s new chief technology officer, Ron McKenzie, said Monday it was a coding error that triggered the failure of the company’s core network on July 8 and took down both wireless and wired services. He described the core as similar to the brain of the network, controlling the flow of data to the larger internet.
The error was introduced in the sixth part of a seven-phase update process that spanned several weeks, and McKenzie said there were no problems during the previous five phases.
Speaking during a hearing of the House of Commons industry committee, which is studying the outage, Rogers CEO Tony Staffieri said the company now plans to physically separate its wireless and internet networks to ensure more redundancy in the future. If one were to go down in the future, he said, the company could move traffic to the other one.
“This added layer of protection will be expensive,” Staffieri said, adding the company estimates it will cost at least $250 million. “But we know it’s the right thing to do.”
Asked about the state of competition and Rogers’ proposed $26-billion deal to acquire Shaw, Staffieri said he does believe Canadians have choice and that the Shaw deal would give the combined company greater scale and ability to invest in network resiliency. He said the deal would allow Rogers to separate its wireless and internet networks in about half the time that it would otherwise take.
Innovation Minister François-Philippe Champagne, who was first up at the hearing Monday morning at 11 a.m., told MPs that he was the one who first reached out to Staffieri during the outage to find out what was happening.
“It should not be (up) to the minister to chase the CEO of a major telco when something like that happens. It should be the other way around,” Champagne said.
Champagne was in Tokyo, Japan, at the time and said he called Staffieri around 4 or 5 a.m. Tokyo time on Saturday morning when he got information from his staff that the Rogers outage was more serious than first believed.
The minister said Staffieri has since been in regular touch to provide updates, but he said the fact that he was the one who had to reach out is part of the reason why he has since demanded better communication from telecoms in the case of outages.
Champagne called Staffieri and the CEOs of other major telecoms to a telephone meeting three days after the outage and demanded they strike formal agreements within 60 days to support each other during network failures in the future.
“The responsibility is solely on Rogers but we all have to be in solution mode,” Champagne said Monday. “The clock is ticking and there is 45 days left to deliver on that commitment.”
Pressed by NDP MP Brian Masse, who questioned the minister on what consequences Rogers or other telecoms could face when it comes to network outages, Champagne said, “When the minister speaks, they listen.”
“They didn’t even call you,” Masse replied, adding that he’s skeptical of an approach to network reliability and public safety that relies on influence and the goodwill of the telecom companies.
“You’ll be able to rely on a formal binding agreement,” Champagne said.
MPs also raised the point of whether there is adequate competition in Canada’s telecom industry, which has become a pivotal issue in Rogers’ proposed $26-billion acquisition of Shaw. The Competition Bureau is going to court to fight the transaction.
The federal government must also approve the takeover and when asked about the deal Monday, Champagne said the companies have not yet submitted a formal proposal to his department. He referenced previous comments he has made saying that he would not allow the “wholesale transfer” of spectrum licences from Shaw to Rogers.
In documents published late Friday evening, Rogers also revealed more details about the coding error that caused the July 8 outage, leaving the carrier unable to route customers’ 911 calls or to deliver four emergency alerts issued that day in Saskatchewan.
As a result of the failure in the core network, Rogers could not shut down its radio network, which could have allowed customers to automatically connect to 911 on another carrier’s network.
The company also said it was unable to transfer traffic to rival carriers despite offers of assistance from Bell and Telus, as those networks would not have been able to handle the extra volume of more than 10 million customers.
Rogers shared that new information about the outage in a lengthy response to questions from the Canadian Radio-Television and Telecommunications Commission (CRTC).
On Sunday, Staffieri said in a letter to customers that the company is working on a formal agreement with competing wireless carriers to “switch 911 calls to each other’s networks automatically — even in the event of an outage on any carrier’s network.”
“I believe this is the only responsible way forward and I am personally committed to making it possible for all Canadians,” Staffieri said.
He also said Rogers is doing a “full review” of its network and plans to invest $10 billion in the next three years as part of its continued “focus on reliability.”
“This includes more oversight, more testing and greater use of artificial intelligence to ensure we’re able to deliver the reliable service you deserve.”
Staffieri will appear before the standing parliamentary committee at noon, along with the company’s new chief technology officer, Ron McKenzie, and its chief regulatory officer, Ted Woodhead.
McKenzie was named CTO last week when the company said Jorge Fernandes would step down from the role in the wake of the network failure.
With files from The Canadian Press
Read more at thespec.com