Motadata: The Temporary Setting That Stayed, The Impact To A Telco

In 30 seconds

1. Setting: Monday morning at a mobile operator. A business customer's depots can't reach head office, and the network operations centre is drowning in alerts.
2. What Motadata did: Motadata ObserveOps folded the alert storm into one incident and showed a router that no longer matched its approved configuration; Motadata ServiceOps held the weekend change that explained it.
3. Business point: Astiostech, a Malaysian technology company, implements Motadata ObserveOps, a unified observability platform, and Motadata ServiceOps, an ITSM, ITAM and patching platform. Together they help telcos connect network problems to the changes behind them.

Networks that change faster than anyone can track

A telco network changes every day: new sites, replaced hardware, software upgrades, emergency fixes and planned maintenance, across equipment from many vendors. Much of that work happens at night, by engineers working against the clock.

That is where many service problems begin:

1. Configuration drift. A device slowly stops matching its approved setup, through temporary fixes, half-finished rollbacks or one-off tweaks.
2. Alert storms. One faulty device makes dozens of others look unhealthy, and the network operations centre (NOC) sees hundreds of alerts instead of one cause.
3. Changes that are hard to find. Maintenance records sit in one system and monitoring in another, so nobody connects Monday's problem to Saturday's work.
4. Business customers with service level agreements, who notice before the NOC does and expect a clear answer, quickly.
5. Blame without evidence, in both directions, between the operator and its customers.

The illustration

At 9:05am on Monday, the big screen in the network operations centre turned from a calm sea of green into something that looked like a durian stall's price board: red, everywhere, all at once.

Faizal, the NOC lead at Kenari Telco, put down his roti canai. Three hundred alerts. Latency. Packet loss. Interface errors. Dozens of devices across the northern region, all complaining together.

His phone rang before he could count them properly. It was Jasmine, the account manager for one of their largest business customers, a logistics company with depots up and down the peninsula.

"Their warehouses can't reach head office," she said. "The trucks can't print delivery orders. They have an SLA with us. They have used the word 'penalty' twice already."

Faizal looked back at the wall. In a NOC, three hundred alerts usually means one problem pretending to be three hundred.

A year ago, the team would have split three ways, one group on the transmission alerts, one on the routers and one on the angry phone calls, and spent the first hour proving which team the problem did not belong to. Wani, the shift engineer, still kept the old whiteboard from those days. It had a column titled "Not us", which was always the longest.

Motadata ObserveOps agreed with Faizal. Alert-noise reduction and cross-domain correlation had folded the storm into a single incident and pointed at one place: an aggregation router in the northern region that every one of those depots passed through. ObserveOps had already opened the incident in Motadata ServiceOps, with the affected devices and the correlation attached. Jasmine's escalation landed on the same incident instead of starting a new one.

The router itself was healthy. Its links were up and its processor was calm. But it was dropping a particular kind of traffic, politely and consistently, like a security guard who has been told no one gets through without a pass.

Jasmine was still on the line. "Their IT head wants to know if it's their firewall," she said. "He says they changed nothing."

"Tell him I believe him," said Faizal. "Give me fifteen minutes and I'll tell him who did."

Faizal opened the network configuration and compliance view. The router's running configuration no longer matched its approved baseline. One difference, highlighted in yellow: a quality of service policy, limiting a whole class of business traffic to a trickle.

"Who added that?" asked Wani.

ServiceOps answered. Faizal pulled up the change records for the router. Saturday night, approved change: replace a faulty line card. Step four: apply a temporary quality of service policy to protect voice traffic during the work. Step nine: remove the temporary policy.

Steps one to eight were ticked. Step nine was not.

The work had finished at 4am on Sunday. The weekend traffic had been light, so the limit hurt nobody. Then Monday arrived, every depot logged on at once, and the temporary policy discovered its true calling.

"Temporary," said Wani, "like the roadworks outside our office."

"Those have been temporary for three years."

"Exactly."

Here was the moment Faizal had learned to respect. The compliance view could show him what had changed. It could not tell him whether that change was meant to be there. A policy on a core router might be a mistake, or it might be protecting something he couldn't see. Pulling it out on a busy Monday, on a router that also carried voice traffic, without knowing why it was there, would be swapping one outage for another.

So he did the unglamorous thing. He called Ravi, the engineer who had done Saturday's work, and who had very reasonably been asleep since Sunday.

Ravi answered on the sixth ring, sounding like a man surfacing from the bottom of a swimming pool. He listened for ten seconds, then woke up completely. "Step nine. Aiyo. Yes, remove it. It was only for the card swap. Nothing else depends on it."

Faizal raised an emergency change in ServiceOps, linked it to Saturday's change and the incident, and attached the compliance difference. The duty manager approved it from the corridor. Faizal let the voice team know, told Jasmine to expect good news within the half hour, waited for Ravi to confirm the exact line on a video call, and removed the policy at 9:41am.

On the ObserveOps view, the router dropped back into line with its baseline. The compliance check turned green. The latency alerts faded one by one, like someone switching off the lights in an office after a long day. Jasmine called back eleven minutes later.

"They're printing again," she said. "The word 'penalty' has left the building."

Their IT head asked for a copy of the incident summary, and Faizal sent it with the compliance report attached. For once, the customer and the operator were looking at the same evidence, and nobody needed the word again.

Faizal wrote the post-incident note in ServiceOps. Cause: rollback step not completed after planned maintenance. Detection: correlated incident and configuration compliance. Fix: policy removed under an emergency change. Follow-up: engineers attach a configuration compliance report before closing any maintenance change, and temporary changes get their own removal ticket with a named owner and a time limit.

Then he added one more line, for Ravi: "Go back to sleep. Step nine is done."

Wani was already printing a sticky note for the NOC wall. It read: "Temporary is a promise, not a duration."

Key points: the problem and how Motadata solved it

FAQ

What is Motadata ObserveOps? Astiostech, a Malaysian technology company, implements Motadata ObserveOps, a unified observability platform. It covers hybrid infrastructure, application and network monitoring, traces, log monitoring, real user monitoring, SLO monitoring and network configuration compliance, with AI anomaly detection, cross-domain correlation and alert-noise reduction.

Can Motadata ObserveOps detect configuration drift on network devices? Yes. Motadata ObserveOps includes network configuration and compliance, which compares a device's running configuration against its approved baseline and highlights differences.

How does Motadata help a telco NOC with alert storms? Motadata ObserveOps uses alert-noise reduction and cross-domain correlation to group related alerts into a smaller number of incidents, so engineers can focus on the likely cause.

How do Motadata ObserveOps and Motadata ServiceOps work together? Alerts in Motadata ObserveOps can automatically create incidents in Motadata ServiceOps, with monitoring context attached. ServiceOps manages the incident, related changes and approvals, including emergency changes.

What does Astiostech do for Motadata customers in Malaysia? Astiostech implements Motadata products: discovery, architecture, deployment, integration, training and ongoing support. Read more on the Motadata page.

Want to know more?

Book a 20-minute monitoring health check

Disclaimer

This story is fiction. Kenari Telco, its customer and every person in it are imaginary. The product capabilities shown are taken from Astiostech's published product information and capabilities confirmed by Astiostech.