Motadata

Motadata: The Results That Never Reached the Ward

A fictional story. The characters, organisations and events are imaginary. Motadata ObserveOps and Motadata ServiceOps are real, and Astiostech implements them in Malaysia

Motadata: The Results That Never Reached the Ward

In 30 seconds

• Setting: morning ward rounds at a large public hospital. The laboratory system and the ward system both look healthy, but no new results have reached the wards since 3am.

• What Motadata did: Motadata ServiceOps grouped the wards' calls into one incident and its CMDB showed the hidden link between the two systems; Motadata ObserveOps showed why that link had stopped.

• Business point: Astiostech, a Malaysian technology company, implements Motadata ObserveOps, a unified observability platform, and Motadata ServiceOps, an ITSM, ITAM and patching platform. Together they help hospital IT teams find hidden dependencies before clinicians feel them.

Clinical systems that depend on things nobody can see

A hospital runs on dozens of connected systems: patient registration, laboratory, pharmacy, radiology, billing and ward systems. Between them sit interfaces and integration servers that pass orders and results from one system to another. Clinicians never see these links. They only see when something does not arrive.

That creates a particular kind of risk:

• Every main system can look healthy while the link between them has quietly stopped.

• Clinicians notice first, usually during ward rounds or clinic sessions, and report problems by phone.

• IT teams lack a map of dependencies, so it takes time to work out which system is behind the symptom.

• Small technical issues have clinical consequences, such as delayed results or medication orders.

• The fix itself needs clinical judgement, because restoring a system can create new risks for patient care.

The illustration

At 7:12am, the first call reached the IT office. At 7:14am, the second. By 7:20am, Wei Jie had stopped answering the phone and started looking at Motadata ServiceOps instead.

Six wards had called. Medical, surgical, two paediatric wards, the cardiac unit and the emergency observation area. The words were different ("no results", "lab system down?", "Dr Kumar is asking again") but ServiceOps' incident clustering had folded them into one incident: laboratory results not appearing in the ward system.

On the medical ward, the round had already stalled. Dr Kumar's team was standing at the third bed of twenty, waiting for a full blood count that the lab insisted it had released at 4am, and that the ward system insisted did not exist.

Wei Jie was the hospital's IT head. His office was beside the server room, which was also where the hospital kept its spare boxes of gloves, three broken wheelchairs and a filing cabinet nobody had opened since 2014. He had asked for the boxes to be moved twice. Each time they had been moved, and each time, by some law of hospital physics, they had come back.

He checked the obvious first. On Motadata ObserveOps, the laboratory system was green: tests were being processed, and results were being saved. The ward system was green: doctors could log in, see patients and write notes. Both sides were healthy. They just weren't talking.

He opened the CMDB in ServiceOps and searched for the ward system's dependencies. There it was, in a box he had nearly forgotten existed: the integration server. Every lab result travelled through it on its way to the wards. So did pharmacy orders and radiology reports.

ObserveOps had the integration server's story ready. Disk space: almost full. Message queue: growing since 3:02am, when the disk had crossed the point where the interface software stopped sending and started waiting. Log monitoring showed the reason. The server had been writing enormous, detailed debug logs since Tuesday, when a vendor engineer had switched on extra logging to troubleshoot a minor issue.

"Just for one day, boss," the vendor engineer had said on Tuesday.

It was Friday.

Forecasting had even drawn the line on Wednesday, showing the disk filling by the end of the week. The alert had opened a ticket in ServiceOps, which had been routed to the vendor's queue, where it was waiting politely for Monday.

The fix was technically simple. Turn off debug logging, clear the old logs, free the disk, and the interface would resume. Wei Jie's hand was literally on the mouse.

Then he stopped.

The integration server was holding four hours of results in its queue. The moment it resumed, those results would pour into the ward system all at once. Doctors halfway through their rounds would see a list that was changing under their eyes. Worse, some of those results might be urgent: a dangerous potassium level, a positive blood culture. If a doctor assumed "no result" meant "no problem" for the past four hours, a patient might already be at risk.

ObserveOps could tell him the disk was full. It could not tell him which patient was waiting for which result. The lab could.

He called Dr Aishah, the head of the laboratory.

"Before I fix it," he said, "I need your help with something that isn't IT."

Dr Aishah understood in a sentence. The lab switched to its downtime procedure at once: urgent and critical results from the past four hours were phoned directly to each ward, and the lab printed a list for every ward of what had been completed since 3am. Wei Jie sent a short message to all six wards: results delayed by a technical fault, critical results being phoned now, full results will appear shortly, please check the printed list before acting on the ward system.

It was slower. It was paper. It was exactly what the downtime drill had practised twice a year, and for once nobody grumbled about the drill.

Only then did he raise an emergency change in ServiceOps, with the ObserveOps disk and queue data, the CMDB dependency and the lab's confirmation attached. The deputy director approved it. The vendor engineer, reached by phone and suitably apologetic, talked him through switching off the debug logging.

At 7:58am, the disk cleared. The interface resumed. On ObserveOps, the message queue began to fall, steadily, then quickly. By 8:15am the backlog had drained, and the lab confirmed that every critical result had been phoned and every printed list delivered. One result had needed that phone call: a patient on the medical ward whose potassium level was high enough that Dr Kumar changed the treatment plan before the round reached the next bed.

Dr Kumar sent a one-line message: "Got the results. Also got the phone call. Thank you for the phone call."

Wei Jie wrote the post-incident note in ServiceOps. Cause: debug logging left on after vendor troubleshooting filled the integration server's disk. Detection: forecast on Wednesday, ward calls on Friday. Fix: logging switched off, disk cleared, after the lab's downtime procedure was in place. Follow-up: alerts on clinical integration servers go to the hospital's own on-call engineer, not the vendor's queue; any temporary vendor change gets its own change record with an end date.

Then he walked to the server room, moved the gloves away from the integration server's air vent, and stuck a note on its rack: "This box is boring. Keep it boring."

Key points: the problem and how Motadata solved it

Common hospital problemWhat happened in the storyHow Motadata helps
Main systems look healthy while the link between them failsThe lab and ward systems were both green; the integration server was notMotadata ObserveOps monitors infrastructure and applications across the whole chain, not just the main systems
Hidden dependencies slow down diagnosisThe CMDB showed the ward system depended on the integration serverMotadata ServiceOps IT asset management and CMDB record which services depend on which systems
Clinicians report the same problem many timesCalls from six wards clustered into one incidentServiceOps AI ticket categorisation and incident clustering group related reports
Warning signs reach the wrong teamThe disk forecast opened a ticket that waited in the vendor's queueObserveOps forecasting and log monitoring raise the signal; ServiceOps routing decides who acts on it
A technical fix can create clinical riskThe lab phoned critical results before the backlog was releasedTools show the technical picture; clinical staff decide what patients need first
Temporary vendor changes are forgottenDebug logging stayed on for three extra daysServiceOps change management keeps temporary changes on record with an owner and an end date

FAQ

What is Motadata ObserveOps? Astiostech, a Malaysian technology company, implements Motadata ObserveOps, a unified observability platform. It covers hybrid infrastructure, application and network monitoring, traces, log monitoring, real user monitoring, SLO monitoring and network configuration compliance, with AI anomaly detection, cross-domain correlation, forecasting and alert-noise reduction.

Can Motadata help hospitals monitor integration servers and interfaces? Motadata ObserveOps monitors hybrid infrastructure, applications and logs, which can include the servers and software that connect clinical systems. Coverage for a specific hospital environment should be confirmed during a health check.

How does a CMDB help hospital IT teams? A configuration management database (CMDB) records IT assets and the relationships between them. Motadata ServiceOps includes IT asset management and a CMDB, so teams can see which services depend on which systems when something fails.

Does Motadata replace a hospital's downtime procedures? No. Motadata helps IT teams find and fix technical problems faster. Clinical downtime procedures, such as phoning critical results, remain a clinical responsibility.

What does Astiostech do for Motadata customers in Malaysia? Astiostech implements Motadata products: discovery, architecture, deployment, integration, training and ongoing support. Read more on the Motadata page.

Reach out to our team

Book a 20-minute monitoring health check

Disclaimer

This story is fiction. The hospital and every person in it are imaginary. The product capabilities shown are taken from Astiostech's published product information and capabilities confirmed by Astiostech.