In my experience recently, and from what I’ve seen documented in Kapehu, the organisation no longer has good information on who/what is impacted by many system problems. For urgent communication with the affected people, we previously relied heavily on service delivery managers and the network of embedded system owners who are no longer with us, or have been absorbed into digital services roles.
Previously an unexpected system outage would have triggered a conversation or communique from the system owner directly (or via SDM) to the impacted charge nurse managers, clerical leaders, rostered SMO, etc.
Now it’s just a broadcast email saying “_____ is unavailable. We’re working on it.”, and often sent too late to be of much use to those affected.
So the important detail (X functions are offline for awhile, but Y functions still work, and use Z workaround if you need it) is no longer being communicated it seems.
Ideally, there would be a self-service option for anyone affected by any major system outage (especially after hours), so that they could check a noticeboard or phone a hotline for a pre-recorded “current outages” notification in the absence of any direct communications. This would enable them to determine whether someone else had already reported the issue, how long the issue might persist, and what alternative options might be available. This info helps enormously when trying to decide when to initiate business continuity plans.
…but such proactive information would require someone involved to understand the details of the impact.
Ideally a ‘status’ page - as other organisations have - ISP’s / Power / Water etc - have outage maps and status of systems - if that was hosted externally with the top level stuff public - but only drill down accessible with username / password authentication - that would be a pretty simple way of showing it - even if it was broken down by Hospital to start with - and the list added to as things came up, then as the helpdesk learnt of the issues - they could be posting updates..
The simple fact is that every outage of hospital systems will cause harm, it is only a matter of time. A minute probably not, an hour likely harm, and 24hours probable harm.
In GP land we passed the threshold of being able to practice safely without a medical record in the early 90’s
GP’s have robust systems that do not tend to fault. I my experience of 35years I can only remember 1 loss of our server.
HNZ needs to step up and put urgent work into getting things right s patients don’t get harmed
Every time tech is outsourced (always to one of the big MNCs although there are NZ companies who could manage well if not better) we lose capability and provenance. It makes me sick that govt depts are using AZURE to transfer files to the cloud – not sure how that squares with the patriot Act either. Basically we’re paying them to deskill our workforce and increase our dependence on a foreign private company to hold national and citizen records. The environment is such that no-one will put their head above the parapet and detail the risks or suggest alternatives.
Well said @hamish.luebbers.
People like to make it sound like a catastrophic nightmare. Actually, many of the applications were tightly integrated through a world-class integration engine (Rhapsody.
Very interesting posts to follow. I can’t help but wonder what’s the each organisations business continuity plan with regard to BAU when there’s an outage so there’s not unnecessary disruption to service etc.
I find it super interesting to see how everyone here is responding to these situations. It is great to see everybody exactly knowing where it fails and what needs to be done about it. Having been on the supplier side for all of my life I look at this from the outside and have no vested interest. I do have a lot of questions when I read all of this.
These systems are created over the course of at least 20 years, expecting them to merge in 2 is unrealistic. I am sure there are a bunch of smart people looking at these issues but lack of documentation, digital asset management and group thinking makes this a nightmare. Certainly if you add budget cuts to that mix.
@CBlairD2
I’m assuming that ‘Southern Network’ means ‘Otago & Southland’. Further ‘Site-Wide’ is not defined.(What site(s)?)
ie. SI-PICS is up but a network problem is preventing it from being used in some sites.
If they are not more detailed about problem definition they risk ‘The sky is falling down’ response.
As clinical users of multiple applications who were affected by the most recent prolonged systems outage, where we in our department could not stand up a paper based replacement because real time data is needed, not historical data, it of real concern that there has been no substantial explanation as to what happened, & why it occurred. More than that, there is no sense of any follow up with the users of the applications as to issues encountered, work impact and clinical risk, nor it would seem any attempt / interest in understanding potential for harm.
It would be very interesting and helpful please to have further information on this …
The reply received did give further technical information with the caveat that could change following further investigation. The later part of the statement was left largely unanswered.
