The two key systems supporting clinic care in the lower South Island are currently down.
I’m not sure if this is SI wide, but suspect it might be. Anyone got any more info?
More details (it seems to be over):
And another one, this time yesterday at Capital & Coast:
Interesting absolute silence here on the eHealth Forum about it. If the culture of fear in Health NZ | Te Whatu Ora (we all know it is very real) is getting in the way of posting about this, we do have the Discussions > Anonymous category.
Valuable / interesting Topics can then be moved to a more general area by @moderators if useful, as this allows more natural discussion.
I noticed that Nathan. Silence. I don’t think it’s fear though. For me? Just not sure what to say that adds value to be honest.
3 Likes
I would be keen to know the details - so far only a suggestion of a modem failure??
Also, in Southern there is about to be the roll out of e-prescribing for outpatients (long overdue). However, I see that the private provider is based in Ireland with a NZ registered subsidiary. I hope the security concerns and retention of data issues plaguing those who use manage my health have been carefully addressed.
Silence is the status-quo, perceived self-preservation. On the contrary, What we are not changing, we are choosing. As registered professionals, we all have a duty of care to rights, ethics, mandates, health quality and safety. Just my 2 cents.
1 Like
I’m not sure that the silence is related to fear, rather with 6000 odd applications across the country, there are outages every other day. If we blew the whistle every time, it would become the boy who cried wolf.
You make a good point, Colin.
However, in this case the systems down were THE two key clinical systems which enable/facilitate almost all health work.
What is remarkable this time was the clinical impact.
There must be a guideline for notifying affected people about an outage, e.g., if longer than a few minutes plus mission critical to specific staff requires notification or if long duration that isn’t mission critical to clinicians doesn’t require notification….
2 Likes
We (clinicians) do get an email after a few hours suggesting “manual processes should be stood up”. However no one knows what the manual processes are, and there’s no obvious place to find out (and the email that tells us to do it doesn’t eg link to a place to find out…).
Fortunately after a few of these events it is now apparent that we can find a lot of the prior clinic letters within the escription app under history, and we have saved an eclair bookmark somewhere for results. PACS seemed unaffected so that helped. There were however a lot of very tidy offices by the end of the day (followed by trying to catch up on a days work that evening etc).
If I were a suspicious man I might wonder if not publicising the workarounds reduces the risk of them failing under the increased workload…
Ben
If it was hardware failure then there’s no response that would be acceptable - replica systems / VMWare or Hyper-V to take over the load - hardware isn’t a huge expense anymore so shouldn’t be an excuse. Even if it’s replica of the ‘core’ systems and the rest are backed up regularly then the ability to recover shouldn’t be the question. as for @colino the fact that there are 6000 applications is absolute insanity - how many of those are duplication? Hospitals don’t do a whole lot of anything ‘different’ - they all deal with patients / drugs / radiology etc - Is there a list of all the applications in use throughout Health NZ ? - the logical path for moving to Health NZ over the DHB’s - would have been to FIRST find all the applications doing the same job, find the one that does it ‘the best’ as perceived by the people using it - and roll it out and remove the other applications. THEN it makes it easier to centralise support when the applications are standardised… At the moment the only thing achieved with the change of name - is exactly that - the change of name. Moved from Wellington DHB → Health NZ Wellington.. If there is a list - can we get hold of it under OIA? or can someone share it to those that are offering to help?
Hi Matthew, this is exactly what the digital team in HNZ have been doing over the last 2 years.
It’s a big job but my understanding is that there is now a far better understanding of the landscape which in turn will allow smarter decisions to be made.
2 Likes
Yes we are doing this where it makes sense but it’s not quite as simple as portrayed.
- we can’t just choose the best one and roll it out removing the rest - we have to ensure govt procurement processes are followed.
- the migration or presentation of historical data can be quite costly. Nearly all of the time we can’t just leave data behind.
- the harmonisation of business processes and clinical configuration can vary from fairly simple to seemingly impossible. It’s not the case that clinical people all agree or that ‘people will fall in line’. We are not a tech company and without well managed business change tech deployments fail.
- we must be accountable for how we spend money.. factoring in where the investment is in its lifecycle, cost benefit analysis, where does this fit with the long term roadmaps etc - it has to be justified and make sense.
- you’d think the vendors would offer economies of scale for us consolidating on their instance of ‘x’ product but sadly sometimes it’s more expensive if we remove the competition
- we don’t always still have the people and documentation with the relevant systems knowledge in detail. We’ve been through several restructures and like any company where this has happened it can introduce challenges
11 Likes
Would that have not made sense to do BEFORE pulling the rug out from under the existing DHB’s?
If they were accountable for how money was spent - then how can the likes of Orion hold everyone to ransom? A private entity - responsible for the behind the scenes of millions of patients - when they were paid a fortune for it - and very little changed inside for a very long time. That’s the biggest headache I have - the big guys are in control - the little guys who want to be involved and help are ignored and ideas totally thrown away. I’ve been to 5 of the conferences where everyone sat around talking about solutions to issues - and then the same questions being asked the following year, if we want to discuss accountable for how money is spent - there seems to be copious amounts of it being wasted on meetings. It is that simple - if you did choose the best one and roll it out - the spend on running the other bunch of applications doing the same thing would drop drastically - as would the man hours to keep one system running instead of multiple. 2 years to know what applications are in use on the existing systems? was there one person walking around the hospitals writing down what was in use? You are right - Health NZ isn’t a tech company - so why did it have 3000 staff in the IT department? Given it outsourced piles of pieces - why didn’t they outsource the IT portion to a Tech company to deal with? I keep seeing the gripes about money being spent, and staff being lost etc - it shouldn’t take an IT staff of that size. I get it - it was a mess of sprawl - so the costs involved in slashing out some of the bloat in replicated software tools should have been a good one to start with. No-one in Health NZ appears to want to look outside the box and risk any of the little guys actually having some ideas on how to help. The little guys have made it work on very little - so maybe look at how it’s been done and if it’s scalable - because Health NZ is meant to be for the healthcare of New Zealanders - and the technology is a tool for them - it’s not the actual thing getting patients health better - that’s the Drs and Nurses and support staff with feet on the ground and right at the coal face.. If the tools aren’t working - get rid of them - wasting money sitting around discussing how to get rid of applications without wasting money - days / weeks / months of meetings involving how many people?
Whew, that is indeed a big job. It’s not only about the ‘landscape’ but the ecosystem that the landscape is part of.
@david.vink so much is involved in decisions about what to keep (and deploy nationally) and what to update (and do so nationally) and what to remove (without breaking the ecosystem so that nothing works). Governance is needed to help make these decisions. The complexity of the ecosystem makes a lot of the work quite difficult, where differing but similar software is required (even if it looks like a duplication, it may not be). @bendun as you say, we are way past the analogue default when a product goes down for an extended period. No-one remembers, and younger staff never knew the analogue ways of working (more than ten years ago I watched a young person hold a credit card machine as though it was a poisonous snake when the system went down in Bunnings). We need new guidelines for when “the system” isn’t working.
3 Likes
I understand that it’s incorrect to say the SI-PICS adn HCS were ‘down’.
I believe that they were unavailable to ‘Southern’ (ie. Otago and Southland) due to coms problem.
One thing I’ve learned over the years is that there is no such thing as “the IT department”. There is an overarching administrative group that acts as a bureaucratic container for dozens or hundreds of different specialties and silos, from security to infrastructure to applications to data services to compliance and so on. Multiply that by 20 organisations, and bringing them all into alignment is a long term project.
As for figuring out which tools are “in use”… I’m only a data user, and I have programmes I use once every few years. I have dozens of programmes I’m unlikely to use again. I have a few I use regularly. And I’ve had colleagues who when asked “what browser do you use” gave the answer “the one the DHB gave us”. They didn’t even know it was Explorer, let alone version number. That wasn’t something they needed to know. Someone walking around a hospital will pick up none of that.
But maybe things would be less fragmented if we hadn’t underfunded health information services for a few decades. Like the rest of the sector. And it also might have enabled more auditing of HISO compliance (including for third party systems), which might have prevented one or two recent data breaches.
2 Likes
I stand corrected! Have edited the title of this topic to reflect that.
It would be great to have some more open disclosure about that so we don’t all speculate wildly.
1 Like
2 years to know what applications are in use on the existing systems?
Not just 2 years; this housekeeping task is attempted - and often failed - every time a major PAS upgrade occurs within a district or region, as a small number of those thousands of systems are integrated in some way with the PASes. Some people get nervous if this isn’t clearly documented.
However it really needs to be stated for the record that those ‘6000 systems’ do not represent 6000-odd distinct systems per se, but in fact is a laundry list of every entity that someone somewhere was willing to put a label on.
I’m talking about every flavour of each system interface, every version of every MS-Office product still in use somewhere in the organisation, every MS-Access database that someone whipped up in the past 30 years, the odd spreadsheet, and every collection of things that one might tentatively call a ‘system’.
The vast majority of those thousands of systems will be unfamiliar to you all, and virtually everyone else in the organisation. They do not necessarily represent technical debt, nor are they a good yard stick by which to measure the consolidation effort ahead of us.
In reality, the number of systems that actually matter to anyone is just a few dozen per district. The rest are just line items on a poorly maintained risk register to keep project managers up at night, and for pedants like me to eventually cross off the list as ‘permanently retired’ once the last person who remembers its name leaves the org.
5 Likes
This has been an interesting discussion, and it highlights how quickly mixed messages can arise when there isn’t clear, shared information about what actually went wrong.
Before I left, work was underway to strengthen the service and system information held in Kapehu, including the mapping between systems, services, and supporting teams. That kind of work is critical in an environment as complex and federated as Health NZ, particularly when issues affect access rather than a full system outage.
From a practical perspective, if you do experience an issue, logging an incident through the Kapehu portal with a clear description of the impact is still one of the most effective ways to ensure it is routed to the right team and investigated promptly. Consistent incident data also helps build a clearer picture over time of where fragility exists.
More broadly, recurring incidents or access issues with the same systems should provide useful signals, not just for immediate remediation, but for longer-term decisions about stabilisation, redesign, or replacement where services are no longer fit for purpose.
Greater transparency about causes and impacts would also help reduce speculation and support shared learning across the sector.
4 Likes