Machine learning and test results - one way to tackle GP burnout?

Discussions
,
1

One of the drivers of GP burnout is dealing with what some describe as a never ending stream of in-box items that they have to apply their minds to. Sometimes this is correspondence with patients, sometimes it’s new work rolling down the hill from the secondary health system, but often GPs talk about it with reference to test results.

Is there a space in here for rules based/machine learning informed systems that would triage the results so that GPs only had to look at those that were of concern?

(apologies if this is treading old ground, I thought I had read about a pilot of this on NZDoc but couldn’t find anything).

I’ll put some thoughts at the end about some of the challenges from a efficacy/implementation point of view, but from a policy perspective I’m interested in how this could be done while addressing very legitimate concerns about liability when something goes wrong.

It seems to me that the Govt could either develop a rules set and then regulate so that Doctors who were using the rules set correctly (including patient records being up to date/code correctly/histories being taken) had some protection in the event of edge cases where there was something wrong.

Alternatively the Govt could set up a standard by which private sector rule sets could be approved.

The idea behind either of these being that you have a rules base that would replicate the decision making process that a normally competent Doctor would go through when reviewing test results. And if a normally competent Doctor wouldn’t have picked up a red flag, because the situation is an outlier, then it seems reasonable that they would be protected.

To be clear - then intent here is not to positively identify health issues, rather it’s to positively identify where the tests and history combine in such a way that a normally competent Doctor would feel no need to follow up.

Efficacy and practical issues:

  • Can we expect to have a robust enough database of historic test results combined with comprehensive and correctly coded patient histories? (I’m working from the assumption that the current work on data cleansing/normalisation/harmonisation is completed)

  • Are there populations which have poorer records or for whom less is understood due to historic underserving by the health system?

  • What happens to liability when patient history items have been incorrectly coded? What happens when those coding errors have been done by a GP three or four GPs ago as the patient has moved around a lot?

  • I’ve specifically not called this AI (partly because I think term is rampantly misused) but is there also a risk that this would be constantly at risk of a public scare around AI/machine learning that would see them close this down in a knee jerk reaction?

  • A GP reviewing a test result for a patient that they know and have ordered the test for is one thing; a GP reviewing a test result that was ordered by someone else (eg a locum or hospital specialist) seems different somehow - is the reason for the test a needed part of the information set?

There are others but I’m keen to hear from people thinking or working in this field.

Note - while I contract to the RNZCGP this is not a specific project on our runway at the moment, this is more just a policy question that’s been hanging around in the back of my mind for a while that I’ve wanted to kick around with people who have more knowledge.

Tom

[edit - found the NZ doc article - https://www.nzdoctor.co.nz/article/news/summer-hiatus/jamie-and-his-band-bots-scaling-paperwork-mountain]

1 Like
2

Hi Tom

This is a topic very close to my heart as I led several years of development under a precision driven health programme. https://precisiondrivenhealth.com/project/feasibility-of-analysis-of-lab-result-patterns-for-patient-results/

However, I have since left Vensa and the programme is still underway in phase five, so commercial sensitivities are still present . However, I’m happy to be DM for a discussion around the themes of the learnings that impact policy and other spin-out implications discovered during this programme. I handed over at the end of Phase 4, so most of the learnings are being applied to ‘phase 5’ automation, which has commercial confidentiality implications.

Basically, most of the practical issues has significant impact on insurance and liabilities either on the provider or on the patient’s future potential for obtaining or being provided insurance policies. We are way off from true automation given the legalities of responsibilities laid out in Chapter 5 of 'Cole’s Medical Practice in New Zealand means you are fairly responsible for all things reviewed.

I can run through other scenarios in a DM.

Kind regards

Samuel

1 Like
3

Thanks Samuel,

I’ll have a read through the information you’ve linked to and check Coles too and will get back in touch once I’ve digest that.

Really appreciate your help here,

Tom

4

Hi Samuel,

I’ve read through the Coles chapter and had a look at the project summary you linked to as well - thank you for both of those.

It certainly looks like uptake of such a tool, assuming it can be designed and integrated into the data systems it relies on, would rely a lot on how comfortable GPs can be that both the tool is reliable and that reliance on the tool is seen as a reasonable course of action by the HDC or other reviewing bodies.

The project looks like it is trying to offer a much fuller service that I was thinking about, which of course would make it more valuable, but also harder to complete. In my mind the rules would simply be trying to identify cases that were within normal ranges for that patient and so did not require any additional actions (other than filing and potentially notification).

The project you contributed to is looking to also identify when action is needed and to supply guidance/advice back to the patient. Positive identification (this is what is wrong with you) seems a much higher bar to pass than negative identification (there is nothing wrong with you).

I’d love to hear more about themes/spin-out implications if you have the time and I appreciate that you are constrained by commercial sensitivity.

Also - earlier this month you provided me with a list of linkedin contacts for portal developers and you were quite right, the level of response from linkedin has been really heartening.

So thank you for that and for being open to having this conversation,

Tom

1 Like
5

@Tom_Broadhead , I did some work in this area a few years ago. One of the struggles was establishing a context for a test result and for the algorithm to interpret the result within that context. For example, if I, as the physician, had made a false positive diagnosis of CHF and the appropriate test was correctly negative. The correct action for the algorithm is not to file the result but to recall the patient to have another think about the cause of their breathlessness. Without a context, the test can not be interpreted by an algorithm, or tragedy was seen to be inevitable if we attempted an interpretation in isolation.

For this reason, we focused on renal function as you could algorithmically interpret the result within the context of the previous results. So how had this result changed relative to the previous ones?

Open to bright ideas, though from bigger brains than mine.

1 Like
6

Hi Tom

Dropped you a message, but in the public forum, I wanted to highlight one particular learning where ‘machine’ generated rules will never do… understanding of laboratory parameters need to have clinical reasoning applied.

‘Normal’ range within a laboratory result is usually a result of a 95% confidence reference range of a sample of the population reviewed as normally expected for that particular assay/test parameter. This is reviewed by a quality committee based on the laboratory, the laboratory devices/test machines, the assay chemicals used or technique applied. What is considered normal will depend on where you are in the country, and what lab you go to. This is only the laboratory medical science component of the issue.

Normal results may change in units over time, depending on refresh of guidelines or new clinical standards.

Then you have the clinical and biological variability of the individual. What is normal for one person may be abnormal for another, subject to the co-morbidity and clinical reasons for the investigation, whether you a titrating a treatment or investigating an acute infection/condition. You may also be following a treatment step-pathway. You may be normal but have been in oncology treatment, so anything has elevated risks.

Therefore between labs, testing devices, assay chemicals/techniques, guidelines of the day, clinical intent and biological variability, it comes down to the ordering doctor’s risk of ‘what may be wrong with you’.

While the future may hold some form of AI opportunities, I don’t think people are willing to carry the risk yet when things go horribly wrong. Let’s see where the research may find the grey area people are willing to navigate.

1 Like
7

I totally agree with @SamuelWong. These were all challenges we had to address along the way. If either of you is interested, here is our solution. I hope it’s of some use.

Godfrey, A. J. R., Russell, G. K., & Betz-Stablein, B. D. (2016). Monitoring acute and chronic kidney failure using statistical process control techniques. Quality Engineering, 28 (2), 184-192.

1 Like
8

Hi Tom, Samuel and Greig

To add to the discussion, there are real life examples of where a normal result was filed by a person without understanding the context and there were adverse outcomes, including from memory some that landed with the HDC. Indeed this has happened also when a result was filed ‘by mistake’.

This is non-trivial issue to overcome. I remember heated discussion around the potential of electronic signing off of lab results in the hospital, complicated by the patient moving between providers - ED to home/GP, ED to ward under different team, ward to rehab with different team.

The area of who is responsible for managing the test result- the orderer of the lab test, the now care provider, the patient, and now we can in AI - the responsibility with AI decision making lies where? The designer, the purchaser, the user, the clinical decision maker, etc.

There are also ethical principles involved where the benefit is for one group, the providers, but the harm is to a different group, the patients.

Happy to continue to discuss. Where i worked clinically for many years in the local sexual health clinic, upto 90% of consultations resulted in lab testing. The big time saver was introducing SMS texting of results to patients.

I suppose one scenario that might work, would be include planned action with ordering the results, depending on the result - in other words what the orderer intends to do with a result, something we should discuss with patients as part of consenting to a test (but then that takes time). Then if the lab order is marked as file if X, then the system could do that.

Cheers Inga

1 Like
9

@i.hunter your last paragraph was the conclusion of the PDH research - phase 5 is how to elegantly do that as part of the clinical workflow, but I can’t go to much into the details of ‘how’ that is optimised, consented etc.

I think clinical inbox items needs to work out where AI makes sense for communication layer (e.g. helping write up content to explain to patients the personalised nature of their result), or to suggest planning options for treatment pathways based on the current guidelines, lab results, clinical variability etc, but to do the filing and confirmation processes around the clinical reasoning of what is expected and unexpected results, where unintended harm is introduced, is where the buck needs to stop with the ordering clinician.

2 Likes