A good story that explains why chasing “high users” or “frequent fliers” rather than a systems approach to care improvement is likely to be ineffective.
Hospital spending and ‘regression to the mean’ — a cautionary tale - STAT
Do we have enough low-cost biostatisticians or expertise distributed across NZ to conduct these evaluation trials as often the continuous quality improvement techniques used to justify changes in operational behaviours/processes don’t provide sufficient rigour to actually prove effectiveness compared with the techniques used for evaluating trials described in this articles.
opengraphobject:[360513095196672 : https://www.statnews.com/2020/01/08/regression-to-the-mean-health-policy/ : title=“” : description=“”]
Interesting question and I am pretty sure the answer is that we don’t. The Health Research Strategy points this out and we are working on how we build infrastructure capability and capacity to support this with HRC and MBIE. That won’t fix the problem tomorrow though.
Have you got any ideas/suggestions?
opengraphobject:[360513095196672 : https://www.statnews.com/2020/01/08/regression-to-the-mean-health-policy/ : title=“” : description=“”]
The traditional problem is that it’s $100k+ usually to commission a research agency/institution to do a thorough evaluation of a programme, which is usually retrospective and not part of a research assessment by design. Therefore many evaluation reports just provides a logical answer to the questions using the answers given by the commissioning agency. You can get by with a $30k commissioned report by an “expert” which will tell you what you want to hear and not much else.
The latest NEAC standard released before Christmas has a lot of best practice theory, and checklists of things to consider, but it doesn’t give the tools that people need to go, and it’s designed for providing guidance for scientific or research applicability to use the right tools for the right circumstances.
I’m wondering as part of nHIP later tranches , would a framework, followed by technology driven decision support assist researcher/CQI developers to ‘calculate’ service and evaluation requirements so everyone’s on the same level of consideration. A few years ago, the government (Treasury) tried to implement a ‘better business case’ framework via a 2003 Word doc template. Could we do something like that but smarter (using dynamic calculators, validated tools and decision trees) for 2020?
opengraphobject:[360513095196672 : https://www.statnews.com/2020/01/08/regression-to-the-mean-health-policy/ : title=“” : description=“”]
Sounds better than what we have - one the problems I have found in this is that much of the information that is gathered tends to the subjective. An example is the prioritisation of projects - while there are a lot of frameworks for prioritising what to do when - it comes down to someones - largely subjective opinion - about how much it will cost and how long will it take for a given scope. To move this there is a need to track very transparently a lot of information that is hidden at the moment about the performance of projects - this is an interesting article about how culture eats data for breakfast:
https://hbr.org/2016/07/how-ceos-can-keep-their-analytics-programs-from-being-a-waste-of-time
opengraphobject:[360513095196672 : https://www.statnews.com/2020/01/08/regression-to-the-mean-health-policy/ : title=“” : description=“”], opengraphobject:[360513156734976 : https://hbr.org/2016/07/how-ceos-can-keep-their-analytics-programs-from-being-a-waste-of-time : title=“How CEOs Can Keep Their Analytics Programs from Being a Waste of Time” : description=“Four challenges to anticipate.”]
Nice article, and I tend to agree regarding the lack of consistency with chief analytics officer roles, if any in major institutions.
Regarding subjective opinions, is a “simple way” you could be objective is applying internationally recognised metrics , such as organisations must calculate Net present value (NPV), Enterprise Value and Terminal Value calculations on any initiatives in healthcare. However, unless you’ve done an MBA or equivalent, those three terms might not be something the standard healthcare innovator, CQI specialist or manager would have an idea or understand the relevance on how to apply when submitting a proposal or evaluating an initiative.
I see the Treasury has recently refreshed their template to a 2018 version with several ‘case’ improvements. For those who want reference https://treasury.govt.nz/information-and-services/state-sector-leadership/investment-management/better-business-cases/guidance
What I’m suggesting is that applications for nHIP access, or ability to guide healthcare innovation standards in NZ is to create a framework with similar themes that document the best practices, suggest guidelines, and calculates the more ‘difficult stuff’ in evaluation metrics such as suggesting research design, including types of evaluation or business case improvements that would standardise the quality of healthcare improvements, so we can be sure a ‘framework compliant’ research outcome can be transferrable or scalable.
There’s a lot of wastage in good-intentioned healthcare improvements but as the first article has explained, but what can we do to collectively learn and build on learnings, rather than consistently repeating the same mistakes, just in a different department/ward/DHB/System OR as the article states: regression to the mean - i.e. doing nothing specific on the intervention would still have resulted in the same outcome so a whole lot of efforts (and resources) results in ineffective value realisation
opengraphobject:[360513095196672 : https://www.statnews.com/2020/01/08/regression-to-the-mean-health-policy/ : title=“” : description=“”], opengraphobject:[360513161871360 : https://treasury.govt.nz/information-and-services/state-sector-leadership/investment-management/better-business-cases/guidance : title=“BBC guidance” : description=“There are a range of types of business cases depending on the decision being sought, the nature of programme or project and the associated scale, risk and uncertainty.”]
Agree - the unfortunate reality with an NPV though is that it is based on assumptions (subjective)…
The most interesting part of the article I saw was that Chief Analytics Officers only last 18 months, on the basis that they tell their colleagues and the collective what they don’t want to hear…
opengraphobject:[360513095196672 : https://www.statnews.com/2020/01/08/regression-to-the-mean-health-policy/ : title=“” : description=“”]
This is not necessarily a surprising outcome - it is worth the review of the whole history of reimbursement and “bounce backs”. A lot of time/effort in health care data science has been spent predicting those who will be readmitted within 30 days. The evidence that any intervention works to prevent this is sparse - and several large studies of the US healthcare system and the impact of changing reimbursement by reducing 30 day readmission have shown it was associated with increased mortality. Just because computers and algorithms may allow us to predict something, doesn’t mean its a useful thing to predict. Similar to my rant about the mammogram study, this emphasizes that efforts to predict and then intervene in healthcare needs to be ensconced in a higher standard of data supporting the intervention is helpful in any way (or at least not harmful).
opengraphobject:[360513095196672 : https://www.statnews.com/2020/01/08/regression-to-the-mean-health-policy/ : title=“” : description=“”]