A paper in pre-print where they gave a number (73 teams) of data research teams the same data to analyse along with hypotheses. Some salient quotes:
13.5% of participating analysts claim[ed] that the target hypothesis [was] “not testable”.
Also 82.4% of the variance in results was unexplained.
This picture is most interesting:
Thanks for sharing this interesting read Jon. It reminds me of Enrico Coiera et al’s article ‘Does health informatics have a replication crisis?’
The article finishes with a call for health informatics (digital health) researchers to increase the development of evidence using rigorous methods to ensure that we build an evidence-based discipline. The call is there – it’s up to us to respond. Who wants to join me?
I think there are probably already useful tools and methods, and it is up to researchers to be transparent in what data, tools and methods they used, with non-destructive approaches.
For example when building our COVID risk score the model has been built in Databricks, in which the code itself is examinable (uses notebooks) from the source data (without necessarily exposing this).
I think a number of researchers see value (read commercialisation) in their thinking and so hide their thinking (not such an issue with publicly funded research?).
We do independent security reviews, and there is independent process/organisational review (eg. ISO) but that is expensive so wonder if that is a big blocker?
Jon
I agree that there are many useful tools for analysing research data. The point the authors were making was about the effect of researcher’s thinking on the analysis outcome. There are many ways of knowing (ontologies of knowledge – different from data ontologies) and epistemologies (how we know what we know) that affect our interpretations of data. The context of the researcher influences the analysis as much as the context of the data collection. There are tools for reducing these influences but we have to be vigilant in their use or our influences slip through.
I like that data are collected from technologies – people too often forget that this can be done and can add rigour to the data analysis. The data are only as good as the tool that is used to collect them, and this applies to data coming from the tech too.
This article is a good reminder for researchers to continuously improve their hypothesis/research question development, data identification and analysis and how they report what they learn.
We should note that this paper depends on the work of sociologists and political scientists.
Richard Feynman would surely have had something to say about this. He would have said “I told you so!” If you haven’t previously read his lecture on cargo cult science now’s the time!
As these data described change over time—which most people, including most statisticians, analyse badly—it’s hardly surprising that their interpretations were effectively random. But it gets worse.
You might also note that the hypothesis they were to test is a causal one. As there is no chance of prospective randomisation here, to test causality one would need to use a directed acyclical graph and Judea Pearl’s do calculus, which isn’t even hinted at, as far as I can see. What did I miss?
They might care to repeat the process, but this time with real scientists and a testable hypothesis 
Dr Jo.
Agree that this was quite a difficult hypothesis to test - how immigration changes public support for social policies. It’s an interesting question but not surprising to me that different teams used different models to try and answer it and that the models didn’t agree with each other. There is probably not a simple relationship and the data available are probably not sufficient to answer the question.
One thing to note is that the study itself doesn’t seem to have been through peer review or published in a major journal - which is one of the ways we have of mitigating the problem of people making conclusions that are not justified by the data/analysis.