This is not directly medical, but is a great explanation of some of the technical hurdles of imaging AI. The concepts can apply to imaging libraries and medicine.
https://www.excavating.ai/
In summary, ImageNet did a bad job on the compilation of the input images and labels for people, due to a range of factors such as the lack of review of people categories in WordNet, the way the images were labelled by an outsourced workforce, the way the images were obtained by scraping websites, and some fundamental assumptions about categorisation. Other image databases have had similar problems.
In healthcare, I think the data collection, categorisation and workforce are all controlled and reviewed to a far better standard. Nevertheless, it’s an important reminder that the model’s performance is predominantly determined by the training data. And the article makes an excellent point that the availability of the training data is an essential part of being able to review machine learning systems.
opengraphobject:[360486293831680 : https://www.excavating.ai/ : title=“The Politics of Images in Machine Learning Training Sets” : description=“An investigation into the politics of training sets, and the fundamental
problems with classifying humans.”]
Is there published data supporting that the data curation, labeling, and workforce is to a higher standard? I ask because I would like to have that reference for a paper I am working on. At least as applied to 30 day prediction of readmission, which has been a heavily pursued endpoint in healthcare, the papers published lack transparency to evaluate this, it falls outside of what regulatory standard the US has in place as pertains to SaMD, and most of the work is being done in fairly uncontrolled ways. I have a former classmate who works in radiology image labeling ontology at UCSF who has indicated that this remains a largely ad hoc process without global standards for film labeling, software development, etc. at this time.
opengraphobject:[360486293831680 : https://www.excavating.ai/ : title=“The Politics of Images in Machine Learning Training Sets” : description=“An investigation into the politics of training sets, and the fundamental
problems with classifying humans.”]
I’m not aware of a published data supporting my opinion that it is a higher standard. What I mean is data drawn from actual clinical practice. Labelling done outside of clinical practice, may have the similar quality problems.
opengraphobject:[360486293831680 : https://www.excavating.ai/ : title=“The Politics of Images in Machine Learning Training Sets” : description=“An investigation into the politics of training sets, and the fundamental
problems with classifying humans.”]
Does your former classmate’s radiology department audit and review image labelling? I assumed this was standard clinical practice.
opengraphobject:[360486293831680 : https://www.excavating.ai/ : title=“The Politics of Images in Machine Learning Training Sets” : description=“An investigation into the politics of training sets, and the fundamental
problems with classifying humans.”]
I think its more about interlinking and level and quality of the coding of data. Consistency of standards use, etc.
uploadedfile:181187862528, opengraphobject:[360486293831680 : https://www.excavating.ai/ : title=“The Politics of Images in Machine Learning Training Sets” : description=“An investigation into the politics of training sets, and the fundamental
problems with classifying humans.”]