Controversy erupts over non-consensual AI mental health experiment

This is is a bit of a bombshell from a non-profit e-mental health provider in the US. Interesting question about whether it is research or not, but my hot hot take is, if you think it needs ethics approval and consent you should get it…

1 Like

That’s what the Unfortunate Experiment was all about in 1987.

The unfortunate aspect is those with power and data custodianship access believes the ends justify the means at all times.

This is something that health system management should have trust but verify approaches to handling data, just like privacy. Ethics have been long ignored in continuous quality improvement and design thinking practitioners because permission means you aren’t genuinely getting feedback without fly on the wall observations.

New tech enables this without anyone being aware of the reasons for ethics.

I went to the same place as well, @i.hunter. Ethics, like management or health informatics, is one of those fields where increasing competence makes one increasingly irrelevant until something really bad happens. Oh, look, it just did.

The project lead was reported to say his error was to discuss on Twitter, not that his error was to push play on this abusive nonsense that deliberately targeted the vulnerable in search of profits for the already rich.

Yet again, disappointed but not surprised.

oof, not a smart move there. Thinking that having a human review the AI generated response before being sent would mean it’s 100% fine is naive at best.

What are you two referring to? Clearly there is a history lesson for many of us to learn!

@NathanK

This was the disastrous experiment at National Women’s on treating cervical Cancer. It led to the Cartwright inquiry and was the key event to the subsequent HDC and patient rights legislation in Aotearoa.

The Cartwright Inquiry

The Wikipedia version is a fuller account in a single spot.

2 Likes

@NathanK

As @Greig said. I had the privilege (and scary) to meet Dame Silvia Cartwright and present on privacy of data issues before her at the annual Cartwright presentation years ago.

@i.hunter, well done. High Court & Court of Appeal judges are truly terrifying, having done several cases and mock courts before them. They are an order of magnitude worse than the QC/KC level for fear-inducing in witnesses, expert or otherwise.