番茄社区

With the rise in patient-facing psychiatric chatbots powered by artificial intelligence (AI), the potential need for patient mental health data could drive a boom in cash-for-data scams, according to mental health experts.

A recent example of controversial data collection appeared on Craigslist when a company called Therapy For All allegedly posted an advertisement offering money for recording therapy sessions without any additional information about how the recordings would be used.

The company's advertisement and website had already been taken down by the time it was highlighted by . However, revealed the company was seeking recorded therapy sessions "to better understand the format, topics, and treatment associated with modern mental healthcare."

Their stated goal was "to ultimately provide mental healthcare to more people at a lower cost," according to the defunct website.

In service of that goal, the company was offering $50 for each recording of a therapy session of at least 45 minutes with clear audio of both the patient and their therapist. The company requested that the patients withhold their names to keep the recordings anonymous.

The website stated that the company was committed to providing "top-quality therapy services" for individuals. And the recordings would be used by its research team "to learn more about approaches to mental healthcare."

There were no further details about how the company planned to use those recordings, and they did not respond to requests from 番茄社区 to clarify their business model.

However, experts suggested this is just one example of an unexpected incentive created from the growth of AI in mental healthcare.

John Torous, MD, director of the digital psychiatry division at Beth Israel Deaconess Medical Center in Boston, told 番茄社区 that misuse of patient data related to AI models is an "extremely legitimate concern," because large language models are only as good as their training data.

"Their chief weakness is they need vast amounts of data to truly be good," Torous said. "Whoever has the best data will likely have the most practical or -- dare I say -- the best model."

He added that high-quality patient data is likely going to be the limiting resource for developing AI-powered tools related to mental healthcare, which will increase the demand and, therefore, the value of this kind of data.

"This is the oil that's going to power healthcare AI," Torous added.

"They need to have millions, if not billions, of examples to train on," he added. "This is gonna become a bigger and bigger trend."

Torous highlighted that mental healthcare technology companies have already been caught crossing this line with unethical use of patient-facing AI tools.

For example, in early 2023, nonprofit mental health platform announced that it used OpenAI's GPT-3 for roughly 4,000 people without their informed consent. The announcement, which came from the , highlighted the lack of understanding around ethical concerns related to patient consent from these companies, Torous said.

Another example, he noted, came when users of the text message-based mental health support tool learned that the company was called Loris.ai. Eventually, the company ended the relationship after .

While concerns around patient data persist, there are also notable clinical implications for patient care and safety, according to Jacob Ballon, MD, MPH, of Stanford University in California.

"I would not want someone to do AI therapy on its own," he told 番茄社区, adding that people seek out psychotherapy to help with complex, sometimes life-threatening, mental health conditions. "These are serious things that people are dealing with and to leave that to an unregulated, unmonitored chatbot is irresponsible and ultimately dangerous."

Ballon added that he doesn't think AI models are capable of producing the nuanced expertise needed to help individual patients address their unique mental health concerns. Even if a company could train their AI chatbot on enough high-quality patient data, it would not be able to appreciate the complexity of each patient, he noted.

Despite those concerns, Torous thinks there will be growth in companies attempting to train AI models on patient data, whether it is collected ethically or not.

"There's probably going to be this whole world where I wonder if patients are going to be pressured or cajoled or convinced to give up their [personal health data]," he said, predicting that the market for patient mental health data will only continue to grow in the coming years.

Comment