Sounds like suppressing research, at the cost of public health/safety.

Some people knew what the tobacco companies were secretly doing, yet they kept quiet, and let countless family tragedies happen.

What are best channels for people with info to help halt the corruption, this time?

(The channels might be different than usual right now, with much of US federal being disrupted.)

Which model exactly? What type of therapy/prompt? Was it a completely dated model, like in the article where they talk about a model from two years ago? We have had massive progress in two years.

What did they use for placebo? Talking to somebody without education, or not talking to anybody at all?

I've heard of some more modern research with llms that had a result that Ai therapist was straight up better than human therapists across all measures.

They should do ELIZA as the control or at least include it to see how far we have or haven’t advanced.

The amount of free money sloshing around the AI space is ridiculous at the moment.

I'm quite curious how the placebo in a study like this works.

As much as I tend to defer to experts, you must also be weary of experts whose very livelihoods are at risk. They may not have your interests at heart.

That one was (genuinely) a bug. OpenAI rolled it back. https://openai.com/index/expanding-on-sycophancy/

(But yeah, relying on systems that can have bugs like that for your mental health is terrifying.)

The very fact that the "world class experts" are warning people not to use it means they have already been replaced in most fields that matter.

They didn't feel threatened by systems like cleverbot or GPT-3.5

“I thought I must be doing something wrong, because people on the replika subreddit were constantly talking about how their replika agent was developing its own personality (I saw no evidence at any point that it had the capacity to do this).”

People really like to anthropomorphize any object with even the most basic communication capabilities and most people have no concept of the distance between parroting phrases and a full on human consciousness. In the 90s Furbys were a popular toy that said started off speaking furbish and then eventually spoke some (maybe 20?) human phrases, many people were absolutely convinced you could teach them to talk and learn like a human and that they had essentially bought a very intelligent pet. The NSA even banned them for a time because they thought they were recording and learning from surroundings despite that being completely untrue. Point being this is going to get much worse now that LLMs have gotten a whole lot better at mimicking human conversations and there is incentive for companies to overstate capabilities.

This actually isn't that surprising.

There are psychological blindspots that we all have as human beings, and when stimulus is structured in specific ways people lose their grip on reality, or rather more accurately, people have their grip on objective reality ripped away from them without them realizing it because these things operate on us subliminally (to a lesser or greater degree depending on the individual), and it mostly happens pre-perception with the victim none the wiser. They then effectively become slaves to the loudest monster, which is the AI speaking in their ear more than anyone else, and by extension to the slave master who programmed the AI.

One such blindspot is the consistency blindspot where someone may induce you to say something indicating agreement with something similar first, and then ask the question they really want to ask. Once you say something that's in agreement, and by extension something similar is asked, there is bleedover and you fight your own psychology later if you didn't have defenses to short circuit this fixed action pattern (i.e. and already know), and that's just a surface level blindspot that car salesman use all the time; there are much more subtle ones like distorted reflected appraisal which are used by cults, and nation states for thought reform.

To remain internally consistent, with distorted reflected appraisal, your psychology warps itself, and you as a person unravel. These things have been used in torture, but almost no one today is taught what the elements of torture are so they can recognize it, or know how it works. You would be surprised to find that these things are everywhere today, even in K12 education and that's not an accident.

Everyone has reflected appraisal because this is how we adopt the cultural identity we have as people from our parents while we are children.

All that's needed for torture to break someone down are the elements, structuring, and clustering.

Those elements are isolation, cognitive dissonance, coercion with perceived or real loss, and lack of agency to remove with these you break in a series of steps rational thought receding, involuntary hypnosis, and then psychological break (disassociation or a special semi-lucid psychosis capable of planning); with time and exposure.

Structuring uses diabolical structures to turn the psyche back on itself in a trauma loop, and clustering includes any multiples of these elements or structures within a short time period, as well as events that increase susceptibility such as narco-analysis/synthesis based in dopamine spikes triggered by associative priming (operant conditioning). Drug use makes one more susceptible as they found in the early 30s with barbituates, and its since been improved so you can induce this is in almost anyone with a phone.

No AI will ever be able to create and maintain a consistent reflected appraisal for the people they are interacting with, but because the harmful effects aren't seen immediately, people today have blinded themselves and discount the harms that naturally result. The harms from the unnatural loss of objective reality.

The other lesson here is that general audience news sites are pretty bad at technology coverage.

> It's called family or a close friend.

It's good you are socially privileged, but a lot of people do not have someone close who they can feel secure to confide in. Even a therapist doesn't help here, as a lot of people have pre-existing conditionings about what a therapist is "I'm not crazy, why do I need a therapist?".

Case in point, my father's cousin lived alone and didn't have any friends. He lived in the same house his whole life, just outside London by himself, with no indoor toilet or hot water. A few years ago, social services came after the neighbours called, because his roof collapsed and he was just living as if nothing was wrong. My father was his closest living family, but they'd not spoken in 20 years or more.

I feel this kind of thing is more common than you think. Especially with older people, they may have friends from the outside, but they aren't close with them that they can talk about whatever is on their mind.

Ultimately what therapists do is lead you through an exploration of yourself, in which you actually do all of the work.

I 100% do not doubt the usefulness of therapy for those who are suffering in some way, but I feel like the idea that "everyone should probably have a therapist" is kinda odd - if you're generally in a good place, you can explore your feelings/motivations yourself with little risk.

I mean, the article addresses exactly your point like one line down?

> In a separate interview last week, Zuckerberg said “the average American has three friends, but has demand for 15” and AI could plug that gap.

And I think we should definitely look on this tech with scrutiny, but I think another angle to look at it is: which is worse? No therapy or AI therapy? You mention suicide, but which would result in a reduction in suicide attempts, a or b? I don't have an answer, but I could see it being possible that because AI therapy provides cheaper, more frequent access to mental care, even if it is lower quality, it could result in a net improvement over the status quo on something like suicide attempts.

I am not a lawyer or a doctor or a counsellor. I will gladly claim that an LLM should not replace any of those professions.

It may be able to assist those professionals, but that is as far as I am willing to go, because I am not blinded by the shine of the statistical turks we are deploying right now.

The other rhetorical hazard is to insist that the new thing has to be a 1:1 replacement for a human therapist in the system we have currently. Who's to say it can't take a different form? There are so many ways a text generator could be used for therapeutic purposes.

Agreed. Also, LLMs are already better than 80% of therapists. I don’t think most people understand the delta between a good therapist and a bad one, and how few really are very good.

UK doesn't protect the term psychotherapy, but there's a distinction between services of counsellors and (regulated) psychologists.

For counselling, people are encouraged to choose counsellors accredited by professional orgs like BACP.

I’ve also experimented with them in that capacity. I like to know first hand. I play the skeptic but I tend to feed the beast a little blood in order to understand it, at least.

As a result, I agree with you.

It gives me pause when I stop to think about anyone without more context placing so much trust in these. And the developers engaged in the “industry” of it demanding blind faith and full payment.

That's not exactly a following reasoning to use for LLMs ... In automation studies things are most dangerous just before full automation due to bias. Why tap the brakes when surly the car will do it on its own when that isn't a guarantee.

The therapist controls the extent of the relationship which determines profits. A disinterested third party should be involved.

> The biggest risk is with privacy

No, the biggest risk is that it behaves in ways that actively harm users in a fragile emotional state, whether by enabling or pushing them into dangerous behavior.

Many people are already demonstrably unable to handle normal AI chatbots in a healthy manner. A "therapist" substitute that takes a position of authority as a counselor ramps that danger up drastically.

> 80% benefit at a fraction of the cost

I'm sure 80% of expert therapists in any modality will disagree.

At best, AI can compete with telehealth therapy, which is known for having practically no quality standards. And of course, LLMs surpass "no quality standards" with flying colors.

I say this very rarely because I think such statements should be used with caution, but in this case: saying that LLMs can do 80% of a therapist's work is actually harmful for people who might believe it and not seek effective therapy. Going down this path has a good probability of costing someone dearly.

> AI is not a substitute for traditional therapy, but it offers an 80% benefit at a fraction of the cost.

That... seems optimistic. See, for instance, https://www.rollingstone.com/culture/culture-features/ai-spi...

No psychologist will attempt to convince you that you are the messiah. In at least some cases, our robot overlords are doing _serious active harm_ which the subject would be unlikely to suffer in their absence. LLM therapists are rather likely to be worse than nothing, particularly given their tendency to be overly agreeable.

Does it offer 80% of the benefit? An AI could match what a human therapist would say 80% (or 99%) of the time and still provide negative benefit.

Therapy seems like the last place an LLM would be beneficial because it’s very hard to keep an LLM from telling you what you want to hear. I can see anyway you could guarantee that a chatbot cause severe damage to a vulnerable patient by supporting their neurosis.

We’re not anywhere close to an LLM which is trained to be supportive and understanding in tone but will never affirm your irrational fears, insecurities, and delusions.

In a field like personal therapy, giving good advice 80% of the time is nowhere near 80% benefit on net.

> simply need informed user choice and responsible deployment

the problem is that "responsible deployment" feels extremely at odds with, say, needing to justify a $300B valuation

What we need is the same thing we've needed for a long time now, ethical standards applied across the whole industry in the same way that many other professions are regulated. If civil engineers acted the way that software engineers routinely do, they'd never work again, and rightly so.

> As with many things in relation to technology, perhaps we simply need informed user choice and responsible deployment.

The average person will never have the required experience to make an informed decision on the efficacy and safety of this.

To be fair a therapist shouldn't be giving you advice either

Sometimes an "unsafe" option is not better than the alternative of nothing at all.

And often, the "unsafe" option is severely worse than nothing at all: https://www.rollingstone.com/culture/culture-features/ai-spi...

You see the same thing with coding. People with actual experience and enough of a perspective to see the problems are ignored because obviously they're just afraid to lose their jobs. Which is not true, it's not even on my list of things that I should be aware of.

Experience matters, that's something we seem to be forgetting fast.

I always liked the theory that we're living in an age where all of our needs can be reasonably met, and we now have enough time to think - in general. We're not working 12 hour days on a field, we're not stalking prey for 5 miles, we have adequate time in our day-to-day to think about things - and ponder - and reflect; and the ability to do so leads to thoughts and epiphanies in people that therapy helps with. We also have more information at our disposal than ever, and can see new perspectives and ideas to combat and cope with - that one previously didn't need to consider or encounter.

We've also stigmatized a lot of the things that folks previously used to cope (tobacco, alcohol), and have loosened our stigma on mental health and the management thereof.

Probably a combination of things, I wouldn't pretend to know, but I have my theories. For men, one half-backed thought I've been having revolved around social circles, friends and places outside work or home. I'm a member in a "men only" sports club (we have a few exceptions due to a special program, but mostly it's men only). One of the older gentlemen, probably in his early 80s, made the comment: "It's important for men to socialise with other men, without women. Young and old men have a lot in common, and have a lot to talk about. An 18 year old woman, and an 80 year old man have very little in of shared interests or concerns."

What I notice is that the old members keep the younger members engaged socially, teach them skills and give them access to their extensive network of friends, family, previous (or current) co-workers, bosses, managers. They give advise, teach how to behave and so on. The younger members help out with moving, help with technology, call an ISP, drive others home, to the hospital and help maintain the facilities.

Regardless of age, there's always some dude you can talk to, or knows who you need to talk to, and sometimes there's even someone who knows how to make your problems go away or take you in if need by.

A former colleague had something similar, a complete ready so go support network in his old-boys football team. Ready to support in anyway they could, when he started his own software company.

The problem: This is something like 250 guys. What about the rest? Everyone needs a support network, if your alone, or your family isn't the best, you only have a few superficial friends, if any, then where do you go? Maybe the people around you aren't equipped to help you with your problems, not everyone is, some have their own issues. The safe spaces are mostly gone.

We can't even start up support networks, because the strongest have no reason to go, so we risk creating networks of people dragging each other down. The sports clubs works because members are from a wider part of society.

From the article:

> > Meta said its AIs carry a disclaimer that “indicates the responses are generated by AI to help people understand their limitations”.

That's a problem, because most likely to turn to an LLM for mental support don't understand the limitations. They need strong people to support and guide them, and maybe tell them that talking to a probability engine isn't the smartest choice, and take them on a walk instead.

How do you figure that it’s “currently”, and the need hasn’t always been there more or less?

It just feels a bit uncertain trusting our feelings to AI we don't truly understand.

They aren’t truly cheap

The study is versus no therapy at all.

In the same way a doctor might step in on your sick plan to give yourself a piercing with your keychain, yeah. They probably should be saying it.

Forget safety for a moment, Zuckerberg's push for Meta AI emotional support looks like a clear play for data and control.

Not what the article is about.

Spreading this bullshit is actively dangerous because someone might believe it and try to rely on a chatbot for their mental health.

Even today, leading LLMS Claude 3.7 and ChatGPT 4, take your questions as "you've made mistake, fix it" instead of answering the question. People consider a much broader context of the situation, your body language, facial expressions, and can come up with unusual solutions to specific situations and can explore vastly more things than an LLM.

And the thing when it comes to therapy is, a real therapist doesn't have to be prompted and can auto adjust to you without your explicit say so. They're not overly affirming, can stop you from doing things and say no to you. LLMs are the opposite of that.

Also, as a lay person how do i know the right prompts for <llm of the week> to work correctly?

Don't get me wrong, i would love for AI to be on par or better than a real life therapist, but we're not there yet, and i would advise everyone against using AI for therapy.

> Leading LLMs in 2025 can absolutely do certain core aspects of cognitive behavioral therapy very effectively given the right prompts and framework and things like journaling tools for the user.

What makes you qualified to assert this?

(Now, I dislike arguments from authority, but as an engineer in the area of life/safety-critical systems I've also learned the importance of humility.)

The failure modes from 2023 are identical to those today. I agree with the now deleted post that there has been essentially no progress. Benchmark scores (if you think they are a relevant proxy for anything) obviously have increased, but (for example) from 50% to 90% (probably less drastically), not the 99% to 99.999% you'd need for real assurance a widely used system won't make mistakes.

Like in 2023, everything is still a demo, there's nothing that could be considered reliable.

> Leading LLMs in 2025 can absolutely do certain core aspects of cognitive behavioral therapy very effectively given the right prompts and framework and things like journaling tools for the user.

But when the situation gets more complex or simply a bit unexpected, would that model reliably recognize it lacks knowledge and escalate to a specialist? Or would it still hallucinate instead?

> 2023 is ancient history in the LLM space.

Okay, what specifically has improved in that time, which would allay the doctors specific concerns?

> do certain core aspects

And not others? Is there a delineated list of such failings in the current set of products?

> given the right prompts and framework

A flamethrower is perfectly safe given the right training and support. In the wrong hands it's likely to be a complete and total disaster in record short time.

> a weak prompt that was not written by a subject matter expert

So how do end users ever get to use a tool like this?

[dead]

The word "can" is doing a lot of work here. The idea that any of the current "open weights" LLMs are outside the capitalist framework stretches the bounds of credulity. Choose the least capitalist of: OpenAI, Google, Meta, Anthropic, DeepSeek, Alibaba.

You trust Anthropic that much?

I think you are right, on one hand we have human beings with own emotions in life and based on their own emotions they might impact negatively others emotion

on the other hand probabilistic/non-deterministic model, which can give 5 different advises if you ask 5 times.

So who do you trust? Until determinicity of LLM models gets improved and we can debug/fix them while keeping their deterministic behavior intact with new fixes, I would rely on human therapists.

How is it possible for a statistical model calculated primarily from the market outputs of a capitalist society to provide an interaction outside of the capitalist framework? That's like claiming to have a mirror that does not reflect your flaws.

I think operating under the assumption that AI is an entity bring itself and comparing it to dogs is not really accurate. Entities (not as in legal, but in the general sense) are beings, living beings that are capable of emotion, of thought and will, are they not? Whether dogs are that could be up to debate (I think they are, personally), but whether language models are that is just is not. The notion very notion that they could be any type of entity is directly tied to the value the companies that created it have, it is part of the hype and capitalist system and I, again personally, don't think anyone could ever turn that into something that somehow ends up against capitalism just because the AI can't directly want something in return for you. I understand the sentiment and the distrust of the mental health care apparatus, it is expensive, it is tied to capitalism, it depends on trusting someone that is being paid to influence your life in a very personal way, but it's still better than trusting it on the judgment of a conversational simulation that is incapable of it, incapable of knowing you and observing you (not just what is written, but how you physically react to situations or to the retelling, like tapping your foot or disengaging) and understanding nuance. Most people would be better served talking to friends (or doing their best trying to make friends they can trust if they don't have any), and I would argue that people supporting people struggling is one way of truly opposing capitalism.