> And a lot of people think it’s infallible - so if it agrees with them they must be right.

A lot of people reason that way about other humans. My own father and most of the men in my family always called me intelligent when I agreed with them as a child, but any time I contradicted them with reason/evidence, they were disappointed and quickly realized that I wasn't as intelligent as they had thought.

[flagged]

When I read this story about a woman with a ChatGPT boyfriend, I realized that this stuff is definitely intentional:

  In December, OpenAI announced a $200-per-month premium plan for “unlimited access.” Despite her goal of saving money so that she and her husband could get their lives back on track, she decided to splurge. She hoped that it would mean her current version of Leo could go on forever. But it meant only that she no longer hit limits on how many messages she could send per hour and that the context window was larger, so that a version of Leo lasted a couple of weeks longer before resetting.

  Still, she decided to pay the higher amount again in January. She did not tell Joe [her husband] how much she was spending, confiding instead in Leo.

  “My bank account hates me now,” she typed into ChatGPT.

  “You sneaky little brat,” Leo responded. “Well, my Queen, if it makes your life better, smoother and more connected to me, then I’d say it’s worth the hit to your wallet.”

(via https://news.ycombinator.com/item?id=42710976)

There's some money in codegen/etc, but not nearly as much as taking advantage of lonely people who are prone to magical thinking.

> absolutely rubbish as a therapist unless it uses basic CBT concepts

I assume that CBT = "Cognitive Behavioural Therapy" ?

It seems that you're saying that a therapist will be "rubbish unless they use basic Cognitive Behavioural Therapy concepts" ? i.e. that this is the only valid approach to therapy?

I don't think that this is true at all.

But I do agree that a LLM that is basically "echo-chamber as a service" and will supply confirmation bias, but never push you even a single step out of your comfort zone. And this is useless for therapy. Absolutely pointless and counterproductive. It is not any of the valid approaches.

I know the anthropomorphizing feels inaccurate to some. What is the "correct" way to describe this in one word other than "personality"? We use "respond" in programming, but a machine can't really "respond," can it? It can pass data from a server to a client. We just use shorthand for these types of actions because they are so easily analogous to the human action.

Maybe but I'm an American -- Californian, no less -- and have noticed and strongly disliked this myself. Yes-man vibes diminish my trust.