Crichton amnesia - Dr. Gell-Mann doesn't enter into it - is a good way to describe how a lot of discourse around these systems goes. Some domain expert picks apart the outputs in their domain but imagines that it's just fine at style or history or what have you, and uses it for that.

I see this especially with STEM experts attempting to automate away activities traditionally associated with humanities subjects.

If you knew a human, who you could text day or night, on any topic, and they’d respond within seconds with a lengthy answer, which was often correct, would you delete their contact number because they sometimes made subtle errors? I would not.

I think LLMs come off as educated, but not really smart enough for their vocabulary.

This seems to be the flimsiest of straw men.

Of the things that the author states as generally agreed upon there is significant debate. I'm not even sure how much of it is considered as a prevalent opinion.

Of the Gell-Mann issue at the end, I have never encountered such a statement, I don't doubt that it has happened here and there, but without supporting evidence to say this is a common occurrence, why bring it up in this manner.

It seems the real intent of this post is to signal that the author does not like AI.

I think there could have been a sensible informative article written about Gell-Mann amnesia warning how it also applies to AI output and suggesting people calibrate their expectations based upon the error rate in fields that they know well.

> And yet I see people who should know better say things like, “I asked a conversational AI some questions that I knew the answers to, and its answers were well-written, but they had the kinds of subtle errors that could lead someone badly astray. But then I asked it some questions that I didn’t know the answers to, and it gave me really good, clear information! What a great learning tool!”

I'd love to see an actual example of this

Is "Gell-Mann amnesia" anything other than a guilt-by-association fallacy?

As Crichton defined it, the person experiencing the effect doesn't even stay in the same section of the newspaper. So outside of small local newspapers, the journalist in the new section is almost always a different person, presumably reporting to a different editor.

> I thought we agreed that all of these “AI” systems are fundamentally just making shit up, and that if they happen to construct coherent sentences more often than your phone’s predictive-text keyboard then the difference is one of degree rather than kind. It’s amazing that the technology works as well as it does, but it’s been clear for a while now that these tools are unreliable, and that that unreliability is inherent.

So are human brains! No shit!

But you know what? I can throw recipes at ChatGPT and it can make some amazing variations! It is actually pretty good at making cocktails if you start off with some initial ideas. Sure it sucks at math, but it is still super useful!

Oh and bug fixing. LLMs can go over dozens of files of code and find weird async bugs.

Mocking out classes! I hate writing Jest mocks, and 100% of the time I get the syntax for proxying objects slightly wrong and spend (literal) hours debugging things. GPT is great at that.

Summarizing meeting transcripts, wonderful!

Or just throw an history entire book into Claude's absurdly large context window and start asking questions.

I also dismissed LLMs given their "accuracy." But I think that's the wrong thing to compare. The sheer fact that a transformer architecture model gets better at scale and that we can feed it at scale is insane. It's what makes it AI and not just ML.

The fact that you can get billions of parameters to do anything useful from a relatively simple algorithm on a relatively small amount (high GBs / low TBs) of text means the algorithm is insane. That's what people miss - they think GPT is trained on "the whole internet" and is similar to some of low-variate regression model that is "approximating things." It is absolutely approximating things - so does all intelligence -- but it is truly sifting / "attending to" what is important over a relatively small corpus and organizing into billions of parameters the way a brain would organize data.

Will it hallucinate details? statistics? Etc.? Yes, and it should not be used in its current form for "truth." But it's very different from a low-variate model that is synthesizing in a low-dimensional space (which is how we gradually learn about the world) and an extremely-high dimensional model that is starting to see "what is important" in ways that are far, far above human intelligence. Similar to a human brain (due to the underlying neural architecture and any type of hierarchical compression of knowledge) but with far more input data, and a simplicity that maybe the brain has maybe it doesn't -- but is far more scalable and capable of hierarchies of information that out-scale us by so many orders of magnitude, and more every 6 months.

3blue1brown's https://www.youtube.com/watch?v=wjZofJX0v4M and upcoming videos I think will show the beauty and simplicity of the algorithm more. To put it another way -- the fact that you get a remotely true outcome with a model that just improves with scale, a remotely true outcome by the algorithm sifting what is important -- means that with time it will know what is more important in ways that far surpass humans.

If you approach interacting with LLM chatbots that way it is absolutely mind-blowing how "on point" the answers are. If you ask ChatGPT why the internet is important? Or why AI/ML models are important? Or why the "Attention is All You Need" paper is important? (yes with some RLHF but that's just to improve a few more percentage points). It will create an incredibly well-sifted, highly compressed answer* all from an algorithm that outputs matrix numbers from fairly limited, fairly shitty internet text, compressed into what is useful in a very eloquent way. That's the excitement of LLMs. Super-human intelligence from an algorithm and low-quality information.

* https://chat.openai.com/share/00a5f9b7-7ee1-4641-92bf-999185...

...and please give me half a billion dollars to.solve that problem, even though I have no proof of concept or even a good hypothesis.

> aside from the considerable ethical concerns with the unauthorized scraping of everybody’s creative work,

If you want to make this argument, I'm "on board."

===

> and the dismal treatment of the people who annotate that work, and the electricity it takes to compile those annotations into models, and the likelihood that companies will see this new technology as a cheaper alternative to their human employees

Agree as well.

====

> those things aside, I thought we agreed that all of these “AI” systems are fundamentally just making shit up, and that if they happen to construct coherent sentences more often than your phone’s predictive-text keyboard then the difference is one of degree rather than kind.

But I'm tired of hearing this argument. I mean, if the LLMs work better and faster than the majority of _human_ assistants at my disposal, then who cares if they are "fundamentally just making shit up". They're better and faster than the competition, no matter how much you damn them with faint praise--end of story as far as I'm concerned with this argument.