this post was submitted on 02 Dec 2023

157 points (85.2% liked)

Technology

59039 readers

3323 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

[email protected]

157

Bill Gates feels Generative AI has plateaued, says GPT-5 will not be any better (indianexpress.com)

submitted 11 months ago by [email protected] to c/[email protected]

78 comments fedilink hide all child comments

Bill Gates feels Generative AI has plateaued, says GPT-5 will not be any better::The billionaire philanthropist in an interview with German newspaper Handelsblatt, shared his thoughts on Artificial general intelligence, climate change, and the scope of AI in the future.

top 50 comments

sorted by: hot top controversial new old

[–] [email protected] 40 points 11 months ago (2 children)

I’m not sure I’d say it’s plateaued today but I definitely think machine learning is going to hit a wall soon. Some tech keeps improving until physical limits stop progress but I see generative AI as being more like self-driving cars where the “easy” parts end up solved but the last 10% is insanely hard.

There’s also the economic reality of scaling. Maybe the “hard” problems could, in theory, be easily solved with enough compute power. We’ll eventually solve those problems but it’s going to be on Nvidia’s timeline, not OpenAI’s.

[–] [email protected] 23 points 11 months ago (1 children)

Generative ai is a bit different from self driving cars in the sense that they're tolerant to failures. This may give more room for improvements when compared to other applications.

[–] [email protected] 2 points 11 months ago

Yes, especially when you consider that the human brain runs on 15W of power!

[–] [email protected] 35 points 11 months ago

Let me save you a click: he doesn’t say anything interesting about why he thinks this.

[–] [email protected] 31 points 11 months ago (1 children)

What does Bill Gates know about GenAI? Is he an expert on the subject?

[–] [email protected] 5 points 11 months ago

He's not. He might pay experts, but he isn't one.

[–] [email protected] 23 points 11 months ago (4 children)

Cool, Bill Gates has opinions. I think he's being hasty and speaking out of turn and only partially correct. From my understanding, the "big innovation" of GPT-4 was adding more parameters and scaling up compute. The core algorithms are generally agreed to be mostly the same from earlier versions (not that we know for sure since OpenAI has only released a technical report). Based on that, the real limit on this technology is compute and number of parameters (as boring as that is), and so he's right that the algorithm design may have plateaued. However, we really don't know what will happen if truly monster rigs with tens-of-trillions of parameters are used when trained on the entirety of human written knowledge (morality of that notwithstanding), and that's where he's wrong.

[–] [email protected] 72 points 11 months ago (14 children)

You got it the wrong way around. We already have a ton of compute and what this kind of AI can do is pretty cool.

But adding more compute power and parameters won't solve the inherent problems.

No matter what you do, it's still just a text generator guessing the next best word. It doesn't do real math or logic, it gets basic things wrong and hallucinates new fake facts.

Sure, it will get slightly better still, but not much. You can throw a million times the power at it and it will still fuck up in just the same ways.

[–] [email protected] 11 points 11 months ago* (last edited 11 months ago) (1 children)

it's still just a text generator guessing the next best word. It doesn't do real math or logic, it gets basic things wrong and hallucinates new fake facts.

If humans are any kind of yardstick here, I’d say all this is true of us too on many levels. The brain is a shortcut engine, not a brute force computer. It’s not solving equations to help you predict where that tennis ball will bounce next. It’s making guesses based on its corpus of past experience. Good enough guesses are frankly our brains’ bread and butter and most of us get through most days on little more than this.

It’s true that we can do more. Some of us, anyway. How many people actually exercise math and logic though? Sometimes it seems like… not a lot. And how many people hallucinate fake facts? A lot.

It’s much like evaluating self-driving cars. We may be tempted to say they’re just bloody awful, but so are human drivers.

[–] [email protected] 2 points 11 months ago (1 children)

I'd say the majority of humans know what 2 + 2 is. Chat GPT doesn't. As it found the answer in some texts it will tell you 4, but all it takes is you telling it that's wrong and suddenly it's 5. So even for the most simple math problem it's extremely easy to throw the whole thing off. Which also means for any prompt you put in it can go in wildly wrong directions at times.

And this is all with good input data, there's plenty of trolls online and the data will only get worse (it already did, the original data up to 2021 was okayish, in the last year tons of crap was put out on top, some of it by Chat GPT itself. So the new model might input the crap it produced before, getting worse over time). The problem on top of that is that you don't know the sources it used. If you ask about a recent event you might receive an insane answer it picked up from a right wing conspiracy site, you simply don't know. There is no fact checking in place.

It's a stunningly good text generator, but that's all it is and it ever will be, at least until they do much more than just add more compute power to it.

[–] [email protected] 4 points 11 months ago (6 children)

Hehe. I’m imagining sitting 100 human test subjects down in a lab setting and asking them what 2+2 is, and then telling them they’re wrong when they answer 4. I don’t know how many of them would guess again but I know it’s not zero. Meanwhile, GPT can probably give a better answer to any advanced math or science query than the majority of humans.

I’m a writer and a language nerd and I watch people all the time use words incorrectly because they think they know what they mean, but they really don’t. They’re just regurgitating them in what they think is the same situation they heard them. They don’t “understand” the word and are just guessing and churning out crap.

I don’t have a dog in this race but I think it’s interesting how people judge artificial intelligence with too much credit given to what goes on with human intelligence. Most people who say it’s “just a next word predictor” read that phrase somewhere and are regurgitating it, not at all dissimilarly to what LLMs do. They use phrases like “it doesn’t actually understand” without being able to define, with any clarity or precision, and without resorting to examples, what would actually impress them as real intelligence.

load more comments (6 replies)

load more comments (13 replies)

[–] [email protected] 7 points 11 months ago* (last edited 11 months ago) (2 children)

Yeah and I think he may be scaling to like true AGI. Very possible LLMs just don't become AGI, you need some extra juice we haven't come up with yet, in addition to computational power no one can afford yet.

[–] [email protected] 12 points 11 months ago (1 children)

Except that scaling alone won't lead to AGI. It may generate better, more convincing text, but the core algorithm is the same. That "special juice" is almost certainly going to come from algorithmic development rather than just throwing more compute at the problem.

load more comments (1 replies)

[–] [email protected] 4 points 11 months ago* (last edited 11 months ago) (3 children)

My hypothesis is that that "extra juice" is going to be some kind of body. More senses than text-input, and more ways to manipulate itself and the environment than text-output. Basically, right now llm's can kind of understand things in terms of text descriptions, but will never be able to understand it the way a human can until it has all of the senses (and arguably physical capabilities) that a human does. Thought experiment: Presumably you "understand" your dog - can you describe your dog without sensory details, directly or indirectly? Behavior had to be observed somehow. Time is a sense too. EDIT: before someone says it, as for feelings I'm not really sure, I'm not a biology guy. But my guess is we sense our own hormones as well

load more comments (3 replies)

[–] [email protected] 6 points 11 months ago

The problem is that between gpt 3 and 4 there is massive increase in number of parameters, but not massive increase in its abilities

[–] [email protected] 3 points 11 months ago

I’ll listen to his opinions more than some, but unfortunately this article doesn’t say anything interesting about why he has this opinion. I guess the author supposes we will simply regard him as an oracle on name recognition alone.

[–] [email protected] 10 points 11 months ago

I hope so. Theyve already got scary implications for creative parts of the economy.

That said, we're in the Cambrian explosion of the tech. As it plateaus, the next step will be enhanced tooling and convenience around it. Better inputs than just text, better, more applications in new spaces, etc.

[–] [email protected] 9 points 11 months ago

"GPT-4 should be enough for anyone." -Bill Gates

[–] [email protected] 9 points 11 months ago (1 children)

bill is a wanker. dont be like bill.

[–] [email protected] 8 points 11 months ago

Now now. He only hired assholes and monsters to execute immoral MS mob style tactics, while he played the great innocent altruist.

[–] [email protected] 7 points 11 months ago (3 children)

Not a single comment yet stating how Gates is a great human being because of his foundation, and how all you haters should fuck the fuck off? sigh, let me the first one.

[–] [email protected] 14 points 11 months ago

Just to make things extremely clear, the above comment has been sarcastic. He's an awful person.

[–] [email protected] 4 points 11 months ago

They're upset he insulted their AI girlfriends.

load more comments (1 replies)

[–] [email protected] 6 points 11 months ago (1 children)

But we have more areas to apply this to. I still can't ask my PC to do some work, like Unistall OneDrive or change a setting in the OS. Send a message on Teams. Where is Jarvis?

[–] [email protected] 5 points 11 months ago (2 children)

Having Generative AI make API calls on your behalf is a work in progress across pretty much every industry. It'll make complex tasks across multiple services a lot easier but it's definitely going to cause weird unpredictable behavior too.

load more comments (2 replies)

[–] [email protected] 4 points 11 months ago

On the one hand, I don't really know enough about AI to comment. What I do remember is that, Bill Gates said the Internet was just a fad in the 90s. This comment caused myself and others problems promoting the Internet in workplaces because those in charge for some reason put some weight to his words. :p

[–] [email protected] 3 points 11 months ago

I've been saying this for years!

[–] [email protected] 2 points 11 months ago

Bill Gates views on AI are about as insightful as Gordon Ramsey’s on orbital mechanics.

load more comments