this post was submitted on 24 Jul 2024
435 points (97.2% liked)
Technology
59381 readers
4116 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
That's what smaller models do, but it doesn't yield great performance because there's only so much stuff available. To get to gpt4 levels you need a lot more data, and to break the next glass ceiling you'll need even more.
Then these models are stupid. Humans don't start as a blank slate. They have an inherent aptitude for language and communication. These models should start out with basics of language, so they don't have to learn it from the ground up. That's the next step. Right now they're just well read idiots.
Yup that is kind of the point. They are math functions designed to approximate human tasks.
I'm not sure what you're pointing at here. How they do it right now, simplified, is you have a small model designed to cut text into tokens ("knowledge of syllables"), which are fed into a larger model which turns tokens into semantic information ("knowledge of language"), which is fed to a ridiculously fat model which "accomplishes the task" ("knowledge of things").
The first two models are small enough that they can be trained on the kind of data you describe, classic books, movie scripts etc... A couple hundred billion words maybe. But the last one requires orders of magnitude more data, in the trillions.