When you buy a TTRPG, essentially, you're getting a bundle of two (or perhaps three) things.
I'm going to put aside the third one, for now, and just talk about what I think are the big two in terms of page count: System Mechanics, and Game World.
Lots of people who are keen on system mechanics seem to feel that game world can pretty much take care of itself.
As long as the game master has a rough idea of what they want to present ("Hey, generic fantasy medieval! D&D-ish. Warcraftish. Lord of the Ringsy. Sorta like The Witcher maybe. Man, you know what I mean!") then the main bulk of the game book is preoccupied with what dice to roll in what circumstances, refering to what game system. To make things simple, adventures are often geographically isolated—an underground complex, a mountain pass, an unexplored island, an abandoned fort. Aaaand with that, we're ready to roll!
In the opposite corner, there are folks who feel that background is vital. They're happy enough with a rules-light system, just as long as the game world adheres to a particular canon. That means EITHER that the game refers to other media—films or books or comics—that the GM really should be familiar with before they get started, OR the bulk of the game book will be dedicated to conveying game world lore.
Okay—so the proposition of many rules light systems is, "rulings not rules"—in other words, the GM can develop detailed or specific rules systems on the fly, to cope appropriately with the particular path the players take.
And the proposition of many open world, or "sandboxy" systems is that the GM can develop the game world as appropriate, often according to complex tables (AD&D wilderness amd random encounter tables, I'm looking at you!) responding both to what the player charaters do, and to the development of the players' skills and objectives.
I suggest that it's quite possible to have a great deal of RPG fun in either circumstance —rulings not rules, or exquisitely defined system mechanics… emergent game world, or fabulously detailed canon. Or anything in between.
Right then. So, if we buy that, we've just established that NEITHER system mechanics NOR game world is vital to a great game. In fact, games can do without either… so can they do without BOTH?
The very existence of one-page, ultra-light RPGs suggests the answer is yes!
Okay, so if that's the case, then why do we sometimes have disappointing game sessions? We can deduce that it must be the case that the disappointment of a poor game session is simply not addressed by EITHER the game world OR the system mechanics.
So what does address the problems that result in poor game sessions? And why isn't that the main focus of game books?
I think it's important to understand the difference between the job of the GM and the game designer. A designers job is to make the GMs job easier and prepare them to run the designers system. I think too many designers forget that and write a book that acts like it assumes a GMing style. At the end of the day a designer can't make a group enjoy their system, but there is such thing as good system writing beyond just making clear rules and an interesting setting.
"Ruling not rules" is a solid idea since that's the strength of a ttrpg, but it often feels like an excuse for the designer not doing their job. The designer gets no credit for the GMs decisions unless they directly influence it through well designed mechanics or writing in such a way to develop the GMs intuition. I've seen designers clearly state their intentions well enough without specific rules so that it helps the GM make a good ruling, but many don't. PbtA systems is were I see this mistake the most. So often they have written a mechanic where it's clear the designer has an assumption in mind but won't explain what it is.
The problem with many setting first design is that it's clear that the person wanted to publish their world building, but didn't want to write a novel so to justify anyone buying it they wrote an RPG. However, designing mechanics is it's own discipline so these games suffer from it if the writer was really just wanted to write a setting guide. In these cases it's a little bit better if you write it as third party content for another ttrpg.
It's rare to see a ttrpg marry both world building and mechanics, but the ones that do tend to be super hyper focused. The real reason I don't think this is more common is because a 600+ rulebook is intimidating even if it's mostly just the setting.