I ... actually kinda agree with you.
I get that 'if alien then sci-fi' is the norm for designating sci-fi (like, if it contains science fiction then it's sci-fi, regardless of the plots focus, the entire thing is classified by the setting).

But my head-canon also focuses on what the story is about.
If I could take out the sci-fi elements & the story wouldn't change (ie could be set in today's Earth), then I only see it as -fi. But also the story could be set on today's Earth but with one single smol sci-fi element (a piece of tech of sorts), and if the entire plot focuses on it, then I understand it as sci-fi.

The most controversial example of this (just in my head) wound be Star Wars. Much later in the extended universe things changed, but the movies started out as pure westerns, like, the same story could have been told as a western and especially the screenplay parts wouldn't have to change, just the backdrop (Im not being literal, but not far from it).

Space sci-fi in general has the tendency to use the dimensions in space like if everything was happening on Earth.