It's called adverse possession and that's why you discard anything not bolted down ASAP under the presumption of illegal dumping, put up a new fencr with big signs, and claim any structures as yours under the presumption that anyone following code (building legal) would hire surveyors to note boundaries. Do not let any other living human past that fence under any circumstance and if they do so against your will videotape all of it.

Of course whoever built it will take you to court but you argue that they can move their structures because you have every right to your own land for all the aforementioned reasons

Edit: and something like this, random dude trying to be a bigger asshole vs Elon Musk, you'd need to hire an attorney just to field calls from attorneys lining up to work pro bono for the free media coverage, and probably wind up with a PR rep working for free for the same reasons, helping you line up slots on random talk shows, because this would go mega fucking viral