Selfhosted

You are right. On the one hand, it's kind of bad, naive distributed architecture (my day job), it could have been done much better. On the other hand, the more important point is that it demonstrates an alternative to centralized. We'll learn a lot about usage patterns here, get new ideas, and either improve Lemmy or build something better from the ground up. Big thanks to Reddit for driving users this way to test scalability and get much better knowledge of usage.

Are all these thousands of lemmy servers useless?

almost. It's actually worse than that - when you subscribe to a community from your server it will fetch like 20 posts and that's it, you'll get only new stuff after that, so there's no possibility to do a full mirror of selfhosted, for example, if you started your instance today and didn't fetch posts and comments manually.

ActivityPub per se is just a spec on s2s/s2c communication, which is not a great thing since in many cases it assumes single source of truth, which potentially puts huge load on more popular instances.

I think a quick and dirty hack to this could be the following - each linked instance may maintain cache of announces (so there would be benefit of just forwarding original http signed requests w/o being afraid of malicious actor), which your instance could pull, this way you could populate your mirror without overloading the original source.
Distributed activities propagation though... Let's say there are some design steps involved to make this truly distributed, however I feel like it's possible.

What's the alternative? You go full-banana decentralised or mega-site Reddit. I think Lemmy is a nice middle ground

They are not useless, if the users would actually spread out among them. Each server has its limits.

I just commented on this in another thread: https://lemmy.world/comment/76011

TL;DR: The server-to-client interactions on Lemmy are a lot heavier than the server-to-server interactions, so even if you're just using your own server to interact with communities on other servers, it should still take load off of the servers you would have been using directly.

I'm quite worried of how well this federation system will work in the long run, especially when more people coming from Rexxit. As people make more post/comments, every federated instance will have to cache more redundant contents from each other, which also will use more storage thus increasing the fee of every instance hoster. There's also another problem of visibility in search engines. Because Lemmy/Kbin can be hosted by anyone, it makes searching on a specific domain impossible, unlike how I can just add "reddit" in the search query. Also since there are multiple Lemmy/Kbin instances, there's a chance there'll be similar communities spread over, fragmenting the communities even further. Until they can find a way to fix those problem, I don't think federation is suited for large scale communities.

As for fragmentation problem, maybe adding a global search for communities like this will help reducing fragmentation. Users can still make their own community in their instance, while other people who don't need to can easily find the community they want.

I don't think that there are thousands. The fediverse stat's show about 300 servers, 200 or so made in the last week.

At that rate, it is not too bad. I expect there will be a plateau at some point, relatively soon, where the need for new ones stop, and the experimental ones disappear.

A network of (“thousands of”) servers has — like most things — pros and cons.

Some of the pros are:

• The network is more resiliant against outages. If lemmy.ml is down, all other users can still access the network.
• It's hard to take legal action against the network or to buy it out (like Big Players™ like to do to get rid of potential competitors).
• It allows various similar or even conflicting moderation policies. The network, i.e. the infrastructure doesn't allow or prohibit any specific opinion (the communities do).
• It allows for different ways to pay the bills: goodwill of the admin, donaitions, ads, fee or selfhosting. The latter also allows great control over the data so you control your privacy.

Some of the cons are:

• Content is replicated across servers, which increases the total amount of data stored.
• Latency and speed suffer.
• Interoperability with the wider Fediverse is less than 100%, which can create confusion and frustration.
• Discovery is more difficult.

This has definitely been a problem with communities being created on the bigger instances and not utilising smaller instances. Happy for someone to say I'm wrong etc, but I think there would be merit in capping instances to x number of users or communities, to force the user base to spread out.

Also, the way signups work, (ie you find a community you like then click sign up but that signs you up to that instance), further exacerbates the issue and the confusion around how federation works. The sign up links on each instance should lead either to a page with an instance finder, or to a random instance that matches the profile of, and is already federated with, the instance you were on. Otherwise the larger instances have a monopoly and are just going to lead to a bad user experience when they can't cope with the traffic.

It's a self defeating prophecy if users only want to sign up to the instances with the big communities, because then everyone is going to keep creating communities there and nobody is going to want to join a smaller instance.

I might be talking nonsense and am happy to be told why that is all wrong :)

I've created my own instance in order to not create more load on others and it took a minute to realise I needed to populate it myself, would be nice to have a default view aggregating popular posts etc. across instances. But maybe I'm just asking for too much hehe

Every server just has a cache … there is no profit for the whole network …

I wouldn't say that caching is no profit. Yesterday there were several times when lemmy.ml was struggling or effectively down for some people, but despite complaints over there I could read lemmy.ml communities just fine through my instance. Caching meant that I was isolated from the service interruption, and the lemmy.ml server was isolated from my contribution to its load.

Based on the bit of research I have done, along with creating https://lemmyonline.com/

It seems you are correct. A small handful of servers contains roughly 95% of the user-base.

I think the intended way for this to work, certain communities can be hosted on their own servers. However, it appears most of the popular communities migrating away from reddit, all flocked to lemmy.world, which is likely contributing to it being overloaded.