this post was submitted on 27 Sep 2024
29 points (100.0% liked)
Programming
17419 readers
27 users here now
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
- Follow the programming.dev instance rules
- Keep content related to programming in some way
- If you're posting long videos try to add in some form of tldr for those who don't want to watch videos
Wormhole
Follow the wormhole through a path of communities [email protected]
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
It's also possible to just pull all posts from an instance. The API is easy to understand.
main.rs
Cargo.toml
Thanks for the suggestion and the code snippets.
i want to see how votes/comments accumulate over time on a post, therefore i would have to poll the "all" posts endpoint in a regular interval. but I would either see new posts with small number of comments/upvoted, or already upvoted post, or i would have to download all posts in a regular time interval which seems impossible to me.
Comments are also easy, the API allows pulling them by latest too. If I was writing a search engine, I would probably just track all known instances and just pull local content from them instead of deduplicating. I haven't really looked at how votes are federated though, so that might be more complicated to keep updated.
I expect just syncing posts and comments from all instances to be mostly easy. In the past I was able to pull all posts and comments from smaller instances in like less than 10 minutes. It's mostly just text so it doesn't take that long. After it's pulled, it can be kept mostly up to date by just pulling to the last date received, and should take much less time than the first sync.
I've noticed there's lots of stuff on Lemmy that fails to federate to other instances. I think there's also actually a 3000€ reward at the moment for improving federation, so if you spend very much time on it, it might be a good idea to see if it can be claimed. Though, I don't really know how the milestone system works, and it might only be available to inside contributors.