overview for marsara9

What’s the best search engine for lemmy? by marsara9 in c/[email protected]

[-] [email protected] 45 points 1 year ago

Thanks for the shout-out.

But FYI I've run into some bugs that's preventing new content from being indexed. So you won't see anything new (from about a week ago) until I can find a new method to fetch new posts.

Feature suggestion: duplicate posts handling by marsara9 in c/[email protected]

[-] [email protected] 48 points 1 year ago* (last edited 1 year ago)

Playing devil's advocate for a bit... So these are just cross-posts. Which existed even on Reddit. ...I assume they weren't handled in any way in Sync or Reddit?

But let's say this is fixed... What to do about the multiple comments threads? How would you reconcile them with each other? Especially since the user can choose different ways to sort the comments as well. Would all of this logic normally handled by the Lemmy back-end now need to run on your phone? Also how do you choose which post / instance to actually display and which ones to hide?

Btw, I'm not trying to dismiss the idea. Just want to call out some of the technical problems that might come up trying to implement such a feature. As well as ask questions to try and determine exactly how such a feature is expected to work.

Welp that answers a lot of why all .ml are down by marsara9 in c/[email protected]

[-] [email protected] 30 points 1 year ago

With ActivityPub all of the primary ids contain the domain of the hosting server. So if you lose your domain none of the other instances know that you're the authority on those communities, posts, comments or users. So essentially federation breaks with all of the old data.

155

A new way to search for communities (www.search-lemmy.com)

submitted 1 year ago by [email protected] to c/[email protected]

28 comments fedilink

I keep see people complaining about not being able to find active communities that match their interests. So I've added a new feature to https://www.search-lemmy.com/ that allows you to search posts for a particular topic and then it tells you which communities have the most posts matching your search query.

And assuming that you've set your home instance correctly, those links will even open up in your home instance, so that you can subscribe directly to them.

For example, if you search for 'linux' (https://www.search-lemmy.com/find-communities/results?query=linux&page=1) it gives you a link to each community, tells you which instance it's on and how many matches it found for your query.

All of the same filters that you can use on the normal search can be used here as well. So if you just want to find the best community that mentions linux on lemmy.world (https://www.search-lemmy.com/find-communities/results?query=linux+instance%3Alemmy.world&page=1), you can filter by just that instance. Click on the Search Tips button to see a list of all of the available filters.

P.S. I'm aware of https://lemmyverse.net/ etc... and while those are great as well, this allows you to search to see what people are actually talking about on the various communities.

Again, if you have any feature requests or find any bugs, PLEASE reach out or ideally go to my github (https://github.com/marsara9/lemmy-search) and log a bug there.

262

Search Lemmy 0.4.0 update (an enhanced search engine for Lemmy) (www.search-lemmy.com)

submitted 1 year ago by [email protected] to c/[email protected]

34 comments fedilink

A couple days ago I updated https://search-lemmy.com/ to 0.4.0.

New features, that several people were asking for:

The UI has been overhauled and it should be much easier to find your home instance now.
Search itself has been overhauled. Increase search performance significantly. I also automatically search for related terms as well. You may now see fewer search results, but ideally they should be more relevant. You can also now include basic syntax like:
- quotes: "some terms that must be together"
- negative terms: cat -dog (shows posts about cats that don't mention dogs)
- either or: cat OR dog (shows posts about either cats or dogs). The default search behavior is now an implicit AND, but order doesn't matter.
I've added several new filters that you can use including:
- !safeoff -- Disables safe search allowing NSFW posts to appear in the search results (NSFW is now hidden by default)
- since:YYYY-MM-DD -- shows only posts that have occurred since the specified date
- until:YYYY-MM-DD -- same as above but in reverse. It will only posts up to the given date.
I've removed the preferred-instance query parameter from the results URL so it should be easier to share links to search results now.
The date the post was created or last updated is now displayed in the search results.

Bug Fixes:

Site performance should now be stable. Fixed a bug related to the database pool that was causing the site to hang.
Fixed a bug that would cause broken links.
Fixed various bugs with the crawler causing posts to be missed.

Known Issues:

If you set your home-instance to a fairly small instance, the number of search results is also relatively small. Once (https://github.com/LemmyNet/lemmy/issues/3259) is resolved. I should be able to show links regardless of what your home instance is set to, allowing you to search the entire Fediverse.
Currently searching only looks at the post title and body. Comments aren't indexed either. This also is dependent on the above issue on Lemmy itself.

Finally some things to note:

I've started to refactor the code to abstract away Lemmy from the actual search engine. As I now start to prepare to search other Fediverse instances like Kbin, and maybe even Mastodon, etc...

Why GitHub? by marsara9 in c/[email protected]

[-] [email protected] 51 points 1 year ago

IMHO federation doesn't bring any real benefits to git and introduces a lot of risks.

The git protocol, if you will, already allows developers to backup and move their repositories as needed. And the primary concern with source control is having a stable and secure place to host it. GitHub already provides that, free of charge.

Introducing federation, how do you control who can and cannot make changes to your codebase? How do you ensure you maintain access if a server goes down?

So while it's nice that you can self host and federate git with GitLab, what value does that provide over the status quo? And how do those benefits outweigh the risks outlined above?

Is there a way to search in a Lemmy community? by marsara9 in c/[email protected]

[-] [email protected] 21 points 1 year ago* (last edited 1 year ago)

https://www.search-lemmy.com

http://www.github.com/marsara9/lemmy-search

Just add community:[[email protected]](/c/[email protected]) at the end of your query.

Announcing a new Search Engine for Lemmy by marsara9 in c/[email protected]

[-] [email protected] 17 points 1 year ago

Not yet but I can add this feature

Announcing a new Search Engine for Lemmy by marsara9 in c/[email protected]

[-] [email protected] 14 points 1 year ago

Ya, now if everyone can stop finding bugs! So I can take some time off. /jk

Announcing a new Search Engine for Lemmy by marsara9 in c/[email protected]

[-] [email protected] 41 points 1 year ago

Yes but that search doesn't take you to the instance that you are logged into already. Which is one of my main goals with this site. While that did give me the inspiration for this and has the power of Google behind it, it lacks knowledge about how the fediverse actually works.

Announcing a new Search Engine for Lemmy by marsara9 in c/[email protected]

[-] [email protected] 15 points 1 year ago* (last edited 1 year ago)

I've already got some complaints about that. You can see one of the issues raised on GitHub.

At the moment, I'm only picking up mastodon posts that are federated to Lemmy, but you can't choose Mastodon as a preferred-instance, yet. When and/if I decide to add Mastodon support, I'll reach out to the admins over there to get feedback first.

Edit and note to any server admin: If you want to block the crawler from hitting your site, just add lemmy-search to your robots.txt and crawling will be prevented. But this doesn't stop cross-federation posts from being picked up on another instance.

1871

Announcing a new Search Engine for Lemmy (www.search-lemmy.com)

submitted 1 year ago* (last edited 1 year ago) by [email protected] to c/[email protected]

153 comments fedilink

I shared bits and pieces of this before, but it's officially up and running now: https://www.search-lemmy.com/

This is an enhanced search engine for Lemmy. With a few primary goals:

You can choose a preferred instance. After choosing what your primary instance is, and performing a search ALL links will open in that instance.
This aims to be a replacement for using site:reddit.com in Google, but just for the fediverse.
You can filter the search results by:
- Instance -- This will filter the results to only show communities that belong to a particular instance. Just type something like instance:lemmy.wrold or instance:https://lemmy.world/. This is separate from your preferred instance, such that you can search for posts on lemmy.world while still opening them on lemmy.ml.
- Community -- You can refine the search by a specific community. You use the same syntax that you'd use here community:[[email protected]](/c/[email protected]).
- Author -- Similar to the above you can also filter by a specific author such as: author:@[email protected].
The entire thing is open-source. You can view the code and even host your own instance... See more details here: https://github.com/marsara9/lemmy-search.

NOTE: This only supports Lemmy instances for now. Other fediverse type instances may be in the future depending on how this works out.

I've been working on this over just the last few weeks, so it hasn't had a chance to crawl much of the fediverse yet. For now it only supports lemmy.world and lemmy.ml but other preferred-instances will come online as time goes by.

If anyone finds any bugs, and I'm sure you will, or if anyone has any suggestions PLEASE raise an issue on GitHub for me to track. Lastly, if anyone wants to help contribute please feel free to reach out.

NOTE TO SERVER ADMINS: You can prevent your site from being crawled by adding lemmy-search to your robots.txt for the user-agent.

Can Lemmy posts be indexed by Google or other search engines? by marsara9 in c/[email protected]

[-] [email protected] 92 points 1 year ago* (last edited 1 year ago)

I'm working on a specialized search engine just for the fediverse. https://github.com/marsara9/lemmy-search

If anyone wants to help out, feel free to reach out, but I hope to have something ready to release soon.

The idea with my version is that it'll search as much of Lemmy / the fediverse as it can and you can select the preferred instance that you want to open any link with.

How to google for content from Fediverse? by marsara9 in c/[email protected]

[-] [email protected] 49 points 1 year ago

I don't mean to keep self promoting but: https://github.com/marsara9/lemmy-search

It's still a work in progress, but it's going to be a search engine specifically built for Lemmy, and maybe eventually the entire fediverse.

Idea is that you'll be able to search just like you normally would in Google but

the links it shows you would take you to your preferred instance
it will search as many lemmy instances as it can find, so it won't find anything outside of the fediverse.

Will Lemmy posts be searchable on Google? by marsara9 in c/[email protected]

[-] [email protected] 37 points 1 year ago

So I've been working on a solution for this.

As I see it Google and others are going to have a hard if not impossible time to incorporate the fediverse, and the fact that the same content can exist on multiple servers.

So I'm working on a search engine specifically build, for Lemmy at least. Where it'll take you to whatever your preferred instance is when tapping on a search result.

I hope to have a MVP up and running in a few more days.