About Lemmit

The server is becoming a tad bit too big for the VM it's running on, so I'll be moving it this weekend. Until it's back up and running, I have paused the bot.

Hi,

I don't post this to be malicious or rude, but simply out of concern.

I believe this bot is killing community growth on Lemmy. I keep coming across would-be-interesting communities only to find a wall of bot posts with zero comments, zero votes, zero engagement. And I walk away disappointed instead of getting into a new community. As, I'm sure, may other users are doing.

Sure, you say, "just block the bot" if I don't like it. But that doesn't stop this thing from stifling any real engagement and growth in communities. Surely if someone can "just...", you can "just" go back to reddit if you want to read reddit content that badly.

I admire the engineering you put into making this thing work. It's impressive, and honestly very cool. But I really think it's actively disengaging users, when Lemmy has enough of a hurdle to overcome in growing new communities.

Thats just my 2 cents. I'm not sure it will mean much, but I felt I had to share it. Again, no ill intent against what you've accomplished in creating this. Best wishes.

Hi, today I just learned about RDX and I realized that it solved my biggest gripe with Lemmit. That being that a lot of the video content I find on Lemmit is hosted on Reddit, and clicking through to it gives Reddit a pageview and puts me back on a site I've sworn never to go back to.

RDX is doing something similar to Lemmit, but rather than publishing reddit content to Lemmy, RDX is just making a whole site with all the reddit content. That makes it the perfect place for Lemmit to direct to for links to content which is hosted on reddit.

Previously I was just downvoting any lemmit content that directed me to reddit, but now on desktop, I'm able to use a redirect plugin to redirect all reddit links to RDX, which is great.

Unfortunately this doesn't work on mobile, because the embedded safari client used in most mobile apps will not run the redirect plugin I have installed and configured in mobile safari.

And then I got to thinking, shouldn't this just be the default behavior of Lemmit anyway? It wouldn't require a lot of coding I don't think, and if I were to hazard a guess, I would think that anyone who's moved off Reddit but still accesses it's content via Lemmit would much prefer to not be redirected back to reddit to view the content.

Could we make rdx redirection the default behavior of Lemmit please?

Some subreddits change their 'title' month-to-month or seasonally with a theme. It would be good if the title was periodically retrieved and the title of the community here updated to reflect that.

I've noticed that new posts aren't coming through from the subreddits, and newly requested subreddits aren't getting the automated response either. Seems to have stopped about 3 days ago.

Apologies if this is known/planned, but seeing as I couldn't see any announcement about it I figured I ought to report it.

Hi, thank you for this bot, it's great!

[email protected] is flagged as nsfw even though the actual reddit ( http://reddit.com/r/formula1/ ) isn't can this be fixed?

Thank you!

As discussed here, I have implemented a minimum level of upvotes that a post needs to have on reddit, as well as a minimum ratio of upvotes to downvotes.

Right now I have those configured to require at least 5 upvotes, and more upvotes than downvotes (0.51). At first glance this already seems to be great improvement. There might be some tweaking later.

As a side note I have now switched from using the reddit RSS feed, to using the JSON feed. This was required in order to get easy access to the upvote/ratio properties. So there might be some new and interesting new bugs introduced because of that. It's a brave new world.

Needless to say, the first thing I'll do after releasing this, is plop down on the couch with a beer, and hope this doesn't crash. Fingers crossed!

Some of posts content were cut off, one example : https://lemmit.online/post/132266

Is there any way to fix that?

I'd like to hear some feedback on this, or approach vectors.

Right now the bot is rather spammy. I was hoping that by using Reddits HOT feed, it would return have some level of quality control (I know, right?). Unfortunately, it seems that in most cases, it will just return anything that's new. The downside of this is that a lot of garbage gets through, and the bot spends a lot of time scraping the underlying page to get the details.

I propose to only archive reddit posts that have a karma score of 5 or higher. In case of subs that hide the karma scores of posts for a certain time, they'd have to be at least 2 hours old, so that the Reddit moderators can weed out garbage on our behalf.

Do you folks have any thoughts on this?

Secondly, I want to put sticky comments on each community, with links to native Lemmy communities that cover the same subject. For this I would need some kind of API, or a master list of... oh, I see sub.rehab has just the thing I need. So expect that somewhere this week :).

So I replied to a comment in a thread the bot posted over in one of the television communities, but I noticed tonight when I was viewing that community in Memmy all of the threads the bot was posting appeared to be empty. Just the title, and the info about where the original post came from and who posted it, but otherwise, pretty useless. I wondered if it was a mistake from the bot or the client I was using, so I tried it over on wefwef, and I saw the links there.

So I guess I don’t know if there’s a bug in how Memmy displays the content, or a bug with how the bot posts, or if it’s just an inconsistency or whatever, but I thought someone might want to know to make some changes.

See you on the other side!

So the update is done, but the bot was offline for 6 hours, and needed to catch up.

Unfortunately, another update slipped through, which switched the default feed from www.reddit.com to old.reddit.com, which has the side effect of changing all the urls in the posts as well. On one hand this is great, because new reddit sucks. On the other hand, this is terrible, because for every post the bot encounters, it checks if it already exists on lemmit... based on the url.

So for every post the bot encountered, it went like "old.reddit.com/r/blabla/123? Haven't seen that one yet, there's an www.reddit.com/r/blabla/123, but that must be something completely different, let's post it again!"

This also meant that the bot took over a minute and a half to update each community because it takes a couple of second per post. When I went to bed last night, I figured it was just posting a lot of content because it had so much catching up to do. But this morning I figured something was off because it still hadn't caught up.

Anyway, the fix is out now. Sorry for all the duplicates. I need coffee now.

ChatGPT, write a post for the stuff that I have in my head and want to get out as an update.

Hmm. No brain implant yet. Guess I'll have to write this the hard way.

Syncing update

It has been an eventful week. I successfully deployed the initial version of smarter content syncing, and have made some adjustments to algorithm since then. Most notably, communities with only 1 subscriber (the bot) will no longer receive updates, and communities with fewer than 5 subscribers or with a low posting frequency will only be updated twice a day. Furthermore, for the highest update priority (every 10 minutes), a community must have a minimum of 50 subscribers. Implementation details can be found in the decide_interval() method over here.

Being a developer is fun

Meanwhile... Damnit, bot is stuck again.

2023-07-08 10:13:39,945 - utils.syncer - INFO - Scraping subreddit: bustynaturals. Last time  2:30:48 ago, interval 120 minutes
2023-07-08 10:13:40,653 - utils.syncer - INFO - 'latina bodies are the best' at https://www.reddit.com/r/BustyNaturals/comments/14twww8/latina_bodies_are_the_best/ updated: 2023-07-08 07:14:13+00:00
2023-07-08 10:13:45,324 - utils.syncer - ERROR - Error trying to retrieve post details, try again in a bit; Couldn't retrieve post detail page
2023-07-08 10:13:46,333 - utils.syncer - INFO - Scraping subreddit: bustynaturals. Last time  2:30:54 ago, interval 120 minutes
2023-07-08 10:13:48,581 - utils.syncer - INFO - 'latina bodies are the best' at https://www.reddit.com/r/BustyNaturals/comments/14twww8/latina_bodies_are_the_best/ updated: 2023-07-08 07:14:13+00:00
2023-07-08 10:13:51,227 - utils.syncer - ERROR - Error trying to retrieve post details, try again in a bit; Couldn't retrieve post detail page
...

1 bugfix and deployment later:

2023-07-08 10:46:42,836 - utils.syncer - INFO - Scraping subreddit: bustynaturals. Last time  3:03:51 ago, interval 120 minutes
2023-07-08 10:46:43,573 - utils.syncer - INFO - 'latina bodies are the best' at https://www.reddit.com/r/BustyNaturals/comments/14twww8/latina_bodies_are_the_best/ updated: 2023-07-08 07:14:13+00:00
2023-07-08 10:46:48,327 - utils.syncer - ERROR - Couldn't find post on https://old.reddit.com/r/BustyNaturals/comments/14told8/latina_bodies_are_the_best/, skipping.

Defederation

Meanwhile, the folks at https://lemmy.world reached out to me to tell me they're defederating Lemmit. They are not fond of high volume of posts made by the bot, and the fact that there are now (quick check) 462 communities on this server all being moderated by a single person. They have already received a couple of complaints about spam, and it didn't help that some requests for NSFW subreddits were not marked as NSFW. Occasionally, those subreddits had explicit thumbnails that appeared in the 'All feed' without warning.

I had a good talk with the LemmyWorld admin, wherein they explained their point of view, and I explained mine. I understand their decision to disassociate with Lemmit, and appreciate their attempt to contact me. Other instances like Beehaw, and some smaller ones have also reached the same decision.

This does mean that you will no longer be able to get new community updates on those servers. So make sure to check the blocked instances list on your home server if you were subscribed to Lemmit. At the same time I have removed all the subscriptions of users from those servers, in order to not affect the sync priority mentioned above. This does mean, that if LemmyWorld, Beehaw, etc ever decide to connect to Lemmit again (however unlikely), you will need to un- and re-subscribe from there.

Meanwhile, I've added a feature in the bot that will remove request posts for NSFW subreddits, if the post itself is not marked for NSFW. This should prevent explicit thumbnails showing up where they are not wanted.

Server growth

Last night I got an alert from my server monitoring that the disk is 80% full. Unfortunately, the disk is only 60 GB, so that doesn't leave much room for expansion. On the bright side, a good chunk of that is from Lemmys very verbose logging (like, 4 GB a day, which gets cleaned up daily), so it should last throughout the weekend if I tune that down. Furthermore, most of the storage growth is from from pictrs, the image upload part of Lemmy, and that can utilize an S3 bucket, rather than using the VM's storage like it is now. Using an S3 bucket offers a cost-efficient solution for expanding storage. Initial estimates indicate a monthly cost of around $5 for 1000 GB of storage, which should be sufficient for a while *fingers crossed*.

In the early days of Lemmit (literally, as the server is less than a month old) image uploads were limited to a default setting, which was something around 40 megabytes. That did add up quickly (thanks to half-minute porn gifs), and so I had to limit the max filesize to 1 MB, and later 0.5 MB. Once the server has switched to S3 storage, I can probably up that limit a little, although not too much.

Finally, Lemmy v0.18.1 has been released, and it contains even more performance boosts compared to v0.18.0, so if there's time left this weekend (and I can verify the Lemmit Bot is compatible), I will probably perform the upgrade.

In particular, posts to NSFW videos hosted on v.redd.it don't work on the www version. The links take you to the comments page, which blocks NSFW content, and nags you to go to the app.

old.reddit.com links just work without logging in.

See the bot in action here!

My instance running Leddit

Click here for a more detailed explanation about the bot's purpose

This bot is intended to be self-hosted. Unfortunately, I can't operate a public instance that takes subreddit requests because of how long syncing comments takes. For comparison, Lemmit takes 21 minutes to sync all of the subreddits on this instance using the old system, but Leddit takes the same amount of time to sync 3 subreddits with around 500k subscribers each once an hour. Smart syncing is planned, but it won't decrease the amount of time taken to sync big and active subreddits.

If you need help setting up an instance, feel free to ask questions in this thread or on the Leddit instance's community.

I want to follow many of the NSFW subreddit-communities from this account and probably request more, but the communities page doesn't show NSFW when not logged in (making an account on lemmit itself just to see the list seems like overkill and could be confusing about what's logged in where, if it's even allowed)

Okay, this one took me a bit longer than I planned (mostly due to sql fun and trying to use integers as minutes, WEEEE!).

Backdrop: Last week I disabled the mirroring of a couple of subreddits to the database, because they were initially requested but the nobody subscribed to them. At the same time, the bot was just crawling in a loop, starting at todayilearned, ending at latestsubreddit. As more subreddits were requested, this loop took longer and longer (21 minutes before I rolled out this update). This wasn't sustainable.

So here's the new situation. The more popular a community is, the more often it will be updated. In this case popular means a mixture between number of subscribers and the amount of posts it receives per day (Link to relevant snippet of source code).

In short, the most popular subs will be synced every 10 minutes, the next tier ever 30 minutes, 120 minutes and the content with either no posts per day or no subscribers (other than the bot), will only be synced every 12 hours. I hope this will hit a good distribution of updates vs popularity, but it will most likely be refined at some point in the future.

Speaking of distribution, we now have over 300 communities on this server 🥳, and their update intervals are spread out as such:

• Every 10 minutes: 22
• Every 30 minutes: 39
• Every 60 minutes: 55
• Every 120 minutes: 143
• Every 720 minutes: 44

With this update running live (I started typing after I deployed it, and it has now gotten through the backlog of 'abandoned' subs), I'm going to step back from feature development for a few days. Any bugs that cause the bot to crash will of course continue to be addressed.

Have a blast!

Just as a disclaimer: I'm not complaining, it's great that all this exists at all :)

I'm just checking if this may be a bug or not: if I compare https://www.reddit.com/r/DotA2/new/ and https://lemmit.online/c/dota2?dataType=Post&page=1&sort=New then the bot is definitely skipping some posts.

For example, between "NothingToSay is called 'responsibility god' in CN Dota2 community." and "TIP: Medusa doesn’t reduce magic damage to her mana when BKB is up" there are 3 other posts on reddit directly, which are missing on lemmit.online

I'm just hoping this can be fixed since this bot makes populating the "real" dota2 community I moderate much easier, but some posts I want to cross-post are missing so I need to do shit manually.

Have you considered doing something similar for Mastodon, to allow interacting with toots within Lemmy UI? I know the opposite is possible and Kbin also has some kind of integration, but that doesn't seem to fully work at the moment either.

For example, it'd be nice for anyone that looks at/finds https://lemmit.online/c/dota2 to also find that https://discuss.tchncs.de/c/dota2 or https://lemmy.world/c/dota2 is an actual community that corresponds to that with user content, not bot content.

I'm sure there's lots of equivalents for other communities as well where that would make sense.

Before was running on the cheapest model (1 core / 1GB mem / 30GB storage) at $12/month. The machine was running pretty low on memory, causing it to start swapping, which in turn caused the cpu to get too busy, and everything to slow down.

Now it has a whopping 2GB of memory, and things seem to have calmed down - cpu is back to around 10-15% usage, and swap is down to 0. Happy times all around.

Because of the amount of subs being archived, it now takes about 15 minutes between updates for each sub (was 18 before I updated the VM).

I'm planning to build some kind of scoring system, based on the amount of posts per subreddit (per day?), and amount of subscribers on the lemmy community. That way communities with little subscribers or that don't see many posts per day, will only be updated once per hour.

At the same time, I feel that subs like AskReddit, OutOfTheLoop and other "question-based" subreddits shouldn't be archived by Lemmit. In my opinion those kind of posts are useless without those answers, but please let me know if you disagree.

[email protected] doesnt match up with r/perchance. its last post was 5d ago.

• Fixed a bug where posts would not be submitted because the title didn't contain long enough words.
• Fixed a bug where posts would not be submitted because the url was too long.
• Fixed a bug where posts would not be submitted when it was linking to a /user subreddit.
• Fixed a bug where the bot would think Every Post Everywhere was a subreddit request, and would reply to it.
• Fixed a bug where the bot would crash without recovering whenever something went wrong during new subreddit requests

A fruitful day all in all, I'd say.