Username:

Thu<remove>leanSn<remove>eed

Is it broken for everyone or just me?

Now that Lemmy 0.19.0 has been out for a few days, we will be proceeding with the update here on Lemmy.tf. I am tentatively planning to kick this off at 4pm EST today (3.5 hrs from the time of this post).

All instance data will be backed up prior to the update. This one will include a handful of major changes, the most impactful being that any existing 2FA configurations will be reset. Lemmy.ca has a post with some great change info - https://lemmy.ca/post/11378137

I PUT IS GOSHDARN CORRECT AND IT DOESN'T THINK IT CORRECT

I can post this because I'm still signed in on jerboa but I can't log in on my browser or reset my password.

My password is only 8 characters and I can't even click login without typing 10 characters and when I go to forgot password and click reset password I never get an email to reset it.

I'm having the same problem on another instance but I can't even use jerboa with that one.

I had the same problem on lemmy.world but password reset worked there.

Any advice?

I wanted to sign up for this instance but it gave me an email not sent error and is now telling me my email already exists on this instance. I haven’t gotten a verification email and I can’t log in. Am I just locked out of signing up now?

I noticed some timeouts and DB lag when I logged in early this afternoon, so I have gone ahead and updated the instance to 0.18.4 to hopefully help clear this up.

We also have a status page available at https://overwatch.nulltheinter.net/status-page/946fd7fd-3ae3-4214-bbbf-dd7206566104 and will soon have this working on status.lemmy.tf.

As I'm sure everyone noticed, the server died hard last night. Apparently, even though OVH advised me to disable proactive interventions, I learned this morning that "the feature is not yet implemented" and that they have proceeded to go press the reset button on the machine every time their shitty monitoring detects the tiniest of ping loss. Last night, this finally made the server mad enough not to come back up.

Luckily, I did happen to have a backup from about 2 hours before the final outage. After a slow migration to the new DC, we are up and running on the new hardware. I'm still finalizing some configuration changes and need to do performance tuning, but once that's done our outage issue will be fully resolved.

Issues-

[Fixed] Pict-rs missing some images. This was caused by an incomplete OVA export, all older images were recovered from a slightly older backup.

[Fixed?] DB or federation issues- seeing some slowness and occasional errors/crashes due to the DB timing out. This appears to have resolved itself overnight- we were about 16 hours out of sync with the rest of the federation when I had posted this.

Improvements-

• VM migrated to new location in Dallas, far away from OVH. CPU cores allocated were doubled during the move.

• We are now in a VMware cluster with the ability to hot migrate to other nodes in the event of any actual hardware issues.

• Basic monitoring deployed, we are still working to stand up full-stack monitoring.

cross-posted from: https://lemmy.world/post/37906

(I'm creating a starting guide post here. Have patience, it will take some time...)

Disclaimer: I am new to Lemmy like most of you. Still finding my way. If you see something that isn't right, let me know. Also additions, please comment!

Welcome!

Welcome to Lemmy (on whichever server you're reading this)

About Lemmy

Lemmy is a federated platform for news aggregagtion / discussion. It's being developed by the Lemmy devs: https://github.com/LemmyNet

About Federation

What does this federation mean?

It means Lemmy is using a protocol (Activitypub) which makes it possible for all Lemmy servers to interact.

• You can search and view communities on remote servers from here
• You can create posts in remote communities
• You can respond to remote posts
• You will be notified (if you wish) of comments on your remote posts
• You can follow Lemmy users/communities on other platforms that also use Activitypub (like Mastodon, Calckey etc) (There's currently a known issue with that, see here

Please note that a server only starts indexing a server/community once it has been interacted with by a user of this server.

A great image describing this, made by @[email protected] : https://imgur.com/a/uyoYySY

About Lemmy.world

Lemmy.world is one of the many servers hosting the Lemmy software. It was started on June 1st, 2023 by @[email protected] , who is also running https://mastodon.world, https://calckey.world and others.

A list of Lemmy servers and their statistics can be found at FediDB

Quick start guide

Account

You can use your account you created to log in to the server on which you created it. Not on other servers. Content is federated to other servers, users/accounts are not.

Searching

In the top menu, you'll see the search icon. There, you can search for posts, communities etc.

You can just enter a search-word and it will find the Post-titles, post-content, communities etc containing that word that the server knows of. So any content any user of this server ever interacted with.

You can also search for a community by it's link, e.g. [[email protected]](/c/[email protected]). Even if the server hasn't ever seen that community, it will look it up remotely. Sometimes it takes some time for it to fetch the info (and displays 'No results' meanwhile..) so just be patient and search a second time after a few seconds.

Creating communities

First, make sure the community doesn't already exist. Use search (see above). Also try https://browse.feddit.de/ to see if there are remote communities on other Lemmy instances that aren't known to Lemmy.world yet.

If you're sure it doesn't exist yet, go to the homepage and click 'Create a Community'.

It will open up the following page:

Here you can fill out:

• Name: should be all lowercase letters. This will be the /c/
• Display name: As to be expected, this will be the displayed name.
• You can upload an icon and banner image. Looks pretty.
• The sidebar should contain things like description, rules, links etc. You can use Markdown (yey!)
• If the community will contain mainly NSFW content, check the NSFW mark. NSFW is allowed as long as it doesn't break the rules
• If you only want moderators to be able to post, check that checkbox.
• Select any language you want people to be able to post in. Apparently you shouldn't de-select 'Undetermined'. I was told some apps use 'Undetermined' as default language so don't work if you don't have it selected

Reading

I think the reading is obvious. Just click the post and you can read it. SOmetimes when there are many comments, they will partly be collapsed.

Posting

When viewing a community, you can create a new post in it. First of all make sure to check the community's rules, probably stated in the sidebar.

In the Create Post page these are the fields:

• URL: Here you can paste a link which will be shown at the top of the post. Also the thumbnail of the post will link there. Alternatively you can upload an image using the image icon to the right of the field. That image will also be displayed as thumbnail for the post.
• Title: The title of the post.
• Body: Here you can type your post. You can use Markdown if you want.
• Community: select the community where you want this post created, defaults to the community you were in when you clicked 'create post'
• NSFW: Select this if you post any NSFW material, this blurs the thumbnail and displays 'NSFW' behind the post title.
• Language: Specify in which language your post is.

Also see the Lemmy documentation on formatting etc.

Commenting

Moderating / Reporting

Client apps

There are some apps available or in testing:

• Memmy (iOS/Android) https://github.com/gkasdorf/memmy/discussions/13
• Mlem (iOS) https://lemmy.world/c/[email protected]
• Jerboa (Android) https://github.com/dessalines/jerboa

Issues

When you find any issue, please report so here: https://lemmy.world/post/15786 if you think it's server related (or not sure).

Report any issues or improvement requests for the Lemmy software itself here: https://github.com/LemmyNet

Known issues

Known issues can be found in the beforementioned post, one of the most annoying ones is the fact that post/reply in a somewhat larger community can take up to 10 seconds. It seems like that's related to the number of subscribers of the community.

I'll be looking into that one, and hope the devs are too.

So after a few days of back and forth with support, I may have finally received some insight as to why the server keeps randomly rebooting. Apparently, their crappy datacenter monitoring keeps triggering ping loss alerts, so they send an engineer over to physically reboot the server every time. I was not aware that this was the default monitoring option on their current server lines, and have disabled it so this should avoid forced reboots going forward.

I am standing up a basic ping monitor to alert me via email and SMS if the server actually goes down, and can quickly reboot it myself if ever needed (may even write some script to reboot via API if x concurrent ping fails, or something). Full monitoring stack is still in progress but not truly necessary to ensure stability at the moment.

OVH has scheduled a maintenance window for 5:00 EST this evening, hopefully they will be able to pinpoint the fault and get parts replaced at the same time. This will likely be an extended outage as they have more diagnostics than I was able to run, so I would expect somewhere around an hour or two of downtime during this.

I am mildly tempted to go ahead and migrate Lemmy.tf off to my new environment but it would incur even more downtime if I rush things, so it'll have to be sometime later.

Update 7:30PM:

I just received a response on my support case, they did not replace any hardware and claim their own diagnostics tool is buggy. We may be having a rushed VM migration over to a new server in the next few days... which would incur a few hours of hard downtime to migrate over to the new server (and datacenter) and switch DNS. Ideally I'd prefer to have time to plan it out and prep for a seamless cutover but I think a few hours of downtime over the weekend is worth ending the random restarts. I'm open to suggests on ideal times for this to happen.

Previous post: https://lemmy.tf/post/393063

UPDATE 07/25 10:00AM:

Support is getting a window scheduled for their maintenance. I've asked for late afternoon/early evening today with a couple hours advance notice so I can post an outage notice.

===========

UPDATE 12:00AM:

Diagnostics did in fact return with a CPU fault. I've requested they schedule the downtime with me but technically they can proceed with it whenever they want to, so there's a good chance there will be an hour or so of downtime whenever they get to my server- I'll post some advance notice if I'm able to.

===========

As I mentioned in the previous post, we appear to have a hardware fault on the server running Lemmy.tf. My provider needs full hardware diagnostics before they can take any action, and this will require the machine to be powered down and rebooted into diagnostics mode. This should be fairly quick (~15-20mins ideally) and since it is required to determine the issue, it needs done ASAP.

I will be taking everything down at 11:00PM EST tonight to run diagnostics and will reboot into normal mode as soon as I've got a support pack. If the diagnostics pinpoint a hardware fault, followup maintenance will need to be scheduled immediately, ideally overnight but exact time is up to their engineers.

I'm also prioritizing prep work to get the instance migrated over to a better server. This has been in the works for a few weeks, but first I'll need to migrate the DB over to a new Postgres cluster and kick frontend traffic through a load balancer to prevent outages from DNS propagation whenever I finally cut over to the new server. I'd also like to get Pict-rs moved up to S3, but this will likely be a separate change down the road.

EDIT 07/24: This is an ongoing issue and may be a hardware fault with the machine the instance is running on. I've opened a support case with OVH to have them run diagnostics and investigate. In the meantime I am getting a Solarwinds server spun up to alert me anytime we have issues so I can jump on and restore service. I am also looking into migrating Lemmy.tf over to another server, but this will require some prep work to avoid hard downtime or DB conflicts during DNS cutover.

==========

OP from 07/22:

Woke up this morning to notice that everything was hard down- something tanked my baremetal at OVH overnight and apparently the Lemmy VM was not set to autostart. This has been corrected and I am digging into what caused the outage in the first place.

I know there is some malicious activity going on with some of the larger instances, but as of this time I am not seeing any evidence of intrusion attempts or a DDoS or anything.

Lemmy 0.18.1 dropped yesterday and seems to bring a lot of performance improvements. I have already updated the sandbox instance to it and am noticing that things are indeed loading quicker.

I'm planning to upgrade this instance sometime tomorrow evening (8/9 around 6-7pm EST). Based on the update in sandbox, I expect a couple minutes of downtime while the database migrations run.

I'm running the Lemmy Community Seeder script on our instance to prepopulate some additional communities. This is causing some sporadic json errors on the account I'm using with the script, but hopefully isn't impacting anyone else. Let me know if it is and I'll halt it and schedule for late-night runs only or something.

Right now I have it watching the following instances, grabbing the top 30 communities of the day on each scan.

REMOTE_INSTANCES: '[
        "lemmy.world",
        "lemmy.ml",
        "sh.itjust.works",
        "lemmy.one",
        "lemmynsfw.com",
        "lemmy.fmhy.ml",
        "lemm.ee",
        "lemmy.dbzer0.com",
        "programming.dev",
        "vlemmy.net",
        "mander.xyz",
        "reddthat.com",
        "iusearchlinux.fyi",
        "discuss.online",
        "startrek.website",
        "lemmy.ca",
        "dormi.zone"]'

I may increase this beyond 30 communities per instance, and can add any other domains y'all want. This will hopefully make /All a bit more active for us. We've got plenty of storage available so this seems like a good way to make it a tad easier for everyone to discover new communities.

Also, just a reminder that I do have defed.lemmy.tf up and running to mirror some subreddits. Feel free to sign up and post on defed.lemmy.tf/c/requests2 with a post title of r/SUBREDDITNAME to have it automatically mirror new posts in a particular sub. Eventually I will federate that instance to lemmy.tf, but only after I'm done with the big historical imports from the reddit_archive user.

cross-posted from: https://sh.itjust.works/post/433151

Just an FYI post for folks who are new or recently returning to Lemmy, I have updated the linked grease/tamper/violentmonkey script for Lemmvy v0.18.

These two scripts (a compact version and a large thumbnail version) substantially rearrange the default Lemmy format.

These are (finally) relatively stable for desktop/widescreen. Future versions will focus a little more on the mobile/handheld experience.

These are theme agnostic and should work with darkly and litely (and variants) themes.

• Greasyfork here: https://greasyfork.org/en/users/1107499-mershed-perderders
• Github here: https://github.com/soundjester/lemmy_monkey
  • If you need the userscript for Lemmy v0.17.4, that can be found here:
    • https://github.com/soundjester/lemmy_monkey/tree/dev-v1.11-Lemmy-v0.17
    • this v0.17.4 userscript is no longer in active development

Screenshot of "Compact" version

main page

-

comments page

As always, feedback is appreciated!

I've been stalling on this but need to get some form of community rules out with the added growth from the Reddit shutdown. These will likely be tweaked a bit going forward but this is a start.

Rules

1. Be respectful of everyone's opinions. If you disagree with something, don't resort to inflammatory comments.

2. No abusive language/imagery. Just expanding on #1.

3. No racism or discrimination of any kind.

4. No advertising.

5. Don't upload NSFW content directly to the instance, use some third party image host and link to that in your posts/comments.

6. Mark any NSFW/erotic/sensitive/etc posts with the NSFW tag. Any local posts violating this rule are subject to removal (but you can repost correctly if this happens).

7. Hold the admins/mods accountable. If we start making changes that you disagree with, please feel free to post a thread or DM us to discuss! We want this instance to be a good home for everyone and welcome feedback and discussion.

NSFW Content Policy

As stated above, please upload any NSFW images to an external site and link them. All NSFW content must be properly tagged, and cannot contain material illegal in the United States.

Additional rules around NSFW content may be added in the future, if necessary. We would prefer everyone use common sense with their posts so we don't have to crack down on this category.

Defederation Policy

Many large instances have started to defederate "problem" instances. We want to avoid doing that unless an instance is causing illegal content to get indexed directly onto our server.

If we encounter the need to block some other Lemmy server, we will engage the community here before taking action.

Bot Policy

Bots are currently allowed on this instance, but we reserve the right to add restrictions if they start getting abused. You're more than welcome to use moderation bots for any communities you run or moderate, and content import/mirroring bots are okay. If you have a bot that is actively creating new posts/comments here, please make sure to use some reasonable rate limits.

Bots are subject to all instance rules.

Obviously with the closing of the Reddit API all of our favourite apps no longer work and you've ended up here looking for alternatives. Lemmy in general isn't Reddit, it's just similar and so it will take a minute to adapt, but as with all communities, it's you that makes it. Thanks especially for choosing Lemmy.tf as your instance. It's nice, speedy and well maintained. I look forward to seeing you all around. Have fun and if you have any questions, feel free to ask.

So Lemmy 0.18.0 dropped today and I immediately jumped on the bandwagon and updated. That was a mistake. I did the update during my lunch hour, quickly checked to make sure everything was up (it was, at the time) and came back a few hours later to everything imploding.

As far as I can tell, things broke after the DB migrations occurred. Pict-rs was suddenly dumping stack traces on any attempt to load an image, and then at some point the DB itself fell over and started spewing duplicate key errors in an endless loop.

I wound up fiddling with container versions in docker-compose.yml until finding a fix that restored the instance. We are downgraded back to the previous pict-rs release (0.3.1), while Lemmy and Lemmy-UI are both at 0.18.0. I'm still trying to figure out what exactly went wrong so I can submit a bug report on Github.

Going forward, I will plan updates more carefully. We will have planned maintenance windows posted at least a few days in advance, and I may look into migrating the instance to my Kubernetes cluster so we can do a rolling deployment, and leave the existing pods up until everything is passing checks. In the meantime, I'm spinning up a sandbox Lemmy instance and will use that to validate upgrades before hitting this instance.

If your feed is sorted by New or includes local content, you may notice a TON of new posts showing up on reddit_ communities here over the next few days- I am attempting to scrape as much content as I can from Reddit prior to the API pricing changes on July 1. All of this content will be limited to the reddit_* communities on this instance, if you don't wish to see this content you can simply block the communities as they appear.

If anyone has requests for a subreddit mirror, drop them in the comments and I'll try to get to your request sometime this week.

Edit: Halted since random other Lemmy instances managed to auto-index my new subs, I don't want to flood any feeds outside of lemmy.tf with this. Since I can't control other instances auto discovering my new communities, all Reddit cloning will now occur in a new, defederated instance.

All import activities are now taking place at https://defed.lemmy.tf/.

I don't want the username password, so I tried to change it to username. Now my name in the top right shows as username but my profile URL is still u/password. Can someone fix this?

Not much has changed today. Email verification is now disabled as we seem to have hit some bug where it just craps out until the Docker pods are restarted, I'm probably going to leave this disabled unless we start getting some large influx of spam users.

Default theme has also been changed to Darkly - Red which feels a bit more reminiscent of Reddit.

Some thoughts on image uploads

Image storage remains my primary concern for instance scalability. I am fairly limited on local storage since the server running this instance is all-NVMe, so if the /pictrs volume fills up too much, I will have to connect a cloud disk. Rather than totally disabling image uploads (which would also mean no avatars), I'm leaning towards setting something like a 400kb limit for all uploads. This is still TBD and may wind up being unnecessary if I can find some cheap option.

In preparation for a (hopeful) influx of users, I've bumped up the resources for the server a bit. We're now running on 4 cores/8gb ram and a 512tb disk on one of my OVH servers, and I may setup a larger disk for image storage if it winds up growing quickly. I've got plenty of resources to spare so there shouldn't be any scalability issues.

Email is also functional and now required for all new signups. No admin validation is required at the moment, but this could change if we start getting a flood of bots or something.

I am also looking for an admin or two to assist with the day-to-day management of this. Not sure what that will look like since I'm pretty new to Lemmy, so any help is appreciated.

Hi, I'm lemmywinks and I'm here to ditch reddit.