Endlesstalk

While working on a small fix to lemmy, that was causing some unneeded cpu usage, I made a change that unfortunately caused the db and pictrs service storage to be deleted.

Thankfully I have backups of everything, so I went to restoring from a backup. However the restoring was very slow, since I used an unoptimal way to backup the db(raw sql dump). After the first backup completed, I found out that it was missing data, so I tried an older backup, but that didn't work either. It was missing data as well. So I tried a backup from another server(Since I backup to 2 different servers), which finally worked.

Usually restoring from backups, haven't taken too long previously, since my backups are faily small, but I will need to look into a quicker way to restore backups for lemmy, since the backup size of lemmy is much bigger.

NOTE: Data from ca. 2 hours before the site went down(16-18 UTC) will be missing and I'm unable to restore it.

I just upvoted a comment in a community hosted on my site, and in the nginx log I can see:

ip.address - - [19/Feb/2024:14:33:31 +0000] "POST /inbox HTTP/1.1" 200 0 "-" "Lemmy/0.19.3; +https://endlesstalk.org"
ip.address - - [19/Feb/2024:14:33:43 +0000] "POST /inbox HTTP/1.1" 200 0 "-" "Lemmy/0.19.3; +https://endlesstalk.org"

Dumping out the activity, I can see:

... (single vote from account on another instance) ...                   
{
  "@context": [
    "https://join-lemmy.org/context.json",
    "https://www.w3.org/ns/activitystreams"
  ],
  "id": "https://endlesstalk.org/activities/like/f102d062-dfe9-4ece-96e8-310a09b249ba",
  "actor": "https://endlesstalk.org/u/freamon",
  "object": "https://sh.itjust.works/comment/9061222",
  "type": "Like",
  "audience": "https://lemmon.website/c/tails"
}
... (single vote from account on another instance) ...        
{
  "@context": [
    "https://join-lemmy.org/context.json",
    "https://www.w3.org/ns/activitystreams"
  ],
  "id": "https://endlesstalk.org/activities/like/f102d062-dfe9-4ece-96e8-310a09b249ba",
  "actor": "https://endlesstalk.org/u/freamon",
  "object": "https://sh.itjust.works/comment/9061222",
  "type": "Like",
  "audience": "https://lemmon.website/c/tails"
}
... (single vote from account on another instance) ...        

It's the same for comments too (if I grep the dump for other people's comments, there's only one, but there's two for comments made by me)

EDIT: can also see dupes from lemmy.ca (they actually sent the same Like 3 times).

The s3 host that pictrs use, have gone down.

I might have to move to another s3 host, which will mean it will take a bit before images are working. This will cause a loss of images for the last 2-3 hours before pictrs stopped working.

EDIT: Have moved to new s3 host. Unsure how many images were lost during the outage.

Local time can be seen here

0.19.3 is mostly bugfixes. See release notes here for more information.

There should be no downtime or very minimal downtime. If there are any issues, check the uptime site here or site status here

Local time can be seen here

0.19.2 contains fixes for outgoing federation and a few other things. See release notes here for more information.

There should be no downtime or very minimal downtime. If there are any issues, check the uptime site here or site status here

EDIT: The server went down / was very very slow to respond. I'm not quite sure why.

The 0.19 version is out and I expect to update sometime during the next week. Since it is a big release, I will need to spend some more time testing, that everything still works fine and ensure that the migration works without problems as well.

I will update with another post, when I know, when the update will take place.

Cross posting

Cross posting from nostr to Lemmy

@0bd4b3c7

Ye ek ḅot simple guide hæ, meri script, yani "hinglish-lipi", ya ise iski hi lipi me "hiṇgliṣ-lipi", ke bare me :

Rules

+ agr koi alphabet normal diḳ rha hæ, ṯo use væse hi pronounce krna hæ jæse normal hinglish (whatsapp ḅaṣa) me krte hæ :
Example : k = क, p = प , m = म

+ ek vyṇjn ki pronounciation krne ke liye 2 alphabets ki koi zrụrṯ nhi hæ :
Example : नमकीन ≠ namakịn, नमकीन = nmkịn

+ agr koi alphabet alg diḳ rha hæ, yani alphabet ke nịẋe dot ho, to use væse hi Use kreṇge jæse normal hinglish (whatsapp ḅaṣa) me kisi alphabet ke ạge "h" lgakr pronounce krte hæ.
Example : खाना = ḳana , शांति = ṣaṇti
+ yhi same alphabet ke nịẋe dot swar ko bṛi mạt'ra me convert kr deta hæ :
Example : कौन = kꜷn , कीमा = kịma , इमली = imli , कैंची = kæṇẋi
Exception : ṛ = strong 'r' ; ẋ = च ; ẍ = छ

+ ट, त ko normal hinglish me "t" aur ठ, थ ko normal hinglish me "th" liḳṯe hæ, isi vjah se ट, त, ठ, थ ko t, ṭ, ṯ, ṱ liḳa jaṯa hæ respectively.

+ kuẍ hinglish ꜷr english ke words same diḳ skṯe hæ, jæse use aur use, iske liye jb mn kre english word ke phle - lga skṯe hæ
Example : use -use krna ṱa

+ अनुस्वार, चन्द्रबिन्दु lgana hæ ṯo ṇ ka -use krna, word ke last me lgana ho ṯo ẍoṛ ḅi skṯe hæ.

+ agar ị word ke end me ạ rhạ ṯo use i ya ị dono liḳ skte hæ, væse agr word ke bịẋ me ạ rhạ hæ to ḅi dono -use kr skṯe hæ lekin isse confusion ho skṯi h

+ same ạ ke liye ḅi hæ, bs ạ me confusion nhi hoṯi generally, to ạ ka -use bs words ki starting me krṯe hæ

+ hlṇṯ yani ạḓe vyṇjn ya ' ka -use krṯe hæ : Example : क्या = k'ya or kya

+ vyṇjn ke bad svr, jæse नई ko ni nhi likh skṯe isliye "n`ai" , agar नाई liḳna hoṯa ṯo "nai" liḳṯe hæ siḓa.

+ pronounciation ke vjh se spelling mistakes acceptable hæ, jæse बहुत ko "bhuṯ" ki jgh "ḅoṯ" liḳna, lekin isse confusion ho skṯi hæ !

v1.0

@nerd

Real Shitpost # 1

Hinglish Plus Pro Script made out of unicode characters

Consonants

क 𑂍
ख ખ *
ग 𑂏
घ 𑂐
ङ ઙ *
च ʧ / 𑂓'
छ 𑂓
ज 𑂔
झ Ɉ / 𑂕
ञ 𑂖
ट 𑂗
ठ ઠ *
ड ડ
ढ 𑂛
ण 𑂢
त ત *
थ 𑂟
द ໔ * Lao digit
ध ຍ * Lao letter
न 𑂢
प 𑂣
फ ફ / ꜰ *
ब ദ * malyalam
भ 𑂦
म 𑂧
य ય *
र ર *
ल ന * malayalam
व વ *
श શ *
ष 𑂭
स 𑂮

1. ह 𑂯ऩ 𑂢ऱ રक़ 𑂍'ख़ ખ'ग़ 𑂏'ज़ 𑂔'ड़ 𑂚ढ़ 𑂜फ़ ફ'य़ ય'क्ष ક્ષज्ञ જ્ઞ

@nerd

Color Combo in Pair that are eye candy to me

Contrasting Red, Blue, Green, Yellow, Black as major and white or silver as minor

@lemmy

Welcome !

This post is crossposted from Mastodon to Lemmy

@lemmy

The update fixes an issue with moderation actions sometimes not federating correctly(See release notes here).

There should be no or very minimal downtime. To convert UTC to your local time, use this

First post. More to come!

I expect a very minimal downtime of ca. 5-15 mins.

I had made some config changes to the database earlier in connection with the move to a new server, that caused the storage usage of the database to grow a lot. Then when it had no more space left, it crashed which caused the downtime. Unfortunately it happened at a time, where I wasn't available to fix immediately, which is why it was down for so long.

It is now fixed and I will keep a watch(setup an alert for database disk usage) to make sure it doesn't happen again.

Seems to be caused by low amount of available storage. This caused k8s to evict/delete pods -> site went down,, then it would fix itself -> site goes up again but then k8s would evict again -> site goes down. This continued untill it stabilized at some point.

This should be fixed and there should be more space available, when I move the server to a new host. I expect to move to a new server sometimes in the comming week. Will annonce the date, when I know, when it will happen.

EDIT: Spoke a little bit too soon, should be fixed now though.

EDIT2:

There was something that kept using storage, so ran into the issue again. Then the volume/storage for the image service(pictrs) stopped worked for some unknown reason(Thankfully, I have backups) and there shouldn't be any images lost.

Good news is that I have reclaimed a lot of storage, so shouldn't be in danger of running out of space for a long time.

While preparing for migration to a new host, I had to setup the db, but during that I deleted a resource in k8s to force a reload of settings in the db. This caused the db to use a different volume and it took a bit before I could revert it back to using the old volume.

No data should have been lost. Let me know, if anything is missing.

Same thing as yesterday.

Unfortunatly the tool for scanning CSAM didn't detect the image, so to ensure there are no CSAM on the server, images for the last 8 hours has been deleted.

The filesystem manager(longhorn) I use reported that multiple volumes were faulted. This caused the site to go down.

I have no idea why the volumes faulted, only that a reboot of the server fixed it. Hopefully this was a strange one off and it doesn't occur again.

To make it easier to deferate from unwanted instances I have switched to using the fediseer. With this tool I can get censures from other trustworthy instances. A censure is a "completely subjective negative judgement"(see more here) and reasons for the censure can be listed.

Currently I'm using the censures from lemmy.dbzer0.com(Can be seen here), that has any of the following reasons for the censure

• Lolicon
• CSAM
• Fascism
• Hate speech
• Bigotry
• Pedophilia
• Bestiality
• MAP

I will still manually deferate from instances, when it is needed, but this makes easier to deferate bad instances I would have missed/didn't know about.

Note: The automated deferation also includes spam instances, which is currently defined by

• More than 30 registered users per local post + comments
• More than 500 registered users per active monthly user.

The main reason is that they allow/support pedophilia, but they also allow zoophilia and biastophilia. They try to label it as MAP(minor-attracted person), but it is still pedophilia. Example of MAP post here.

There have been a report of CSAM and unfortunately the lemmy-safety doesn't go through the images quickly enough(on my hardware) to be of use in this case.

I think there exists a tool to purge a image via information from a post, but wasn't unable to find it now. In the future I can hopefully use that tool, when reports of CSAM come in.