datahoarder

1. Copy the document ID. For example, if the URL is https://docs.google.com/document/d/16Asz8elLzwppfEhuBWg6-Ckw-Xtf/edit, then the ID is 16Asz8elLzwppfEhuBWg6-Ckw-Xtf.
2. Open this URL: https://docs.google.com/document/u/1/d/poop/revisions/load?id=poop&start=1&end=1 (replace poop with the ID from the previous step). You should see a json file.
3. Add 0 to the end of the number after end= and refresh. Repeat until you see an error page instead of a json file.
4. Find the highest number that makes a json file instead of an error page appear. This involves repeatedly trying a number between the highest number known to result in a json file and the lowest number known to result in an error page.
5. Download the json file. You might need to remove the characters before the first {.

I found the URL format for step 2 here:

https://features.jsomers.net/how-i-reverse-engineered-google-docs/

I am working on an easy way. Edit: here it is https://lemmy.ca/post/21281709

cross-posted from: https://programming.dev/post/13631943

Firefox Power User Keeps 7,400+ Browser Tabs Open for 2 Years

cross-posted from: https://leminal.space/post/6179210

I have a collection of about ~110 4K Blu-Ray movies that I've ripped and I want to take the time to compress and store them for use on a future Jellyfin server.

I know some very basics about ffmpeg and general codec information, but I have a very specific set of goals in mind I'm hoping someone could point me in the right direction with:

1. Smaller file size (obviously)
2. Image quality good enough that I cannot spot the difference, even on a high-end TV or projector
3. Preserved audio
4. Preserved HDR metadata

In a perfect world, I would love to be able to convert the proprietary HDR into an open standard, and the Dolby Atmos audio into an open standard, but a good compromise is this.

Assuming that I have the hardware necessary to do the initial encoding, and my server will be powerful enough for transcoding in that format, any tips or pointers?

So it's been a a few years since I've bought hard drives for my little home server and wanted to get a bead on what's the target on dollar to TB in the post Covid world. Thanks!

Hello, I'm wondering what do you guys use and recommend for efficient book, comic, manga and lightnovel file management, tagging, directory structures and automated tools for all that.

My collection is mostly made from humble bundle book bundles, for getting tags into comics I use comictagger and as for file structure, it was mostly just me just putting something to separate the books.

I wan't to hear you guys input because most of you are a lot more efficient or have a lot more experience in saving big ammounts of data, and I wan't to make my process as painless and future proof as possible as my collection starts to grow.

Edit: I use linux so software like comicrack which I heard a lot about isn't really accessible to me. The files also need to be accessible to my kavita server.

How to store digital files for posterity? (hundreds of years)

I have some family videos and audios and I want to physically save them for posterity so that it lasts for periods like 200 years and more. This allows great-grandchildren and great-great-grandchildren to have access.

From the research I did, I found that the longest-lasting way to physically store digital content is through CD-R gold discs, but it may only last 100 years. From what I researched, the average lifespan of HDs and SSDs is no more than 10 years.

I came to the conclusion that the only way to ensure that the files really pass from generation to generation is to record them on CDs and distribute them to the family, asking them to make copies from time to time.

It's crazy to think that if there were suddenly a mass extinction of the human species, intelligent beings arriving on Earth in 1000 years would probably not be able to access our digital content. While cave paintings would probably remain in the same place.

What is your opinion?

I formatted my PC recently and I'm reinstalling some stuff I forgot to make a backup of settings like yt-dlp so I started searching for what is a good config to download the best mp4 quality and found some interesting setups so I figured I'd make a thread for people to share what they use.

Here's the most best setup I found so far which downloads a 1080p mp4 with the filename and includes metadata, english subtitles, and chapters if available: yt-dlp -f 'bestvideo[height<=1080][ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best' -S vcodec:h264 --windows-filenames --restrict-filenames --write-auto-subs --sub-lang "en.*" --embed-subs --add-metadata --add-chapters --no-playlist -N 4 -ci --verbose --remux-video "mp4/mkv" URL

Ideally it would also mark the sponserblock section and download to a specified folder

@datahoarder The Internet Archive Just Backed Up an Entire Caribbean Island
https://www.wired.com/story/internet-archive-backed-up-aruba-caribbean-island/

Hi, anyone could point me at an ELI5 about SAS hardware? I'd like to assemble a NAS using an old HP Z200, I want SAS because I'd get also a tape drive for backups and I cannot find SATA tape drives. For example, is a Dell Perc H310 pci-e card good for me? Can I avoid hardware RAID?

While clicking through some random Lemmy instances, I found one that's due to be shut down in about a week — https://dmv.social. I'm trying to archive what I can onto the Wayback Machine, but I'm not sure what the most efficient way to go about it is.

At the moment, what I've been doing is going through each community and archiving each sort type (except the ones under a month, since the instance was locked a month ago) with capture outlinks enabled. But is there a more efficient way to do it? I know of the Internet Archives save from spreadsheet tool, which would probably work well, but I don't know how I'd go about crawling all the links into a sitemap or csv or something similar. I don't have the know-how to setup a web crawler/spider.

Any suggestions?

Seems the SSD sometimes heats up and the content disappears from the device, mostly from my router, sometimes from my laptop.
Do you know what I should configure to put the drive to sleep or something similar to reduce the heat?

I'm starting up my datahoarder journey now that I replaced my internal nvme SSD.

It's just a 500GB one which I attached to my d-link router running openwrt. I configured it with samba and everything worked fine when I finished the setup. I just have some media files in there, so I read the data from jellyfin.

After a few days the content disappears, it's not a connection problem from the shared drive, since I ssh into the router and the files aren't shown.
I need to physically remove the drive and connect it again.
When I do this I notice the somewhat hot. Not scalding, just hot.

I also tried this connecting it directly to my laptop running ubuntu. In there the drive sometimes remains cool and the data shows up without issue after days.
But sometimes it also heats up and the data disappears (this was even when the data was not being used, i.e. I didn't configure jellyfin to read from the drive)

I'm not sure how I can be sure to let the ssd sleep for periods of time or to throttle it so it can cool off.
Any suggestion?

What in your hoard do you treasure the most? I imagine to a lot of us it is photos and videos of our families, which I'd love to hear about, but also interested in rare bits of media or information that makes your collection unique.

Last year Elon Musk accidentally revealed that he has a burner account on Twitter called @ErmnMusk.

Now that account is gone.

I'm looking for an archive: its tweets, its likes, anything and everything. Does anyone know where to find?

So I’ve been consolidating all of my storage and removing all the duplicates and junk files.

In actual physical storage, this was spread across 12TB worth of hard drives, all partially full.

After everything was said and done, I’m using 1.3TB of space if you don’t include games. ¯\_(ツ)_/¯

This is stuff dating back to 2015. Sometimes it’s actually worth it to just clean up your junk files.

I imagine a lot of people who are into data hoarding already know a lot of this but I thought the video was pretty neat. It briefly talks about the history of different compression formats and provides a brief blurb about why you may want to use one or the other.

I'd recommend checking it out if you want 15 minutes of background noise.

For anyone new to data compression TechQuickie and CrashCourse have videos on it. If you really want to go down the rabbit hole you could check out media compression and see how things like JPEGs and PNGs work.

cross-posted from: https://sh.itjust.works/post/14280067

What is the best tool to get URLs for all tweets within a given date range?

The ideal behaviour I'm looking for would be something like this:

Input: https://twitter.com/SpaceX 2023-09-01 2024-02-08

Output:

• https://twitter.com/SpaceX/status/1755763378449183003#m
• https://twitter.com/SpaceX/status/1755759459765567825#m
• https://twitter.com/SpaceX/status/1755752291578302545#m
• ...

What would be the best tool to achieve this? Thanks in advance!

Not sure if this is better fit for datahoarder or some selfhost community, but putting my money on this one.

The problem

I currently have a cute little server with two drives connected to it running a few different services (mostly media serving and torrents). The key facts here is that 1) it's cute and little, 2) it's handling pretty bulky data. Cute and little doesn't go very well with big raid setups and such, and apart from upgrading one of the drives I'm probably at my limit in terms of how much storage I can physically fit in the machine. Also if I want to reinstall it or something that's very difficult to do without downtime since I'd have to move the drive and services of to a different machine (not a huge problem since I'm the only one using it, but I don't like it).

Solution

A distributed FS would definitely solve the issue of physically fitting more drives into the chassi, since I could basically just connect drives to a raspberry pi and have this raspi join the distributed fs. Great.

I think it could also solve the issue of potential downtime if I reinstall or do maintenance, since I can have multiple services read of the same distributed FS and reroute my reverse proxy to use the new services while the old ones are taken offline. There will potentially be a disruption, but no downtime.

Candidates

I know there are many different solutions for distributed filesystems, such as ceph, moosefs, glusterfs and miniio. I'm kinda leaning towards ceph because of it's integration in proxmox, but it also seems like the most complicated solution in the bunch. Is it worth it? What are your experiences with these, and given the above description of my use-case which do you think would be the best fit?

Since I already have a lot of data it's a bonus if it's easy to migrate from my current filesystem somehow.

My current setup uses a lot of hard links as well, so it's a big bonus if the solution has something similar (i.e. some easy way of storing the same data in multiple places without duplicating it)

cross-posted from: https://lemmy.dbzer0.com/post/13532369

DDoS secrets responsible for hosting leaks such as EpikFail and BlueLeaks will stop its activities, I would like help from anyone who has space left so we can download everything and keep seeding.

Torrent download links: https://data.ddosecrets.com/

I'm going to archive some youtube videos, what's the proper way to change it from mp4 to webm to etc etc or vice versa?

In the past when I couldn't run a video file for whatever reason I would just rename the file, but I'm assuming there's better ways to do it. And is there a specific order I have to go in? (e.g. with audio going from .mp3 to .flac doesn't make sense.)

Thanks in advance.