The original post: /r/datahoarder by /u/LeviAEthan512 on 2024-12-26 04:25:28.

Hi, does anyone have recommendations for an expansion card that's good for data hoarding?

Right now, I've got a USB HDD enclosure. I hear it's best to go for eSATA, so I think I'll switch to that. Unless the latest USB advances have made that irrelevant)

I also have a USB expansion card, but it's kinda sketchy, so that's another reason I want to upgrade.

Now I have two problems. I'm using a couple of those ports on my USB card, and I only have one available PCIe slot (the rest are blocked by my graphics card). So whatever card I get needs to have both USB and eSATA. I'm not sure this exists. I can't find it in any case. I've found cables that go eSATA to USB-C, but they're all crazy expensive compared to USB cables, and I don't recognise any of the brand names either.

If I can't get both, then I'm stuck on USB for the foreseeable future. In that case, I would like a proper card from a reputable brand. Does anyone know any? The Chinese one I'm using works find 90% of the time, but causes a wide range of problems infrequently. I know it's the card because I've used it in 3 different systems and they've all encountered these problems, only when the card is in use.

The original post: /r/datahoarder by /u/Imaginary_War9923 on 2024-12-26 00:53:01.

Hi strangers,

I have been working on a media scraper for a few months for my own archiving and it is finally ready to be released to the public~

This bot is designed as a comprehensive tool for managing and automating downloads using a self-reliant database. It offers both manual and automatic entry through a command line tool and a batch file. As well as a guided command line interface to add, manage, and delete entries as well as modify a global blacklist system to block tags on BOORU sites. It also currently supports Pixiv downloads.

Note that this program doesn't currently support anything but gelbooru and Pixiv but that will be changing within the next week or so as I refine the bot :)

Anyway, here is a link to the project, keep an eye on it because I am working on improving it, Happy hoarding!

https://github.com/Waffles-54/scraping-bot-manager

The original post: /r/datahoarder by /u/olo99 on 2024-12-26 00:09:07.

I got some questions that I hope to get some clarification on:

I have 8 x WD RED SSD (4TB) that I want to set up with Drivepool + SnapRAID.

1. Is 2 disk parity for SnapRAID still recommended even when using SSDs and not regular HDDs, or is 1 disk good enough?

2. Is there any point in getting a PLP SSD to use as cache drive in addition to the other ones? (at the moment I don't have a UPS, would PLP give a little protection for sudden shut downs?)

3. I have seen some using 2 SSDs as cache drives (1 with duplication). And then "dumping" the data over to the storage drives and then have SnapRAID just run off the storage drives. In my case all drives will be SSDs, would this still benefit in any way? Or would it be better just having all drives (without parity) in just one big storage pool? I know SnapRAID doesn't like to be run when anything is active.

For example would this be ideal:

disk 1 cache drive

disk 2 duplicate of 1

disk 3-7 storage

disk 8 parity (SnapRAID).

This would then give 24TB total storage. (down to 20 if double parity).

Or what would you recommend? :)

The original post: /r/datahoarder by /u/22marks on 2024-12-26 03:06:20.

So, the newest full English Wikipedia ZIM files are 102GB, which is just too large for a 100GB Bluray. Does anyone know of a version that strips out just 2GB of more obscure entries. Or perhaps obscure multimedia, but keeps the majority? I'd like to have a local backup on Bluray, but don't know the best way to go about truncating/pruning a ZIM.

I know I can use various tools to do this manually, but is anyone aware of a "100GB ZIM" already made?

The original post: /r/datahoarder by /u/Chief_James592 on 2024-12-26 02:49:44.

With the whole Ban coming up in January I need something that can download videos in bulk preferably free but I donk know how to use java scrips so is there any solutions ?

The original post: /r/datahoarder by /u/Chief_James592 on 2024-12-26 02:48:02.

With the whole Ban coming up in January I need something that can download videos in bulk preferably free but I donk know how to use java scrips so is there any solutions ?

The original post: /r/datahoarder by /u/Chief_James592 on 2024-12-26 02:48:02.

With the whole Ban coming up in January I need something that can download videos in bulk preferably free but I donk know how to use java scrips so is there any solutions ?

The original post: /r/datahoarder by /u/GorillaRimjob on 2024-12-26 02:43:13.

I have some movies that were shared to me on Koofr, but the download option was disabled. The file doesn't allow hotlinking. I cant download from developer tools (or at least I cant figure it out). Can't find anything on github. I tried multiple chrome extensions. I have no idea what to do other than screen record the entire movie. Please help!

The original post: /r/datahoarder by /u/hellojello2016 on 2024-12-26 02:19:33.

Like the title says, I don’t want to buy from Amazon from past bad experience. Where do you all buy enterprise grade drives besides Amazon?

The original post: /r/datahoarder by /u/Flashy-Ad-591 on 2024-12-26 00:24:00.

I'm looking to expand my storage. At the moment I have 3 4TB drives connectioned to a multi-usb. As you can imagine, the speeds are atrocious. I'm looking into NAS, but I have no experience whatsoever with it. Could anyone give some advice on a good NAS solution (I'd be looking for something around the ~80 TB mark). Recommended storage drives would also be really appreciated.

The original post: /r/datahoarder by /u/NXGZ on 2024-12-25 22:46:51.

The original post: /r/datahoarder by /u/realdealneal18 on 2024-12-25 21:23:58.

The original post: /r/datahoarder by /u/BobbythebreinHeenan on 2024-12-25 20:54:00.

im Going through the process of upgrading my videos to 4k. I’ve been using 1080p for a long time. I’d been storing everything on 16 tb western digital mybook external drive. But with this 4k upgrade I’m working on, I bought two WD 20tb external,drives. And I’ve got maybe 5tb left.

I kept the original 16tb as my main backup for essential stuff that I can’t recover. rare videos and files. The rest is expendable on my 20tb drives. Though I’d be sad if it was damaged. But everything is u;timately replaceable.

my main question is do I need to quit Mickey Mousing around playing with these WD drives? Every couple years I upgrade my drives and get a bigger external and transfer everything over. Leaving the old ones for backup.

is this an ok setup for video content? I’d like to set up a plex server soon. Or even run Infuse. Do I need to bite the bullet and buy something more serious?

The original post: /r/datahoarder by /u/x0b0t on 2024-12-25 20:24:53.

Very basic, reverse engineered, Google Photos mobile API client.

Made for uploading files as Pixel XL without relying on a physical device/emulator.

Media uploaded with it will not take up account's storage.

https://github.com/xob0t/gphotos_mobile_client

The original post: /r/datahoarder by /u/icebreaker374 on 2024-12-25 19:57:32.

The original post: /r/datahoarder by /u/milkygirl21 on 2024-12-25 04:35:47.

I just bought a 12TB Barracuda Pro and currently doing a hard format on it. Is SeaTools or Victoria (or any other tool recommended) to scan & repair bad sectors before I start using the drive for backup? Anything else I should check for? Thank you.

The original post: /r/datahoarder by /u/slippy_3 on 2024-12-25 04:34:16.

Hello!

I'm hoping to get some feedback on my new backup/archival solution. I'd appreciate any thoughts y'all have. I'm trying to see if there's anything I've overlooked or could improve.

Background

I'm a new filmmaker and have a few projects under my belt, so about 4TB of data, and I have will be growing my video footage archive. The plan is to hang onto the footage for now at least, and will be adding to it with subsequent projects.

Equipment/Software

• Macbook M1 Pro
  • I'm keeping a copy of most of my files in iCloud
• 4 Bay Terramaster DAS with support for USB 3.1 Gen2
• 3x 8TB Toshiba N300
• 1x 6TB older WD Elements drive
  • I've been using this as my storage drive until recently
• 1x 6TB new WD MyBook drive
• Apple Time Machine
• Carbon Copy Cloner
• 2TB iCloud plan

Storage/Archival Plan

Drive One - Archive 8TB

• Using the DAS, I'll be storing all my footage for completed projects on one of the 8TB Toshiba drives.

Drive Two - Backup #1 8TB

• A second 8TB drive will be in the DAS but only serves as a backup to the first drive.
• Using Carbon Copy Cloner, I run a task to copy all data from the Archive drive to the Backup #1 and have it check the files to make sure everything is copied correctly.

Drive Three - Backup #2 8TB

• A third drive also dedicated to backing up the Archive drive will with Carbon Copy Cloner, but after this initial backup, it will be moved to an offsite location.

Personal Backup Plan

• Both external WD drives will be backed up using Time Machine.
• One of the external drives will be moved offsite.
• Most files on my personal computer will also be synced to iCloud.

Offsite Plan

• Periodically, after making sure everything is up to date with the active drives, I will take them ( the backup of the Archive drive from the DAS, as well as the active external WD drive) to the offsite location where I store the second Archive backup and Time Machine backup drives, and swap them out.
• Leaving the most updated copy offsite, I'll bring the out of sync drives back to my house and update them, then use them as the new active Archive backup and Time Machine backup drives.
• Rinse and repeat.

This means I'll always have the Archive drive at home with my personal data on the computer + iCloud, even when moving the secondary drives to the offsite location.

Thoughts on cloud storage, RAID, and NAS

• I was using Backblaze rather than using the offsite location to store the backup drives, but after some testing, I hated the lack of detail the software provided on the current status of my backup. I also was able to confirm that things were not backed up to the cloud after the initial backup, even though I gave them overnight to be backed up (with a Gigabit connection over ethernet). I read other users concerns around Backblaze not updating, and after my own personal experience, decided to stop using their service.
• I started using IDrive, who's software I liked much better (though that's not saying much), and I appreciated knowing I am paying for a specific amount of storage, and not feeling like I'm getting away with something like I did on the Backblaze personal plan. However, I really don't like the fact that IDrive charges an exorbitant amount when you go over your capacity, and they don't keep the external drives in sync, i.e. if you make changes to your external drives, it only ever copies over new files, it doesn't remove deleted files or rename existing files, creating duplicates, with no way to automatically sync them. This would lead to it being much easier to go over your storage allotment I've paid for and end up with surprise charges.
• I've never used RAID, and in research for my situation, I'm not sure it would really benefit me in any way, as I'm not looking for speed, just resiliency of my data. Plus I couldn't use the snapshot feature integrated with Carbon Copy Cloner.
• I ultimately decided that I'd rather be completely in control of my data, and not have to trust a cloud company with (in my experience) mediocre customer support.
• As cool as a NAS would be, I don't really have any use to access the data over the network. It's much faster to plug it into my computer directly to transfer files. Plus I didn't like the idea of wearing out the drives by leaving them on all the time, not to mention the increased risk of a power outage or surge damaging my equipment.

Concerns with my setup

• Since the Archive and Backup drives are offline the majority of the time, is this going to cause any issues with the drives? I read something about them losing magnetism or something when not in use. But I'm assuming that's for long periods of time, like years. Whereas with me they will be off at most a couple of months.
• I'm using a hard case with foam inserts to transport the drives to the offsite location, but will the packing and unpacking of the drives increase chances of damage?
• The offsite location is still within the same state and not that far from me, so I know it won't completely protect my data if there was some major disaster. But if that happens, I don't think I'll care much about some old footage. I'm also using iCloud for my personal data, so I'll have a copy in the cloud somewhere at least.

I know that was a lot! So if you made it this far, I appreciate it. Please let me know if you see any flaws in my plan or ways to increase efficiency. Thanks so much!

The original post: /r/datahoarder by /u/Beavisguy on 2024-12-25 19:28:48.

I am looking at getting a new internet SSD as My boot drive look at getting a 2tb Samsung 980 Pro, Crucial P3 Puls or WD Black SN770 ??

The original post: /r/datahoarder by /u/nig8mare on 2024-12-25 19:00:26.

Gbatemp is a forum that I want to archive some profiles but any attempt to do so using save page now on archivedotorg will just archive the login page. Archivewebdotpage would be a good solution for this as I already have the account but I'm not sure if the login cookies could get saved alongside the archive which worries me a bit. Can someone tell me if I'm wrong? Searching up anything about this didn't give any results. I'm using the electron app of archivewebdotpage specifically

The original post: /r/datahoarder by /u/TheCountofNotreDame on 2024-12-25 18:48:34.

Hi All!

Is anyone else using windows storage pool for Radarr/Plex? I have verified my hardlinks are working, yet windows is showing different capacity in Storage pool vs. File Explorer for my drive D below (which is three 22 TB HDDs pooled on a windows PC). Has anyone else had this issue?

https://preview.redd.it/4oyxnhgzbv8e1.png?width=1070&format=png&auto=webp&s=21096f231912e83e11ba010b56c9203988441a66

https://preview.redd.it/5j8kxw90cv8e1.png?width=1397&format=png&auto=webp&s=19b79d6bc852125eb725b42b936e50a5e2527b85

The original post: /r/datahoarder by /u/randdude220 on 2024-12-25 18:03:58.

tl;dr:

Looking for a system to tag files in such a way that the information is accessible from all devices. Eg. photos of a specific family member will be tagged #person_name_surname then I can find all photos, videos and documents of this person regardless of the folder they are located in or what device OS I use. Maybe some program that adds "tag this file" to right click context menu and then adds the tag(s) to file metadata somehow?

Details:

I have multiple TB of data currently in Google Drive. I am thinking about migrating to Dropbox for various reasons. You hardcore veteran datahoarders will probably disapprove of this decision but my #1 priority of cloud storage is speed of usage and convenience. I want the experience of regular uploading to be as frictionless as possible in Mac, Windows and IOS because I do it daily and if it's even a minor hassle I'm going to start procrastinating on it unfortunately. I basically use it to catalog my everyday life photos and videos like a diary of some sorts. Needless to say I will incorporate some kind of automatic offline backup systems too.

Anyway I have my files very meticulously organized into folders but there are many files that I want to group additionally so some kind of labeling/tagging system that is independent from folders seems to be the way to go but I'm not sure how to approach this. Should I find some kind of application that adds some metatags to files or...?

An example process: I want to tag some files in my Mac folder that is synced to Dropbox. Now I would like to see these tags also in the offline backup storage that automatically one-way syncs from Dropbox to my home backup server HDD (and of course preferably to other locations in the future) while using Windows PC, so OSX's own tagging system is out of the question. I don't want to use Dropbox's own tagging system either in case I want to stop using it someday.

Is that kind of thing even possible?

Some additional info to maybe help find a solution:

• I will do most of the tagging only from Mac and Windows so mobile device support is only optional.
• Only files will be tagged not folders.
• All types of files must be taggable, not only photos and videos.
• Bulk tagging should be possible to shorten the time spent on this whole process.
• Finding files by multiple tag names should be possible (eg. #person_name & #location_home).
• Tag information must not get lost in any process of file transferring.

I would be greatly appreciated by any info or tip! Thanks in advance!!

The original post: /r/datahoarder by /u/Renrut23 on 2024-12-25 17:37:52.

I know there's a lot of debate about these cables. Looking to use molex to sata to bypass the 3.3v power disable feature on my drives. I ordered these

Cable Matters 3-Pack 4 Pin Molex... https://www.amazon.com/dp/B00VJ9V8NY

But now I'm not sure of the pigtails and if it's better to go 1 to 1. I figured cable matters is a good brand so should be safe, or is there better ones to go with.

The original post: /r/datahoarder by /u/hasanbeder on 2024-12-25 17:07:42.

Hey fellow directory explorers! 👋

I've been working on something cool that I think you'll love - M3Unator! It's a userscript that makes creating playlists from open directories a breeze. You know those times when you find an awesome media directory but manually creating playlists is a pain? That's exactly why I built this.

https://github.com/hasanbeder/M3Unator

✨ What's Cool About It:

• 🎬 Works with pretty much any media format you can think of (40+ formats!)
• 🔍 Smart enough to find all your media files automatically
• 🌲 Can dig through subdirectories (you control how deep)
• 📊 Shows you exactly what's happening while it works
• 🎨 Clean, modern interface that doesn't get in your way
• 🔒 100% private - everything happens in your browser

🚀 Want to Try It?

1. Grab your favorite userscript manager (I recommend Tampermonkey)
2. Install M3Unator from:
   • Greasy Fork
   • GitHub

💡 Super Simple to Use:

1. Find an open directory with some media
2. Click the M3Unator button
3. Pick your settings
4. Hit generate and you're done!

I'm actively working on this and would love to hear what you think! Any feedback, feature requests, or bug reports are super welcome. Hope this makes your media organizing life a bit easier! 🎉

P.S. It works with Apache, Nginx, Lighttpd, and pretty much any standard directory listing. Give it a shot and let me know how it goes!

The original post: /r/datahoarder by /u/Desperate_Eye_09 on 2024-12-25 16:40:36.

Hi i am enrolled in a course which is provided from a online coaching institute, which is paid. now it is about to end in next month, but i want to download all the content from that site, so anybody can help me out. Content is officially locked and we can not download, so i need who can break this.

The original post: /r/datahoarder by /u/Rugta on 2024-12-25 16:18:03.

Assuming no redundancy, what's the fastest sequential and random read/write speeds you've gotten?