Linux

The old answer is a chroot jail, the new answer is a Docker container or VM if Docker won't cut it.

I'm lazy, so Virtualbox is my VM software of choice. I keep a machine with a fresh debian install and just Clone it to make throwaway VMs.

Keep in mind that malicious software on a VM might be isolated from the host in many ways, but if it's allowed to communicate on your network then it can still be dangerous, especially if you have samba shares, or services you don't expose to the outside internet with weak or default passwords. (Did you change the admin password on your router's Web interface?)

Creating a VM with no network interfaces is "mostly safe", but you hear about VM bust out exploits now and then.

In reality, gold standard is a separate physical computer with no network connections to anything but other untrusted physical computers, and no wireless adapters (Bluetooth or WiFi). This is an "air gapped" network, but if you're dealing with shit that makes you want an air gap, either you already know more than you're gonna learn on Lemmy, or you're bout to get your door kicked in by men in black suits :D

You might want to look into Firejail, kinda complicated to setup but it's made for this.

I think chroot could achieve this too but I don't know how secure it is

Depending on what level of malware you’re worried about, running it in a Docker container could be a solution.

I've created a tool for similar of use-cases: https://codeberg.org/contr/contr
You could run your workload inside, say, an alpine container:

cd path/to/evil/dir
contr alpine
❯ # inside container, run dangerous program
❯ ./dangerous_program

If the program needs extra dependencies, you'll have to write a Containerfile and build an image with the dependencies installed – there's an example in the repository. Just installing the dependencies at runtime inside the container is also an option, but all changes inside the container are lost on exit.

Look up chroot

To be completely safe I'd just use a VM, if you think that is an overkill then I suggest Bubblewrap.