Commenting incase someone bumps into this issue in the future:
I couldn't figure it out and nothing worked. Later I wanted to delete commits so I made a new repo and pushed the files there and just out of curiosity tried disabling "Releases" and "Packages" in the new repo and it worked!
I guess it's a bug. Just deleted the old repo.