This article will describe how to download an image from a (docker) container registry.

Manual Download of Container Images with wget and curl

Intro

Remember the good `'ol days when you could just download software by visiting a website and click "download"?

Even apt and yum repositories were just simple HTTP servers that you could just curl (or wget) from. Using the package manager was, of course, more secure and convenient -- but you could always just download packages manually, if you wanted.

But have you ever tried to curl an image from a container registry, such as docker? Well friends, I have tried. And I have the scars to prove it.

It was a remarkably complex process that took me weeks to figure-out. Lucky you, this article will break it down.

Examples

Specifically, we'll look at how to download files from two OCI registries.

1. Docker Hub
2. GitHub Packages

Terms

First, here's some terminology used by OCI

1. OCI - Open Container Initiative
2. blob - A "blob" in the OCI spec just means a file
3. manifest - A "manifest" in the OCI spec means a list of files

Prerequisites

This guide was written in 2024, and it uses the following software and versions:

1. debian 12 (bookworm)
2. curl 7.88.1
3. OCI Distribution Spec v1.1.0 (which, unintuitively, uses the '/v2/' endpoint)

Of course, you'll need 'curl' installed. And, to parse json, 'jq' too.

sudo apt-get install curl jq

What is OCI?

OCI stands for Open Container Initiative.

OCI was originally formed in June 2015 for Docker and CoreOS. Today it's a wider, general-purpose (and annoyingly complex) way that many projects host files (that are extremely non-trivial to download).

One does not simply download a file from an OCI-complianet container registry. You must:

1. Generate an authentication token for the API
2. Make an API call to the registry, requesting to download a JSON "Manifest"
3. Parse the JSON Manifest to figure out the hash of the file that you want
4. Determine the download URL from the hash
5. Download the file (which might actually be many distinct file "layers")

One does not simply download from a container registry

In order to figure out how to make an API call to the registry, you must first read (and understand) the OCI specs here.

• https://opencontainers.org/release-notices/overview/

OCI APIs

OCI maintains three distinct specifications:

1. image spec
2. runtime spec
3. distribution spec

OCI "Distribution Spec" API

To figure out how to download a file from a container registry, we're interested in the "distribution spec". At the time of writing, the latest "distribution spec" can be downloaded here:

• https://github.com/opencontainers/distribution-spec/releases/tag/v1.1.0
• https://github.com/opencontainers/distribution-spec/releases/download/v1.1.0/oci-distribution-spec-v1.1.0.pdf

The above PDF file defines a set of API endpoints that we can use to query, parse, and then figure out how to download a file from a container registry. The table from the above PDF is copied below:

ID	Method	API Endpoint	Success	Failure
end-1	GET	/v2/	200	404/401
end-2	GET / HEAD	/v2/<name>/blobs/<digest>	200	404
end-3	GET / HEAD	/v2/<name>/manifests/<reference>	200	404
end-4a	POST	/v2/<name>/blobs/uploads/	202	404
end-4b	POST	/v2/<name>/blobs/uploads/?digest=<digest>	201/202	404/400
end-5	PATCH	/v2/<name>/blobs/uploads/<reference>	202	404/416
end-6	PUT	/v2/<name>/blobs/uploads/<reference>?digest=<digest>	201	404/400
end-7	PUT	/v2/<name>/manifests/<reference>	201	404
end-8a	GET	/v2/<name>/tags/list	200	404
end-8b	GET	/v2/<name>/tags/list?n=<integer>&last=<integer>	200	404
end-9	DELETE	/v2/<name>/manifests/<reference>	202	404/400/405
end-10	DELETE	/v2/<name>/blobs/<digest>	202	404/405
end-11	POST	/v2/<name>/blobs/uploads/?mount=<digest>&from=<other_name>	201	404
end-12a	GET	/v2/<name>/referrers/<digest>	200	404/400
end-12b	GET	/v2/<name>/referrers/<digest>?artifactType=<artifactType>	200	404/400
end-13	GET	/v2/<name>/blobs/uploads/<reference>	204	404

In OCI, files are (cryptically) called "blobs". In order to figure out the file that we want to download, we must first reference the list of files (called a "manifest").

The above table shows us how we can download a list of files (manifest) and then download the actual file (blob).

Examples

Let's look at how to download files from a couple different OCI registries:

1. Docker Hub
2. GitHub Packages

Docker Hub

To see the full example of downloading images from docker hub, click here

GitHub Packages

To see the full example of downloading files from GitHub Packages, click here.

Why?

I wrote this article because many, many folks have inquired about how to manually download files from OCI registries on the Internet, but their simple queries are usually returned with a barrage of useless counter-questions: why the heck would you want to do that!?!

The answer is varied.

Some people need to get files onto a restricted environment. Either their org doesn't grant them permission to install software on the machine, or the system has firewall-restricted internet access -- or doesn't have internet access at all.

3TOFU

Personally, the reason that I wanted to be able to download files from an OCI registry was for 3TOFU.

Verifying Unsigned Releases with 3TOFU

Unfortunaetly, most apps using OCI registries are extremely insecure. Docker, for example, will happily download malicious images. By default, it doesn't do any authenticity verifications on the payloads it downloaded. Even if you manually enable DCT, there's loads of pending issues with it.

Likewise, the macOS package manager brew has this same problem: it will happily download and install malicious code, because it doesn't use cryptography to verify the authenticity of anything that it downloads. This introduces watering hole vulnerabilities when developers use brew to install dependencies in their CI pipelines.

My solution to this? 3TOFU. And that requires me to be able to download the file (for verification) on three distinct linux VMs using curl or wget.

⚠ NOTE: 3TOFU is an approach to harm reduction.

It is not wise to download and run binaries or code whose authenticity you cannot verify using a cryptographic signature from a key stored offline. However, sometimes we cannot avoid it. If you're going to proceed with running untrusted code, then following a 3TOFU procedure may reduce your risk, but it's better to avoid running unauthenticated code if at all possible.

Registry (ab)use

Container registries were created in 2013 to provide a clever & complex solution to a problem: how to package and serve multiple versions of simplified sources to various consumers spanning multiple operating systems and architectures -- while also packaging them into small, discrete "layers".

However, if your project is just serving simple files, then the only thing gained by uploading them to a complex system like a container registry is headaches. Why do developers do this?

In the case of brew, their free hosing provider (JFrog's Bintray) shutdown in 2021. Brew was already hosting their code on GitHub, so I guess someone looked at "GitHub Packages" and figured it was a good (read: free) replacement.

Many developers using Container Registries don't need the complexity, but -- well -- they're just using it as a free place for their FOSS project to store some files, man.

hello! For University I need to use a remote machine with a very very VERY weak password I cannot change, and I have to use that machine to edit some code with a few other students of my team. All the code should then be pushed to a repo of my personal github. I'd like to be able to grant access to only that repo, so that if someone guesses the password it cannot touch my other stuffs. What options do I have?

[SOLVED] EDIT:
as suggested by @[email protected] I created a github fine grained access token setting its only permission as read/write only that repo. Then I cloned the repo on the remote machine and set the url to include the token:

git remote set-url origin https://myusername:[email protected]/myusername/myrepo.git 

I then set the user and email:

git config user.name myusername
git config user.email [email protected]

and voilà! I can now simply push without any password requested! And in case someone gained access to the token (that is stored in plain text inside the .git folder) it would only grant access to that specific repo, limiting the damages

As far as I understand it, Forgejo is a soft-fork of Gitea, and, as far as I am aware, Gitea includes both the backend and frontend. But then I came across Codeberg, which appears to state:

Self-Hosting Forgejo, the software that powers Codeberg.

This makes it sound like Forgejo is the backend, and Codeberg is the frontend, but I'm not 100% sure. If so, did Forgejo separate Gitea's UI, and just soft-fork the backend?

Can you not remove "Releases" and "Packages" section from your repository in GitHub?

There is a gear icon on the repository page "Edit repository details" and it seemingly allows you to remove those sections from the page but they don't do anything. Is it just me / is this limited with a free account or just a bug?

Couldn't find anything about this by googling. Any answers much appreciated!

My account was flagged because I forked and contributed to the project Eaglercraft, and that means my account is basically useless. I have had enough of Microsoft's exploitation of power and want to switch to another alternative.

I tried GitLab, but I need to signup with a credit card and I am not comfortable giving my personal info out.
I tried Gitea and the experience is great, but I am limited to 5 repos. I tried Source Forge, but I cannot verify my phone number when creating a repo. The prompt just returns an API error.

What other alternative should I try?

cross-posted from: https://programming.dev/post/223663

Hey folks!

I've noticed that it's often difficult for newcomers to git to understand what the heck is happening and how the commands work.

Here's a flowchart that has helped me explain things in the past, and (more than once) folks have asked me for a copy of it to use as a cheat sheet. Hope it's helpful!

cross-posted from: https://lemmy.ml/post/568420

In reaction to the surprise announcement of the creation of Gitea Ltd and the transfer of domains and trademark to this company, worried members of the Community have written an Open Letter to the elected Owners of the project.

The request is to return the assets and manage them by a community-led non-profit organization and furthermore improve the community organization, so that the Trust and Health of the project is restored.

The Open Letter can be signed by sending a PR to the Codeberg repository.

So I have this exact need:

There is an upstream project doing their own thing over git and I want to build container images locally and commit them to my image repository all while following the same version system as upstream.

To be more precise (perhaps abstract) about my need, what is the best way to apply the same patch when upstream release a new version.

Any input and best practices or lessons learned are welcome.

I hate github with a passion. I have a slightly different name for it that I won't use here because I'm a polite c**t.

They've sunk to a new low now though, in not displaying the URLs for git repos. Not if I allow their (non-free) Javascript to run, and certainly not if I don't. Maybe I'm not using an "approved" browser.

Well at least MS' reason for buying github are clear now - if people can't get at the code then open-source dies.

cross-posted from: https://lemmy.ml/post/77351

Join the FedeProxy vidcall and help bring Gitea to the Fediverse

Whether you are technical or not, there's many ways you can help. By doing so you'll contribute to offering real and open alternatives to the dominant position that Github has on the open source movement. Decentralized FOSS development on the Fediverse, no less!

Agenda:

• Proofreading of grant proposal
• Dev bounty: Generate gitea private keys
• Find individuals & orgs to support grant application and/or federation in Gitea
• Facts / articles that demonstrate the popularity of Gitea
• Where to advertise the effort towards federation?
• First grant application must be sent before October 1st, 2021 for the @NGIZero Discovery call

Provide your availability for the vidcall here: https://framadate.org/jO19mi38nMKWNYbt

Read these other Lemmy posts and learn how you can earn money now:

• Opportunity: Bring Gitea to the Fediverse (funding available)
• Opportunity: Diversity audit for the open-source FedeProxy project

Additional information:

• The proposed Gitea federation design
  • FedeProxy forum discussion
• Details about available Funding, Bounties and planning for additional grants
  • FedeProxy forum discussion

This photo is from Where Good Ideas Come From: A Natural History of Innovation. In it, there's a chapter dedicated to studying 'fluid networks'. Fluid networks are characterized by (1) high density and (2) malleability. These are the characteristics that make coral reefs, cities, universities, and the internet innovation machines.

Not only do innovations happen incredibly quickly in those fluid networks, but they are evidently much better at innovating than lonesome geniuses or groups who are innovating for profit, which is what the image I mentioned earlier points out.

These characteristics of the fluid networks are also present, I argue, in Git. Perhaps not in all of Git, but in projects dense enough, with enough users. Get enough users in a project, and to the extent that the code maintainers can make the repository malleable, you will get innovation at incredible speed.

Because of this, we can say that Git is indeed a version control system for projects without much activity, but with projects with many users and enough capacity to merge commits, Git is also an innovation machine. This is why Git has not only changed the world, but will continue doing it.

This is a front-end to a start of an app store I was thinking to create with a friend (that would do the back-end) but due to school we never advanced more than this. If anyone is interested to use my HTML and CSS feel free to, just please credit me and put the github link to it and/or my mastodon