Containers are surging right now. This series of blog posts will explore a small corner of that universe by building a Docker Registry that adheres to the Docker Registry HTTP V2 API. The information contained in these posts will take a conceptual approach rather than a step-by-step approach. The code will be available in full on GitHub, as the Superhuman Registry.
For this project we'll use Haskell as the implementation language so that we can use Servant.
Servant is a set of packages for declaring web APIs at the type-level and then using those API specifications to:
Servant allows us to specify everything from request bodies to Headers at the type level, which will help us be explicit as we explore Manifests, Tags and Digests. Since the purpose of this set of articles is informative, the types will help ground our conversations.
Firstly, we'll need a new Haskell project. Stack provides nice templating functionality, so we'll use that to scaffold a new project.
Since we aren't focusing on Servant itself for this series, we'll skip a bunch of the boilerplate and backing code to focus in on the handlers and business logic. The code for this section is on GitHub for those that want to investigate further.
One of the benefits of working from a spec is that there are a full set of routes already penned out for us to implement so we can achieve compatibility with the wider ecosystem of tools, such as the Docker Engine.
To start, we'll translate the routes pretty loosely. Then we'll go back and fill in the return types as we write each of the route handlers. Translating the V2 API into types looks like the following.
This produces a set of routes that lay out as follows:
This matches up with the spec quite well and gives us a nice base to start writing more specific code without worrying about whether we'll miss a route.
If we take a closer look at the types we just wrote out we
see a bunch of concepts including
Digests. Interestingly, we don't
Name to represent an repository name.
adhere to a specific regex (
and be less than 256 characters. In plain english from the
A repository name is broken up into path components. A component of a repository name must be at least one lowercase, alpha-numeric characters, optionally separated by periods, dashes or underscores.
Tags are strings that reference images. For example, if we
debian and wanted to only use the tag
we could pull using the format
An image manifest provides a configuration and a set of layers for a container image. It looks like the following JSON:
Layers are stored in the blob portion of the registry, keyed by digest.
The first route we'll look at implementing is also the most simple. It's the route that lets clients know that this registry implements the V2 APIs.
The type of the
/v2 route is
Which breaks down to a
GET request with an
application/json content type. The response has a single
Docker-Distribution-API-Version, which is what
lets a client know which API version our registry
implements. We also send back no body content. Finally, we
can dig a bit deeper into
Get, which is a type alias for
Verb 'GET 200. This tells us that a successful response
will have a 200 code.
The only other valid codes for this route
429 Too Many Requests but since we haven't implemented
authorization or rate-limiting, we'll skip that for now.
Our logging is pretty basic right now. We'll worry about bulking it up later. For now, we're going to leave the default Katip stdout which leaves us with time, loglevel, hostname (container id), thread id and source location:
The primary purpose of a Registry is to store layers and manifests so a client (such as a Docker Engine) can pull images. We'll avoid supporting legacy versions of the registry for security and simplicity reasons, which means our registry will only work for docker 1.10 and above. Benefits of this include not having to rewrite v2 manifests into the v1 format.
We need to figure out what the docker is doing on a push.
Since I'm running Docker for Mac, booting a server to act as
a registry is pretty simple. We'll use
nc for a first
Now that we have a server acting as a "registry", we need to tag and push an image to it.
Great! Our server is listening and the engine is pushing to the right place. If it wasn't, we could've seen something like this:
There's a problem though,
nc doesn't implement the
endpoint, so the docker client falls back to v1 of the api.
Luckily, we've implemented the v2 endpoint already so we'll
skip netcat and jump back into Haskell.
We can use
Wai.Middleware.RequestLogger to log out
everything docker tries to do to our registry. Using
docker-compose to boot up our registry and re-attempting the
From the information, we see that the
/v2/ route is
working as expected, but we hit
src/SR/Blobs.hs:26:14, which is totally expected because
we haven't implemented
uploadBlob yet. Notice that the
engine retries the upload request.
If the route didn't exist, we would have seen a 404 in the logs.
This matches with what we know about the upload process.
We can throw a couple print statements in to replace the undefined as such:
Which will yield us some progress when trying to push.
This is good progress, but we clearly have some issues since the docker engine is still retrying the endpoint.
There are two approaches to blob upload
resumeable. The docs for
/v2/<name>/blobs/uploads detail that the digest query
param is the differentiator between monolithic and resumable
Initiate a resumable blob upload. If successful, an upload location will be provided to complete the upload. Optionally, if the digest parameter is present, the request body will be used to complete the upload in a single request.
Let's take a look at an implementation for the
<>/blobs/uploads) route. We modifiy the type to reflect
the various headers and response codes (docker engine is a
picky client). All of the relevant information is
communicated through headers, so we return
PostAccepted is a shortcut for
Now the handler code. We generate a new uuid to send back in
the response. Our first go is just trying to get the docker
client to continue to the next request but in the future we
should do something with the uuid so we can respond to
uploadAPI might look scary, but it's just
specifying the route we want to generate for the
header. We do this so that Servant will automatically check
that the route is valid for the
api we are serving and we
get a compile error if it doesn't typecheck.
We add 3 headers, setting the Range to
"0-0" because we
are only responding to resumable upload requests for now.
(Otherwise we'd have to handle the case of an extra query
string parameter). Once we generate the
Location and the
Docker-Upload-UUID, we send them back so the docker engine
can start uploading blobs at the specified
We also need a couple instances which allow us to render
UUID into path components and headers. (note:
these are orphan instances, but we could fix that by using a
newtype and declaring the instances for the newtypes
We push again to test the route
And voilà, we get the desired effect. The docker engine
accepts the UUID and tries to upload blobs to
That's totally not the right URI though. We've accidentally
used a relative URI in our
Location header. We'll fix that
The next route, as shown in the logs
above, is the
PATCH to the
Location header we sent back
down. The type for the
PATCH route changes to:
We need to accept and echo back the
Range header, while
the request body comes in as an
OctetStream. We take this
information and just write out the
OctetStream to a file
With this code (and another upload attempt from the engine),
we can see that the next request is a
PUT, which indicates
the last request for this layer.
PATCHs finish flowing in, the client sends a
request with the digest and potentially any final layer
content. Note that at this point, we have not implemented
any append functionality so our
PATCH endpoint will only
work for small layers. Likewise, our
PUT and the
that comes after it will be very minimal, omitting critical
functionality. This is so that we can get through all of the
requests and confirm a full upload flow.
Digest for a layer is a sha256 hash as such:
Our handler will just emit
NoContent so we can skip the
validation code and get on with groking the entire request
This is getting familiar, so we will move on to a minimal
HEAD which is a request to check to see if a particular
layer (identified by
Digest) has been uploaded. Our
version responds "yes" to every single
indicating that the layer exists in the registry already.
Luckily for us the Docker client doesn't seem to validate
Content-Length header which means we can just echo the
Digest back in a header and call it done.
With the rest of the pieces in place, we receive a
Manifest for an image. When uploading the
Manifest for an image, it is interesting to note that this
is the first reference to a
Tag that we have seen so far.
In this case, it is
It is also interesting to remind ourselves that
docker pull works if we use the sha256 hash of the
Manifest. To see this in action let's grab the sha for
hello-world, which we've been using to test our registry.
This is useful because a
Reference, which is the final
path segment in the
PUT URL, can be a
Digest OR a
We need to know this because the response headers need the
hello-world is a
style manifest. The other major option for us is going to be
a Manifest List, aka a Fat Manifest. Our
Manifest looks as
the following, with a single layer.
We can parse the above
Manifest JSON into the following
Haskell datatype. In the future we can also do digest
validation for each digest in the manifest.
For us, it is important to note how the mechanics behind backward compatibility work.
When pushing images, clients which support the new manifest format should first construct a manifest in the new format.
Which is great for us because that means we only have to code support for v2 manifests and compatible clients will behave appropriately.
At this point, it's useful to set up a proxy. After doing
that, we can log out arbitrary parts of the
requests/responses to find the following headers coming from
the docker engine request to
The one we really care about is the
The request identifies the type of
Manifest based on the
Content-Type, so we can easily handle different
content. To add support for the
we can write the implementation for a new Servant
Note that we have also included the ability to hash the
incoming content with this
Content-Type. This means our
handlers don't have to worry about dealing with hashing.
Our route with the new
Content-Type looks like the
CH comes from the
cryptonite package. The request body is
specified as a tuple of
(Digest, Manfiest) which comes
We are still doing as little work as possible in the handler.
Now that we have a fully "working" image upload (at least, the docker engine is convinced we handled everything correctly), we have two ways forward.
pushcompatibility complete with resumable upload, etc.
Since we eventually we want to support multiple backends, Option 1 will have to happen in the future. To do this, we would start off with a GADT that defines the various operations that need to be implemented for a new backend.
To refresh, here is the list of handlers we need to
What we want in the end is a Generalized Algebraic Data Type
that defines what it means to be a
example declaration for the
uploadBlob functionality shows
that a compliant backend would need to declare a function
which too a
Maybe Digest and returned
Either UploadError NoContent. We would use
UploadError to restrict the types of errors which come
back so we can communicate with exists clients such as the
This involves a couple of pieces including changing our
handler types from the following (which uses
to something more general. Possibly just a
HasRegistryBackend constraint on the monad (
App is also
This would allow us to move configuration of the server into the executables while still allowing the library to provide guarentees about what it needs to be able to function. In effect, allowing us to build binaries that target Postgres, File Systems, and other interesting data stores.
We will keep this in mind as we move forward with a concrete implementation based on Postgres. This concrete implementation will inform us as to which abstractions make sense. An interesting choice for the second store implementation would be something without transactions and different consistency guarentees such as S3 or a KV store.
Since we now need a "real" backend, we'll spin up Postgres using sqitch for migrations. The official Postgres image allows us to use a shell script to initialize the db so all we need to do in addition is install sqitch:
initdb.sh is the following script to execute sqitch:
The migrations (in order) are the following three. The first sets up our SR schema.
The second registers uuid-ossp, which we will use when tracking upload requests.
Finally, we implement a table with automatic
modified_at timestamps. These timestamps will help us
implement garbage collection for abandoned uploads. We let
Postgres handle generation of unique UUIDs on
leave the creation of a Large Object for the future.
To handle the initialization of new uploads, we use the following query which grabs a connection from the pool, executes the statement and returns the UUID of the new upload.
runPG is a utility function which handles any connection
At this point I figured out that Docker will ask to mount if you push an image, then retag it and push the retagged image.
The last push produces the following request to blob upload:
putBlob we'll need to make our hacky
headBlob return 404s. This gives us a nice spot to
blobs table which will form the basis of our
TODO: implement CREATE TABLE
HEAD handler will check to see if the blob exists in
catalog. If it does, we need to get the
Content-Length of the blob to return. If it does not exist
we just 404.