2017-11-27_-_cleaning_up_docker-registry

anonymous@undisclosed.example.com (Anonymous) — Tue, 15 Jun 2021 20:09:47 +0000

2017-11-27 - cleaning up docker-registry

it has been 2 months since last post and over 5 months since anything more verbose. Code::Dive is now over and looks like i'll have a bit of time to catch up blogging some ideas and implementations i've being doing around, but had no time to post about. today we'll start with… docker-registry.

when setting up a new service, one must typically accept a bit of maintenance as well. :) there are 3 things you need to cover on your own, that might not come very obvious, if you have not set up a docker-registry instance before, namely:

HTTPS access
authentication/authorization
removal of (old?) images

first one is simple – just put nginx as a reverse-proxy, to terminate HTTPS, and use HTTP internally to contact to the instance.

second one is not very complex either - for many cases it should be enough to configure the very same nginx instance you're using for HTTPS, to enforce basic-auth on each and every path to reverse-proxied service.

the third thing however is FAR from obvious. more over there is very little help on the internet. some ppl write a lot of voodoo scripts, that manipulate registry's internals – this is definitely not a thing i wanted to have on my instance. it turns out that registry's 2.0 version of API defined a REST call to untag images, BUT it was not stated clearly how to do this! more over only the most recent versions on registry (since 2.5 i think) actually have this call implemented.

but how to use it? i've spent way more time i'd like to admit, to solve this issue. if you hit this page with the same problem – how to remove an image from a private registry, here's the procedure.

configuration

first thing, that is very easy to miss, is configuration. by default docker-registry does NOT allow any explicit removals! even if you set up everything else correctly, w/o this option the feature will just not work. what you need to do is edit /etc/docker/registry/config.yml file and add the following section:

storage:
  delete:
    enabled: true

remove request

the API looks very simple – just send a DELETE request to http://myregistry:5000/v2/<image>/manifest/<blob-id>. cool – now how do i get the blob-id thing? while most API calls do accept either ID, or just a name… delete does not. id is the only way to go.

in order to get the ID, the tricky part is that special header must be set in the GET query, namely Accept: application/vnd.docker.distribution.manifest.v2+json. if you won't set it, you'll get a lot of different pieces of information, aside from the single one you're actually looking for. :/ to make things even funnier - the response also comes in a header form…

so the example call via curl, that gets the blob-id for a given image and a tag pair, can look like this:

curl -q -v \
  -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
  -X GET \
  http://myregistry:5000/v2/<image>/manifests/<tag> 2>&1 | grep Docker-Content-Digest | awk '{ print $3 }'

the output will look similar to this: sha256:40506065be119bfc51093a65bf0123147cd2af72c767168dafdcb2d131f72e4d.

now that we finally have the blob-id, we can delete it, from the registry:

curl -v -X DELETE http://myregistry:5000/v2/<image>/manifests/sha256:40506065be119bfc51093a65bf0123147cd2af72c767168dafdcb2d131f72e4d

remove data

if you made it this far, you'll notice that while you no longer can docker pull the removed image, your disk space usage have not decreased either! the procedure above only removes a tag of the image, but does not remove data blobs, to free disk space!

to actually wipe the data from the disk, garbage collector must be called, inside the docker-registry image, with a configuration file as a parameter. the call looks like this:

docker exec -it my-registry-container \
  registry garbage-collect /etc/docker/registry/config.yml

…and wait. for bigger registry this can take some minutes to complete and will flood your screen with messages.

when the command will exit, your cleanup has just finished.

oh – and btw: don't run this whole procedure in parallel with adding stuff to your registry. even though it only uses the official API and provided commands, it still tends to leave registry in an inconsistent state! :/

aftermath

despite the procedure being from hell, pretty much undocumented (well – there is a documentation for all of it, but left on your own it is anything but trivial to glue the above procedure up!) and problematic in terms of stability, it is not the end of the story.

when you use registry to store CI-build microservices images (which seems very natural), it is very common to have cleanup procedure of old images. explicit deleting some images is only a one step of a bigger run in there. of course getting list of images ordered by creation time is tricky as well, since tags do not have a creation date – only images have it. and all of the sudden automation of the cleanup becomes even bigger pain.

long story short – as of writing this post, the procedure is terrible and barely usable. it think there is huge opportunity for improvements here and i hope we'll see better API in the future. in particular it would be nice to have a registry-operating API embedded directly into docker client application. push and pull are there, but removal is unfortunately still missing.

2017-11-27_-_strings_and_sso

anonymous@undisclosed.example.com (Anonymous) — Tue, 15 Jun 2021 20:09:47 +0000

2017-11-27 - strings and SSO

since many years C++ standard not only prohibits copy-on-write (aka: CoW) strings implementation, but even allows implementations to use small-string-optimization (aka: SSO). i've recently came across an interesting bug, “using” this corner case, in a code similar to this:

std::string value_;
// ...
std::string one{"foo"};
char const* ptr = one.data(); // 1
std::string two{ std::move(one) };
// ... 'one' no longer exists here ...
// ... 'two' is still here ... 
cout << ptr; // valid or not?

it used to work on one of the compilers, but not on the other.

it turned out that one of the implementations used SSO, while other did not. when there was no SSO, line (1) was ok, since one.data() returned the very same, heap-allocated pointer, that later on was passed for ownership to the string two.

however when SSO kicks in, short sting may be kept on stack! no memory allocation is performed, but also one.data() will never be equal to two.data(), regardless if two is copy- or move- initialized. the net effect is dangling pointer and UB is unleashed.

so is the code correct or not? NO – it is NOT correct! why? because SSO is valid, standard-compliant implementation feature of std::string and one may not rely on it being (or not being) there. even if your compiler does not implement it yet, it might still break badly with next compiler update. and to make things worse, it will break silently.

lesson learned is that one should never assume neither any particular state of moved-from object, nor a state of pointer/reference generated out of it, unless this is explicitly stated inside the library. one famous example of such special case are std::shared_ptr and std::unique_ptr that, when moved from, guarantee to be nullptr.

by default however, there are no guarantees for other objects. technically speaking moved-from object must only be able to safely destroy itself (i.e. call d-tor). nothing more.

BaSzErr - blog:2017:11:27