BaSzErr - blog:2017:03:16

2017-03-16_-_docker_cache_vs._docker_registry

anonymous@undisclosed.example.com (Anonymous) — Tue, 15 Jun 2021 20:09:46 +0000

2017-03-16 - docker cache vs. docker registry

i've spent most of the last working day exploring issue on our CI servers. here is the story…

disease's history

after adding jobs to rebuild docker images with each revision, i've added an internal docker registry, to make sure we have a common place to store generated images (easy, convenient, reproducible, no need to build anything manually, etc…). since we have multiple agents, i've quickly noticed that even though i explicitly do docker pull on images, they still get rebuild over and over again, even though Dockerfile (nor its dependencies) did not changed.

time went by (images too some time to build) and there was not apparent reason why caching does not work. starting docker daemon with debug logs enabled indicated that cache is not used (“thank you captain obvious”). so what's going on?

world is a changing place

querying the internet revealed an interesting fact – in docker 1.10.0 caching has drastically changed. TL;DR. the problem is that, if you allow remote images to add stuff to your local cache (as docker used to do, until some time ago), it will effectively allow anyone to misguide your cache, leading to potential security breach. example provided was sth along this:

FROM debian
RUN apt-get update

let's say that this will generate cache entries #1 (“FROM”) and #2 (“RUN”). now malicious image could pollute your cache, putting anything behind “RUN”, and just pretending this was achieved with a declared “RUN” command. fix is simple – do not trust remote cache.

there is however a problem. you no longer can rely on the fact that builds on different machines will generate the same images. in fact it's directly opposite – it will always generate new image! this means, that now each build host must (sooner or later) rebuild image to populate own cache. it also means that pushing to docker registry will take N-times as much space (where N i number of build machines)! so you loose both in terms of time and size.

what can be done?

there are different ways to overcome this issue. one idea is to share caches between different machines and/or propagate caches via docker save + docker load (still shared storage is needed). it however requires another infrastructure element and extra complexity.

another workaround is to explicitly allow caching entries, from a given source, with --cache-from "image:tag" entry. it has entered docker 1.13. this looks the most promising at the moment.

one more alternative i was thinking about was to use cache info on what has already been built locally and store it for extended periods of time. this way it would be enough to keep one hash per layer locally. even though each host still need to rebuild the image at least once, it does not need to keep it forever, just to have caches populated. instead it could push it to (local) instance of docker registry and remove locally. next time it is needed, it's enough to pull it back from registry and re-validate hashes of layers and content match expected ones.

2017-03-16_-_starting_a_script

anonymous@undisclosed.example.com (Anonymous) — Tue, 15 Jun 2021 20:09:46 +0000

2017-03-16 - starting a script

some time ago we had an issue on CI build machines. on test environment, failed builds did not fail. all CI did was to run a simple bash script – sth along these lines:

#!/bin/bash -ex
cd /build/path
cmake /source/dir
ninja
ctest

we double-checked that build errors failed the script locally, but not on our CI. after a bit of digging it turned out that CI is running scripts like this:

bash /path/to/ci/script

while we run these “the usual way”:

cd /path/to/ci
./script

difference? when source-running, bash-bang is ignored! :) this in turn means that stopping on error was not honored – the rest you already know.

answer? update your script:

#!/bin/bash
set -ex       # NOTE: changing shell settings inside the script, not bash-bang
cd /build/path
cmake /source/dir
ninja
ctest

2017-03-16_-_starting_container_as_non-root

anonymous@undisclosed.example.com (Anonymous) — Tue, 15 Jun 2021 20:09:46 +0000

2017-03-16 - starting container as non-root

around the end of last year i spent some time investigating how to run a script, that runs some command in docker container, with a user/group of a local user, that has started the script. this was important to me, so that files that docker container create on a mounted volume are not owned by root:root, but me:me instead (i.e. user that started the container). the problem is, that different users have different UIDs/GIDs, on different machines. how to unify this?

there is a -u switch do docker run, that allows to pass in user and a group. it looked very promising at first:

docker run -it --rm -u oops debian:testing

…but it failed:

docker: Error response from daemon: linux spec user: unable to find user oops: no matching entries in passwd file.

user must exist in the image, in order to start this way. i've played around a lot with different options, helper proxy scripts, parameters/UIDs deduction, etc… finally it turned out there is a super simple, but not widely known solution: -u switch also accepts UIDs and GIDs, and then they do not need to map to any user inside the container!

docker run -it --rm -u 666:999 debian:testing

I have no name!@07bf48e8b622:/$ id
uid=666 gid=999 groups=999

from here on it was simple. for the sake of example let's assume we want to mount users ~/data directory, to /mount directory in the container, while making sure that files generated by a container map to user who runs the command. the spell is:

docker run -it --rm -u "$(id -u):$(id -g)" -v ~/data:/mnt my_image some_cmd

hope this will save you some time. enjoy! :)