Using docker history command

I love when a tool I know but I have never found an use for becomes useful. This is the story of when docker history became useful.

The problem

Every now and then we would face this problem that our testing database was on a different schema than our testing services expected.

The database gets created and populated with dummy data from one of the docker containers, while there are some that use the database in different ways.

So clearly the problem was that these containers were not using the same code (this was confirmed by going inside the containers and looking at the code). There were many options that were ruled out, the docker containers and images were created about at the same time, the same configuration would work locally and on other test environments. Yet, when deploying to this particular environment, somehow, some services code was not pulled from the expected git branch.

Previous experience

We had similar issues in the past and were mostly caused by caching docker layers. Those times I had resolved the issue by pruning the cache

docker prune -a
docker buildx prune -a

But this time it had no effect, the system kept building with an unexpected git branch.

Using docker history

I decided to inspect the layers of the docker images with docker history

And it was here inspecting the output, that I could see that the layer which was been used to create the database schema was been pulled fresh from git, so the problem had nothing to do with caching. I could also see that the dependency layers were taken from the cache, so caching was working fine.

Tracing down the problem

From here on, became the typical debugging process begun. The whole mess was due to a missing line continuation character on the bash command that called the make file to trigger the image built process of this particular service. Only the git tag was left out of the make target variables. A tiny error difficult with a big impact.