Docker Secrets and MySQL Password Management

In this posting we will look at currently recommended ways of managing passwords in MySQL Docker containers and explore whether the recently introduced concept of Docker Secrets could play a role in this area.

Managing runtime secrets in Docker has traditionally been hard to do securely. The MySQL Docker images have typically offered various ways to set the MySQL root password, where some methods are recommended over others. The typical ways to set the root password are 1) specifying the password directly using the MYSQL_ROOT_PASSWORD environment variable 2) bind-mounting a password file into the container, and have MYSQL_ROOT_PASSWORD point to this file, and 3) setting the MYSQL_RANDOM_ROOT_PASSWORD in order to have MySQL generate a random root password. The recommended way on MySQL 5.6 and newer is to use MYSQL_RANDOM_ROOT_PASSWORD in conjunction with MYSQL_ONETIME_PASSWORD, and we’ll briefly explain why this is so.

Specifying the password directly using MYSQL_ROOT_PASSWORD is the least secure option. When running a Docker container, its environment variables are exposed to both the host system and to the container itself, leaving the password at very high risk of exposure. We’ll leave it as an exercise for the reader to find out how and why; suffice it to say that we strongly discourage this way of doing it in any kind of setting where security is of any concern whatsoever.

Bind-mounting a password file will avoid some of the exposure, but the file would still have to be stored on the host system. The environment variable would also expose where it can be accessed on the host system. Not really ideal. Now, the recommended way is to generate a one-time password upon first run using the MYSQL_RANDOM_ROOT_PASSWORD and MYSQL_ONETIME_PASSWORD variables, and then set a secure password after the container initialization is complete. This will lead to only limited exposure of the password in the presumably short interval between container init and first time use, and is thus strongly recommended over the other available options.

Because managing secrets securely in Docker containers is a relatively common need for many Docker users, it was to widespread acclaim that Docker introduced a new mechanism for managing sensitive data with Docker Secrets in version 1.13. It has been stated that this feature only works with Docker Swarm, but the Docker Compose documentation gives the impression that you can leverage the secrets framework on ‘regular’ containers through the use of Docker Compose. We thought we’d check out whether this could be used for management of passwords in MySQL containers.

The Docker Secrets documentation states that when running in Swarm mode, secrets are securely stored in the encrypted Raft log and replicated to the other Swarm managers. Docker also provides tools for granting access to additional secrets, revoking access, and rotating secrets. However, when run with ‘regular’ containers, the secrets are much less secure, as we shall see.

Setting up a container with a secret using Docker Compose is relatively straightforward. We’ll define our secret in the bottom section, and tell MySQL to use that secret as the root password.

version: "3.1"
services:
mysql:
image: mysql/mysql-server:latest
container_name: mysql
environment:
MYSQL_ROOT_PASSWORD: /run/secrets/my_secret
secrets:
- my_secret

secrets:
my_secret:
file: ./my_secret.txt

The observant reader may have spotted that the secret points to a file on the host computer’s file system. The weakness here is that this file has to hold the secret in plain text. This is obviously not desirable. We overlook this weakness for now and create the ./my_secret.txt file with a test password and use docker-compose up -d to create our container.

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
aaf5d45eb2fe mysql/mysql-server:latest "/entrypoint.sh my..." 24 seconds ago Up 24 seconds 3306/tcp, 33060/tcp mysql

All apparently seems well, our container is running and we can connect to the MySQL server with the root password we set. However, a docker inspect shows us what’s really happening under the hood. As can be seen, it’s a simple bind-mount that mounts the password file into the container, with a few special permissions such as making the file read-only.

This isn’t much more than what is already available in Docker, only that it’s now branded under the secrets name, giving it a (dangerous) sense of security when it really isn’t secure. All the typical problems with regards to mounting password files would apply here as well. In addition, Docker’s surrounding mechanisms for dynamically managing secrets aren’t available when running ‘regular’ containers.

As it currently stands, the recommended way to launch MySQL containers is very much the same as before: Use the MYSQL_RANDOM_ROOT_PASSWORD and MYSQL_ONETIME_PASSWORDmechanisms for MySQL 5.6 and newer. Better support for managing secrets in ‘regular’ containers would be a welcome addition to the Docker ecosystem.