← Back to overview

Solving the problem of growing Docker and journald logfiles

Every once in a while, I get monitoring alerts from servers whose filesystems tend to continuously grow until they reach a critical point, even though nothing significant is happening on them. Over time, I’ve identified two main culprits responsible for this growth in usage: Docker and journald.

Unfortunately, neither of them ships with sensible defaults when it comes to log rotation. As a result, I often find myself implementing short-term solutions for long-term problems, which is, of course, not very efficient. To permanently fix the issue with Docker and journald logging, two small configuration adjustments do the trick.

Docker

When looking at /var/lib/docker, you can find Docker container logs in the sub directories, usually stored in JSON format. Depending on what’s going on with your containers, these logs can get huge, like many, many gigabytes. Their path and filename is built like this:

/var/lib/docker/containers/{container-id}/{container-id}-json.log

To manually clean up the logs, while the containers are still running, usually you can just echo an empty string into the file, which will immediately remove all contents but keep the file there and writable for Docker.

echo > /var/lib/docker/containers/{container-id}/{container-id}-json.log

However, to permanently solve the issue, you can indeed configure logrotation in the Docker Daemon configuration file which then will take care of preventing the logs from growing too big:

/etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "10"
  }
}

You may create the file, if it doesn’t exist, yet. This configuration will limit the file size of a single log file to 100 MB while instructing Docker to logrotate after that size and only keep the last 10 of these log files. You can, of course adjust the values for your use case accordingly.

One downside of this configuration is that you need to restart the Docker daemon after configuring it:

sudo systemctl restart docker

Ansible

You can simply automate this task by creating a template (yes, you could also do it as as simple file, but in case you plan to automate further settings, like e.g., registry proxies, it’s a good idea to store it as template) and deploy it to the affected servers. Just create it in your templates directory under templates/docker/daemon.json.j2 and put in the contents above.

tasks/main.yml
- name: Configure Docker Daemon
  ansible.builtin.template:
    src: docker/daemon.json.j2
    dest: /etc/docker/daemon.json
  notify:
    - Restart Docker

To perform the restart of the Docker Daemon, you need a corresponding handler:

handlers/main.yml
- name: Restart Docker
  ansible.builin.service: name=docker state=restarted

journald

Symptomatic of an issue with journald logging is a ever-so-slightly growing size of the following directory:

/var/log/journal/

For reference, you can manually clean this directory up:

Terminal window
sudo journalctl --vacuum-size=1G # Delete everything but the last 1GB of logs
sudo journalctl --vacuum-time=7d # Delete everything but the last 7 days of logs

However, this is manual work and while you theoretically could automate this command by stuffing it in a cron job file, it’s a bit cleaner to just properly configure journald. I tend to use 1G as size limit, but your mileage may vary, depending on how much is going on in the system, what your regulatory requirements are and how much disk size the system is able to use.

/etc/systemd/journald.conf
[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
#SystemMaxUse=
SystemMaxUse=1G
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100

After changing the configuration file, you need to restart journald by running:

sudo systemctl restart systemd-journald

Ansible

Of course I don’t want to rollout this change on more than 3 servers by hand. To achieve the same thing with Ansible, you can use this little task:

tasks/main.yml
- name: Configure journald for reduced max logsize
  ansible.builtin.lineinfile:
    path: /etc/systemd/journald.conf
    regexp: '^SystemMaxUse='
    insertafter: '^#SystemMaxUse='
    line: SystemMaxUse=1G
  notify:
    - Restart systemd-journald

In order to restart systemd-journald afterward, you need a handler:

handlers/main.yml
- name: Restart systemd-journald
  ansible.builtin.service: name=systemd-journald state=restarted