Why are sysdig's docker logs filling up my filesystem?

Docker log management is primarily a Docker management issue, and external to the sysdig agent or on-premises service.  However, users are likely to run into this regardless because of some unfortunate defaults.

 

Docker, by default, uses the json-file logging driver, which by default stores the text from stderr and stdout of the initial process to /var/lib/docker/containers/<container_id>/<container_id>-json.log.  These files default to growing without any limit, so provided data continues to come from the container, they will eventually fill any storage.

 

We recommend that you choose to either use an alternate logging driver that you configure to suit your needs, or that you cap the amount of data stored per container as described in in the docker documentation:

https://docs.docker.com/engine/admin/logging/overview/

 

If you choose to keep using the json-file logging driver, we suggest you pick a ceiling on log storage such as

 

# dockerd ... --log-opt max-size=10m --log-opt max-file=10

Which requests a maximum of 10 files with a maximum of 10MB each.

 

If you wish to configure the logs retained with differing amounts on a per-container basis, you can use the same log options at the docker run commandline used when bringing up the agent.

 

Further details about docker log configuration are fully out of scope for the Sysdig agent and on-premises installation both.

Have more questions? Submit a request