How Can I Keep My Agent And Docker Logs Lean?

 

Depending on the logging settings used for your container based applications and Docker itself, you may find a large amount of disk space is consumed by your running containers. This includes the Sysdig Monitor agent in specific scenarios such as when debug mode is enabled for the agent.

This guide shows how to find the amount of disk space Docker's logging of the Sysdig Monitor agent is using and now to reduce that amount. This example can be applied to any Docker container.

  1. Find the "sysdig-agent" container's working directory with:
    docker ps -a

  2. Find the "sysdig-agent" container's working directory using the ID:
    docker inspect [SYSDIG_AGENT_CONTAINER_ID]
    finding the hostname path, ex)
    "HostnamePath": "/var/lib/docker/containers/3534ba018c34195f0c. . . 

  3. Check the space used by /var/lib/[SYSDIG_AGENT_CONTAINER_ID]-json.log

 

The Sysdig Monitor agent itself logs output to both the console and its log file /opt/draios/logs/draios.log  rotating the file up to 10 times with 100MB used each rotation. Total disk space used will be a maximum of 1GB plus the existing open draios.log file.

Docker will also be writing any container console output (our regular agent console log messages and any other console output) to the file  /var/lib/[SYSDIG_AGENT_CONTAINER_ID]-json.log  You will want to check if that json log file is consuming too much space. Again, in the case of the Sysdig Monitor agent, this can happen if you have 'debug' or 'trace' logging turned on for the console.

To reduce the amount of logging by the agent and Docker, you can turn down the agent's console and log file message frequency using our FAQ: How-can-I-change-the-agent-log-level.  The default message setting for the console and log file is 'info'. You can reduce the console logging if needed but we suggest keeping the file logging level at 'info' to help trouble-shooting should issues arise.  Remember, the agent's logs are rotated and should not take up more then a little over 1GB at any time.

To prevent any Docker container logs from consuming too much disk space, you can create a 'logrotate' script if your host has the 'logrotate' utility installed (many Linux distros do).  For example, create a file 'docker' in /etc/logrotate.d with the content below:

/var/lib/docker/containers/*/*.log {
  rotate 7
  daily
  compress
  size=1M
  missingok
  delaycompress
  copytruncate
}

Cron is typically setup to run logrotate automatically and this script example will archive up to seven 1MB files before deleting the oldest logs.  Use man logrotate for details on adjusting the settings to your use case.

Have more questions? Submit a request