Implementing the LogWatcher Chisel

These instructions show how to create a metric from strings found in log files using a new feature of the Sysdig agent. Installing the opensource sysdig utility into your host is required in addition to the already installed Sysdig agent. The agent will call on the utility to implement the logwatcher 'chisel' and look for a defined string in a file. The agent then collects a StatsD metric for how many times the string was found and sends it to your account for reporting.

You can install the sysdig utility either natively in the host (easiest) or install it in your existing Sysdig agent's container (only a little less easy). The instructions below assume the containerized Sysdig agent is being used so examples are shown for slipping the utility into the agent's container.

See http://www.sysdig.org/install/ for more detail on installing the utility in general.

 

1) Be sure your docker container is already running
docker ps | grep sysdig-agent

 

2) Step inside the sysdig-agent container

docker exec -it sysdig-agent bash

 

3) Install the sysdig free utility
apt-get update
apt-get -y install sysdig

 

4) Exit the container
exit

 

5) Copy the Sysdig agent's config file out of the container
docker cp sysdig-agent:/opt/draios/etc/dragent.yaml dragent.yaml

 

6) Edit the dragent.yaml confg file

Add the chisel entry below with your log file name and string. Replace <> items with the file name and string to search. Be wary of the formatting - use 2 space indents - and do not specify a path with the filename:

chisels:
  - name: logwatcher
    args:
      filespattern: <YOUR-FILE>
      term: <STRING-TO-SEARCH>

 

Here is a sample filled-out configuration file. I'm looking for the string 'Sent' in my agent's log file:

customerid: 831f2-your-key-here-d69401
tags: acct:dev,linux:centos,local:nyc
chisels:
  - name: logwatcher
    args:
      filespattern: draios.log
      term: Sent

 

7) Copy the edited config file back into the agent container
docker cp dragent.yaml sysdig-agent:/opt/draios/etc/dragent.yaml

 

8) Restart the agent
docker restart sysdig-agent

 

On the Sysdig Monitor web interface, select your host/instance on the Explore tab, (choose real-time mode 'Go Live') and then look under the Metrics list > StatsD section for a metric name of "logwatcher.<LOGFILE>.<STRING>". In the example above, we would see metric logwatcher.draios.log.Sent and the number of 'Sent' items appearing per second.

You can also create an alert to be notified when an important log entry appears by watching your new string metric. Lastly, you can add multiple  -name  sections in the config file if you need to look for more strings. Be careful not to have too many entries (over a dozen) as resource utilization will go up on your host.

If you need to use the logwatcher chisel to monitor the output of docker logs <container-name> find the container's docker log file with:

docker inspect <container-name> | grep LogPath

 

Note 1: Currently, if you supply a string with spaces in it, the metric generated will also have spaces and it cannot be used to create an alert. Alerts do not support metric names with spaces at this time.

 

Note 2: There are several ways to edit the agent's configuration file. Please see our FAQ on other ways to add configuration options to /opt/draios/etc/dragent.yaml:

How-can-I-edit-the-agents-configuration-file

Have more questions? Submit a request