These instructions show how to create a metric from strings found in log files using a new feature of the Sysdig agent. Installing the opensource sysdig utility into your host is required in addition to the already installed Sysdig agent. The agent will call on the utility to implement the logwatcher 'chisel' and look for a defined string in a file. The agent then collects a StatsD metric for how many times the string was found and sends it to your account for reporting.
You can install the sysdig utility either natively in the host (easiest) or install it in your existing Sysdig agent's container (only a little less easy). The instructions below assume the containerized Sysdig agent is being used so examples are shown for slipping the utility into the agent's container.
See http://www.sysdig.org/install/ for more detail on installing the utility in general.
1) Be sure your docker container is already running
docker ps | grep sysdig-agent
2) Step inside the sysdig-agent container
docker exec -it sysdig-agent bash
3) Install the sysdig free utility
apt-get -y install sysdig
4) Exit the container
5) Copy the Sysdig agent's config file out of the container
docker cp sysdig-agent:/opt/draios/etc/dragent.yaml dragent.yaml
6) Edit the dragent.yaml confg file
Add the chisel entry below with your log file name and string. Replace <> items with the file name and string to search. Be wary of the formatting - use 2 space indents - and do not specify a path with the filename:
chisels: - name: logwatcher args: filespattern: <YOUR-FILE> term: <STRING-TO-SEARCH>
Here is a sample filled-out configuration file. I'm looking for the string 'Sent' in my agent's log file:
customerid: 831f2-your-key-here-d69401 tags: acct:dev,linux:centos,local:nyc chisels: - name: logwatcher args: filespattern: draios.log term: Sent
7) Copy the edited config file back into the agent container
docker cp dragent.yaml sysdig-agent:/opt/draios/etc/dragent.yaml
8) Restart the agent
docker restart sysdig-agent
On the Sysdig Monitor web interface, select your host/instance on the Explore tab, (choose real-time mode 'Go Live') and then look under the Metrics list > StatsD section for a metric name of "logwatcher.<LOGFILE>.<STRING>". In the example above, we would see metric
logwatcher.draios.log.Sent and the number of 'Sent' items appearing per second.
You can also create an alert to be notified when an important log entry appears by watching your new string metric. Lastly, you can add multiple
-name sections in the config file if you need to look for more strings. Be careful not to have too many entries (over a dozen) as resource utilization will go up on your host.
If you need to use the logwatcher chisel to monitor the output of
docker logs <container-name> find the container's docker log file with:
docker inspect <container-name> | grep LogPath
Note 1: Currently, if you supply a string with spaces in it, the metric generated will also have spaces and it cannot be used to create an alert. Alerts do not support metric names with spaces at this time.
Note 2: There are several ways to edit the agent's configuration file. Please see our FAQ on other ways to add configuration options to /opt/draios/etc/dragent.yaml: