This guide shows how to create a metric from a string search in any text/log file using the 'logwatcher' chisel available in the Sysdig Monitor agent. You configure the chisel by listing the string to be searched for and the file to look for it in.
When found, you will see a metric in the Explore tab's Metrics > StatsD section with the name of the chisel, the file being searched and the string.
1) Edit the /opt/draios/etc/dragent.yaml confg file
Add the chisel entry below with your log file name and string you are looking for. Replace <> items with the file name and string to search. Be wary of the formatting - use 2 space indents - and do not specify a path with the filename:
There are several ways to edit the agent's configuration file. Please see our FAQ on other ways to add configuration options to /opt/draios/etc/dragent.yaml: How-can-I-edit-the-agents-configuration-file
chisels: - name: logwatcher args: filespattern: <YOUR-FILE> term: <STRING-TO-SEARCH>
Here is a sample filled-out full configuration file which looks for the string 'Sent' in the Sysdig agent's own log file:
customerid: 831f2-your-key-here-d69401 tags: tagname.tagvalue chisels: - name: logwatcher args: filespattern: draios.log term: Sent
2) Restart the agent to affect the configuration:
For the Container agent:
docker restart sysdig-agent
service dragent restart
On the Sysdig Monitor web interface, select your host/instance on the Explore tab, (choose real-time mode 'Go Live') and then look under the Metrics list > StatsD section for a metric name of "logwatcher.<LOGFILE>.<STRING>". In the example above, we would see metric
logwatcher.draios_log.Sent and the number of 'Sent' items appearing per second.
You can also create an alert to be notified when an important log entry appears by watching your new string metric. Lastly, you can add multiple
-name: sections in the config file if you need to look for more strings/files.
TIP: If you need to use the logwatcher chisel to monitor the output of
docker logs <container-name> find the container's docker log file with:
docker inspect <container-name> | grep LogPath
Note: Currently, if you supply a string with spaces, forward-slashes, or back-slashes in it, the metric generated will also have these characters and so cannot be used to create an alert.
Note2: Logwatcher is implemented as a LUA script and, due to resources consumed with this chisel, it is not recommended to have more then a dozen string searches configured per agent/host.