I am trying to generate alert when the Cisco router's interface throughput exceeds certain value. Please have a look at my code.
I am using telegraph to collect the data. My measurement "monitor usage" contains multiple field keys and values. How to separate them and alert on each of them? Am I using "derivative" node correctly? Is it the field key I should use in the "derivative" node?
Is there a way to log each node output to help troubleshooting? In my case, how do I see "bytes_recv_per_sec" value as being calculated by the "alert" node?
What does "log" node do exactly? Where is the log file? I don't see much valuable information in "/var/log/kapacitor/kapacitor.log"
I have tried playing with different nodes with different parameter methods without much success. Thanks for any help.
dbrp "telegraf"."autogen"
stream
|from()
.measurement('monitor usage')
.where(lambda: "agent_host" == '10.92.192.15')
|window()
.period(1m)
.every(1m)
|httpOut('dump')
|stats(1m)
|httpOut('stats')
|derivative('Envision_Po1.891_ifHCInOctets')
.unit(1s).nonNegative()
.as('bytes_recv_per_sec')
|alert()
.warn(lambda: "bytes_recv_per_sec" > 40.0)
.crit(lambda: "bytes_recv_per_sec" > 80.0)
// post to logfile
.log('/tmp/alerts.log')
At this stage I want to see "/tmp/alerts.log" to generate some alert message when the alert threshold is met. Eventually I want to send an alert email.
As per request, here is info about my database on the "monitor usage" measurement
> show series from "monitor usage";
key
---
monitor\ usage,agent_host=10.85.1.5,host=dzhao-trex,sw_name=it-na-wan-pro-01-11p.int.pason.com
monitor\ usage,agent_host=10.85.1.6,host=dzhao-trex,sw_name=it-na-wan-pro-02-11p.int.pason.com
monitor\ usage,agent_host=10.92.192.15,host=dzhao-trex,rtr_name=irt01-00rm255r2r1.int.pason.com
monitor\ usage,agent_host=10.92.192.17,host=dzhao-trex,sw_name=wsw01-00rm255r2r1.pason.com
monitor\ usage,agent_host=172.19.254.200,host=dzhao-trex,sw_name=ssw01-04rm120r7.int.pason.com
monitor\ usage,agent_host=192.168.195.1,host=dzhao-trex,sw_name=wrt01-06.int.pason.com
> show field keys from "monitor usage";
name: monitor usage
fieldKey fieldType
-------- ---------
Envision_Po1.891_ifHCInOctets integer
Envision_Po1.891_ifHCOutOctets integer
Gi0/0/0_ifHCInOctets integer
Gi0/0/0_ifHCOutOctets integer
GigabitEthernet0/0/0_ifHCInOctets integer
GigabitEthernet0/0/0_ifHCOutOctets integer
GigabitEthernet0/0/2_ifHCInOctets integer
GigabitEthernet0/0/2_ifHCOutOctets integer
HE_Po1.892_ifHCInOctets integer
HE_Po1.892_ifHCOutOctets integer
Po1003_ifHCInOctets integer
Po1003_ifHCOutOctets integer
Rogers_Po1.894_ifHCInOctets integer
Rogers_Po1.894_ifHCOutOctets integer
Shaw_Po1.893_ifHCInOctets integer
Shaw_Po1.893_ifHCOutOctets integer
Te2/5/6_ifHCInOctets integer
Te2/5/6_ifHCOutOctets integer
Tunnel100_ifHCInOctets integer
Tunnel100_ifHCOutOctets integer
Tunnel1_ifHCInOctets integer
Tunnel1_ifHCOutOctets integer
Tunnel2_ifHCInOctets integer
Tunnel2_ifHCOutOctets integer
YYCIX_Gi0/0/1_ifHCInOctets integer
YYCIX_Gi0/0/1_ifHCOutOctets integer
> show tag keys from "monitor usage";
name: monitor usage
tagKey
------
agent_host
host
rtr_name
sw_name