标签：

How do your servers talk to you?

Most of us monitor what our servers do, in some way or another. We might use Icinga/Nagios  for some tasks, or use any of what feels like a myriad different tools  for other tasks. I assume, that in most cases, people use e-mail, SMS,  etc. for sending out warning or critical alerts when they occur.

Each of these monitoring services needs to be configured to send  alerts via a particular channel (e.g. SMTP, SMS) to particular users,  and if the need to change these associations arises, most people have to  reconfigure their software in order to do so. For Icinga/Nagios, there are "recipes" on getting particular notification systems configured; that works, of course, but it‘s painful, error-prone, and the configuration is very static.

Some specialized tools have other mechanisms. Ansible for example, has a slew of notification modules  (some of which I even wrote), which can announce to your deployment  team what‘s going on while it runs. Here again, each time you use a  particular notification module in Ansible, you have to configure it to  notify a particular service‘s account. What if that changes? Do you  really want to change all the Ansible playbooks ...? No.

Let me show you what I mean by listing the source of an alert, its type, and its destination:

icinga       disks          jane@example.com icinga       mail queue     jane@example.com icinga       dnsbl          SMS:1234567890 ansible      playbook       twilio:john18 ...

So, when Icinga reports a problem with a DNS blacklist, an SMS will  be sent, and when Icina warns about a mail queue, an e-mail to Jane is  fired off.

Suppose we want to change that: in addition to having an e-mail sent  off, we wish to alert a bunch of mobile phones via Pushover. Or we want  to change the content of the message, etc. For most of us that means  chasing down all instances of the configured notification method, and  modifying / adding to that. No thanks. Really not.

Can we set up some sort of notification "bus" which services use to  send notifications to, and which then decides, using a central config,  what to do with those messages, maybe even reformat them on a  per-destination basis? Yes we can.

We created that recently. It‘s called mqttwarn, and it‘s pretty easy to set up. All you need is an MQTT broker, which is a lightweight "bus" for your network. Your services then "talk" to that broker (i.e. send it messages). mqttwarn  grabs those messages, and dispatches them on to a notification service  (SMTP, NNTP, Twilio, Pushover, HTTP, ...), stores them in files or in  MySQL, publishes them to Redis, pipes them to programs, etc. All  messages, some only (filtered by content, say).

I‘ll show you a small example to hopefully whet your appetite.

mqttwarn is configured in an ini-type file. The difficult part is setting up the so-called service targets which are notification services. I‘ll configure mqttwarn to provide a target called smtp:jplocal which delivers via SMTP, and another one called pushover:icinga which delivers a message to my mobile phone via Pushover.net.

[defaults] hostname  = ‘localhost‘ port      = 1883 ; name the service providers you will be using. launch   = smtp, pushover [config:pushover] targets = {     ‘icinga‘      : [‘xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx‘, ‘yyyyyyyyyyyyyyyyyyyyyyyyyyyyyy‘],   } [config:smtp] server  =  ‘localhost:25‘ sender  =  "MQTTwarn <jpm@localhost>" starttls  =  False targets = {     ‘jplocal‘     : [ ‘jpm@localhost‘ ],     } [monitoring/+] targets = smtp:jplocal, pushover:icinga

When I launch mqttwarn, it will subscribe to the MQTT topic monitoring/+. (The + is a single-level wildcard which matches any one subtree.)

If I then publish a message to my MQTT broker, mqttwarn  will receive that message and act upon it. In this example I‘ll use a  command-line utility to do so, but we‘d normally configure our  monitoring system (or Ansible, etc.) to publish via MQTT, and there are  all manner of language bindings to accomplish that.

mosquitto_pub -t monitoring/warning -m ‘Disk utilization: 94%‘

As you‘ve probably guessed, -t specifies the topic name, and -m the payload to publish.

Do we have e-mail?

Date: Thu, 03 Apr 2014 09:26:36 +0200 From: MQTTwarn <jpm@localhost> To: jpm@localhost Subject: mqttwarn X-Mailer: mqttwarn Disk utilization: 94%

Simultaneously, mqttwarn notified the message via Pushover, and my phone alerts me accordingly:

Coming back to Ansible, if I wanted to post that same notification from a playbook, I would use the mqtt notification module. (The strange-looking quotes are necessary to protect the colon in the string.)

--- - hosts: all   tasks:   ...   - local_action: ‘mqtt                   topic=monitoring/warning                   payload="Disk utilization: 94%"‘

If I later change my mind and want to modify one of the targets, say,  I want to add another e-mail recipient, store messages, etc., I just  modify mqttwarn‘s  configuration file; no need to change the system which actually  initiated the notification alert (i.e. I don‘t have to alter my Icinga configuration).

Looking at Icinga (or Nagios), we can very easily add something similar. Consider your fugly notification module

define command{     command_name    notify-service-by-email     command_line    /usr/bin/printf "%b" "***** Icinga *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$     }

and now look at mine:

define command{     command_name notify-service-by-mqtt     command_line /usr/bin/notify-by-mqtt.py     }

The rather simple notify-by-mqtt program extracts the values I‘m interested in from the environment (note: requires setting enable_environment_macros=1 in icinga.cfg) and produces a JSON payload which is published via MQTT to mqttwarn. The payload looks like this:

{     "_type": "icinga",      "date": "2014-04-04",      "eventstarttime": "1396621975",      "hostdisplayname": "localhost",      "hostname": "localhost",      "hoststatetype": "HARD",      "servicedesc": "JPtest",      "servicedisplayname": "JPtest",      "serviceoutput": "file /tmp/f1: ENOENT",      "servicestate": "CRITICAL",      "servicestateid": "2",      "shortdatetime": "2014-04-04 16:38:25",      "time": "16:38:25",      "timet": "1396622305" }

On the receiving end, mqttwarn  can filter messages, and I change their formatting on a per/service  basis (e.g. I can be more verbose in an e-mail than in a tweet). mqttwarn can transform data on the fly, and it can apply templates for format the message.

Let me show you a simple example. I‘ll take the above mqttwarn configuration and add the title and format lines to it:

[monitoring/+] targets = smtp:jplocal, pushover:icinga title = {servicestate} {servicedisplayname} format = {time}: {hostname}:{servicedisplayname} is {servicestate}: {serviceoutput}

mqttwarn attempts to  decode JSON from the payload received over MQTT, and it can use the  JSON elements to populate format-type strings. The result as seen on my  phone via Pushover is like this:

Seen from bottom to top:

1. our first example

2. an Icinga alert with the raw JSON in it

3. the final result with formatted title and body

We‘ve had a lot of very positive feedback on mqttwarn, and I‘m confident you‘ll be able to put it to good use!

Using MQTTWarn to monitor server or device

标签：

原文地址：http://my.oschina.net/u/2306127/blog/419380