monitoring | vcloudnine.de

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

While searching for a simple monitoring für my root servers, I’m stumbled over a python-based software called Simplemonitor. Other alternatives, like Nagios, or forks like Incinga etc., were a bit too much for my needs.

What is SimpleMonitor?

SimpleMonitor is a Python script which monitors hosts and network connectivity. It is designed to be quick and easy to set up and lacks complex features that can make things like Nagios, OpenNMS and Zenoss overkill for a small business or home network. Remote monitor instances can send their results back to a central location.

My requirements were simple:

Ping monitoring
TCP monitoring
HTTP monitoring
Service monitoring
Disk space monitoring

Monitoring is nothing without alerting, so I was pretty happy that Simplemonitor is able to send messages into a Slack channel! But it can also send e-mails, SMS, or it can write into a log file. To get a full feature overview, visit the Simplemonitor website.

The project is hosted on GitHub. If you are familiar with Python, you can contribute to the project, or you can add features as you need.

Installation & configuration

The installation is pretty simple: Just fetch the ZIP or the tarball from the project website, and extract it.

The configuration is split into two files:

monitor.ini
monitors.ini

The naming is a bit confusing. The monitor.ini contains the basic monitoring configuration, like the interval for the checks, the alerting and reporting settings. The monitors.ini contains the configuration of the service checks. That’s confusing, that confused me, and so I changed the name of the monitors.ini to services.ini.

[monitor]
interval=60
monitors=services.ini

The services.ini (monitors.ini) contains the service checks. This is a short example of a ping, a service check, a port check, and a disk space check.

[ping-host1]
type=host
host=host1.tld.de
tolerance=3

[svc-postfix-host1]
type=rc
runon=host1.tld.de
service=postfix

[port-postfix-host1]
type=tcp
host=host1.tld.de
port=25

[diskspace]
type=diskspace
partition=/
limit=4096M

The alerting is configured in the monitor.ini. I’m using only the Slack notification. All you need is a web hook and the corresponding web hook URL.

[slack]
type=slack
channel=#monitoring
limit=1
url=https://hooks.slack.com/services/afjnsdifnsdfnsdf

In case of a service fail, or service recovery, a notification is sent to the configured Slack channel.

To start Simplemonitor, just start the monitor.py. It expects the monitor.ini in the same directory.

root@host1 /opt/simplemonitor # python2 monitor.py -v
SimpleMonitor v1.7
--> Loading main config from monitor.ini
--> Loading monitor config from services.ini
Adding host monitor ping-host2
Adding rc monitor svc-postfix-host1
Adding rc monitor svc-nginx-host1
Adding rc monitor svc-mysql-host1
Adding rc monitor svc-fail2ban-host1
Adding rc monitor svc-postgrey-host1
Adding rc monitor svc-phpfpm-host1
Adding rc monitor svc-named-host1
Adding diskspace monitor diskspace
--> Loaded 9 monitors.

Adding logfile logger logfile
Adding slack alerter slack

--> Starting... (loop runs every 60s) Hit ^C to stop
php_fpm is running as pid 33937.
Passed: svc-phpfpm-host1
named is running as pid 566.
Passed: svc-named-host1
fail2ban is running as pid 41306.
Passed: svc-fail2ban-host1
Passed: diskspace
postgrey is running as pid 649.
Passed: svc-postgrey-host1
mysql is running as pid 23726.
Passed: svc-mysql-host1
Passed: ping-host2
postfix is running as pid 53332.
Passed: svc-postfix-host1
nginx is running as pid 52736.
Passed: svc-nginx-host1

Summary

I really like the simplicity of Simplemonitor. Download, extract, configure, run, done. That’s what I’ve searched for. It is still under development, but you should not expect that it will gain much complexity. Even if features will be added, it should be a simple monitoring.

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

The Simple Network Management Protocol (SNMP) is a protocol for monitoring and configuration of network-attached devices. SNMP exposes data in the form of variables and values. These variables can then be queried or set. A query retrieves the value of a variable, a set operation assigns a value to a variable. The variables are organized in a hierarchy and each variable is identified by an object identifiers (OID). The management information base (MIB ) describes this hierarchy. MIB files (simple text files) contain metadata for each OID. These are necessary for the translation of a numeric OID into a human-readable format. SNMP knows two devices types:

the managed device which runs the SNMP agent
the network management station (NMS) which runs the management software

The NMS queries the SNMP agent with GET requests. Configuration changes are made using SET requests. The SNMP agent can inform the NMS about state changes using a SNMP trap message. The easiest way for authentication is the SNMP community string.

SNMP is pretty handy and it’s still used, especially for monitoring and managing networking components. SNMP has the benefit, that it’s very lightweight. Monitoring a system with WBEM or using an API can cause slightly more load, compared to SNMP. Furthermore, SNMP is a internet-protocol standard. Nearly every device supports SNMP.

Monitoring host hardware with SNMP

Why should I monitor my ESXi host hardware with SNMP? The vCenter Server can trigger an alarm and most customers use applications like VMware vRealize Operations, Microsoft System Center Operations Manager, or HPE Systems Insight Manager (SIM). There are better ways to monitor the overall health of an ESXi host. But sometimes you want to get some stats about the network interfaces (throughput), or you have a script that should do something, if a NIC goes down or something else happens. Again, SNMP is very resource-friendly and widely supported.

Configure SNMP on ESXi

I focus on ESXi 5.1 and beyond. The ESXi host is called “the SNMP Agent”. We don’t configure traps or trap destinations. We just want to poll the SNMP agent using SNMP GET requests. The configuration is done using esxcli . First of all, we need to set a community string and enable SNMP.

[root@esx1:~] esxcli system snmp set -c public -e true
[root@esx1:~] esxcli system snmp get
   Authentication:
   Communities: public
   Enable: true
   Engineid: 00000063000000a100000000
   Hwsrc: indications
   Largestorage: true
   Loglevel: info
   Notraps:
   Port: 161
   Privacy:
   Remoteusers:
   Syscontact:
   Syslocation:
   Targets:
   Users:
   V3targets:

That’s it! The necessary firewall ports and services are opened and started automatically.

Querying the SNMP agent

I use a CentOS VM to show you some queries. The Net-SNMP package contains the tools snmpwalk and snmpget. To install the Net-SNMP utils, simply use yum .

[root@web ~]# yum install net-snmp-utils.x86_64

Download the VMware SNMP MIB files, extract the ZIP file, and copy the content to to /usr/share/snmp/mibs.

[root@web mibs]# ls -lt
total 3852
-rw-r--r--. 1 root root  50968 Jun  3 17:05 BRIDGE-MIB.mib
-rw-r--r--. 1 root root  59268 Jun  3 17:05 ENTITY-MIB.mib
-rw-r--r--. 1 root root  52586 Jun  3 17:05 HOST-RESOURCES-MIB.mib
-rw-r--r--. 1 root root  10583 Jun  3 17:05 HOST-RESOURCES-TYPES.mib
-rw-r--r--. 1 root root   7309 Jun  3 17:05 IANA-ADDRESS-FAMILY-NUMBERS-MIB.mib
-rw-r--r--. 1 root root  33324 Jun  3 17:05 IANAifType-MIB.mib
-rw-r--r--. 1 root root   3890 Jun  3 17:05 IANA-RTPROTO-MIB.mib
-rw-r--r--. 1 root root  76268 Jun  3 17:05 IEEE8021-BRIDGE-MIB.mib
-rw-r--r--. 1 root root  89275 Jun  3 17:05 IEEE8021-Q-BRIDGE-MIB.mib
-rw-r--r--. 1 root root  16082 Jun  3 17:05 IEEE8021-TC-MIB.mib
-rw-r--r--. 1 root root  44543 Jun  3 17:05 IEEE8023-LAG-MIB.mib
-rw-r--r--. 1 root root  71747 Jun  3 17:05 IF-MIB.mib
-rw-r--r--. 1 root root  16782 Jun  3 17:05 INET-ADDRESS-MIB.mib
-rw-r--r--. 1 root root  46405 Jun  3 17:05 IP-FORWARD-MIB.mib
-rw-r--r--. 1 root root 185967 Jun  3 17:05 IP-MIB.mib
-rw-r--r--. 1 root root    229 Jun  3 17:05 list-ids-diagnostics.txt
-rw-r--r--. 1 root root  77406 Jun  3 17:05 LLDP-V2-MIB.mib
-rw-r--r--. 1 root root  16108 Jun  3 17:05 LLDP-V2-TC-MIB.mib
-rw-r--r--. 1 root root  23777 Jun  3 17:05 notifications.txt
-rw-r--r--. 1 root root  39918 Jun  3 17:05 P-BRIDGE-MIB.mib
-rw-r--r--. 1 root root  84172 Jun  3 17:05 Q-BRIDGE-MIB.mib
-rw-r--r--. 1 root root   1465 Jun  3 17:05 README
-rw-r--r--. 1 root root 223872 Jun  3 17:05 RMON2-MIB.mib
-rw-r--r--. 1 root root 148032 Jun  3 17:05 RMON-MIB.mib
-rw-r--r--. 1 root root  22342 Jun  3 17:05 SNMP-FRAMEWORK-MIB.mib
-rw-r--r--. 1 root root   5543 Jun  3 17:05 SNMP-MPD-MIB.mib
-rw-r--r--. 1 root root   8259 Jun  3 17:05 SNMPv2-CONF.mib
-rw-r--r--. 1 root root  31588 Jun  3 17:05 SNMPv2-MIB.mib
-rw-r--r--. 1 root root   8932 Jun  3 17:05 SNMPv2-SMI.mib
-rw-r--r--. 1 root root  38048 Jun  3 17:05 SNMPv2-TC.mib
-rw-r--r--. 1 root root  28647 Jun  3 17:05 TCP-MIB.mib
-rw-r--r--. 1 root root  93608 Jun  3 17:05 TOKEN-RING-RMON-MIB.mib
-rw-r--r--. 1 root root  20951 Jun  3 17:05 UDP-MIB.mib
-rw-r--r--. 1 root root   3175 Jun  3 17:05 UUID-TC-MIB.mib
-rw-r--r--. 1 root root   2326 Jun  3 17:05 VMWARE-CIMOM-MIB.mib
-rw-r--r--. 1 root root  22411 Jun  3 17:05 VMWARE-ENV-MIB.mib
-rw-r--r--. 1 root root  53480 Jun  3 17:05 VMWARE-ESX-AGENTCAP-MIB.mib
-rw-r--r--. 1 root root   2328 Jun  3 17:05 VMWARE-HEARTBEAT-MIB.mib
-rw-r--r--. 1 root root   1699 Jun  3 17:05 VMWARE-NSX-MANAGER-AGENTCAP-MIB.mib
-rw-r--r--. 1 root root 146953 Jun  3 17:05 VMWARE-NSX-MANAGER-MIB.mib
-rw-r--r--. 1 root root  15641 Jun  3 17:05 VMWARE-OBSOLETE-MIB.mib
-rw-r--r--. 1 root root   2173 Jun  3 17:05 VMWARE-PRODUCTS-MIB.mib
-rw-r--r--. 1 root root   8305 Jun  3 17:05 VMWARE-RESOURCES-MIB.mib
-rw-r--r--. 1 root root   3736 Jun  3 17:05 VMWARE-ROOT-MIB.mib
-rw-r--r--. 1 root root  11142 Jun  3 17:05 VMWARE-SRM-EVENT-MIB.mib
-rw-r--r--. 1 root root   3872 Jun  3 17:05 VMWARE-SYSTEM-MIB.mib
-rw-r--r--. 1 root root   7017 Jun  3 17:05 VMWARE-TC-MIB.mib
-rw-r--r--. 1 root root   7611 Jun  3 17:05 VMWARE-VA-AGENTCAP-MIB.mib
-rw-r--r--. 1 root root   8777 Jun  3 17:05 VMWARE-VC-EVENT-MIB.mib
-rw-r--r--. 1 root root  38576 Jun  3 17:05 VMWARE-VCOPS-EVENT-MIB.mib
-rw-r--r--. 1 root root  26952 Jun  3 17:05 VMWARE-VMINFO-MIB.mib

Now we can use snmpwalk to “walk down the hierarchy “. This is only a small part of the complete output. The complete snmpwalk output has more than 4000 lines!

[root@web mibs]# snmpwalk -m ALL -c public -v 2c esx1.lab.local
SNMPv2-MIB::sysDescr.0 = STRING: VMware ESXi 6.0.0 build-3825889 VMware, Inc. x86_64
SNMPv2-MIB::sysObjectID.0 = OID: VMWARE-PRODUCTS-MIB::vmwESX
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (402700) 1:07:07.00
SNMPv2-MIB::sysContact.0 = STRING:
SNMPv2-MIB::sysName.0 = STRING: esx1

Now we can search for interesting parts. If you want to monitor the link status of the NICs, try this:

[root@web mibs]# snmpwalk -m ALL -c public -v 2c esx1.lab.local IF-MIB::ifDescr
IF-MIB::ifDescr.1 = STRING: Device vmnic0 at 03:00.0 bnx2
IF-MIB::ifDescr.2 = STRING: Device vmnic1 at 03:00.1 bnx2
IF-MIB::ifDescr.3 = STRING: Device vmnic2 at 04:00.0 bnx2
IF-MIB::ifDescr.4 = STRING: Device vmnic3 at 04:00.1 bnx2
IF-MIB::ifDescr.5 = STRING: Device vmnic4 at 06:00.0 bnx2
IF-MIB::ifDescr.6 = STRING: Device vmnic5 at 06:00.1 bnx2
IF-MIB::ifDescr.7 = STRING: Distributed Virtual VMware switch: DvsPortset-0
IF-MIB::ifDescr.8 = STRING: Virtual interface: vmk0 on port 33554442 DVS 6b a0 37 50 c6 24 04 b8-25 08 f5 ea 32 ef 48 27
IF-MIB::ifDescr.9 = STRING: Virtual interface: vmk1 on port 33554443 DVS 6b a0 37 50 c6 24 04 b8-25 08 f5 ea 32 ef 48 27
IF-MIB::ifDescr.10 = STRING: Virtual interface: vmk2 on port 33554444 DVS 6b a0 37 50 c6 24 04 b8-25 08 f5 ea 32 ef 48 27
IF-MIB::ifDescr.11 = STRING: Virtual interface: vmk3 on port 33554445 DVS 6b a0 37 50 c6 24 04 b8-25 08 f5 ea 32 ef 48 27

As you can see, I used a subtree of the whole hierarchy (IF-MIB::ifDescr). This is the “translated” OID. To get the numeric OID, you have to add the option -O fn to snmpwalk .

[root@web mibs]# snmpwalk -O fn -m ALL -c public -v 2c esx1.lab.local IF-MIB::ifDescr
.1.3.6.1.2.1.2.2.1.2.1 = STRING: Device vmnic0 at 03:00.0 bnx2
.1.3.6.1.2.1.2.2.1.2.2 = STRING: Device vmnic1 at 03:00.1 bnx2
.1.3.6.1.2.1.2.2.1.2.3 = STRING: Device vmnic2 at 04:00.0 bnx2
.1.3.6.1.2.1.2.2.1.2.4 = STRING: Device vmnic3 at 04:00.1 bnx2
.1.3.6.1.2.1.2.2.1.2.5 = STRING: Device vmnic4 at 06:00.0 bnx2
.1.3.6.1.2.1.2.2.1.2.6 = STRING: Device vmnic5 at 06:00.1 bnx2
.1.3.6.1.2.1.2.2.1.2.7 = STRING: Distributed Virtual VMware switch: DvsPortset-0
.1.3.6.1.2.1.2.2.1.2.8 = STRING: Virtual interface: vmk0 on port 33554442 DVS 6b a0 37 50 c6 24 04 b8-25 08 f5 ea 32 ef 48 27
.1.3.6.1.2.1.2.2.1.2.9 = STRING: Virtual interface: vmk1 on port 33554443 DVS 6b a0 37 50 c6 24 04 b8-25 08 f5 ea 32 ef 48 27
.1.3.6.1.2.1.2.2.1.2.10 = STRING: Virtual interface: vmk2 on port 33554444 DVS 6b a0 37 50 c6 24 04 b8-25 08 f5 ea 32 ef 48 27
.1.3.6.1.2.1.2.2.1.2.11 = STRING: Virtual interface: vmk3 on port 33554445 DVS 6b a0 37 50 c6 24 04 b8-25 08 f5 ea 32 ef 48 27

You can use snmptranslate to translate an OID.

[root@web mibs]# snmptranslate .1.3.6.1.2.1.2.2.1.2
IF-MIB::ifDescr
[root@web mibs]# snmptranslate -O fn IF-MIB::ifDescr
.1.3.6.1.2.1.2.2.1.2

So far, we have only the description of the interfaces. With a little searching, we find the status of the interfaces (I stripped the output).

IF-MIB::ifOperStatus.1 = INTEGER: up(1)
IF-MIB::ifOperStatus.2 = INTEGER: up(1)
IF-MIB::ifOperStatus.3 = INTEGER: down(2)
IF-MIB::ifOperStatus.4 = INTEGER: down(2)
IF-MIB::ifOperStatus.5 = INTEGER: up(1)
IF-MIB::ifOperStatus.6 = INTEGER: up(1)

ifOperStatus.1 corresponds with ifDescr.1 , ifOperStatus.2 corresponds with ifDescr.2 and so on. The ifOperStatus corresponds with the status of the NICs in the vSphere Web Client.

If you want to monitor the fans or power supplies, use these these OIDs.

HOST-RESOURCES-MIB::hrDeviceDescr.35 = STRING: POWER Power Supply 1
HOST-RESOURCES-MIB::hrDeviceDescr.36 = STRING: POWER Power Supply 2
HOST-RESOURCES-MIB::hrDeviceDescr.37 = STRING: FAN Fan Block 1
HOST-RESOURCES-MIB::hrDeviceDescr.38 = STRING: FAN Fan Block 2
HOST-RESOURCES-MIB::hrDeviceDescr.39 = STRING: FAN Fan Block 3
HOST-RESOURCES-MIB::hrDeviceDescr.40 = STRING: FAN Fan Block 4

HOST-RESOURCES-MIB::hrDeviceStatus.35 = INTEGER: running(2)
HOST-RESOURCES-MIB::hrDeviceStatus.36 = INTEGER: running(2)
HOST-RESOURCES-MIB::hrDeviceStatus.37 = INTEGER: running(2)
HOST-RESOURCES-MIB::hrDeviceStatus.38 = INTEGER: running(2)
HOST-RESOURCES-MIB::hrDeviceStatus.39 = INTEGER: running(2)
HOST-RESOURCES-MIB::hrDeviceStatus.40 = INTEGER: running(2)

Many possibilities

SNMP offers a simple and lightweight way to monitor a managed device. It’s not a replacement for vCenter, vROps or SCOM. But it can be an addition, especially because SNMP is an internet-protocol standard.

vcloudnine.de

… virtualization on cloud 9

Tag Archives: monitoring

Simplemonitor – Python-based monitoring

How to monitor ESXi host hardware with SNMP