Authoring, PowerShell, SCOM

The default SCOM web availability monitoring can be a thing of beauty, but it is not easy to integrate in custom management packs. I needed a simple web resource monitor that can check for response code and content length.

This is a script that will do just that. It can be used with a custom two state PowerShell script monitor. The question is how to scope this monitor…maybe my previous post about unhosted classes (click) comes in handy here…

Please note that the cmdlet Invoke-WebRequest is introduced in PowerShell 3.0.

HTTPS is supported, but if the web server has SSLv3/TLS1.0 disabled, you will get an Unexpected Error. This can be easily fixed by using TLS 1.2.

Authoring, SCOM

What to do if you want to monitor, for whatever reason, an application from another monitored machine. Normally one of the options is to discover an unhosted class of the application, and start monitoring from an instance of that class. The reason why you must choose an unhosted class is that, when the monitored object goes critical, it wil not affect the health state of the server that monitors it.

Unhosted class instances live on the management servers. This poses a problem when the instance of the class lives behind a SCOM gateway. In that case the management servers cannot reach the application, and monitoring will not be possible.

It would be great if you could “host” the unhosted class on another machine. This could be the SCOM gateway itself, or another server with a SCOM agent installed in the same subnet/branch office/etc as the application.

This is one of the rare cases where the Microsoft.SystemCenter.HealthServiceShouldManageEntity relationship comes in handy. This functionality enables you to discover unhosted class entities, and let the discovery know what agent should manage it.

TLDR; You can control which agent will monitor the unhosted class instance, and when the monitored object goes critical, it wil not affect the health state of the server that monitors it.

Great. Now how do you do this…

First, we need to discover the instance like we normally do. This example gives the relevant snippets of code from a PowerShell discovery script.

Second, we find the health service of the machine the discovery runs against:

Third, we tell SCOM the discovered instance should be managed by the health service found in the second step.

This approach has two quirks.  The agent of the “watcher node” needs to have agent proxy enabled. And when the watcher node enters Maintenance Mode…the workflows of the unhosted class will remain active, but the information will not be sent to the management group.

I hope this will help you to make your SCOM monitoring even more versatile.

Authoring, SCOM

In this example I will explain how to extend the SCOM Unix/Linux class with an attribute for the amount of physical memory. Strangely enough, this is available for Windows Machines by default, but not for Unix/Linux machines. Sometimes this information is needed for an export from SCOM to a CMDB, or just handy during the daily work of Linux sysadmins.

CIM and the SCX Agent:

SCOM discovers information about the Unix/Linux machines via a CIM server that is provided by the cross platform (xPlat), or SCX, agent. This agent is already installed if the Unix/Linux server is monitored by SCOM.

The CIM server, hosted by the agent, provides a set of namespaces. The purpose of a namespace is grouping a set of classes that relate to a specific subject. The classes contain properties, this is the actual information (of the instance) of an object (name, type and value).

Step 1. Extending the class:

We like to discover the amount of physical memory. This newly discovered value must be stored in a property. Therefore we will create a new class, based on Microsoft.Unix.Computer. This class has a property with the name Memory that will hold the discovered value.

Step 2. Discovering physical memory:

The first way is to discover the value by a shell command discovery. I planned on using the ExecuteShellCommand RunAs provider. This provider will execute a shell command, and return the value as StdOut.

The other way is to make use of the CIM namespace root/scx, which contains the class SCX_OperatingSystem. This class contains the property TotalVisibleMemorySize, which happens to be the physical memory (yay!).

Because reading the value directly from the namespace is more efficient, as the value is already known, I chose that as the way to go.

Step 3. Building the discovery:

Now the way of discovering is clear, I could start to build the discovery module. The discovery collects the information via CIM using a URI. The URI specifies the namespace root/scx and the class SCX_OperatingSystem.

SCOM will receive all the properties in the class…so we will have to filter out the information that is needed. It is possible to filter the value also, but in this case we will accept any value. The string ‘.*’ will do just that.

When the correct infomation is collected, the next step is to put the value of TotalVisibleMemorySize in the class property with the name Memory:

The complete discovery will look like this:

That is it for the discovery, the complete management pack can be downloaded here: Custom_Linux_Attributes

Just import this MP and after the next discovery cycle all Linux machines will show their physical memory as a property of the class Custom Linux Attributes. Always test in a test environment, and/or scope non-production servers to be safe. Happy authoring!

ZABBIX

I am in the process of upgrading ZABBIX to 3.2 for the environments I manage it for. It is a fairly easy process, and I’ve not run into any big problems yet.

The order for upgrading the various components of ZABBIX is as follows:

  1. Server
  2. Web Interface
  3. Proxy
  4. Agents

This post describes how to upgrade ZABBIX Server.

Most of the environments I manage are virtualized and have a good backup strategy in place. Make a good backup before starting this procedure, you dont want to lose historical data, custom monitoring, screens and views etc. With a virualized server, with a local DB, you could get away with a snapshot.

When there is a good backup in place…proceed with the following steps:

Stop ZABBIX Server:

Remove the package zabbix-release:

Add the repository for ZABBIX 3.2 to your configuration:

Start the actual update, this will update all packages on your system to the latest version. It can take a while.

Check if ZABBIX is actually installed:

Restart both agent and server services:

After starting the services, ZABBIX wil upgrade the database. In some bigger environments this can take a while. The web interface will not be available until this conversion is completed.

If all went well… you will be welcomed by a new ZABBIX web console 🙂

Now you will have to upgrade the proxies, and it is advised to install the latest agent on the managed servers.

ZABBIX

In this first post of the series I describe how to install the ZABBIX agent on a RHEL or CentOS 7 machine.

Zabbix agent installation

First add the Zabbix repository. It is best to install agents that are of the same version as your ZABBIX server installation, so possibly you will need to change the URL. The best way is to browse the following URL to find the repo for the right version: https://repo.zabbix.com/zabbix/.

I typically do no not install agents that are newer than the currently installed server, but older versions do not seem to be a problem (quote from zabbix.com):

Older agents from Zabbix 1.x, 2.x and previous versions of Zabbix 3.x can still be used with Zabbix 3.4. It generally does not require any configuration changes on the agent side, apart from parameters related to logging for versions before 3.0, which you may want to review. However, to take full advantage of new and improved items, improved performance and reduced memory usage, use the latest 3.4 agent.

The URL below is the one for ZABBIX 3.2:

Now the repository is known, install the Zabbix agent:

After the agent is installed, you will have to make a couple of changes to the zabbix agent configuration file. Configure the following items:

  • server – IP or hostname. agent multi-homing is supported, provide a comma delimited list of servers
  • serveractive – optional: when the server will perform active checks.
  • hostname – Unique, case sensitive hostname. When not configured HostnameItem will be used.

When the machine is added to ZABBIX, this name will be used to create a host object. Please note that the hostname is case sensitive.

Firewall configuration (firewalld)

Determine the firewall profile currently active for the interface you intent to use for the ZABBIX traffic:

Create the firewall rule, change the zone to the one active on your interface.

After the rule has been created, reload the firewall config:

Start the Zabbix agent, also at startup:

Check if the service runs, and check the logging for failures:

Now the installation is completed you will have to create a new Host object in ZABBIX, add the right templates, after that monitoring will commence. This will be posted as a next part of this series. In the meanwhile happy monitoring!

PowerShell, SCOM

This week I encountered an issue recreating a missing SCOM Connector. This time it was the Savision Dashboards Library Connector, but it could be any connector. During the creation of the connector the following error message was displayed:

In the case of Live Maps, when the connector is missing, it will be created during logon with an administrative account. But this time Live Maps failed to create it. The error appeared on screen and in the event log of one of the Management Servers.

First I tried to create the connector manually using the following PowerShell. This failed with the same message.

So…it seemed that the connector only existed partially. By selecting the class “Connector” in discovered inventory, this got confirmed. The connector still exists in the database, but was not shown in the console in the connector view. Only discovered inventory and Get-SCOMClassInstance displayed the connector.

Next I tried to remove the connector with the Remove-SCOMConnector cmdlet. But as expected it could not find the connector for the same reason.

So…how to remove the remnants of the old connector? By editing the database directly? By the IsDeleted = 1 property? Both options are (very much) not supported by Microsoft. After a while I found a GitHub post from Jan van Meirvenne:

https://github.com/JanVanMeirvenne/SCOMRunbooks/blob/master/SCOMRunbooks/Modules/CustomSCOM.psm1

Warning: This is a very powerful function. You could easily render your management group useless by deleting the wrong objects. Please check and double check. Use at your own risk!

Jan received this information from a Microsoft field engineer as a supported way to remove an instance. I loaded the psm and executed the following command:

After that the remnants were deleted, I could logon to Live Maps Unity. Live Maps automagically created the new connector…and all was good again.  This was not a Savision specific issue but they were very helpful none the less, great company with great products.

Please note: All ID’s and GUIDs in this post are unique to this specific issue, so you need to replace them with your own.

SCOM

If you dont like your SCOM console crashing, then dont install update KB3185331. This update is contained in the following rollups:

https://support.microsoft.com/en-us/kb/3192392
https://support.microsoft.com/en-us/kb/3185331

October 2016 security only quality update for Windows 8.1 and Windows Server 2012 R2

If you already installed the update, and experience instability, just uninstall the update and reboot.

Microsoft also warned about this update: https://blogs.technet.microsoft.com/germanageability/2016/10/13/october-2016-windows-patch-kb3192392-might-cause-scom-2012r2-console-to-crash/

PowerShell

You want a script to bulk create AD users? Here you have it.

The input CSV needs to be in the following format:
upn;Firstname;Lastname;Password;Department;Office

Other

The HNAS web interface doesn’t provide per session TCP statistics, this needs to be done via the CLI. With the TCP session statistics you will be able to identify the most bandwidth consuming clients that are accessing the HNAS.

Logon to the HNAS CLI using Putty. I used the default ID: manager.

Now you are logged in, start a TCP connection statistics gathering. Specify a reasonable amount of seconds for the collection to run:

After the statistics are collected, display the results with the following command. The parameter ‘show-peer’ is important in this example, it will display the IP address of the machine talking to the HNAS:

output: