HP OpenView System Administration Handbook [Electronic resources] : Network Node Manager, Customer Views, Service Information Portal, HP OpenView Operations

8.2 CREATING THRESHOLD AND REARM EVENTS

NNM provides default threshold and rearm events when custom thresholds are exceeded. The default events can be viewed in NNM's Alarm Browser or in the Exception Report but take no action by default. You can configure custom events to perform additional tasks such as sending a pop-up message to the management station or passing the information to an external application, such as a paging system. The purpose of a threshold event is to perform some action based on exceeding the custom threshold value. The rearm event allows you to configure another event based on the variable being monitored returning to normal.

Specific event numbers available to you are in the range of 1 to 10,000. Numbers outside this range have been reserved for vendor SNMP traps. Your organization should come up with a numbering convention for your enterprise. Threshold events should always be an odd number. The threshold event used in this example is 1001. The rearm event will always be an even number equal to one plus the threshold event number. 1002 will be the rearm event for the first example.

8.2.1 Defining a Threshold Event for Data Collection

When performing data collection, you should determine whether or not to alarm on custom thresholds for the data being monitored. You may wish to store the data for trend analysis and/or simply alarm based on exceeding a threshold without storing the data.

The following steps allow you to define custom threshold and rearm values for the previously defined data collection

ieee3023MacTransmitted :

1. Select

   OptionsData Collection & Thresholds from the menu bar (see Figure 8-1).

2. Modify the collection

   ieee3023MacTransmitted by selecting it from the lower section of the

   Data Collection & Thresholds window. Then select

   EditModifyMIB Collection… from the Data Collection & Thresholds menu bar.

3. Change the Collection Mode to

   Store, Check Thresholds . Notice that the bottom part of the dialog box is no longer grayed out. It activates when you check thresholds. It is only grayed out when the Collection mode is set to Store, No Thresholds. Fill in the following fields, as shown in Figure 8-8:

   Figure 8-8. Set the

   Collection Mode to

   Store, Check Thresholds to generate events caused by exceeding a threshold value. Fill in the

   Threshold and

   Rearm fields. Change the

   Threshold Event Number to 1001.

   Collection Mode	Store, Check Thresholds
   Threshold	> 2000
   Rearm	< 1500
   Threshold Event Number	1001

   The values for threshold and rearm can be accurately determined by displaying the data described previously (

   ActionsShow Data ). While viewing the data, use the

   snmpwalk command to generate interface traffic. The idea is to make sure you can exceed the threshold value to test the threshold and rearm values. To use the

   snmpwalk command, type

   snmpwalk hostname .

   After populating the specified fields, click

   [Configure Threshold Event… . The dialog boxes shown in Figures 8-9 and 8-10 are displayed.

   Figure 8-9. When adding a custom event, you will be prompted to add a new event configuration for the Threshold event. Click

   [OK] .

   Figure 8-10. A second prompt is displayed with instructions on how to handle specific sources for the event configuration.

4. Acknowledge the dialog box

   "No currently configured event for Event Identification "1.3.5.8.1.4.1.11.2.17.1.0.1001" sources: "hostname". Would you like to add this event configuration?" shown in Figure 8-9 by clicking

   [OK] .

5. Acknowledge the dialog box

   "Please add event configuration for Enterprise Identifier "OpenView". If you would like to have special event handling for sources "hostname," add them into the "Source" field of the "Add Event" dialog shown in Figure 8-10 by clicking

   [Close] . The second dialog box indicates what to do if you would like to customize events on a per-node basis. In this example, we will not customize the event for a specific node.

6. Upon acknowledgement of the two dialog boxes, two additional dialog boxes are displayed: the

   Event Configuration (Figure 8-11) and the

   Event Configurator (Figure 8-12). In order to propagate the Event number, select

   ViewEvent Identifiers)Display as

   SNMP

   Traps from the Event Configuration menu bar, as shown in Figure 8-11. The Specific Trap number 1001 is displayed, as shown in Figure 8-12. Complete the following fields in the

   Event Configuration dialog box.

   Figure 8-11. Select

   ViewEvent IdentifiersDisplay as SNMP Traps and the Event Number will be populated automatically for the Event Configuration.

   [View full size image]

   Figure 8-12. Provide the

   Event Name and an optional

   Pop-up Notification for the event. The variables listed in the

   Event Description can be referenced using $1, $2, $3, etc. $2 passes the hostname and $8 passes the sampled value to the pop-up notification.

   [View full size image]

   Event Name	TooManyPackets
   Pop-up Notification	Too many packets $8 on $2

   If the event is displayed as an OID instead of a trap, you can change the events to be displayed as SNMP traps (described previously) or you must supply the event number in the format 0.event# (0.1001) in the OID field. This is due to the differences between SNMP version 2 and version 2C traps.^[2]

   ^[2] The originally proposed SNMP version 3 was intended to provide encryption. Because the decision makers could not come to an agreement on which encryption method to implement in SNMP version 3, we now have version 2C with no encryption.

   The

   Event Description shown in Figure 8-12 describes the variables available to be passed by the collection/configuration. These variables can be referenced in the

   Event Log Message, Pop-up Notification , and

   Command for Automatic Action . Use the dollar sign ($) in front of the number to reference the variable. For example, $2 references the hostname of the system on which the event happened.

   1. The ID of application sending the event.

   2. The name of the host that caused the threshold event.

   3. The HP OpenView object identifier, if available.

   4. The MIB variable in dotted numeric format.

   5. The name of the collection.

   6. The MIB instance.

   7. The threshold value.

   8. The sampled value.

   9. The highest sampled (peak) value.

   10. The time the highest value was sampled.

   11. The lowest sampled (trough) value.

   12. The time the lowest value was sampled.

   13. The threshold operator.

   14. The threshold count.

   Additional variables are available for use in defining events. Select the Event Log Message field and press the

   F1 key. Scroll down and select

   [Variables] . You will see the list of pre-defined variables available for use in the event configuration.

7. Save and close the Event Configuration dialog box by selecting

   FileSave , then

   FileClose from the Event Configuration dialog box, as shown in Figure 8-13. Only one Event Configuration dialog box can be opened at a time. After creating or modifying an event, always save and close the Event Configuration dialog box.

   Figure 8-13. Select

   FileSave and

   FileClose from the Event Configuration dialog box after defining threshold and rearm events.

   [View full size image]

After creating a threshold event, you probably want to create a rearm event to indicate that things have returned to a normal state. The rearm event can be configured similarly to the threshold event to popup messages passing values to into the event message and the popup window.

8.2.2 Defining a Rearm Event for Data Collection

The steps required in configuring a rearm event for data collection are very similar to configuring to those of configuring a threshold event:

1. Select from the menu bar

   OptionsData Collection & Thresholds (see Figure 8-1).

2. Modify the collection

   ieee3023MacTransmitted by selecting it and select

   Edit)ModifyMIB Collection… from the Data Collection menu bar (Figure 8-2).

3. Click the

   [Configure Rearm Event…] button (Figure 8-8).

4. Acknowledge the dialog box

   "No currently configured event for Event Identification "1.3.5.8.1.4.1.11.2.17.1.0.1002" sources: "hostname". Would you like to add this event configuration?" shown in Figure 8-14 by clicking

   [OK] .

   Figure 8-14. When adding a custom event, you are prompted to add a new event configuration. A second prompt is displayed with instructions on how to handle specific sources for the rearm event. The rearm event is always an even number: the threshold event number plus one.

5. Acknowledge the dialog box

   "Please add event configuration for Enterprise Identifier "OpenView". If you would like to have special event handling for sources "

   hostname"

   , add them into the "Source" field of the "Add Event" dialog" by clicking

   [Close] (Figure 8-10).

   Note

   The rearm event number will always be an even number equal to the threshold event number incremented by 1.

6. Provide the

   Event Name and a

   Pop-up Notification in the Event Configuration dialog box and click

   [OK] , as shown in Figure 8-15.

   Figure 8-15. Provide the

   Event Name and the

   Pop-up Notification in the Rearm dialog box.

   [View full size image]

   Event Name	TooManyPacketsRearm
   Pop-up Notification	All is well on $2

Because the threshold and rearm events are tied to data collection via the

[Configure Threshold/Rearm Event…] buttons, predefined event messages already exist for both threshold and rearm events. The event message may be modified if you like, but it is not necessary. The default category for threshold and rearm events is the

Threshold Alarms category. The event category may be modified by selecting from the drop-down list in the

Category, shown in Figures 8-12 and 8-15.

After defining the rearm event, save it and perform the following steps in order to display the data collected:

7. Select

   FileSave from the Event Configuration dialog box (Figure 8-5).

8. The steps Select

   ActionsResume Collection from the Data Collection & Thresholds dialog box (Figure 8-5).

9. Select

   FileSave from the Data Collection & Thresholds dialog box (Figure 8-5).

10. Select

    ActionsShow Data from the Data Collection & Thresholds dialog box (Figure 8-5).

This example triggers the popup notification "

Too many packets <sampled value> on <hostname> " on the management station when the threshold value is exceeded. The sampled value is the value of the MIB variable

ieee3023MacTransmitted for the hostname being monitored. The popup notification is generated every polling interval (5 seconds) until the value has crossed below the rearm value.

When the sampled value has dropped below the rearm value, the popup notification "

All is well on <hostname> " is generated on the management server. If you want a threshold event to occur but do not want a rearm notification to occur, set the threshold and rearm values to the same number. After validating the data and the popup notification, remember to go back and change the polling interval to a more reasonable value (Figure 8-8). Depending on the number of collections defined and the severity of the collections, you may want to set the polling interval to 15 minutes, 30 minutes, or 1 hour. Always save the collection (

FileSave ) after making modifications.

For each threshold and rearm event configured and violated, an alarm occurs in the

Threshold Alarms category.

8.2.3 Generating Actions Based on Custom Thresholds

When defining custom events, you can supply the command to be executed on the NNM system when threshold/rearm values are exceeded. Essentially, anything you type from the command line on the system may be used as an automatic action. This may be a script or a binary executable. Automatic actions are frequently used to send email, trigger audio alerts, alert paging devices, or pass information to a trouble ticketing system.

8.2.3.1 Automatic Actions for UNIX systems

Automatic actions are implemented in the

Commands for Automatic Action field of the Threshold (Figure 8-12) and Rearm Event (Figure 8-15) configuration notification boxes. The command used in this field is executed on the management system. Any of the variables in the description field of the event may be passed in the automatic action. For example, the following action sends an email to root on a UNIX management station. The variable $2 is the nodename and $8 is the sampled value as described previously.

echo "$2 exceeded packet threshold: $8." | mailx s "$2 
Threshold Exceeded" root 

The automatic action is executed on the management system unless you use a utility, such as

remsh , to run the command on a remote UNIX system. ^[3]If the management server was configured to execute actions on the managed node with root access, you could issue the following as an automatic action. Assume that a process, such as

sendmail , needed to be restarted after a particular threshold was exceeded. You could define an automatic action such as this to restart the

sendmail daemon:

^[3] The remsh (REMote SHell) command requires a

.rhosts file or

/etc/hosts.equiv to be configured on a UNIX system. In many environments this is considered to be a security risk. For more details, refer to the UNIX man page on

remsh for HP-UX and

rsh for Solaris.

remsh  
remoteHost  
/sbin/init.d/sendmail start 

Note

The OpenView Operations product actually is more capable of application monitoring than NNM. The point here is that NNM has the capability to execute automatic actions both locally and remotely if properly configured.

8.2.3.2 Automatic Actions for Windows Systems

Actions for Windows systems can also be defined in Data Collection and Threshold Events. For example, if you want to send a message to a remote windows system, include the following automatic action in the Threshold dialog box. The $2 variable translates to the hostname of the system on which the packet threshold has been exceeded. $8 is the sampled value.

"net send  " Node $2  " exceeded packet threshold:" $8 "." 

Given a hostname of winxp256 and a packet threshold of 99999, the resulting command displays a popup message on the remote system similar to that shown in Figure 8-15a.

The capability to execute remote commands on Windows systems is available from the Windows Resource Kit command

rcmd . The Remote Command Service (RCMD.EXE) provides a secure, robust way to remotely administer and run command-line programs. RCMD consists of client and server components. The client is a command-line program, RCMD.EXE. The server end, RCMDSVC.EXE, is installed and run as a service. Issued from the management server, the following example command starts the task scheduler service on the target system:

rcmd  \\ 
hostname  
net start "task scheduler" 

This command can be used in the automatic action field of threshold event configuration. NNM can automatically restart the task scheduler on the remote node without human intervention. Assuming that a MIB variable exists to indicate whether the task scheduler service is running, you could create a threshold event to monitor that MIB variable and configure an automatic action to restart the task scheduler service.

Note

By default, NNM only performs commands that are trusted commands. You must specify the command to be trusted in a file that resides in the trusted commands configuration directory:

UNIX: $OV_CONF/trustedCmds.conf

Windows:

install_dir\ conf\trustedCmds.conf

The format of this file is

Keyword=Absolute Path and can include environment variables listed in the configuration file

ov.envvars.sh . The following are sample entries for trusted commands file:

snmpnotify=$OV_BIN/snmpnotify
ovIfIndexRemap.ovpl=$OV_BIN/ovIfIndexRemap.ovpl

If the commands are not specified in the directory and are used in event configuration, NNM generates an error event and the action is not executed. You can override the trusted command feature by creating a file named ALLOW_ALL in the trusted commands configuration directory. After making modifications to the trusted commands directory, you must force the

ovactiond process to re-read the configuration. This is accomplished by typing the following command:

xnmevents -events 

8.2.4 Creating Custom Alarm Categories

As mentioned previously, the default alarm category for both threshold and rearm events is

Threshold Alarms . When creating or modifying to an event, you can specify the category to which you would like it to be written. Custom alarm categories may be created for storing custom events. Follow these steps to create a custom category:

1. Open the Event Configuration dialog box by selecting

   OptionsEvent Configuration .

2. Select

   EditConfigureAlarm Categories from the Event Configuration dialog box, as shown in Figure 8-16.

   Figure 8-16. To add a custom alarm category, select

   EditConfigureAlarm Categories… from the Event Configuration dialog box.

   Type the name of the new alarm category and click

   [Add] and

   [Close] , as shown Figure 8-17.

   Figure 8-17. Provide the category name (Tammys Alarms) and click the

   [Add] button. Then click the

   [Close] button.

3. Select

   FileSave from the Event Configuration dialog box.

4. Verify that the new alarm category (Tammys Alarms) exists, as shown in Figure 8-18.

   Figure 8-18. The new alarm category (Tammys Alarms) is displayed in the

   Alarm Categories window.

5. Modify the category for your custom threshold and rearm events by using the drop-down list in the

   Category field, as shown in Figure 8-19.

   Figure 8-19. The

   Category of a custom

   Threshold Event may be modified to send Event Log Messages to a custom alarm category, such as Tammys Alarms.

   [View full size image]

By default, events are sorted by Event Identifiers. Events may be sorted by name by selecting the Enterprise ID (

OpenView ) and selecting

ViewSortSort By Event Name , as shown in Figure 8-20. Sorting by name makes it easier to locate an event by name.

8.2.5 Accessing Events from the Alarm Browser

Another way to access an event is via the

Alarm Browser . You can locate the event configuration that generated the message by selecting a message in the Alarm Browser and selecting

ActionsConfigure Event… as shown in Figure 8-21. You may then make modifications to the event, such as the

Category .