8.2 CREATING THRESHOLD AND REARM EVENTS
NNM provides default threshold and rearm events when custom thresholds are exceeded. The default events can be viewed in NNM's Alarm Browser or in the Exception Report but take no action by default. You can configure custom events to perform additional tasks such as sending a pop-up message to the management station or passing the information to an external application, such as a paging system. The purpose of a threshold event is to perform some action based on exceeding the custom threshold value. The rearm event allows you to configure another event based on the variable being monitored returning to normal.Specific event numbers available to you are in the range of 1 to 10,000. Numbers outside this range have been reserved for vendor SNMP traps. Your organization should come up with a numbering convention for your enterprise. Threshold events should always be an odd number. The threshold event used in this example is 1001. The rearm event will always be an even number equal to one plus the threshold event number. 1002 will be the rearm event for the first example.
8.2.1 Defining a Threshold Event for Data Collection
When performing data collection, you should determine whether or not to alarm on custom thresholds for the data being monitored. You may wish to store the data for trend analysis and/or simply alarm based on exceeding a threshold without storing the data.The following steps allow you to define custom threshold and rearm values for the previously defined data collection ieee3023MacTransmitted :
- Select Options
Data Collection & Thresholds from the menu bar (see Figure 8-1).
- Modify the collection ieee3023MacTransmitted by selecting it from the lower section of the Data Collection & Thresholds window. Then select Edit
Modify
MIB Collection… from the Data Collection & Thresholds menu bar.
- Change the Collection Mode to Store, Check Thresholds . Notice that the bottom part of the dialog box is no longer grayed out. It activates when you check thresholds. It is only grayed out when the Collection mode is set to Store, No Thresholds. Fill in the following fields, as shown in Figure 8-8:
Figure 8-8. Set the
Collection Mode to Store, Check Thresholds to generate events caused by exceeding a threshold value. Fill in the Threshold and Rearm fields. Change the Threshold Event Number to 1001.
The values for threshold and rearm can be accurately determined by displaying the data described previously (ActionsCollection Mode Store, Check Thresholds Threshold > 2000 Rearm < 1500 Threshold Event Number 1001 Show Data ). While viewing the data, use the snmpwalk command to generate interface traffic. The idea is to make sure you can exceed the threshold value to test the threshold and rearm values. To use the snmpwalk command, type snmpwalk hostname .After populating the specified fields, click [Configure Threshold Event… . The dialog boxes shown in Figures 8-9 and 8-10 are displayed.
Figure 8-9. When adding a custom event, you will be prompted to add a new event configuration for the Threshold event. Click
[OK] .height="166" SRC="/image/library/english/10090_08fig09.jpg" >
Figure 8-10. A second prompt is displayed with instructions on how to handle specific sources for the event configuration.
- Acknowledge the dialog box "No currently configured event for Event Identification "1.3.5.8.1.4.1.11.2.17.1.0.1001" sources: "hostname". Would you like to add this event configuration?" shown in Figure 8-9 by clicking [OK] .
- Acknowledge the dialog box "Please add event configuration for Enterprise Identifier "OpenView". If you would like to have special event handling for sources "hostname," add them into the "Source" field of the "Add Event" dialog shown in Figure 8-10 by clicking [Close] . The second dialog box indicates what to do if you would like to customize events on a per-node basis. In this example, we will not customize the event for a specific node.
- Upon acknowledgement of the two dialog boxes, two additional dialog boxes are displayed: the Event Configuration (Figure 8-11) and the Event Configurator (Figure 8-12). In order to propagate the Event number, select View
Event Identifiers
)Display as SNMP Traps from the Event Configuration menu bar, as shown in Figure 8-11. The Specific Trap number 1001 is displayed, as shown in Figure 8-12. Complete the following fields in the Event Configuration dialog box.
Figure 8-11. Select
ViewEvent Identifiers
Display as SNMP Traps and the Event Number will be populated automatically for the Event Configuration.
[View full size image]height="513" SRC="/image/library/english/10090_08fig11.jpg" >
Figure 8-12. Provide the
Event Name and an optional Pop-up Notification for the event. The variables listed in the Event Description can be referenced using $1, $2, $3, etc. $2 passes the hostname and $8 passes the sampled value to the pop-up notification.
[View full size image]height="562" SRC="/image/library/english/10090_08fig12.jpg" >
If the event is displayed as an OID instead of a trap, you can change the events to be displayed as SNMP traps (described previously) or you must supply the event number in the format 0.event# (0.1001) in the OID field. This is due to the differences between SNMP version 2 and version 2C traps.[2]Event Name TooManyPackets Pop-up Notification Too many packets $8 on $2 [2] The originally proposed SNMP version 3 was intended to provide encryption. Because the decision makers could not come to an agreement on which encryption method to implement in SNMP version 3, we now have version 2C with no encryption.
The Event Description shown in Figure 8-12 describes the variables available to be passed by the collection/configuration. These variables can be referenced in the Event Log Message, Pop-up Notification , and Command for Automatic Action . Use the dollar sign ($) in front of the number to reference the variable. For example, $2 references the hostname of the system on which the event happened.- The ID of application sending the event.
- The name of the host that caused the threshold event.
- The HP OpenView object identifier, if available.
- The MIB variable in dotted numeric format.
- The name of the collection.
- The MIB instance.
- The threshold value.
- The sampled value.
- The highest sampled (peak) value.
- The time the highest value was sampled.
- The lowest sampled (trough) value.
- The time the lowest value was sampled.
- The threshold operator.
- The threshold count.
- Save and close the Event Configuration dialog box by selecting File
Save , then File
Close from the Event Configuration dialog box, as shown in Figure 8-13. Only one Event Configuration dialog box can be opened at a time. After creating or modifying an event, always save and close the Event Configuration dialog box.
Figure 8-13. Select
FileSave and File
Close from the Event Configuration dialog box after defining threshold and rearm events.
[View full size image]height="492" SRC="/image/library/english/10090_08fig13.gif" >
8.2.2 Defining a Rearm Event for Data Collection
The steps required in configuring a rearm event for data collection are very similar to configuring to those of configuring a threshold event:
- Select from the menu bar Options
Data Collection & Thresholds (see Figure 8-1).
- Modify the collection ieee3023MacTransmitted by selecting it and select Edit
)Modify
MIB Collection… from the Data Collection menu bar (Figure 8-2).
- Click the [Configure Rearm Event…] button (Figure 8-8).
- Acknowledge the dialog box "No currently configured event for Event Identification "1.3.5.8.1.4.1.11.2.17.1.0.1002" sources: "hostname". Would you like to add this event configuration?" shown in Figure 8-14 by clicking [OK] .
Figure 8-14. When adding a custom event, you are prompted to add a new event configuration. A second prompt is displayed with instructions on how to handle specific sources for the rearm event. The rearm event is always an even number: the threshold event number plus one.
- Acknowledge the dialog box "Please add event configuration for Enterprise Identifier "OpenView". If you would like to have special event handling for sources " hostname" , add them into the "Source" field of the "Add Event" dialog" by clicking [Close] (Figure 8-10).NoteThe rearm event number will always be an even number equal to the threshold event number incremented by 1.
- Provide the Event Name and a Pop-up Notification in the Event Configuration dialog box and click [OK] , as shown in Figure 8-15.
Figure 8-15. Provide the
Event Name and the Pop-up Notification in the Rearm dialog box.
[View full size image]height="562" SRC="/image/library/english/10090_08fig15.jpg" >
Event Name TooManyPacketsRearm Pop-up Notification All is well on $2
- Select File
Save from the Event Configuration dialog box (Figure 8-5).
- The steps Select Actions
Resume Collection from the Data Collection & Thresholds dialog box (Figure 8-5).
- Select File
Save from the Data Collection & Thresholds dialog box (Figure 8-5).
- Select Actions
Show Data from the Data Collection & Thresholds dialog box (Figure 8-5).
8.2.3 Generating Actions Based on Custom Thresholds
When defining custom events, you can supply the command to be executed on the NNM system when threshold/rearm values are exceeded. Essentially, anything you type from the command line on the system may be used as an automatic action. This may be a script or a binary executable. Automatic actions are frequently used to send email, trigger audio alerts, alert paging devices, or pass information to a trouble ticketing system.
8.2.3.1 Automatic Actions for UNIX systems
Automatic actions are implemented in the Commands for Automatic Action field of the Threshold (Figure 8-12) and Rearm Event (Figure 8-15) configuration notification boxes. The command used in this field is executed on the management system. Any of the variables in the description field of the event may be passed in the automatic action. For example, the following action sends an email to root on a UNIX management station. The variable $2 is the nodename and $8 is the sampled value as described previously.
The automatic action is executed on the management system unless you use a utility, such as remsh , to run the command on a remote UNIX system. [3]If the management server was configured to execute actions on the managed node with root access, you could issue the following as an automatic action. Assume that a process, such as sendmail , needed to be restarted after a particular threshold was exceeded. You could define an automatic action such as this to restart the sendmail daemon:
echo "$2 exceeded packet threshold: $8." | mailx s "$2
Threshold Exceeded" root
[3] The remsh (REMote SHell) command requires a .rhosts file or /etc/hosts.equiv to be configured on a UNIX system. In many environments this is considered to be a security risk. For more details, refer to the UNIX man page on remsh for HP-UX and rsh for Solaris.
NoteThe OpenView Operations product actually is more capable of application monitoring than NNM. The point here is that NNM has the capability to execute automatic actions both locally and remotely if properly configured.
remsh remoteHost /sbin/init.d/sendmail start
8.2.3.2 Automatic Actions for Windows Systems
Actions for Windows systems can also be defined in Data Collection and Threshold Events. For example, if you want to send a message to a remote windows system, include the following automatic action in the Threshold dialog box. The $2 variable translates to the hostname of the system on which the packet threshold has been exceeded. $8 is the sampled value.
Given a hostname of winxp256 and a packet threshold of 99999, the resulting command displays a popup message on the remote system similar to that shown in Figure 8-15a.
"net send " Node $2 " exceeded packet threshold:" $8 "."
Figure 8-15a. The
net send command can be used to display a popup message on a Windows system.
This command can be used in the automatic action field of threshold event configuration. NNM can automatically restart the task scheduler on the remote node without human intervention. Assuming that a MIB variable exists to indicate whether the task scheduler service is running, you could create a threshold event to monitor that MIB variable and configure an automatic action to restart the task scheduler service.NoteBy default, NNM only performs commands that are trusted commands. You must specify the command to be trusted in a file that resides in the trusted commands configuration directory:UNIX:
rcmd \\ hostname net start "task scheduler"
$OV_CONF/trustedCmds.confWindows:
install_dir\ conf\trustedCmds.confThe format of this file is Keyword=Absolute Path and can include environment variables listed in the configuration file ov.envvars.sh . The following are sample entries for trusted commands file:
If the commands are not specified in the directory and are used in event configuration, NNM generates an error event and the action is not executed. You can override the trusted command feature by creating a file named ALLOW_ALL in the trusted commands configuration directory. After making modifications to the trusted commands directory, you must force the ovactiond process to re-read the configuration. This is accomplished by typing the following command:
snmpnotify=$OV_BIN/snmpnotify
ovIfIndexRemap.ovpl=$OV_BIN/ovIfIndexRemap.ovpl
xnmevents -events
8.2.4 Creating Custom Alarm Categories
As mentioned previously, the default alarm category for both threshold and rearm events is Threshold Alarms . When creating or modifying to an event, you can specify the category to which you would like it to be written. Custom alarm categories may be created for storing custom events. Follow these steps to create a custom category:
- Open the Event Configuration dialog box by selecting Options
Event Configuration .
- Select Edit
Configure
Alarm Categories from the Event Configuration dialog box, as shown in Figure 8-16.
Type the name of the new alarm category and click [Add] and [Close] , as shown Figure 8-17.Figure 8-16. To add a custom alarm category, select
EditConfigure
Alarm Categories… from the Event Configuration dialog box.
height="492" SRC="/image/library/english/10090_08fig17.gif" >
Figure 8-17. Provide the category name (Tammys Alarms) and click the
[Add] button. Then click the [Close] button. - Select File
Save from the Event Configuration dialog box.
- Verify that the new alarm category (Tammys Alarms) exists, as shown in Figure 8-18.
Figure 8-18. The new alarm category (Tammys Alarms) is displayed in the
Alarm Categories window. - Modify the category for your custom threshold and rearm events by using the drop-down list in the Category field, as shown in Figure 8-19.
Figure 8-19. The
Category of a custom Threshold Event may be modified to send Event Log Messages to a custom alarm category, such as Tammys Alarms.
[View full size image]height="458" SRC="/image/library/english/10090_08fig20.jpg" >
Figure 8-20. By default, event configurations are sorted by the Event Identifier. Event configuration may be sorted by selecting
View[View full size image]
8.2.5 Accessing Events from the Alarm Browser
Another way to access an event is via the Alarm Browser . You can locate the event configuration that generated the message by selecting a message in the Alarm Browser and selecting Actions
Figure 8-21. Event Configuration may be accessed from the Alarm Browser by selecting a message and selecting
Actions[View full size image]