Introduction to Intrusion Detection
It is becoming increasingly important for network security personnel to defend company resources, not only passively by using firewalls, virtual private networks (VPNs), encryption techniques, and whatever other tricks they have up their sleeves, but also by deploying proactive tools and devices throughout the network. This is where IDSs come in.Chapter 5, "Security Policies," for more details.Recall from Chapter 1, "Network Security Overview," that two types of potential intruders exist:
- Outside intruders
- Inside intruders
Although the majority of intrusion attempts actually occur from within the organization or by inside intruders, the most common security measures that are put in place protect the inside network from the outside world. Outside intruders are often referred to as crackers .It's clear that a mechanism is desirable and required to detect both types of intrusions continuously. IDSs are effective solutions for both types of attacks. These systems run constantly in your network, notifying network security personnel when they detect an attempt they consider suspicious. IDSs have two main components, namely, IDS sensors and IDS management.IDS sensors can be software and hardware based used to collect and analyze the network traffic. These sensors are available in two varieties, network IDS and host IDS.
- A host IDS is a server-specific agent running on a server with a minimum of overhead to monitor the operating system.
- A network IDS can be embedded in a networking device, a standalone appliance, or a module monitoring the network traffic.
IDS management, on the other hand, acts as the collection point for alerts and performs configuration and deployment services for the IDS sensors in the network.
IDS Fundamentals
A solid understanding of the fundamentals and different IDS technologies is required before the actual analysis and deployment discussions can start.
Notification Alarms
The overall purpose of IDSs is to trigger alarms when a given packet or sequence of packets seems to represent suspicious activity that violates the defined network security policy. Although alarms are essential, it is critical for network security personnel to configure the IDS to minimize the occurrence of false negative and false positive alarms.Let's start with a definition of these terms. A false positive is a condition in which valid traffic or a benign action causes the signature to fire.NOTEA signature can be best described as a set of events and patterns that is recognized from a protocol-decoded packet. This set defines an alarm-firing condition when offending network traffic is seen.A false negative is a condition in which a signature is not fired when offending traffic is transmitted. False negative alarms occur when the IDS sensor does not detect and report a malicious activity, and the system allows it to pass as nonintrusive behavior. This can be catastrophic for network operation. Therefore, minimizing false negatives has the highest priority. In general, there are two main reasons for a false negative to occur:
- The first results from the sensor lacking the latest signatures.
- The second can occur because of a software defect in the sensor.
The IDS configuration should be continuously updated with new exploits and hacking techniques upon their discovery.False positive alarms occur when the IDS sensor classifies an action or transaction as anomalous (a possible intrusion) although it is actually legitimate traffic. A false alarm requires an unnecessary intervention to analyze and diagnose the event. Clearly, network administrators try to avoid this type of situation because a large number of false positives can significantly drain resources, and the specialized skills required for analysis are scarce and costly.As a central warehouse of security knowledge, Cisco has developed an encyclopedia to provide security professionals with an interactive database of security vulnerability information.The Cisco Secure Encyclopedia can be accessed at the following location:http://www.cisco.com/pcgi-bin/front.x/csec/csecHome.plAs stated previously, the process of updating the IDS configuration is a continuous activity because it is virtually impossible to completely eliminate false positives and false negatives. For instance, if new applications are deployed throughout your organization, retuning the sensors might be required to minimize false positives. Most sensors provide flexible tuning capability during steady state operations, so there is no need to take them off-line at any point.
Signature-Based IDS
The signature-based IDS monitors the network traffic or observes the system and sends an alarm if a known malicious event is happening. It does so by comparing the data flow against a database of known attack patterns. These signatures explicitly define what traffic or activity should be considered as malicious. Various types of signature-based IDSs exist, including the following:
The pattern-matching systems look for a fixed sequence of bytes in a single packet, which has three advantages: It is simple, it generates reliable alerts, and it is applicable to all protocols. The weakness of pattern-matching systems is that any slightly modified attack leads to false negatives. Multiple signatures may be required to deal with a single vulnerability in stateful pattern-matching systems because matches are made in context within the state of the stream.Protocol decode-based systems decode very specific protocol elements, such as header and payload size and field content and size, and analyze for Request for Comment (RFC) violations. These systems have the advantage of being highly specific and, as a result, minimize the chance for false positives.NOTEProtocol-specific documentation is in the form of RFCs. These documents are published and reviewed by the Internet Engineering Task Force (IETF) working groups. For example, RFC 791 describes version 4 of the TCP/IP protocol.Table 10-1 gives a general overview of the pros and cons of signature-based IDSs.
Pros | Cons |
---|---|
Low false positive rate (reliable alerts) | Single vulnerability may require multiple signatures |
Simple to customize | Continuous updates required |
Applicable for all protocols | Modifications lead to misses (false negatives) |
Cannot detect unknown attacks | |
Susceptible to evasion |
Figure 10-1. Attack That Can Be Prevented Using Signature-Based IDS
Figure 10-2 displays the Network Security Vulnerability Index. Figure 10-3 is a typical example of an exploit signature and how it is formatted in the database.
Figure 10-2. Network Security Database
[View full size image]

Figure 10-3. A Smurf Attack Signature (Name, Signature ID, and Description)
[View full size image]

Policy-Based IDS
The policy-based IDSs (mainly host IDSs) trigger an alarm whenever a violation occurs against the configured policy. This configured policy is or should be a representation of the security policies (for more detail, see Chapter 5). For instance, a network access policy defined in terms of access permissions is easy to implement. The marketing department on network x is allowed to browse only engineering websites and has no access to FTP software directories on segment y. This is a fairly simple example of network policy; other policies are much harder to implement. If, for instance, a company's management team does not allow the browsing of game sites, the IDS must be able to communicate with a database of blacklisted sites to check whether a policy violation has occurred.Figure 10-4 illustrates this violation, which can be prevented by using a policy-based IDS. Employees from the engineering department should not be able to access either the marketing department VLAN or its servers.
Figure 10-4. Attack That Can Be Prevented Using Policy-Based IDS

Pros | Cons |
---|---|
Low false positive rate (reliable alerts) | Network administrator must design a set of policy rules from scratch |
Simple to customize | Long deployment time |
Anomaly-Based IDS
The anomaly-based IDS looks for traffic that deviates from the normal, but the definition of what is a normal network traffic pattern is the tricky part. Once the definition is in place, the anomaly-based IDS can monitor the system or network and trigger an alarm if an event outside known normal behavior is detected. An example of abnormal behavior is the detection of specific data packets (routing updates) that originate from a user device rather than from a network router. This technique is known in the world of crackers as spoofing, as described in Chapter 2, "Understanding VulnerabilitiesThe Need for Security."Table 10-3 gives a general overview of the pros and cons of anomaly-based IDS.
Pros | Cons |
---|---|
Unknown attack detection | High false positive rate |
Easy deployment for networks with well-defined traffic patterns | Interpretation of generated alarms is difficult |
Network IDS versus Host IDS
The previous sections outlined different analysis technologies. A good IDS has to be built around a solid implementation of these various technologies. Host IDSs and network IDSs are currently the most popular approaches to implement analysis technologies. A host IDS can be described as a distributed agent residing on each server of the network that needs protection. These distributed agents are tied very closely to the underlying operating system and are covered more in detail during the course of this chapter.
Figure 10-5. Host IDS

Figure 10-6. Network IDS

IDS Type | Pros | Cons |
---|---|---|
Host IDS | Verification of success or failure of an attack possible.Has a good knowledge of the host's context and, as a result, is more focused on a specific system.Not limited by bandwidth restrictions or data encryption. | Operating system/platform dependent. Not available for all operating systems.Impact on the available resources of the host system.Expensive to deploy one agent per host. |
Network IDS | Protects all hosts on the monitored networkcost effective.Independent of the operating system and has no impact on the host (runs invisibly).Especially useful for low-level attacks (network probes and DoS attacks). | Deployment is very challenging in switched environment.Network traffic may overload the NIDS (CPU intensive).Not effective for single packet attacks, and hidden attacks in encrypted packets. |
Evasion and Antievasion Techniques
Network IDSs have a fundamental problem whereby a skilled attacker can evade the detection mechanism by exploiting ambiguities in the traffic patterns, network topology, and the IDS architecture. Network IDS evasion enables the attacker to use techniques that challenge the detection mechanisms and therefore allow certain attacks to pass unnoticed.Chapter 2.As you can imagine, most vendors are aware of these evasion techniques and combat them by using antievasion countermeasures. Antievasion techniques can range from fragmentation alarms, packet loss alarms, and protocol decodes to tunable TCP stream reassembly options, alarm summarization, and others.
Organizational Issues and Complications
Intrusion detection spans many business functions within an organization. Organizational issues and complications are a direct result of the required interaction between the different groups.Chapter 6, "Secure Design," these constraints can be markedly different in nature and can include technological constraints, social constraints, and political constraints.
Technological Constraints
The changing needs of consumers and society in general are obvious. All these developments cause Internet traffic to double every few months, whereas CPU processing speed is only doubling about every year to year-and-a-half. Because of the far more rapid increase of Internet traffic levels, computation is still a constraint for network designers, particularly in the case of routers and switches. Typically, the computation (processing) limitations that apply to network design are associated with the processing of the routing table calculations, encryption and decryption of secured packets, accounting, incoming and outgoing access lists, or even normal packet forwarding. The processing of network traffic from IDSs may overload the sensor or appliance (such processing is CPU intensive) because it sniffs all packets being sent on a specific segment.Technological issues also include the bandwidth of the interfaces, tap placement, and switch configuration.
Social Constraints
Manpower or labor in general is clearly a concern in any network design. The more often a task must be executed, the more the design should focus on making that particular task simple and efficient to manage. Considering that 24 hours a day, 7 days a week, 365 days a year (24 x 7 x 365) monitoring and response capabilities are required for a proper IDS, a good IDS management design reduces labor costs. Network security personnel in charge of the IDSs require a cross-functional skill set, ranging from networking and security to operating systems. Staffing and personnel training should be considered as a top priority when designing an IDS for your network.Some larger enterprises can consider outsourcing their IDS management so that internal resources can be employed elsewhere. But when you consider the complexity of tuning the IDS according to the security policy, service-level agreements are not easy to negotiate.
Political Constraints
A company should have an incident response policy and procedure in place that has been approved by the senior management team. This policy includes recovery procedures in case of a severe attack. In addition, the following should be absolutely clear to the network administrator: the circumstances that require senior management notification and the stage at which the company's legal department calls for law enforcement.