Lesson 2 Introduction to the ISA Server Firewall

The Internet provides organizations with new opportunities to connect with customers, partners, and employees. While this presents great opportunities, it also opens new risks and concerns in areas such as security, performance, and manageability. As the use of the Internet continues to expand, so do the security and performance challenges. Many security problems can be addressed with the ISA Server Firewall service, which allows you to control access to and from network resources through rules based on any number of configurable policy elements.

After this lesson, you will be able to


Describe how ISA Server provides three layers of filtering methods

Describe ISA Server's built-in application filters

Describe the types of packet-level or application-level attacks that can be detected by ISA Server


Estimated lesson time: 35 minutes

Filtering Methods

A firewall enhances security by using various filtering methods, including packet filtering, circuit-level (protocol) filtering, and application filtering. Advanced enterprise firewalls, such as ISA Server, combine these methods to provide protection at multiple network layers.

IP Packet Filtering

Packet filters allow you to control the flow of IP packets to and from ISA Server, as shown in Figure 1.4. When Packet Filtering is enabled (this setting can be viewed or modified on the IP Packet Filters Properties dialog box), all packets on the external interface are dropped unless they are explicitly allowed. With IP packet filtering, your system intercepts and evaluates packets before they are passed to higher levels in the firewall engine or to an application filter.

Figure 1.4 IP packet filtering

If you configure IP packet filters to allow only certain specified packets to pass through the ISA Server, you can greatly enhance the security of your network. IP packet filtering also allows you to block packets originating from specific Internet hosts and reject packets associated with many common attacks. With IP packet filtering, you can also block packets destined to any service on your internal network, including the Web Proxy service, Web server, or an SMTP server.

IP packet filters can filter packets based on service type, port number, source computer name, or destination computer name. IP packets filters are static; they apply to specific ports, and they are always either allowed or blocked. Allow filters allow the traffic through, unconditionally, at the specified port. Block filters always prevent the packets from passing through the ISA Server computer.

Even if you do not enable packet filtering, communication between your local network and the Internet is allowed only when you explicitly configure protocol rules that permit access.

Circuit-Level (Protocol) Filtering

You can configure circuit-level or protocol filtering in ISA Server through access policy rules and publishing rules. As shown in Figure 1.5, this feature lets you inspect sessions as opposed to connections or packets. A session can include multiple connections, which provides a number of important benefits for Windows-based clients running Firewall Client software.

Figure 1.5 Circuit-level or protocol filtering sessions may comprise more than one connection.

Dynamic Filtering

ISA Server supports dynamic filtering through access policy rules and publishing rules. With dynamic filtering, ports open automatically only as required for communications, and ports close when the communication ends. This approach minimizes the number of exposed ports in either direction, and it provides a high level of hassle-free security for your network.

Support for Session-Based Protocols

Circuit-level filtering provides built-in support for protocols with secondary connections, such as FTP and streaming media. It also allows you to define the protocol's primary and secondary connection in the user interface without any programming or third-party tools. You can achieve this by specifying the port number or range, protocol type, Transmission Control Protocol (TCP) or User Datagram Protocol (UDP), and inbound or outbound direction.

Application Filtering

The most sophisticated level of firewall traffic inspection is application-level security. Good application filters allow you to analyze a data stream for a particular application and provide application-specific processing including inspecting, screening or blocking, redirecting, or modifying the data as it passes through the firewall. As illustrated in Figure 1.6, this mechanism is used to protect against hazards such as unsafe SMTP commands or attacks against internal Domain Name System (DNS) servers. Third-party tools for content screening, including those used for virus detection, lexical analysis, and site categorization, all use application and Web filters to enhance the functionality of your firewall.

Figure 1.6 Application-level filtering

ISA Server includes the following built-in application filters:

HTTP Redirector Filter. The HTTP redirector filter forwards HTTP requests from the firewall and SecureNAT clients to the Web Proxy service. This creates transparent caching for clients that do not have their browser configured to direct to the Web Proxy service.

FTP Access Filter. The FTP filter intercepts and checks FTP data. A kernel-mode data pump gives you high-performance data transfer for approved traffic.

SMTP Filter. The SMTP filter intercepts and checks your SMTP e-mail traffic, protecting mail servers from attack. The filter recognizes unsafe commands and can screen e-mail messages for content or size, rejecting unapproved e-mail before it ever reaches the mail server.

SOCKS Filter. For clients without Firewall Client software, the SOCKS filter forwards requests from SOCKS 4.3 applications to the ISA Firewall service. The access policy rules determine whether the SOCKS client application communicates with the Internet. Unlike Winsock, SOCKS can support any client platform, including Unix, Macintosh, and non-standard computer devices.

RPC Filter. The RPC filter allows sophisticated filtering of RPC requests based on specific interfaces. You select RPC interfaces to expose.

H.323 Filter. The H.323 filter directs H.323 packets used for multimedia communications and teleconferencing. It provides call control, including the capability to handle incoming calls and to connect to a specific H.323 gatekeeper.

Streaming Media Filter. The streaming media filter supports industry-standard media protocols, including Microsoft Windows Media Technologies and both streaming media protocols from RealNetworks, Progressive Networks Audio (PNA) and Real-Time Streaming Protocol (RTSP). It also allows users to split live Windows Media streams, thus saving bandwidth.

POP and DNS Intrusion Detection Filters. These two filters recognize and block attacks against internal servers, including DNS Host Name Overflow, DNS Zone Transfer, and Post Office Protocol (POP) Buffer Overflow.


H.323 Gatekeeper

H.323 Gatekeeper works together with the H.323 protocol filter to provide full communications capabilities to H.323-registered clients that use applications compliant with H.323 Gatekeeper, such as NetMeeting 3.x. H.323 Gatekeeper provides registered clients with call routing and directory services and enables others to reach them using their well-known alias. Clients registered with H.323 Gatekeeper can use H.323 Gatekeeper to participate in video, audio, and data conferences in local area networks (LANs) and wide area networks (WANs); across multiple firewalls; and over the Internet. As shown in Figure 1.7, H.323 Gatekeeper is configured in ISA Management through the H323 Gatekeepers node.

Figure 1.7 Configuring H.323 Gatekeepers

Broad Application Support

ISA Server predefines about 100 application protocols and allows administrators to define additional protocols based on port number, type, TCP or UDP, and direction. Protocols with secondary connections are supported using Firewall Client software or an application filter.

Bandwidth Rules

Bandwidth rules determine which connection gets priority over another. ISA Server bandwidth control does not limit how much bandwidth can be used. Rather, it informs the Windows 2000 QoS packet scheduling service how to prioritize network connections. Any connection that does not have an associated bandwidth rule receives a default scheduling priority. On the other hand, any connection with an associated bandwidth rule will be scheduled ahead of default-scheduled connections.

Integrated Virtual Private Networking

ISA Server helps administrators set up and secure a virtual private network (VPN). As illustrated in Figure 1.8, a VPN is an extension of a private network that encompasses links across shared or public networks like the Internet. A VPN enables you to send data between two computers across a shared or public intranet in a manner that emulates the properties of a point-to-point private link. ISA Server can be configured as a VPN server to support secure, gateway-to-gateway communication or client-to-gateway remote access communication over the Internet.

Figure 1.8 VPN integration with ISA Server

The local VPN wizard runs on ISA Server on the local network. The local ISA VPN computer connects to its Internet Service Provider (ISP). The remote VPN wizard runs on the ISA Server on the remote network. The remote ISA Server VPN computer connects to its ISP. When a computer on the local network communicates with a computer on the remote network, data is encapsulated and sent through the VPN tunnel. Windows 2000 standards–based VPN supports PPTP and L2TP/IPSec tunneling technology. A tunneling protocol, such as PPTP or L2TP, is used to manage tunnels and encapsulate private data. Data that is tunneled must also be encrypted to be a VPN connection.

Integrated Intrusion Detection

ISA Server features an integrated intrusion-detection mechanism. This identifies when an attack is attempted against your network. The firewall administrator can set alerts to trigger when an intrusion is detected. You can also specify, with alerts, what action the system should take when the attack is recognized. This may include sending an e-mail message or page to the administrator, stopping the Firewall service, writing to the Windows 2000 Event Log, or running any program or script. ISA Server implements intrusion detection at both the packet filter and the application filter level.

ISA Server's intrusion detection feature is based on technology licensed from Internet Security Systems (ISS), Inc., Atlanta, GA, http://www.iss.net.

Packet Filter Intrusions

At the packet filter level, ISA Server detects the following attacks:

All Ports Scan Attack. An attempt is made to access more than the preconfigured number of ports.

Enumerated Port Scan Attack. An attempt is made to count the services running on a computer by probing each port for a response.

IP Half Scan Attack. Repeated attempts are made to connect to a destination computer, but no corresponding connection is established. This indicates that an attacker is probing for open ports while evading logging by the system.

Land Attack. A land attack involves a TCP connection that was requested by a spoofed source IP address and port number that match the destination IP address and port number. If the attack is successfully mounted, it can cause some TCP implementations to go into a loop that crashes the computer.

Ping of Death Attack. A large amount of information is appended to an Internet Control Message Protocol (ICMP) echo request/ping packet. If the attack is successfully mounted, a kernel buffer overflows when the computer attempts to respond, and crashes the computer.

UDP Bomb Attack. This is an attempt to send an illegal UDP packet. A UDP packet that is constructed with illegal values in certain fields causes some older operating systems to crash when the packet is received.

Windows Out of Band Attack. This means an out-of-band, denial-of-service attack is attempted against a computer protected by ISA Server. If mounted successfully, this attack causes the computer to crash or causes a loss of network connectivity on vulnerable computers.


POP and DNS Application Filters

ISA Server also includes POP and DNS application filters that analyze all incoming traffic for specific intrusions against the corresponding servers. The DNS intrusion detection filter helps you to intercept and analyze DNS traffic destined for the internal network. The POP intrusion detection filter intercepts and analyzes POP traffic destined for the internal network. The administrator can configure the filters to check for the following intrusion attempts.

DNS Hostname Overflow. A DNS hostname overflow occurs when a DNS response for a host name exceeds a certain fixed length. Applications that do not check the length of the host names may return overflow internal buffers when copying this host name, allowing a remote attacker to execute arbitrary commands on a targeted computer.

DNS Length Overflow. DNS responses for IP addresses contain a length field, which should be four bytes. By formatting a DNS response with a larger value, some applications executing DNS lookups will overflow internal buffers, allowing a remote attacker to execute arbitrary commands on a targeted computer.

DNS Zone Transfer from Privileged Ports (1-1024). A DNS zone transfer from privileged ports (1-1024) occurs when a client system uses a DNS client application to transfer zones from an internal DNS server. The source port number is a privileged port number (between 1 and 1024), indicating a client process.

DNS Zone Transfer from High Ports (above 1024). A DNS zone transfer from high ports (above 1024) occurs when a client system uses a DNS client application to transfer zones from an internal DNS server. The source port number is a high port number (above 1024) that indicates a client process.

POP Buffer Overflow. A POP buffer overflow attack occurs when a remote attacker attempts to gain root access of a POP server by overflowing an internal buffer on the server.


Secure Publishing

ISA Server uses server publishing to process incoming requests to internal servers, such as SMTP servers, FTP servers, database servers, and others. Requests are forwarded downstream to an internal server, which is located behind the ISA Server computer.

Server publishing allows virtually any computer on your internal network to publish to the Internet. Security is not compromised, because all incoming requests and outgoing responses pass through ISA Server. When a server is published by an ISA Server computer, the IP addresses that are published are actually the IP addresses of the ISA Server computer. Users who request objects think that they are communicating with the ISA server—whose name or IP address they specify when requesting the object—while they are actually requesting the information from the publishing server.

For example, when you use Microsoft Exchange Server with ISA Server, you can create server-publishing rules that specifically allow the e-mail server to be published to the Internet. In this scenario, the ISA Server firewall intercepts the Exchange Server's incoming e-mail. This makes ISA Server appear as an e-mail server to clients. With ISA Server, you can filter the traffic and forward it to the Exchange Server according to whatever rules and policies you configure. Your Exchange Server is never exposed directly to external users and sits in its secure environment, maintaining access to other internal network services.

Figure 1.9 illustrates how you can use ISA Server in a similar way to publish securely to Web servers. When a client on the Internet requests an object from a Web server, the request is actually sent to an IP address on the ISA Server. Web publishing rules configured on the ISA Server forward the request as applicable to the internal Web server.

Figure 1.9 ISA Server protects internal publishing servers.

Lesson Summary

The ISA Server firewall provides filtering at three separate levels. First, through IP packet filters, ISA Server either blocks or allows a connection based on service type, port number, source computer name, or destination computer name. IP packets filters are static; they apply to specific ports, and they are always either allowed or blocked. Second, ISA Server provides session-aware circuit filtering in the form of access policy rules and publishing rules. This capability allows for dynamic packet filtering and provides support for protocols with secondary connections. Finally, ISA Server's application filters allow you to analyze a data stream for a particular application and provide application-specific processing including inspecting, screening or blocking, redirecting, or modifying the data as it passes through the firewall.

The sophisticated, multilayer nature of ISA Server's Firewall service allows you to configure powerful and flexible access control policies, intrusion detection, secure server publishing, bandwidth prioritizing, and VPN integration.