Foundation Topics
What Causes a Failover Event?
In a PIX Firewall failover configuration, one of the PIX Firewalls is considered the active unit, and the other is the standby unit. As their names imply, the active unit performs normal network functions and the standby unit monitors and is ready to take control should the active unit fail to perform its functionality. A failover event occurs after a series of tests determines that the primary (active) unit can no longer continue providing its services, at which time the standby PIX Firewall assumes the role of the primary. The main causes of failover are shown in Table 10-2.
Table 10-2. Possible Failover Event Situations Failure Condition | Reasons that Standby becomes Active |
|---|
No Failure | Failover active An administrator can force the standby unit to change state by using the failover active command, which causes failover to occur. This is the only situation in which failover occurs without the primary (active) unit having any problems. | Power loss or reload | Cable errors The cable is wired so that each unit can distinguish between a power failure in the other unit and an unplugged cable. If the standby unit detects that the active unit is turned off (or resets), it takes active control. | Loss of power When the primary (active) unit loses power or is turned off, the standby unit assumes the active role. | PIX Firewall hardware failure | Memory exhaustion If block memory exhaustion occurs for 15 straight seconds on the active unit the standby unit becomes the active unit. | Network failure | Failover communication loss If the standby unit does not hear from the active unit for more than twice the configured poll time (or a maximum of 30 seconds), and the cable status is OK, a series of tests is conducted before the standby unit takes over as active. |
What Is Required for a Failover Configuration?
The hardware and software for the primary and standby PIX Firewalls must match in the following respects for failover configuration to work properly:
Note Failover for 501 and 506E models is not supported. The only additional hardware that is needed to support failover is the failover cable. Both units in a failover pair communicate through the failover cable. The failover cable is a modified RS-232 serial link cable that transfers data at 115 kbps. It is through this cable that the two units maintain the heartbeat network. This cable is not required for LAN-based failover. Some of the messages that are communicated over the failover cable are the following: Hello (keepalive packets) Configuration replication Network link status State of the unit (active/standby) MAC address exchange
It is also important to examine the labels on each end of the failover cable. One end of the cable is labeled "primary," and the other end is labeled "secondary." To have a successful failover configuration, the end labeled "primary" should be connected to the primary unit, and the end labeled "secondary" should be connected to the secondary unit. Changes made to the standby unit are never replicated to the active unit. In addition to the hardware and software requirements, it is also important to correctly configure the switches where the PIX Firewalls directly connect. Port Fast should be enabled on all the ports where the PIX Firewall interface directly connects, and trunking and channeling should be turned off. This way, if the PIX Firewall's interface goes down during failover, the switch does not have to wait 30 seconds while the port is transitioned from a listening state to a learning state to a forwarding state.
Port Fast
Many Cisco switches provide a Port Fast option for switch ports. Configuring this option on a switch port enables a simplified version of the Spanning-Tree Protocol that eliminates several of the normal spanning-tree states. The pre-forwarding states are bypassed to more quickly transition ports into the forwarding states. Port Fast is an option that you can enable on a per-port basis. It is recommended only for end-station attachments. |
Failover Monitoring
The failover feature in the Cisco PIX Firewall monitors failover communication, the power status of the other unit, and hello packets received at each interface. If two consecutive hello packets are not received within an amount of time determined by the failover feature, failover starts testing the interfaces to determine which unit has failed and transfers active control to the standby unit. At this point, the "active" LED on the front of the standby PIX Firewall lights up and the "active" LED on the failed PIX Firewall unit dims.
Note The failover poll seconds command enables you to determine how long failover waits before sending special failover hello packets between the primary and standby units over all network interfaces and the failover cable. The default is 15 seconds. The minimum value is 3 seconds, and the maximum is 15 seconds. Failover uses the following tests to check the status of the units for failure: Link up/down test If an interface card has a bad network cable or a bad port, is administratively shut down, or is connected to a failed switch, it is considered failed. Network activity test The unit counts all received packets for up to 5 seconds. If any packets are received at any time during this interval, the interface is considered operational and testing stops. If no traffic is received, the ARP test begins. Address Resolution Protocol test The unit's ARP cache is evaluated for the ten most recently acquired entries. One at a time, the PIX Firewall sends ARP requests to these machines, attempting to stimulate network traffic. After each request, the unit counts all received traffic for up to 5 seconds. If traffic is received, the interface is considered operational. If no traffic is received, an ARP request is sent to the next machine. If at the end of the list no traffic has been received, the ping test begins. Ping test A broadcast ping request is sent out. The unit then counts all received packets for up to 5 seconds. If any packets are received at any time during this interval, the interface is considered operational and testing stops. If no traffic is received, failover takes place.
Configuration Replication
Configuration changes, including initial failover configurations to the Cisco PIX Firewall, are done on the primary unit. The standby unit keeps the current configuration through the process of configuration replication. For configuration replication to occur, the two PIX Firewall units should be running the same software release. Configuration replication usually occurs when: The standby unit completes its initial bootup and the active unit replicates its entire configuration to the standby unit. Configurations are made (commands) on the active unit and the commands/changes are sent across the failover cable to the standby unit. Issuing the write standby command on the active unit forces the entire configuration in memory to be sent to the standby unit.
When the replication starts, the PIX Firewall console displays the message Sync Started. When the replication is complete, the PIX Firewall console displays the message Sync Completed. During the replication, information cannot be entered on the PIX Firewall console. The write memory command is important, especially when failover is being configured for the first time. During the configuration replication process, the configuration is replicated from the active unit's running configuration to the running configuration of the standby unit. Because the running configuration is saved in RAM (which is unstable), you should issue the write memory command on the primary unit to save the configuration to Flash memory.
Stateful Failover
In stateful failover mode, more information is shared about the connections that have been established with the standby unit by the active unit. The active unit shares per-connection state information with the standby unit. If and when an active unit fails over to the standby unit, an application does not reinitiate its connection because stateful information from the active unit updates the standby unit.
Note Some applications are latency-sensitive. In some cases, the application times out before the failover sequence is completed. In these cases, the application must reestablish the session. Replicated state information includes the following: TCP connection table, including timeout information for each connection Translation (xlate) table and status Negotiated H.323 UDP ports, SIP, and MGCP UDP media connections Port allocation table bitmap for PAT HTTP replication
Because failover cannot be prescheduled, the state update for the connection is packet-based. This means that every packet passes through the PIX Firewall and changes a connection's state, and triggers a state update. However, some state information does not get updated to the standby unit in a stateful failover:
Most UDP state tables are not transferred, with the exception of dynamically opened ports that correspond to multichannel protocols such as H.323. In addition to the failover cable, stateful failover setup requires a 100-Mbps or Gigabit Ethernet interface to be used exclusively for passing state information between the active and standby units. IP 105 is used to pass data over this interface. The stateful failover interface can be connected to any of the following: Category 5 crossover cable directly connecting the primary unit to the secondary unit 100BASE-TX full duplex on a dedicated switch or a switch's dedicated VLAN 1000BASE-SX full duplex on a switch's dedicated VLAN
A Cisco PIX Firewall with two FDDI cards cannot use stateful failover because an additional Ethernet interface with FDDI is not supported in stateful failover.
LAN-Based Failover
The distance restriction of 6 feet of serial cable between two PIX Firewall devices in a failover configuration is no longer a limitation starting with PIX Firewall Version 6.2. LAN-based failover is a new feature (available only on PIX Firewall 6.2 or higher) that extends PIX Firewall failover functionality to operate through a dedicated LAN interface without the serial failover cable. This feature provides a choice of failover configuration on the PIX Firewall. The obvious benefit of LAN-based failover is that it removes the 6-foot distance limitation from the PIX Firewall devices in a failover configuration. If the LAN-based failover command interface link goes down, the PIX Firewall notifies the peer through "other" interfaces, and then the standby unit takes over. If all connectivity between the two PIX Firewall units is lost, both PIX Firewalls could become active. Therefore, it is best to use a separate switch for the LAN-based failover command interface, so that a failed switch will not cause all connectivity to be lost between the two PIX Firewall units. The weakness of LAN-based failover is the delayed detection of its peer power loss, consequently causing a relatively longer period for failover to occur.
Note Crossover Ethernet cables cannot be used to connect the LAN-based failover interface. Additionally, it is recommended that you dedicate a LAN interface for LAN-based failover, but the interface can be shared with stateful failover under lightly loaded configurations. Cisco PIX Firewall Version 6.2 enhances failover functionality so that the standby unit in a PIX Firewall failover pair can be configured to use a virtual MAC address. This eliminates potential "stale" ARP entry issues for devices connected to the PIX Firewall failover pair in the unlikely event that both firewalls in a failover pair fail at the same time.
Configuring Failover
To configure failover, you need to become familiar with a few key commands. Table 10-3 shows the commands used to configure and verify failover.
Table 10-3. PIX Firewall Failover Commands Command | Description |
|---|
failover lan enable | Enables LAN-based failover. | failover | Enables the failover function on the PIX Firewall. Use this command after you connect the failover cable between the primary and secondary unit. Use the no failover command to disable the failover feature. | failover lan key key-secret | Specifies the shared secret key. | failover active | Makes the PIX Firewall unit it is issued on the active unit. This command is usually used to make the primary unit active again after repairs have been made to it. | failover ip address if-name ip-address | Issued on the primary unit to configure the standby unit's IP address. This is the IP address that the standby interface uses to communicate with the active unit. Therefore, it has the same subnet as the system address.[a] The if-name argument is the interface name, such as outside . The ip-address is the interface name's IP address. | failover link stateful-if-name | Enables stateful failover on the specified interface. | show failover | This popular command displays the status of the failover configuration. | failover poll seconds | Specifies how long failover waits before sending special hello packets between the primary and secondary units. The default is 15 seconds. The minimum is 3 seconds, and the maximum is 15 seconds. | failover reset | Can be entered from either unit (active or standby), preferably the active unit. This forces the units back to an unfailed state and is used after repairs have been made. | write standby | Enter the write standby command from the active unit to synchronize the current configuration from RAM-to-RAM memory to the standby unit. | failover lan interface interface-name | Configures LAN-based failover. | failover lan unit primary | secondary | Specifies the primary or secondary PIX Firewall to use for LAN-based failover. | failover replicate http | Allows the stateful replication of HTTP sessions in a stateful failover environment. |
[a] The system address is the same address as the active unit IP address. When the active unit fails, the standby assumes the system address so that there is no need for the network devices to be reconfigured for a different firewall address.
Figure 10-1 shows two PIX Firewall units in a failover configuration. Example 10-1 shows a sample configuration for a PIX Firewall Failover configuration.
Figure 10-1. Network Diagram of Failover Configuration
Example 10-1. Sample Configuration for primary-PIX
hostname primary-PIX
nameif ethernet0 outside security0
nameif ethernet1 inside security100
nameif ethernet2 failover security10
interface ethernet0 100full
interface ethernet1 100full
interface ethernet2 100full
ip address outside 192.168.1.1 255.255.255.0
ip address inside 10.10.10.1 255.255.255.0
ip address failover 172.16.10.1 255.255.255.224
failover ip address outside 192.168.1.2
failover ip address inside 10.10.10.2
failover ip address failover 172.16.10.2
global (outside) 1 192.168.1.15-192.168.1.40 netmask 255.255.255.224
nat (inside) 1 0.0.0.0 0.0.0.0 0 0
Configuring failover involves defining your configuration on the primary PIX Firewall. This configuration is then replicated to the standby PIX Firewall. The following steps illustrate the tasks needed to define a basic PIX Firewall configuration utilizing a serial failover deployment.
Note Before you begin the failover configuration, be sure that you connect the failover cable to the units correctly. Also be sure that the standby unit is not powered on.
Step 1. | Enable failover:
Primary-pix (config)# failover
| Step 2. | Assign interface ethernet2 a name for stateful failover:
Primary-pix (config)# nameif ethernet2 failover securitry10
| Step 3. | Set the interface speed:
Primary-pix (config)# interface ethernet2 100full
| Step 4. | Assign an IP address to the interface:
Primary-pix (config)# ip address failover 172.16.10.1 255.255.255.240
| Step 5. | Verify your failover configuration:
Primary-pix (config)# show failover
| Step 6. | Configure the secondary unit IP address from the primary unit by using the failover ip address command. Add the failover ip address command for all interfaces, including the one for the dedicated failover interface and any unused interfaces:
Primary-pix (config)# failover ip address outside 192.168.1.2
Primary-pix (config)# failover ip address inside 10.10.10.2
Primary-pix (config)# failover ip address failover 172.16.10.2
| Step 7. | Save your configuration:
Primary-pix (config)# write memory
| Step 8. | Use the show ip address command to view the addresses you specified:
Primary-pix (config)# show ip address
System IP Addresses:
ip address outside 192.168.1.1 255.255.255.0
ip address inside 10.10.10.1 255.255.255.0
ip address failover 172.16.10.1 255.255.255.240
Current IP Addresses:
ip address outside 192.168.1.1 255.255.255.0
ip address inside 10.10.10.1 255.255.255.0
ip address failover 172.16.10.1 255.255.255.244
The current IP addresses are the same as the system IP addresses on the failover active unit. When the primary unit fails, the current IP addresses become those of the standby unit. | Step 9. | Enable stateful failover:
Primary-pix (config)# failover link failover
| Step 10. | Power up the secondary unit. At this point, the primary unit starts replicating the configuration to the secondary. | Step 11. | Verify your failover configuration:
Primary-pix (config)# show failover
Failover On
Serial Failover Cable status: My side not connected
Reconnect timeout 0:00:00
Poll frequency 15 seconds
Last Failover at: 22:19:11 UTC Mon Jan 19 2004
This host: Primary - Active
Active time: 345 (sec)
Interface failover (172.16.10.1): Normal
Interface outside (192.168.1.1): Normal
Interface inside (10.10.10.1): Normal
Other host: Secondary - Standby
Active time: 0 (sec)
Interface failover (172.16.10.1): Normal
Interface outside (192.168.1.1): Normal
Interface inside (10.10.10.1): Normal
Stateful Failover Logical Update Statistics
Link : failover
Stateful Obj xmit xerr rcv rerr
General 0 0 0 0
sys cmd 0 0 0 0
up time 0 0 0 0
xlate 0 0 0 0
tcp conn 0 0 0 0
udp conn 0 0 0 0
ARP tbl 0 0 0 0
RIP Tbl 0 0 0 0
Logical Update Queue Information
Cur Max Total
Recv Q: 0 0 0
Xmit Q: 0 0 0
The show failover command displays the last occurrence of a failover. The first part of the show failover command output describes the cable status. Each interface on the PIX Firewall unit has one of the following values: Normal The active unit is working, and the standby unit is ready. Waiting Monitoring of the other unit's network interfaces has not yet started. Failed The PIX Firewall has failed. Shutdown The interface is turned off. The second part of the show failover command describes the status of the stateful failover configuration. Each row is for a particular static object count: General The sum of all stateful objects. Sys cmd Refers to logical update system commands, such as login and stay alive . Up time The value for PIX up time that the active PIX Firewall unit passes on to the standby unit. Xlate The PIX Firewall translation information. Tcp conn The PIX Firewall dynamic TCP connection information. Udp conn The PIX Firewall dynamic UDP connection information. ARP tbl The PIX Firewall dynamic ARP table information. RIF tbl The dynamic router table information. The Stateful Obj has these values: Xmit Indicates the number of packets transmitted. Xerr Indicates the number of transmit errors. Rcv Indicates the number of packets received. rerr Indicates the number of receive errors. | Step 12. | Enter the write memory command from the active unit to synchronize the current configuration to the Flash memory on the standby unit. |
|