Definitive MPLS Network Designs [Electronic resources] نسخه متنی

Figure 5-25. Regional Customer Attachment Example

[rt-constrain], to restrict distribution of routes based on extended community attributes.

[rt-constrain] provides a new SAFI (RT filter) that is used to advertise the route target extended community attributes used at the PE routers for import/export policy. For example, consider Figure 5-26. Globenet has a customer called ABC Inc. in California and another customer called XYZ Inc. in New York. Each PE router advertises the route target values used by these VPNs to the route reflectors using the RT filter SAFI. Using this information, the route reflectors can filter the advertisement of the routes to unnecessary regions of the network.

As illustrated in Figure 5-27, this filtering results in the route reflectors in California not storing the routes that are used only on the East Coast. Likewise, the route reflector in New York doesn't store the routes that are used only in California.

In the case of ASBRs that have no VRFs configured, an extended community list filter may be used to generate the updates to the RT filter SAFI.

Globenet Routing Convergence Strategy

Convergence can be defined as the time taken for routers in a particular routing domain to learn about the complete topology and to recompute an alternative path (assuming that one exists) to a particular destination after a network change has occurred. This process involves the routers adapting to these changes by synchronizing their view of the network with other routers in the same domain. Because Globenet uses a number of protocols that all interact, it was important within the design to define a routing convergence strategy that covered both its backbone and its services' convergence.

Broadly speaking, convergence may be split into two subcategories:

Convergence of the internal infrastructure This category includes backbone IGP convergence and MP-BGP control plane convergence.

Convergence of external services This category includes convergence of routing information between external customer sites, such as between sites using the Layer 3 MPLS VPN service.

Globenet identified that the convergence of its backbone network was dependent on which protection mechanisms it deployed at different layers. The strategy chosen was to combine MPLS fast reroute and fast IGP convergence, as detailed in the "Network Recovery Design" section later in this chapter.

Layer 3 MPLS VPN ServiceRouting Convergence

The Layer 3 MPLS VPN architecture uses the services of BGP (with multiprotocol extensions) to distribute VPN routing information between the edges of the network. BGP was essentially invented to solve a route distribution problem in a very scalable manner and to provide a mechanism to achieve a loop-free routing topology. Because of its distance vector-like behavior, and the features implemented to provide stability, BGP by its very nature does not converge as quickly as a link-state IGP. Interaction between MP-BGP and the IGP can have a significant effect on routing convergence. For this reason, additional implementation timers have been added to speed up the default routing convergence times. These timers may be adjusted to provide an optimal deployment of a Layer 3 MPLS VPN service.

From a backbone network perspective, Globenet tunes its IS-IS timers to gain subsecond IGP convergence. However, because the Layer 3 MPLS VPN service relies heavily on the MP-BGP protocol for route distribution, subsecond IGP convergence is insufficient to ensure fast failure detection affecting a set of VPN routes. Instead, next-hop reachability of all routes is checked, by default, every 60 seconds (this is driven in IOS by the "scanner" process and is a configurable timer). If a particular link fails in the backbone network, the IGP can detect this very quickly. However, the MP-BGP process may take up to 60 seconds to determine that a given next hop is no longer available. Therefore, the routes previously learned via that next hop are invalid. For this reason, Globenet chose to deploy the next-hop tracking (NHT) feature on all its PE routers.

The NHT feature is enabled by default in the IOS version deployed at the Globenet PE routers. It allows MP-BGP to register next-hop addresses (PE router loopback interface addresses) for MP-BGP routes with the RIB Address Tracking Filter (ATF) feature. This feature provides an efficient routing update notification service and monitors all route changes in the PE router's Routing Information Base (RIB). When a route changes in some way (for example, it becomes unreachable, or the metric changes) the ATF immediately notifies MP-BGP so that it has current routing information and therefore can react to the change. If a next hop for a given set of VPNv4 routes changes, the router can react immediately rather than waiting for the periodic "scanner" process as previously described.

Although backbone convergence clearly was important, Globenet noted that it needed to define convergence characteristics of its Layer 3 MPLS VPN service from its customers' point of view. It defined this as site-to-site convergence. To this end, it wanted to make sure that convergence times (in terms of route distribution from customer site to remote customer site(s)) were as close as possible to those obtained by the customer when Globenet was using its previous Layer 2 overlay technology. To achieve this goal, Globenet realized that a certain amount of protocol tuning would be necessary, primarily within MP-BGP.

To understand each of the components of the overall site-to-site convergence time, Globenet analyzed in the laboratory how a routing update was sent from one site to another (including new and withdrawn routes). Using this information, Globenet defined eight individual convergence points in the total end-to-end convergence time. These are illustrated in Figure 5-28, as highlighted by T1 through T8.

These are defined as follows:

Convergence point T1 PE router receipt of a routing update from an attached customer site.

This convergence point is very dependent on the routing protocol used on the PE-CE link. However, the most noticeable figure relates to the case in which BGP-4 is used on the PE-CE link because it relies on a timer to advertise routes, the maximum value of which is 30 seconds.

Convergence point T2 Import of local routing information received at point T1 into the corresponding ingress VRF(s).

This value is typically close to 0, except in the case of OSPF where an SPF-delay timer is used, the default of which is 5 seconds.

Convergence point T3 Receipt of local routes received at point T1 into MP-BGP on the local PE router.

This value is always close to 0 because routes are placed into MP-BGP immediately on receipt.

Convergence point T4 Advertisement of newly inserted VPNv4 routes to MP-BGP peers (the Globenet route reflectors).

This convergence point is driven by the MP-BGP advertisement-interval timer for internal BGP. By default it is 5 seconds, although this timer is initiated only when an update is sent. Therefore, if no updates have been sent in the last 5 seconds preceding the receipt of another route, the update for this route is sent immediately. Because Globenet deploys VPNv4 route reflectors, this timer is always 10 seconds. Convergence point T4 is relevant to the initial PE router that first advertises a route plus the route reflector that subsequently advertises it between the edge of the network.

Convergence point T5 Processing of advertised VPNv4 routes from point T4 on a remote PE router.

MP-BGP updates are processed immediately; therefore, this convergence point is always close to 0.

Convergence point T6 Import of newly received routes at point T5 into local VRF(s).

In IOS the PE router runs a timer called the import scanner every 15 seconds by default (although this can be configured). This timer is not applied in case of route withdrawals because these are processed immediately, or for next-hop accessibility checks. However, it is relevant in terms of when newly received routes are imported into interested local VRFs.

Convergence point T7 Advertisement of routes received from the MPLS backbone to CE routers.

The value of this convergence point is the same as for T1. Assuming that the Globenet customer uses BGP-4 on the PE-CE link, it uses the same advertisement interval default of 30 seconds in BGP-4.

Convergence point T8 Processing of incoming updates by the relevant attached CE router(s).

Generally this is close to 0 except in the case of OSPF on the PE-CE link, which uses an SPF-delay timer with a default of 5 seconds.

Using these convergence points, Globenet determined that the default theoretical convergence times, depending on the PE-CE protocols, were as detailed in Table 5-3.

These convergence times were clearly the worst-case figures and therefore were likely to be substantially lower in practice. They were also based on intra-AS connectivity. If the routing information needed to cross AS boundaries, these times would increase based on the additional BGP-4 convergence. However, Globenet felt that some parameter tuning was necessary to reduce these figures, especially in the case of external BGP on the PE-CE links.

Having decided to tune protocol timers, Globenet realized that the values set would depend on a number of factors (such as the number of routes, total number of VRFs, total number of BGP-4 sessions, and so on) because of their potential processing impact. Globenet also realized that careful validation of the settings was necessary. After various lab tests, Globenet noted that the biggest gains in terms of convergence time reduction could be obtained by tuning the operation of the BGP protocol, both within the backbone network and on the PE-CE links (assuming that BGP-4 was in operation on these links).

Tuning the BGP Protocol

The main delay in route convergence with the BGP protocol is the time taken to advertise a new or deleted VPN route. This time is primarily driven by the advertisement interval timer. This is set by default to 5 seconds for internal BGP (convergence point T4) and 30 seconds for external BGP (convergence points T1 and T7).

Globenet chose to reduce the internal BGP timer to 1 second and the external BGP timer to 5 seconds. These new timer values allow routes to be distributed across the backbone network more quickly. They also provide a small delay for the advertisement of these routes to external peers to allow a certain amount of packing of routes into the updates.

Using these new timer values, Globenet was able to drop the theoretical maximum convergence time (when BGP-4 is used on the PE-CE links) to 27 seconds. (This is the default theoretical maximum of 85 seconds minus twice a 4-second saving for internal BGP and twice a 25-second saving for external BGP.) This time is more inline with the other routing protocols. Globenet monitors the routers' available resources on a regular basis to make sure that these new timer values do not negatively affect its routers' scalability.

Edge Router Capabilities

Clearly the edge router capabilities are a major factor in scaling the Layer 3 MPLS VPN service. The main components that drive the edge router's scalability are processing power and available memory space.

When reviewing the CPU and memory characteristics, Globenet noted several points that affected the overall scale. In general, it found that CPU usage is driven by a number of factors, including (but not limited to) the following:

Quality of Service Design" section, Globenet minimized the QoS features to be supported on the PE routers in the following ways:

In the case of managed CE routers, the classification, marking, and policing functions are currently entirely performed by the CE router's completely offloading this task from the PE router. This greatly facilitates scaling. It is much easier for each CE router to support the QoS functions relevant to a single site than for a PE router to support the QoS function for all the sites attached to it.

Globenet chose a DSCP/EXP value scheme such that EXP values get automatically marked correctly based on the Differentiated Services Codepoint (DSCP) values via the IOS default mapping so that no DSCP-to-EXP mapping needs to be performed by the PE router.

In the case of unmanaged CE routers, a relatively light input policy is applied because traffic classification doesn't involve any fancy packet inspection and is based only on the DSCP field.

In a nutshell, Globenet ensured that the QoS features on the input side have no, or minimal, impact on the PE router performance. Hence, the QoS features to be considered for PE router performance impact are primarily the ones related to scheduling on the egress (when forwarding packets onto the PE-CE link). These features really cannot be avoided, because they are essential to the enforcement of Globenet's five classes of service over this congestion point. They also have been designed as light as possible because they also only rely on DSCP-based classification.

"Managed" or "unmanaged" service For customers that use the "managed" service (they have managed CE routers), the CPU requirement may be distributed between the PE router and the attached CE routers, as just discussed for the specific aspect of QoS. This helps scale the overall system to a greater extent. This is not possible for unmanaged CEs.

PE-CE protocol connectivity type Each routing protocol has different requirements and therefore needs more or fewer processing cycles. For example, OSPF requires the processing of initial LSA generation but may remain relatively quiet assuming no routing changes, except for database refresh every 30 minutes. In contrast, RIP sends periodic updates at regular intervals.

Number of attached VPNs and associated VRFs Clearly the number of VRFs at the PE router drives CPU requirements because the amount of routing information to be processed and stored increases with the number of VRFs.

From a memory perspective, Globenet noted that the amount of memory required at the PE router increases based on the following:

Number of VRFs/routes Each VRF and every route within the VRF requires memory. As the number of VRFs and routes increases, so does the required memory. Note that a VPNv4 route requires more memory than an IPv4 route.

Internet routes at the PE routers Because Globenet stores Internet routes at some of its PE routers, the amount of memory available to store VPN routes is clearly reduced. This may be quite significant on some of its older router platforms that have limited memory.

OSPF/RIPv2 connectivity OSPF and RIPv2 both rely on databases to store their routing information. This memory is in addition to that used to store the routes in the VRF and forwarding tables. Globenet therefore restricts the number of these types of customers that may be deployed on any given PE router.

BGP paths The number of remote sites for a given VPN increases the number of routes and paths received from remote locations at the local PE router. Given this, Globenet chose to use the maximum routes command in each VRF to restrict the number of routes and to be able to engineer the routers appropriately.

Globenet has several different edge router platforms, each of which had its maximum scale characterized during lab verification testing in terms of the maximum number of services it can support (assuming typical service distribution and associated features). The parameters used to determine the maximum scale were based on the typical split of Internet/Layer 3 VPN customer attachments, typical access speeds, typical CoS portfolio, maximum continuous CPU load of 50 percent, and so on. Based on this, Globenet defined provisioning rules for each platform it specified as 70 percent of the maximum scale tested.