Definitive MPLS Network Designs [Electronic resources] نسخه متنی

Virtual POP Design

In parts of the world where Globenet had customers but did not have the possibility or the desire to extend the Globenet network, it elected to deploy virtual POPs (VPOPs). A VPOP is a POP managed by Globenet but colocated in the premises of another service provider. It is connected to the rest of Globenet's network via inter-AS TE LSPs providing bandwidth guarantees. More specifically, not only do such TE LSPs span multiple AS boundaries, but those autonomous systems belong to different service providers, which requires close cooperation.

This section, which describes the VPOP design, is one of the most specific aspects of the Globenet network design.

The motivation for relying on an inter-AS TE LSP to connect the VPOP to the rest of the network is as follows:

Fast convergence in case of inter-ASBR link failures When interconnecting two autonomous systems via a pair of ASBRs, the failure of such an inter-ASBR link usually provokes a traffic disruption that may last for a few tens of seconds or sometimes a few minutes. This is especially true when the pair of ASBRs is interconnected by means of a single link, because in this case the link failure requires that BGP converge (with a change of BGP next hop). One of Globenet's requirements was to be able to offer equivalent service availability to the end customers that are connected to a VPOP.

Bandwidth and QoS guarantees The second motivation for the use of inter-AS TE LSPs was to get bandwidth guarantees between the VPOP and the rest of the network. You will see in the rest of this section that MPLS Traffic Engineering was an ideal candidate to reach that objective.

Globenet managed to establish agreements with several regional service providers, including Africa Telecom, involving a private inter-AS TE peering. The case of Africa Telecom is covered in detail in this section.

Africa Telecom is a regional service provider with tens of POPs in Africa and three POPs in Europe (in Paris (France), London (UK), and Frankfurt (Germany)), interconnected with links from 2 Mbps to OC-3. MPLS Traffic Engineering has been deployed in Africa Telecom's core (ATC) for fast recovery with Fast Reroute for link protection.

Globenet had already deployed two POPs in Africa: Algiers (Algeria) and Johannesburg (South Africa). For the Johannesburg POP, the decision was made to convert it to a VPOP.

Conversion of the Johannesburg POP to a VPOP

Figure 5-47 shows the Johannesburg POP before its conversion to a VPOP (when the POP was part of Globenet's network).

Globenet followed these migration steps:

Globenet's customers are connected to the VPOP via leased lines or Frame Relay PVCs provided by Africa Telecom.

Attributes of the Inter-AS TE LSPs

As with any other TE LSP, several attributes must be determined for an inter-AS TE LSP:

Inter-AS TE LSP types Various types of inter-AS TE LSPs exist. A contiguous TE LSP is by definition an end-to-end TE LSP that traverses multiple domains (IGP area, AS). A stitched TE LSP is made of multiple segments in each domain that are stitched at the domain boundaries. Globenet and Africa Telecom decided to deploy contiguous TE LSPs so that Globenet could keep strict control of their inter-AS TE LSPs. Indeed, with a stitched TE LSP, each domain is in charge of its own segment. This implies that the headend router of such inter-AS TE LSPs loses control of the LSP reoptimization, for instance. Considering that such TE LSPs are really critical to Globenet because it carries all the traffic for each VPOP, contiguous TE LSPs were much more appropriate. Moreover, stitched TE LSPs have several limitations in terms of Fast Reroute. (See the IETF relevant documents for more details).

Bandwidth Although Globenet elected to use a dynamic bandwidth adjustment mechanism in each of its regions, such a model would have been difficult to apply for the case of inter-AS TE LSP because TE LSP costs are determined by their bandwidth. Hence, the agreement made with Africa Telecom was to statically define the inter-AS TE LSP bandwidth with the possibility to revisit each TE LSP bandwidth requirement once a month at Globenet's request. Consequently, Globenet developed some internal scripts gathering the amount of traffic sent to each inter-AS TE LSP once every 10 minutes. Each bandwidth sample is collected and the TE LSP bandwidth is adjusted monthly to the 95th percentile by Globenet.

TE/DS-TE Class Type Because the ATC network uses MPLS TE only for Fast Reroute and not for constraint-based routing, regular TE and its single class type (CT0) can be used in the ATC network to perform admission control and constraint-based routing of the inter-AS TE LSPs. Hence, these are signaled as CT0 LSPs, and ATC need not deploy DSTE. The section "VPOP QoS Design" has more details on the mechanisms involved in the ATC network to guarantee appropriate levels of performance to the traffic carried over the inter-AS TE LSPs.

Recovery Each inter-AS TE LSP is configured as fast reroutable to benefit from Fast Reroute in case of a network element failure within the ATC network or at AS boundaries. More details related to network recovery of inter-AS TE LSPs are discussed later in this section.

Globenet VPOP Migration Strategy

There were two main motivations for converting the Johannesburg POP into a VPOP:

The two leased lines Johannesburg-Lisbon and Johannesburg-Rome could be cancelled, leading to substantial cost savings. They were instead replaced by inter-AS TE LSPs whose cost per Mbps was significantly lower and whose bandwidth could be adjusted as needed with a significantly finer granularity.

This provided an opportunity for Globenet to get its equipment hosted in the Africa Telecom POP and hence decommission the premises it was running in Johannesburg. The costs there were very high, considering the need for power supply protection, air conditioning, and so on.

Four additional VPOPs were deployed in Africa following the same interconnection model as for the Johannesburg VPOP: Brazzaville (Congo), Cairo (Egypt), Khartoum (Sudan), and Abuja (Nigeria).

For instance, the two Globenet PE routers of the Brazzaville VPOP have been connected by means of inter-AS TE LSPs to Globenet's Algiers POP, as shown in Figure 5-50.

When the number of customers in a specific area gets large, Globenet can decide to deploy an additional VPOP so as to connect those customers to a closer VPOP, thus reducing the cost of the access lines. However, this requires adding and managing a new VPOP. Such economic analysis is conducted on a case-by-case basis.

Path Computation for Inter-AS TE LSPs

An inter-AS TE LSP path can be computed through various techniques that differ in terms of optimality. In a nutshell, the first method is called loose hop routing or per-domain routing. It consists of configuring the path on the headend router as a set of loose hops where each hop is an ASBR and the last hop is the final destination. Upon receiving the RSVP-TE Path message, the ASBR computes the shortest path in its AS, obeying the set of constraints to reach that hop. (This operation is sometimes called the ERO expansion.)

Although it's quite simple, the limitation of such an approach lies in its inability to compute a shortest path end to end. Indeed, when configuring the set of loose hops, the network administrator does not know which set of loose hops will provide the shortest path to reach the destination. The network topology in the other AS is unknown and might change as links and nodes are added in the future and network element failures unavoidably occur.

Consider the case of the inter-AS TE LSP between Globenet's Paris POP and Johannesburg VPOP. As shown in Figure 5-51, in steady state, the shortest inter-AS path between the two is via Paris' ATC ASBR (the solid line).

Let's now consider various failure cases.

In the first case, the link between Paris and N'Djamena fails. As shown in Figure 5-52, with the simple loose hop routing approach, Globenet's Paris router would retry along the same ATC ASBR of Paris. This would lead to the traversing of traffic across the 2-Mbps link between Paris and Algiers within the ATC network. Such a path would, of course, be suboptimal. However, Globenet's Paris ASBR could not have detected such suboptimality because of its lack of visibility to the available resources and topology in the ATC network.

Now consider the case of ATC router failure in Paris. Because of the lack of visibility on the ATC network topology and state, Globenet tends to configure on the Paris ASBR a backup loose path through the closest ASBR connected to the ATC network. Doing so reroutes the affected inter-AS TE LSPsin this case, to the Frankfurt ASBR connected to the Paris POP by means of an OC-48 link. Unfortunately, as shown in Figure 5-53, such a choice is not optimal because the Frankfurt ASBR of the ATC network is in fact connected to the N'Djamena node by a 2-Mbps link. In this failure case it would be more desirable to route the inter-AS TE LSP via Algiers (by means of OC-3 links in Globenet's network) so as to follow a path in the ATC network exclusively made of OC-3 links.

The two previous failure examples highlight that the loose hop routing path computation method cannot guarantee the selection of the shortest end-to-end path. Furthermore, even if a static configuration can be performed to get an optimal path in the majority of cases, this would require constant adjustments to cope with network upgrades, failures, and reoptimization within both networks.

Globenet conducted a detailed analysis to determine whether being able to compute the shortest path was a strict requirement. Globenet concluded that because Africa is a large continent and, more importantly, because the ATC network is made up of links having significantly different link speeds, there were many situations in which the difference in propagation delays and bandwidth (and consequently QoS) between the shortest path and an alternate path would be significant. Hence, it was required to select the end-to-end shortest path whenever possible.

Moreover, considering the number of peerings between Globenet and Africa Telecom (four for the moment, with plans to increase that number), any manual solution would be cumbersome and ineffective in handling dynamic and unpredictable changes (such as network element failures).

Consequently, Globenet and Africa Telecom elected to use a path computation element (PCE)-based computation approach for their inter-AS TE LSP path computation. One of the main advantages of the PCE-based computation approach described here of a PCE lies in its ability to compute the path of a TE LSP for which it is not the headend. In some cases, as shown next, the computation of an end-to-end path requires the cooperation of multiple PCEs. Ideal candidates for PCE are ASBRs (or ABRs in the case of interarea MPLS TE). The path computation algorithm consists of using a backward recursive approach whereby each PCE computes a set of shortest path(s) to the destination in its own domain. These are passed to the requesting upstream PCE, which itself computes a set of shortest paths to the destination, and so on.

In the Globenet-Africa Telecom case, two autonomous systems are involved. Let's discuss how such optimal paths are computed in this environment. Figure 5-54 shows Steps 1 and 2 in the computation of the inter-AS TE shortest path using a PCE-based approach between Globenet and Africa Telecom.

This example considers the case of an inter-AS TE LSP between the Globenet ASBR in Paris and a PE router residing in the Johannesburg VPOP. Note that a similar procedure applies to the case of the inter-AS TE LSP between a Globenet PE router residing in a VPOP and one of Globenet's ASBRs.

As shown in Figure 5-54, all the ASBRs of Globenet and ATC act as PCEs (Paris, Frankfurt, and Algiers).

For the sake of illustration, assume the following link metric assignment in both networks:

OC-192Cost = 1 (currently there is no OC-192)

OC-48Cost = 4

OC-3Cost = 16

2 MbpsCost = 1232

Let's now review the PCE-based inter-AS TE LSP path computation process step by step.

Let's now go back to the failure case of the link between Paris and N'Djamena. Upon receiving the failure notification by means of an RSVP-TE Path Error message, the Globenet ASBR in Paris would reinitiate a path computation request. ATC's ASBR in Paris would return the following set of three paths:

From the ATC ASBR of Algiers to the destinationCost = 48

From the ATC ASBR of Paris to the destinationCost = 1280

From the ATC ASBR of Frankfurt to the destinationCost = 1264

Thus, when the Globenet ASBR in Paris computes the shortest path from itself to the destination, it finds three possible paths:

Path 1 (via Paris)Cost = 1 + 1280 = 1281

Path 2 (via Frankfurt)Cost = 1264 + 4 + 1 = 1269

Path 3 (via Algiers)Cost = 16 + 1 + 48 = 65

Hence, the best end-to-end path is via Algiers, not Frankfurt (although this is the closest ASBR from Paris from the Globenet ASBR viewpoint).

This example demonstrates that the PCE-based path computation method always allows for the computation of the shortest end-to-end path across multiple domains.

[SECOND-METRIC] specifies the possibility of using either the IGP or the TE metric for the path computation. Every computed path cost returned by the PCEs would then be based on the TE link metric, thus ensuring the computation of the shortest end-to-end path.

Note

Such a PCE-based path computation approach can also be used in a context where the headend router is located anywhere in the originating AS. In such a case, a very similar procedure is used, whereby the headend router first initiates the request to the PCE residing in its domain. For example, consider a TE LSP that originates on a node X in the Globenet network and terminates on a PE router in Johannesburg. Then, there would just be an additional phase corresponding to the sending of the path computation request to the PCE in the Globenet network. The rest of the procedure would be identical.

Reoptimization of Inter-AS TE LSPs

In contrast with intra-area TE LSPs, where the triggers for reoptimization can be either timer- or event-driven, Globenet chose to elect a timer-based-only reoptimization approach for its inter-AS TE LSP. Globenet decided to trigger the reoptimization of its inter-AS TE LSPs every 2 hours. The reoptimization process is similar to the computation of an inter-AS TE LSP path. The only difference is that the path of the active TE LSP is provided in the path computation request to the PCE to avoid double booking of bandwidth when evaluating the existence of a better path. If a more optimal (shorter) path exists, the headend router resignals the TE LSP using the nondisruptive make-before-break procedure. We must highlight a very important aspect of PCE-based path computation concerning reoptimization. Consider again the case of failure of the link between Paris and N'Djamena in the ATC network. With a loose-hop routing path computation, there is no way to discover whether the initial preferred path (because the set of loose paths has to be manually configured) has been restored (if the current path does not go through the same set of ASBRs) other than by resignaling the TE LSP and seeing whether it succeeds. Of course, this is very undesirable. Conversely, the PCE-based path computation allows the headend to issue a new path computation request mentioning that such a request is related to a reoptimization. The headend router thus reroutes the inter-AS TE LSP if and only if a shorter path exists.

Routing onto Inter-AS TE LSPs

Routing the traffic onto inter-AS TE LSPs requires a bit of extra configuration. Indeed, an automatic mechanism so as to use such TE LSPs when computing the routing table is not currently available (but is being investigated at the time of writing). By definition, routing protocols hide some information across domains so that the headend of an inter-AS LSP does not have topology information about the domain in which the TE LSP's destination belongs. Note that this generally applies to interdomain MPLS Traffic Engineering, where a domain may be an IGP area or an autonomous system.

Consider the example shown in Figure 5-56. From the Globenet VPOP to Globenet's ASBR1, several static routes are required on PE1:

A static route pointing to LSP1 for the ASBR1 loopback address (destination of the inter-AS TE LSP)

A static route pointing to LSP1 for the loopback addresses for the set of PE routers and route reflectors residing in the Globenet network (just one route reflector is shown in Figure 5-56)

Note that PE1 learns Globenet's IPv4 and VPNv4 routes by means of the MP-BGP sessions between PE1 and the relevant set of Globenet route reflectors.

From the Globenet core network to VPOP, just one static route to the PE1's loopback address pointing to LSP2 is required on ASBR1. Note that such a static route is then redistributed into IS-IS.

The aim of the MP-BGP session is for ASBR2 to learn the inter-AS TE LSP's destination addresses and for ASBR1 to learn the inter-ASBR link addresses for management purposes. At this point, the regular BGP route recursion is applied to steer the traffic to the appropriate inter-AS TE LSP.

VPOP QoS Design

Globenet wanted to offer to customers connected to a VPOP the exact same QoS services, including support for its five VPN CoSs and the associated comprehensive SLA commitments. To allow this, Africa Telecom uses the inter-AS TE LSPs discussed earlier to give Globenet a service of Virtual IP Leased Line (VLL). Such a VLL service is characterized by the following:

Packets carried over the inter-AS TE LSPs experience a delay that is comparable to the delay provided by a classical leased line. (In other words, it is dominated by the propagation delay.)

Packets carried over the inter-AS TE LSPs experience a very small jitter (say, 30 to 50 ms). This is compatible with the jitter objectives that Globenet wants to commit to for its most demanding CoS (VPN voice) over that route.

Packets carried over the inter-AS TE LSPs experience a negligible loss.

QoS marking transparency is provided to the packets carried over the inter-AS TE LSPs. In other words, Globenet can set the packets' QoS markings according to its scheme on one side of Africa Telecom and can be sure that these markings will be preserved when the packets reenter Globenet on the other side.

To provide such a service, Africa Telecom decided first to handle the VLL traffic separately from the rest of its traffic from a scheduling perspective. At every hop of the ATC network, Africa Telecom activated a new queue dedicated to the VLL traffic. Because Africa Telecom was not using the EF queue to schedule its own traffic, it decided to use the EF queue for the VLL traffic and instantiate it as a strict priority queue to optimize delay and loss. A new EXP value of 5 is also dedicated to the VLL traffic.

Then, Africa Telecom takes full advantage of the TE capabilities to perform admission control and constraint-based routing of the inter-AS TE LSPs within its network. It configured BC0 to 50 percent on its OC-3 links and to 30 percent on the lower-speed links. First, this ensures that the load of VLL traffic routed on any link is sufficiently low so that the strict priority EF queue can guarantee the targeted delay/jitter/loss. Also, this ensures that sufficient capacity is kept for the rest of Africa Telecom traffic.

Should Africa Telecom need to schedule some of its own traffic into the EF queue in the future, it would carry the corresponding traffic onto appropriately sized TE tunnels. This would ensure that consistent admission control is performed across the inter-AS TE LSPs used for the VLL service and the intra-AS TE LSPs used for its own EF traffic.

To make sure that packets received from Globenet in the inter-AS TE LSPs can be classified easily in the ATC network, Africa Telecom applies an input QoS policy on its ASBRs on the interfaces attaching the Globenet routers. This policy unconditionally remarks the EXP field of the outmost label stack entry (the one containing the label for the inter-AS TE LSP) to EXP=5 (the VLL value in the Africa Telecom network). It leaves all the other QoS markings untouched (the EXP field of LDP and MP-BGP labels applied by Globenet as well as the DSCP of carried packets). This way, the packets entering Africa Telecom have EXP=5 in the outmost label stack entry but carry the Globenet marking in all the QoS fields. At the egress of Africa Telecom, the ASBR performs penultimate hop popping, which, by default, leaves the EXP value of all the exposed label stack entries untouched. Hence, when reentering Globenet's network, packets arrive with the regular Globenet QoS marking. Also, at the egress of Africa Telecom, the ASBR applies an egress QoS policy specified by Globenet and based on Globenet's EXP scheme. The QoS marking and QoS service policies at every step across the operators are illustrated in Figure 5-57 from the Globenet VPOP to the Globenet core.