Definitive MPLS Network Designs [Electronic resources] نسخه متنی

Globenet Network Environment

Several characteristics are specific to international service provider networks. Typically they rely on a very broad set of physical link types and related Layer 2 technologies. This is primarily because of the wide range of pricing per Mbps of bandwidth (for instance, link costs are significantly higher in Asia than in the U.S.). Consequently, this unavoidably results in limited capacity in certain regions, such as Asia and intercontinental links.

This led Globenet to carefully optimize the design in some parts of its network using various technologies such as QoS, MPLS Traffic Engineering, and ATM to efficiently engineer its network and reduce the recurrent bandwidth costs. Specifically, Asia takes advantage of ATM with Private Network-Network Interface (PNNI) routing and signaling protocol. POPs are interconnected through Globenet's ATM switches with carefully sized and carefully routed ATM variable bit rate-real time (VBR-rt) virtual circuits (whose size is determined based on traffic requirements). Conversely, high-speed SONET-DWDM links (OC-3 and OC-48) have been deployed between the POPs in the U.S.

Globenet does not own any fiber. Instead, it leases every circuit (protected and unprotected SONET-Synchronous Digital Hierarchy (SDH) circuits, dense wavelength division multiplexing (DWDM) links, time-division multiplexing (TDM) circuits, and ATM permanent virtual circuits (PVCs)) around the world and has to deal with many other carriers and service providers.

In larger POPs, Globenet uses separate PE routers and P routers, just like the service providers described in earlier chapters. However, in smaller POPs, Globenet has a number of routers that play the role of both P router and PE router, without any separation between the two functions. Shared PE routers are used to support all IP services, including Internet access and IP VPNs.

Virtual POP Design."

This highlights the level of complexity required by such international networks in terms of interoperations with regional and sometimes international service providers. This entails elaborate arrangements at the commercial level, at the technical level, such as those necessary to provide tight Service Level Agreements (SLAs) and high availability in all regions, and at the operational level, such as appropriate interactions across network management systems from different providers. The Network Operations Centers (NOCs) have to deal with many other service providers while continuing to be the single point of contact. Moreover, the operational arrangements also involve sophisticated monitoring and troubleshooting tools to quickly and efficiently determine a problem's root cause in case a failure occurs in some part of the world.

Globenet Service Portfolio

Globenet has developed a broad data service portfolio over the years. Originally it offered X.25, Frame Relay, and ATM international services. The X.25 and Frame Relay traffic has significantly decreased over the last three years, and Globenet actively encourages its X.25 and Frame Relay customers to migrate toward other VPN services such as Layer 3 MPLS VPN. Nevertheless, ATM traffic continues to increase slightly, especially in Asia. Therefore, Globenet continues to provide native ATM services.

Internet access services have been offered for many years and continue to increase at a rate of approximately 40 percent a year. Internet connectivity occurs via private and public peering in Network Access Points (NAPs) as well as transit providers.

Layer 3 MPLS VPN service has undoubtedly been the most successful service over the last three years. Globenet has a current installed base of 16,000 dedicated VPN ports grouped into 500 VPNs. It has a traffic growth rate of 80 percent per year and increases its number of customer sites by 120 percent per year. It also offers multicast and voice VPN services.

Moreover, Globenet supports a rich set of five classes of service (CoSs) in the context of the Layer 3 MPLS VPN offering.

Remote access to both VPN and Internet services has been very successful, primarily because of the high population of mobile workers in international companies.

Recently, Globenet started expanding its Layer 3 MPLS VPN service offering to support IPv6 VPNs in addition to the existing IPv4 VPNs.

Globenet POP Network Structure

Globenet POPs are of four different typesType 1a, Type 1b, Type 2, and Type 3based on the density of customer access and aggregated traffic throughput.

Type 1 POP Structure

Type 1 POPs are located in large cities and are made up of a set of two P routers and multiple PE routers. P routers are connected to other Type 1 POPs via E3, OC-3, or OC-48 links and to the Type 2 POPs via n * 2 Mbps and OC-3 links. In Asia, they are connected via ATM PVCs supported over Globenet's ATM switches. The multiple PE routers are connected to the P routers via Gigabit Ethernet switches, as shown in Figure 5-1. Furthermore, several virtual LANs (VLANs) are used for separation:

Figure 5-1. Globenet Type 1a POP Design (Every Region Except Asia-Pacific)

[View full size image]

One VLAN connects the Network Access Server/Broadband Access Server (NAS/BAS) to the various PE routers in their respective VRFs.

One VLAN connects the core-facing interfaces of the PE routers to the P routers.

P routers connect PE routers as well as NASs. In the Type 1 POPs located in countries where asymmetric digital subscriber line (ADSL) service is offered, BASs are also connected to the P routers.

Figure 5-2 shows the Type 1a POP design in the Asia-Pacific region (AsiaPac). The difference with the Type 1a POP in other regions is that connectivity to Type 2 POPs in AsiaPac is supported through ATM PVCs provided by Globenet ATM switches.

Figure 5-2. Globenet Type 1a POP Design for the AsiaPac Region

Chapter 1, "Technology Primer: Layer 3 VPNs, Multicast VPNs, IPv6, and Pseudowire.") Such Type 1b POPs are deployed in the North America and EuropeMiddle EastAfrica (EMEA) regions. Here, Globenet decided to carry its ATM traffic over its MPLS infrastructure because of the dominance of IP traffic over native ATM traffic in these regions. The transport of the ATM traffic by means of pseudowire service is illustrated in Figure 5-4.

Figure 5-3. Globenet Type 1b POP Design

[View full size image]

Figure 5-4. Transport of ATM by Means of Pseudowire Service in Type 1b POPs

[View full size image]

Type 2 POP Structure

Smaller POPs are classified as Type 2. They have only a single set of routers that act as both P routers and PE routers and are called P/PE routers. There are at least two such P/PE routers per POP, each connected to other Type 1 or Type 2 POPs. P/PE routers are also connected to CE routers, NASs, and occasionally BASs. The interconnection between devices is identical to a Type 1 POP and relies on Layer 2 Gigabit Ethernet switches. The structure of a Type 2 POP is shown in Figure 5-5 (all regions except AsiaPac) and Figure 5-6 (AsiaPac).

Figure 5-5. Globenet Type 2 POP Design (All Regions Except AsiaPac)

[View full size image]

Figure 5-6. Globenet Type 2 POP Design for the AsiaPac Region

[View full size image]

Type 3 POP Structure

In Asia, Globenet has very small POPs limited to one P/PE router connecting the NAS and several CE routers. Such Type 3 POPs are interconnected to the closest Type 1 or Type 2 Globenet POP by means of ATM PVCs provided by its ATM network or leased from a regional service provider. The structure of a Type 3 POP is shown in Figure 5-7.

Figure 5-7. Globenet Type 3 POP Design (Only in Asia)

[View full size image]

Globenet Worldwide Network Architecture

The total number of routers residing within the 77 POPs run by Globenet worldwide slightly exceeds 250, including 200 PE routers sharing the Layer 3 MPLS VPN and Internet services.

To help illustrate the overall network architecture, the next few sections describe the network topology built by Globenet on a per-region basis.

Note

For the sake of simplicity, for the regions with a large number of POPs, just a subset of representative POPs is shown in each figure.

EMEA Region

[MLPPP]) is used. The structure of the EMEA region is shown in Figure 5-8.

Figure 5-8. Globenet Network Design for the EMEA Region

[L2VPN]. If you want to know more about this resource, look up the code in the "References" appendix to find out specific information about the resource.

In EMEA, the IP network and the ATM network (deployed earlier for support of native ATM services) operate using either of two models in different parts of the network:

In "ships in the night" mode, the ATM network and the IP/MPLS network each have their own dedicated bandwidth capacity and operate independently of each other.

In "inverted overlay" mode, in some cases, ATM switches are trunked over the IP/MPLS core network via MPLS pseudowires. In this mode, all the bandwidth is managed by the IP/MPLS network, and the bandwidth needed by ATM is provided by the IP/MPLS network through the MPLS pseudowires.

Globenet's strategy in EMEA is to migrate toward a full inverted overlay mode in the long run considering the increasing IP/ATM ratio in this region.

Asia-Pacific Region

The total number of POPs in Asia is 30, with two additional POPs in Australiaone in Sydney (Type 1) and another in Perth (Type 2). The network topology is (partially) shown in Figure 5-9.

Figure 5-9. Globenet Network Design for the AsiaPac Region

[View full size image]

Only a few of the POPs in Asia are Type 1 POPs, and they are interconnected with unprotected OC-3 links. The rest are predominantly Type 2 POPs with a few Type 3 POPs.

Because of significant bandwidth costs in this region, Globenet recognized the need for careful bandwidth dimensioning and for tight traffic engineering. This is why it decided to interconnect the Type 2 and Type 3 POPs in this region via ATM virtual circuits (VCs) supported over Globenet's ATM switches instead of via separate dedicated links (as is done in EMEA). This allows Globenet to make fine-grain optimization of bandwidth usage, because VCs can be individually sized to match current demand without any granularity limitations that come with direct links.

Hence, in Asia, Globenet effectively uses an overlay model in which all the capacity is used to interconnect the ATM switches and where Type 2 POPs are interconnected by ATM VCs supported off of the ATM switches. The ATM VC sizes vary between 1 Mbps (for the smallest POP, such as the Type 3) and 40 Mbps. Above such rates it is usually more cost-effective to interconnect routers by means of dedicated leased lines.

For operational and cost reasons, Globenet resizes each ATM VC only once every six months (based on measured load multiplied by an overbooking factor). Because of this low frequency, Globenet elected to use an additional traffic engineering technique (MPLS-based TE) on top of its ATM PVCs to further optimize the use of the bandwidth provided by the ATM network between ATM VC resizing periods. Moreover, as explained in detail in this chapter, MPLS TE allows for the use of dynamic TE LSP dimensioning every two hours, which could not be easily achieved by means of ATM PNNI.

North America Region

There are 15 POPs in Globenet North America region (the U.S. and Canada). The majority of those are Type 1b, interconnected by unprotected OC-48 (based on leased DWDM links). The rest are Type 2 POPs connected to Type 1 POPs via unprotected OC-3 links. The North America topology is shown in Figure 5-10.

Figure 5-10. Globenet Network Design for the North America Region

[View full size image]

Because Globenet's ATM switches are interconnected via ATM pseudowires over the MPLS network in North America, you see that Globenet uses the inverted overlay model in this region. In this model, all the capacity is managed by the IP/MPLS network, and ATM switches are interconnected via MPLS pseudowires supported off of the IP/MPLS network.

South America Region

Globenet has six Type 2 POPs in South America located in Caracas (Venezuela), Bogota (Columbia), Rio de Janeiro (Brazil), Santiago (Chile), Buenos Aires (Argentina), and Lima (Peru).

Until recently, all these POPs were interconnected via ATM PVCs supported over the ATM switches using the overlay model (as deployed in the Asia region). However, a recent upgrade changed this topology to dedicated 34-Mbps links directly connecting the POP routers, thereby migrating from the overlay model to the "ships in the night" model also used in parts of the EMEA region. The topology for this region is shown in Figure 5-11.

Figure 5-11. Globenet Network Design for the South America Region

QoS Design in the Core Network in the EMEA, AsiaPac, and South America Regions" and "MPLS Traffic Engineering in the AsiaPac, EMEA, and South America Regions."

It is worth observing that because of each region's specificities in terms of bandwidth service costs and availability, Globenet ended up using three different models for IP/MPLS and ATM interworking:

A "ships in the night" model was chosen in some places in EMEA as well as in South America, where IP/MPLS and ATM run independently of each other.

An overlay model was chosen in AsiaPac, with IP/MPLS running over ATM.

An inverted overlay model was chosen in North America and in some places in EMEA, with ATM running over IP/MPLS.

The ratio between the IP and ATM traffic was a key decision-making factor for selecting the appropriate model for IP/MPLS and ATM interworking. Indeed, in places where the majority of the traffic is IP, the inverted model is the most appealing and cost-effective. Conversely, in regions where such a ratio is in favor of ATM, the overlay model is perfectly adequate. Finally, the "ships in the night" model adopted in some places in EMEA and in South America is motivated by a situation in which neither the IP/MPLS traffic nor the ATM traffic is predominant. In the future, Globenet foresees a general trend toward generalizing the inverted overlay model considering the considerable growth of the IP traffic.

Intercontinental Connectivity

All the previously described regions are interconnected via a mix of protected E3 and OC-3 links, as shown in Figure 5-12.

Figure 5-12. Globenet Network Design for Interregion Connectivity

[View full size image]

Globenet Routing Architecture

Globenet elected to use a multi-AS routing architecture in which each region constitutes a separate autonomous system. The rationale for this was primarily driven by scale, where regional routing information could be bounded. Within each AS the routing protocol of choice is Intermediate System-to-Intermediate System (IS-IS) with a single IS-IS Level 2 topology.

Internet routes are distributed to the P routers. However, this is purely a failsafe procedure so that Internet traffic may continue to be forwarded in the event of a label-switching failure. The design of Globenet's Internet route reflection is as discussed in Chapter 4. However, each region is an independent autonomous system. Internet access is provided locally within each region, as well as through the distribution of IPv4 routes between different regions.

The total number of Interior Gateway Protocol (IGP) routes varies from AS to AS but never exceeds 1000 in any given AS. For IPv4 Internet, Globenet currently carries approximately 140,000 routes. For its Layer 3 MPLS VPN service, the total number of VPNv4 routes worldwide is 80,000, including routes that are received from regional inter-AS partners. Because of the increasing number of VPNv4 routes, Globenet chose to filter routes between regions, even though in terms of number of routes the current savings at the Autonomous System Boundary Routers (ASBRs) is minimal.

Interoperator Partnerships

Chapter 4) is one example of a regional service provider with whom Globenet is partnering for seamless Layer 3 MPLS VPN service.

In some countries, however, Globenet is either unable (because of local regulations) or unwilling (because of a large number of customers) to use any of the connectivity models in the inter-AS VPN solution set to extend its network. In these cases, Globenet might use a virtual POP (VPOP) in which one of its routers is colocated in the POP of a regional service provider. The router is connected to Globenet's main network by means of inter-AS traffic-engineered label-switched paths (TE LSPs) providing bandwidth guarantees for transit through the regional service provider. Such interconnection is discussed in detail in the "Virtual POP Design" section.

Link Types and Protection Details

Table 5-1 shows the different link types and characteristics in the Globenet network.

Table 5-1. Link Types and Characteristics in the Globenet Backbone
Link Type	Speed	Protection	Localization
OC-48 DWDM	2.5 Gbps	None	North America and EMEA: Type 1 to Type 1 POPs
OC-3	155 Mbps	Protected and unprotected, depending on the region	All regions: Type 1 to Type 1 POPs Type 1 to Type 2 POPs
E3	34 Mbps	Protected	In South America: Type 2 to Type 2 POPs
ATM	From 1 Mbps to 40 Mbps	Via ATM PNNI	Asia Pacific: Type 1 to Type 2 POPs Type 2 to Type 2 POPs Type 2 to Type 3 POPs
n * 2 Mbps	From 2 Mbps (n=1) to 8 Mbps (n=4)	Protected	EMEA: Type 1 to Type 2 POPs Type 2 to Type 2 POPs
Gigabit Ethernet	1 Gbps/10 Gbps	None	All regions: Within POPs

Note

In general, Globenet always tries to deploy unprotected OC-3 links (and to rely on MPLS Traffic Engineering Fast Reroute functionality to provide fast local protection). However, there are several regions where only protected OC-3 links are available. This explains the mix of protected and unprotected OC-3 links.

During the past ten years, Globenet has gathered various network failure statistics that vary greatly within each region in terms of both frequency and duration. The results are summarized in Table 5-2. These statistics have greatly influenced the routing and MPLS Fast Reroute network design.

Table 5-2. Link Failure Statistics in the Globenet Network
Failure Type	Link/Router Type	Occurrence	Duration
Link failure	OC-48 links (EMEA and U.S.)	Unknown	Unknown
Link failure	OC-3 SONET links (unprotected)	Between once a week (EMEA) to several times a day (South America)	From a few seconds to several days (fiber cut) or even weeks for the transatlantic links
Link failure	ATM	On average, twice a week	From 1 second to 1 minute
Router interface failure	Edge+core	Negligible	A few hours
Unplanned router failure (such as power supply, router software failure with traffic impact)	Edge+core	Once every 3 months	Variable
Router reboot (planned failure)	Edge	Once every 10 months	10 minutes
Router reboot (planned failure)	Core	Once a year	10 minutes

Definitive MPLS Network Designs [Electronic resources] نسخه متنی

Globenet Network Environment

Globenet Service Portfolio

Globenet POP Network Structure

Type 1 POP Structure

Figure 5-1. Globenet Type 1a POP Design (Every Region Except Asia-Pacific)

Figure 5-2. Globenet Type 1a POP Design for the AsiaPac Region

Figure 5-3. Globenet Type 1b POP Design

Figure 5-4. Transport of ATM by Means of Pseudowire Service in Type 1b POPs

Type 2 POP Structure

Figure 5-5. Globenet Type 2 POP Design (All Regions Except AsiaPac)

Figure 5-6. Globenet Type 2 POP Design for the AsiaPac Region

Type 3 POP Structure

Figure 5-7. Globenet Type 3 POP Design (Only in Asia)

Globenet Worldwide Network Architecture

EMEA Region

Figure 5-8. Globenet Network Design for the EMEA Region

Asia-Pacific Region

Figure 5-9. Globenet Network Design for the AsiaPac Region

North America Region

Figure 5-10. Globenet Network Design for the North America Region

South America Region

Figure 5-11. Globenet Network Design for the South America Region

Intercontinental Connectivity

Figure 5-12. Globenet Network Design for Interregion Connectivity

Globenet Routing Architecture

Interoperator Partnerships

Link Types and Protection Details

Table 5-1. Link Types and Characteristics in the Globenet Backbone

Table 5-2. Link Failure Statistics in the Globenet Network