The Business Case for Storage Networks [Electronic resources] نسخه متنی

Implementation

While continuing to search for the technology enablers to support the storage vision, Cisco reorganized the storage support team to better align with the vision's goals. The first move was to trim the operational team from 11 part-time storage administrators to six full-time storage administrators (and three full-time project managers) led by Scott Zimmer, senior IT manager, and Brian Christensen, director of IT business systems hosting. As the new team, named Enterprise Storage Services (ESS), became a cohesive unit, its focus shifted from ad-hoc firefighting to a more strategic direction of enabling standards and repeatable installations. Accordingly, the new storage team was tasked with critical business-impacting initiatives, such as consolidation of DAS and SAN islands to an enterprise-wide SAN infrastructure.

At the same time, Cisco started its negotiations to purchase Andiamo Systems, a startup venture that was building the next-generation Fibre Channel director, in which Cisco was the sole outside investor.

The Cisco SAN islands were comprised of a mix of fixed Brocade and McDATA Fibre Channel switches, which had grown to over 100 in number. As the impending release of the first generation MDS 9000 switches from Cisco/ Andiamo approached, however, the team halted additional SAN switch purchases long enough to certify the MDS 9000 switches for use in its own datacenter. This certification process entailed participation in a lengthy beta process as well as interoperability and performance testing with numerous products from various disk, software, and HBA vendors.

The Enterprise Storage Services team wasted no time implementing the new solution. The first deployment of the MDS 9509 switches in the Cisco datacenters came directly on the heels of completing the MDS 9000 beta program. As soon as the MDS hardware was released, it was deployed in the Cisco development and disaster recovery datacenters in Research Triangle Park (RTP), North Carolina.

Note

The Cisco IT department has a management objective to be the "first and best" Cisco customer, whereby Cisco IT provides product feedback early and often during each phase of the development and deployment life cycles. In addition, the Cisco-on-Cisco initiative dictates that wherever possible, Cisco run its IT operations on Cisco products. It is important to note, however, that the MDS platform was chosen, not mandated, as a technology enabler for the storage vision, based solely on its performance and its high availability (HA) capabilities.

In January of 2003, four MDS 9509s were deployed directly into the ERP and data warehousing development and disaster recovery environments in RTP, NC. In this environment, 11 HP-UX hosts and approximately 100 TB of storage were split across two SAN islands. The first SAN island, composed of seven HP hosts and 11 EMC Symmetrix frames, was built on a core of four McDATA, 32-port switches, two on each path, connected via ISLs. The second SAN island with the remaining four hosts was built on a core of two 32-port McDATAs attached to one Hewlett-Packard XP XP512 storage array. This environment is shown in Figure 8-1.

The cutover from the legacy McDATA environment to the MDS 9000 was done in two stages. In the first stage, one path in each SAN island was patched through one Cisco MDS 9509 director. Each director was configured with four 16-port Fibre Channel line cards and three 32-port Fibre Channel line cards to allow for maximum bandwidth to multiple environments in the future. The first stage of the cutover is shown in Figure 8-2.

Because the MDS 9000 was still a relatively new product, the team decided to run the applications dual-pathed through both the MDS 9000 and the McDATA switches for four weeks. This was long enough to demonstrate assured stability and performance, particularly for the ERP application, long considered one of the "crown jewels" of IT.

After four weeks with no issues, the second and final stage of the migration was completed when the team removed the connections to the McDATA switches and collapsed both SAN islands into one MDS fabric with four MDS 9509s (connected with two four-port ISLs).

Each cutover was completed during business hours with no disruption in service to the applications in question, one of which was a heavily utilized manufacturing development environment accessed by developers around the clock.

The final stage of the migration is shown in Figure 8-3.

To build a similar fabric that provided the capacity, performance, and reliability required to support the ever-growing ERP environment, the team would have needed to deploy a minimum of ten McDATA directors and dedicate many of those ports to ISLs. In short, the team would have needed to purchase two and a half times more McDATA switches to simulate the fabric built with the four MDS 9509s.

In addition to being scalable, the Enterprise Storage Services team found the Cisco MDS solution to be easy to manage and monitor with Cisco Fabric Manager, an integrated set of software tools that allows management and real-time monitoring of the Cisco MDS 9000 Family devices.

In addition to providing highly-available and robust SAN connectivity for the foundation of the storage utility vision, the MDS platform was also leveraged as part of an overall plan to lower operational expenditures and to reduce the total points of management. In the third and fourth quarter of 2003, multiple MDS units were aggressively deployed as the backbone of a major storage consolidation project.

Since the first migration, all SAN islands based on McDATA and Brocade switches have been replaced with MDS switches in Cisco datacenters around the world. In total, 142 Brocade and McDATA fixed switches and two McDATA directors were replaced with only 32 director-class MDS platform switches.

Long-term plans for SAN connectivity include collapsing all independent SAN fabrics into a single consolidated SAN infrastructure per datacenter, and linking all campus datacenters in each location over intercampus dark fiber with Fibre Channel over IP (FCIP). In addition, a major rollout of iSCSI is planned for 2005 and 2006.

Consolidation

Storage growth at Cisco remained unchecked across business units in IT from the latter part of the 1990s until early 2003. Given the disparate nature of the storage management initiatives and of the groups managing the storage, a significant number of purchases continued to be made, even though allocation efficiency hovered around 20 to 30 percent.

As the Enterprise Storage Services team began to gain traction and to share best practices, management determined that a significant cost savings could be achieved with a major consolidation effort. Management determined that, in addition to saving on maintenance bills and deferring datacenter expansions, a consolidation effort could increase economies of scale by reducing the number of points of management.

After negotiations with its primary storage vendor, Cisco agreed to purchase eight external RAID arrays in January, 2003 to consolidate 80 older frames, which were fully depreciated but carried a maintenance expense of over 4 millions dollars per year. The additional depreciation expense of the eight new frames was offset by immediate relief of the budgeted maintenance expense for the first year (and additional subsequent years for the life of the hardware). Although the project itself was monumentalcrossing multiple business units and involving over 30 outages to install FC HBAs into hosts which were previously DAS SCSIit was completed at budget and only two months behind schedule.

Initially, significant delays faced the project team stemming from two datacenter-related issues. The first was that the RTP datacenter was found to be at its maximum weight capacity in some locations. One month previous to the arrival of the consolidation hardware, management formed a "tiger team" that was dedicated to moving storage frames that were not on the consolidation list to make room for the new hardware.

The second issue causing delays in the project was the decision to deploy structured fiber cabling as part of the overall SAN infrastructure. Structured fiber offers significant management advantages over ad-hoc runs of fiber under the datacenter floor. When a structured fiber installation is completed, all fiber drops are patched through centralized patch panels and additional SAN storage capacity can be added without installing additional fiber.

The deployment of structured cabling even without the consolidation project would have been a significant undertaking. The two projects combined required monumental effort on the part of the storage and datacenter staff. After the structured cabling design was approved, and the cabling had begun, the new environment began to take shape. The consolidation hardware was deployed, and system administrators started copying host environments to the new locations. Unfortunately, a large number of the new fiber connectors were faulty and had to be replaced. During this time, the host environments could not be copied and the consolidation project was temporarily put on hold.