Software Upgrades
Upgrading existing software applications from time to time incorporates new features and fixes any defects that existed in the earlier software versions. The upgrade planning and procedures vary from one IPT component to another. Hence, you should read the upgrade documentation and release notes for new versions thoroughly before you perform an upgrade.Also, before you perform upgrades in the production network, test the new software versions in the lab network to ensure that you do not encounter new problems, such as lack of interoperability with other products in the network.You can make the following types of upgrades in Cisco IPT solutions:Upgrade the operating systemUpgrade application software, such as Cisco CallManager, Cisco Unity, and Cisco Customer Response Applications (CRA)Upgrade the software on IPT endpoints, such as gateways and Cisco IP phones
The next few sections provide the best practices and recommendations for performing various software upgrades.
Operating System Upgrades
Again, a general rule is to test any OS upgrades or application software upgrades in the lab network prior to applying them in the production network. This section describes the following recommended practices for OS upgrades:Subscribe to notification alerts from Cisco Systems.Stop services such as antivirus scanning and monitoring tools.Use recommended server access methods to perform upgrades.Schedule the upgrade.Document registered device counts.
Subscribe to Notification Alerts
It is important to ensure that all the people in the organization who are responsible for managing the IPT network are notified of new software releases related to voice products as and when they become available. You can sign up for the Cisco Voice Technology Group subscription tool, available at the following URL, to receive notifications on the new OS service releases, CallManager, and other applications availability:http://www.cisco.com/cgi-bin/Software/Newsbuilder/Builder/VOICE.cgiYou also should sign up to receive security field notices from Cisco. For procedures on subscribing, refer to the following URL:http://www.cisco.com/en/US/products/products_security_vulnerability_policyl#SecurityInfo
Stop Services
Before you perform software upgrades on CallManager servers or any other Cisco AVVID application servers, such as Cisco Unity or CRA, stop the antivirus software scanners and third-party monitoring tools and disable Cisco Security Agent (CSA) if it is running.Use the following steps to disable CSA:
Step 1. | Right-click the CSA icon on the program bar and click Suspend Security. Click OK when asked whether you want to suspend it. |
Step 2. | Go to the Start > Programs > Administrative Tools > Services menu option on the CallManager or on the application server. From the list of services, select the Cisco Security Agent Service and click the Stop Service button to stop the service. |
To disable McAfee VirusScan:
Step 1. | Double-click the VirusScan icon on the program bar. |
Step 2. | Click the <Disable> button. |
NoteYou must complete both of the steps described to disable CSA. Otherwise, CSA prevents the installation process from stopping IISADMIN or any other required services and installation fails. If you are using a virus scanner other than McAfee, follow the vendor instructions to disable it.
Use Recommended Server Access Methods to Perform Upgrades
Cisco recommends that you not perform upgrades via Microsoft Terminal Services Client. Instead, perform upgrades via Virtual Network Computing (VNC) or directly from the console. VNC software is bundled along with CallManager. It is available under the C:\Utils\VNC directory on the CallManager servers. For security reasons, you should disable VNC during normal operations. When you need to perform upgrades, use Windows Terminal Services to enable VNC and then disable it after the upgrade is complete. For upgrades, or whenever a reboot is required, always make sure to have a local resource around to work directly on the server console, if required.
Schedule the Upgrade
In most cases, software upgrades to servers require a reboot. This causes some interruptions to the call processing. Therefore, make sure to do the following:Plan to do the upgrades during off-peak hours.Notify your management, network operations center (NOC) staff, and end users about the anticipated service disruptions and the upgrade timeframe.Prepare a backout plan so that you can quickly revert to older versions if the upgrade fails.
Document Registered Device Counts
The server reboots during upgrades cause devices such as IP phones, gateways, and media resource devices to unregister and reregister with the servers. Therefore, before you begin the upgrade, make a note of the number of devices registered to each CallManager server by monitoring the following performance counters, the names of which are self-explanatory:FXOPortsInServiceFXSPortsInServiceRegisteredHardwarePhonesRegisteredMGCPGatewayRegisteredOtherStationDevicesTranscoderResourcesTotalPRISpansInServiceT1SpansInService
You can use either the Performance Monitor application that comes with Windows 2000/2003 or the Real-Time Monitoring Tool (RTMT; refer to the "Real-Time Monitoring Tool" section, later in this chapter) to collect this information. The registered device counts taken after the upgrade should closely match the counts taken before the upgrade.To run Performance Monitor on CallManager servers, select Start > Programs > Administrative Tools > Performance. After the application opens, as shown in Figure 9-1, you can add the counters in the preceding list and any other counters by selecting the appropriate performance object. After you add all the desired counters to monitor, you can save the view as a Microsoft Management Console (MMC) file by selecting Save from the Console menu. Running this saved MMC file before you perform an upgrade opens the Performance Monitor screen with all the configured counters.
Figure 9-1. Monitoring Performance Counters
[View full size image]

Windows Operating System Upgrade
CallManager and many other IPT applications run on the Microsoft Windows 2000 Server platform. To install any of these IPT applications (except Cisco Unity) on the Windows 2000 Server, you use special Windows installation CD-ROMs (Media Convergence Server [MCS] Operating System [OS] CD-ROM) with special installation screens that prompt for user input and configure the IPT applications during the installation.On the second Tuesday of every month, Microsoft releases security hot fixes to resolve defects in any of the windows operating system versions. This does not apply to critical fixes, which are released immediately after the vulnerability is reported and the fix is available. Do not install the hot fixes for IPT application servers by directly downloading them from the Microsoft website unless Cisco recommends doing so.For critical hot fixes released by Microsoft, Cisco policy is to test and then post them on Cisco.com within 24 hours. Cisco consolidates the noncritical fixes in the form of Operating System Upgrade Service Releases and posts them on Cisco.com on the third Tuesday of every month.Because Cisco Unity servers are loaded with regular Windows 2000/2003 OS CD-ROMs, you can load the hot fixes or apply the OS service packs that are available from Microsoft. Check the following URL for the recommended way to apply these service packs for Cisco Unity and Cisco Unity Bridge products:http://www.cisco.com/univercd/cc/td/doc/product/voice/c_unity/cmptblty/msupdateNoteCisco does not modify the actual security hot fix released by Microsoft, although Cisco might add its own wrapper to simplify the installation and provide a consistent user interface to the CallManager administrator for all OS upgrades.The following URL provides the CallManager security patch process:http://www.cisco.com/application/pdf/en/us/guest/products/ps556/c1167/ccmigration_09186a0080157c73.pdfThe best practice is to apply the OS hot fixes and OS upgrade service releases at a regular interval on all servers. You should apply the critical hot fixes immediately after the patch is available. This process aids in hardening the OS. In most cases, the OS patches are independent of the application running on the servers. Hence, the changes in the hot fix will not affect the application.
CallManager Software Upgrade
In the Cisco IPT solution, CallManager is the heart of the network, and the availability of the IPT system depends on the uninterrupted running of the CallManager servers. Thus, the major task in software upgrades is to upgrade the software on the CallManager servers. This section covers the following information that you need before you can perform the CallManager software upgrade:Upgrade planning and impactsFailover procedureRecovery methods
Upgrade Planning and Impacts
Before you decide to upgrade the CallManager software version, ask yourself the following questions:Does the current version have any critical defects?What defects are fixed in the new version? Do these defects apply to the existing network?Does the new version include any new features or support for additional hardware that would be useful to deploy in the existing network?Did Cisco announce the End of Life for the current version?
If no critical defects are affecting your existing deployment and product support is continued for your deployment, you do not need to rush for the upgrade. You also need to consider the amount of time and resources required to complete the upgrade before you make a decision.As mentioned earlier, if you do decide to upgrade, you should perform the upgrade in the lab network before you apply it in the production cluster. You should develop a test plan to test the features and full functionality that deployed in your production environment. Document any changes that end users will notice because of the upgrade, and communicate this information to them ahead of time. If necessary, develop a plan to educate or train the users on new features that you want them to know about.
Failover Procedure
All software upgrades require that you reboot the servers at the end of the process. This causes some interruptions to call processing. However, you can minimize these interruptions by following this failover procedure:
Step 1. | Back up the Publisher database before you attempt an upgrade. |
Step 2. | Perform the upgrades on the Publisher server and reboot. |
Step 3. | Perform the upgrades on the TFTP server and reboot. |
Step 4. | Upgrade the backup subscriber and reboot. Fail over all devices to the backup subscriber, leaving the primary subscriber idle, and perform upgrades. |
Step 5. | Set the startup type for the Cisco CallManager service (ccm.exe) to Manual on the primary subscribers. This ensures that the CallManager service does not start automatically after a reboot. This avoids phones cycling between primary and backup subscribers if the upgrade requires multiple reboots. |
Step 6. | Perform the upgrade on the primary subscriber and reboot. Start the CallManager service and set it to start automatically if you have set it to start manually in the previous step. Fail over the devices back to the primary subscriber after the upgrade is complete and successful. |
You should perform the upgrades sequentially, one at a time in the cluster. In a Cisco CallManager cluster, you must upgrade all the servers in the cluster to ensure that all of them run the same CallManager version, OS version, and SQL version. Ensure that you reboot the server after the upgrade.
Recovery Methods
You should plan for the restoration of the system to its original state in case you encounter any problems during the upgrades. The following are the three methods to speed up the recovery:Use backup data.Use mirrored hard drives.Uninstall newly installed software.
The following sections describe these recovery procedures.
Recovery Using Backup Data
Before you make changes to the system, make sure the Backup and Restore System (BARSthe software component that runs on CallManager servers to back up and restore the data) has successfully completed the backup of data. Store the backup data on a tape drive or a network drive. Successful backup helps you to restore the system to the same state that it was in before the upgrade. Remember that BARS does not back up the OS files or the data files of other, third-party applications.Recovery using the backup data requires more time because it involves many steps, as described here:You have to build the server freshly starting with the operating system install and bring it to the same level as the system from which the backup was taken. The IP address and server name of this new server should match with the old system.Install the CallManager application and install any additional service packs to bring it up to the same version level as the old system.Install the BARS application and start the data restoration process.After successfully restoring the data, install the tools and any other products that existed on the old system, such as Bulk Administration Tool (BAT), CDR Analysis and Reporting (CAR), antivirus software, and CSA.
Recovery Using Mirrored Hard Drives
A few Cisco Media Convergence Server (MCS) platforms, such as 7835 and 7845, are equipped with two or more hard drives configured with redundant array of independent disks (RAID) 1, also called drive mirroring. In the second method of recovery, you can pull the mirrored hard drive on each machine before you make the upgrade. This recovery method takes less time than using backup data, but it requires spare hard drives.This process requires careful planning. If you have a server with two hard drives (one in slot 0 and the other in slot 1), remove the drive in slot 1 before starting the upgrade. Label the drive with the machine name and the slot number from which it was taken out. Keep the removed drive in a safe place. If you need to use it for recovery, follow these steps:
Step 1. | Power off the server. |
Step 2. | Remove the hard drive from slot 0. |
Step 3. | Insert into slot 1 the hard drive that was pulled before the upgrade. |
Step 4. | Power on the server. The server will boot from the drive in slot 1. |
Step 5. | At the bootup screen, press F2 to select the Interim Recovery mode option. |
After the system successfully boots up, insert the drive in slot 0. The system starts mirroring the data from slot 1 to slot 0. You can monitor the progress of mirroring through the HP Array Configuration Utility on the CallManager servers. To access this utility, select Start > Programs > HP System Tools > HP Array Configuration Utility.
Recovery by Uninstalling the Software
After you perform the upgrade to the latest version, if you notice any issues, you can uninstall the recently installed software upgrades by using Add/Remove Programs, accessed from the Control Panel. You should also refer to the release notes or installation instructions for the software to get the exact steps that are required to uninstall the software.
Application Software Upgrades
CallManager integrates with many other Cisco IPT applications IPT applications and third-party third-party products. Before you make an upgrade decision about CallManager or any application software, you should check whether the new version of the CallManager software is compatible with the other software products that are running in the cluster. Refer to the Cisco CallManager Compatibility Matrix on http://www.cisco.com/en/US/partners/pr46/pr13/partners_program_solution09186a00800a3807l
Upgrade Planning and Impacts
Because OS installation steps are the same for the majority of the Cisco AVVID applications and CallManager, the best practices recommended performing OS upgrades for CallManager that also apply to the application servers.However, the process of upgrading software versions for each application varies from one product to another. Refer to the specific product documentation and release notes before the upgrade.The list that follows includes a few commonly omitted tasks after a CallManager upgrade affects the functioning of the application servers:Whenever you upgrade the CallManager version, download the new version of the JTAPI plug-in from the Cisco CallManager Plug-Ins page on the application servers that use a JTAPI connection to communicate with CallManager. For CRA servers, you can use the JTAPI Update tool to upgrade the JTAPI version. To access this tool, on the CRA server, select Start > Programs > Cisco CRA Administrator > JTAPI Update Tool. Examples of such applications are CRA servers (Cisco IP-IVR, Cisco IPCC Express), Cisco Conference Connection (CCC), Cisco Emergency Responder (CER), and CallManager Peripheral Gateway (PG) devices in a Call Center network deployed using Intelligent Contact Management (ICM).Download the new version of TAPI Service Provider (TSP) from the CallManager Plug-Ins page to the application servers that use a TAPI connection to communicate with CallManager. Examples of such applications are third-party voice mail, Cisco IP SoftPhone, or other applications that use TSP drivers.If you have deployed Cisco Unity in the network, you might have to upgrade the Cisco Unity-CM TSP drivers on the Unity servers. Check the Cisco CallManager Compatibility Matrix, download the recommended version onto Unity servers, and perform the upgrade. TSP upgrade on the Unity servers requires reboot of Unity servers. Check the latest Unity TSP Compatibility Matrix with CallManager versions on Cisco.com athttp://www.cisco.com/univercd/cc/td/doc/product/voice/c_unity/cmptblty/tspmtrx.pdfIf you have integrated CallManager with a corporate Lightweight Directory Access Protocol (LDAP) directory such as Microsoft Active Directory or Netscape Directory Server, check the CallManager documentation and perform the schema updates on the directory servers if required. Changes made to the directory schema cannot be reverted. Hence, the best practice is to take a backup of the corporate directory database before you upgrade the schema. CallManager schema updates are backward compatible, so after you update the schema, even if you downgrade the CallManager version, it still works with the new schema.Sometimes new CallManager versions also include updates to the applications, such as BAT, CAR, and RTMT. Make sure to check the release notes and update these applications accordingly.
IP Phone and Gateway Upgrades
IP phones and gateways are the other components in the Cisco IPT solution that might require change in the firmware loads to fix some defects or implement new feature functionality. Typically, a CallManager upgrade also upgrades the firmware on the IP Phones and on the Catalyst MGCPbased voice gateways. Thus, you do not have to follow separate installation procedures.The next section discusses the following three topics:
Step 1. | Understand the procedure to upgrade the firmware on a few Cisco IP phones or on a few gateways. |
Step 2. | Discuss the planning steps to perform the phone/gateway loads and impact of the upgrade process. |
Step 3. | Identify the steps to revert back to the old firmware in case the upgrade operation fails. |
Upgrading the Firmware on a Few Cisco IP Phones or Gateways
You might need to upgrade the firmware on some Cisco IP Phones or Gateways to test the functionality and features in the new firmware versions before rolling out the changes to all the Cisco IP Phones and Gateways in the production network. The best practice when upgrading the firmware is to load it on a few devices and run it for a few days. If the results are good, you should proceed with upgrading the firmware on all devices.You can specify the device firmware loads in two places in CallManager. One is on the Device Defaults page. To access this page from the Cisco CallManager Administration page, select System > Device Defaults. Before performing the upgrade, make a note of the device load versions specified in this page. The second place is at the individual IP phone or the gateway level. The device firmware specified at the individual device level takes precedence over the specified firmware on the device defaults page.To perform a selective upgrade of firmware on devices, download the latest firmware and copy the binary files onto the TFTP server. The default path is C:\Program Files\Cisco\TFTPPath. If you are downloading an executable wrapper that contains the loads, the wrapper automatically copies the files to the TFTP server and updates the Device Defaults page with the latest load information. To perform the selective upgrade, you need to update the Device Defaults page configuration that points back to old load IDs.To upgrade the load on a Cisco IP phone, from the Cisco CallManager Administration page, search for the IP phone for which you want to perform the upgrade and, in the phone device configuration, specify the IP phone Load ID, as shown in Figure 9-2, and reset the phone. After the Cisco IP Phone reboots, verify that the Cisco IP Phone has the latest phone load. On 7960/7940 Cisco IP Phone models, to verify the load, press the Settings button on the Cisco IP Phone and then choose Status > Firmware Versions menu options to verify that the App Load ID displayed matches the configured Load ID on the Cisco IP Phone device page, as shown in Figure 9-2. After the Cisco IP Phone registers with CallManager, make a note of the IP address from the CallManager Administration page and browse to the phone to check the software version when you are not local to the IP phone.
Figure 9-2. Selective Upgrade of Firmware on Cisco IP Phones and Gateways
[View full size image]

Example 9-1. Verifying the Gateway Load on a Catalyst WS-6608-T1 Gateway Port
cat6k-voice> (enable) show version
WS-C6509 Software, Version NmpSW: 8.1(3)
Copyright © 1995-2003 by Cisco Systems
NMP S/W compiled on Oct 10 2003, 12:38:01
System Bootstrap Version: 5.3(1)
System Boot Image File is 'bootflash:cat6000-supk8.8-1-3.bin'
System Configuration register is 0x2
Hardware Version: 2.0 Model: WS-C6509 Serial #: SCA0443045D
PS1 Module: WS-CAC-1300W Serial #: SON04472655
Mod Port Model Serial # Versions
--- ---- ------------------- ----------- ------------------------------------
1 2 WS-X6K-SUP1A-2GE SAD04480PZW Hw : 7.0
Fw : 5.3(1)
Fw1: 5.4(2)
Sw : 8.1(3)
Sw1: 8.1(3)
WS-X6K-SUP1A-2GE SAD04480PZW Hw : 7.0
Sw :
3 48 WS-X6348-RJ-45 SAL05031N6H Hw : 1.4
Fw : 5.4(2)
Sw : 8.1(3)
4 8 WS-X6608-T1 SAD04320KZA Hw : 1.1
Fw : 5.4(2)
Sw : 8.1(3)
HP1: D00404000008; DSP1: D0054133 (4.1.33)
HP2: D00404000007; DSP2: D0054133 (4.1.33)
HP3: D00404000007; DSP3: D0054133 (4.1.33)
HP4: C00103010014; DSP4: C002E031 (3.3.2)
HP5: C00103010014; DSP5: C002E031 (3.6.15)
HP6: C00103010014; DSP6: C002E031 (3.6.15)
HP7: C00103010014; DSP7: C002E031 (3.6.33)
HP8: D00404000007; DSP8: D0054133 (3.6.33)
6 5 WS-SVC-CMM SAD075303F1 Hw : 2.2
Fw : 12.2(13)ZP2,
Sw : 12.2(13)ZP2,
7 5 WS-SVC-CMM SAD0640008A Hw : 2.1
Fw : 12.2(13)ZP,
Sw : 12.2(13)ZP,
15 1 WS-F6K-MSFC SAD04480KXC Hw : 1.4
Fw : 12.1(1)E,
Sw : 12.1(1)E,
DRAM FLASH NVRAM
Module Total Used Free Total Used Free Total Used Free
------ ------- ------- ------- ------- ------- ------- ----- ----- -----
1 65408K 48723K 16685K 16384K 11459K 4925K 512K 280K 232K
Uptime is 88 days, 0 hour, 22 minutes
cat6k-voice> (enable)
Upgrade Planning and Impacts
After you have successfully tested the new loads on a few devices and are ready to upgrade the firmware for the entire cluster, follow these steps:
Step 1. | Update the Device Defaults page to reflect the new firmware load ID. |
Step 2. | Remove the device-specific load information from the phones or the gateways, as configured in Figure 9-2. If you do not perform this step, when you later upgrade the whole cluster with new device loads, these devices will still have the old load. |
Step 3. | Reset the phones or gateways based on device pools to minimize the number of TFTP requests sent to the TFTP server. In a CallManager cluster with many IP phones, performing a firmware upgrade on all the devices at once affects the performance of the TFTP server. This results in connection timeouts for TFTP requests. |
Failover Procedures and Recovery Methods
You can always revert to the old device loads if you have to do so. The old firmware images still exist in the TFTP server. Hence, all you need to do is update the Device Defaults page to point back to the old firmware load ID and reset the phones and gateways based on the device pool.NoteStarting with CallManager 3.3.3 Software Release 1, Cisco has added image authentication to its various IP Phone protocols. This prevents tampering with the binary image prior to its being loaded into the phone. Any tampering with the image causes the phone to fail the authentication process and reject that image. The image authentication occurs through signed binary files. After the phones are loaded with the signed binary loads, you cannot revert to unsigned loads.If you want to load the unsigned loads on a phone that was previously booted with signed binary loads, you are out of luck. Your only option is to send the phones back to Cisco and receive replacements.
BIOS Upgrades
The BIOS contains all the code required to control the keyboard, display screen, disk drives, serial communications, and several miscellaneous functions. Server vendors release upgrades to the BIOS to add additional functionality into the BIOS. Cisco posts on Cisco.com the upgrades to the BIOS for various server platforms. Upgrading the BIOS requires rebooting the servers. Hence, you should follow the same guidelines described earlier in the "Operating System Upgrades" section to reduce the impact on the call processing.