Business Continuity and HIPAA Business Continuity Management in the Health Care Environment [Electronic resources] نسخه متنی

Team Functions:

Coordinates recovery operations for Computers, Telecommunications, and Data Communications.

Alternate Site:

Recovery Center

FIRST NAME

(First Names)

LAST NAME

(Last Names)

Home Phone #

(555-555-5555)

Mobile Phone Beeper

(444-444-4444)

Communications Task Force

FIRST NAME

(First Names)

LAST NAME

(Last Names)

Home Phone

(555-555-5555)

Mobile Phone Beeper

(444-444-4444)

Specialty

(Specialty)

IS Support Task Force

FIRST NAME

(First Names)

LAST NAME

(Last Names)

Home #

(555-555-5555)

Mobile # Beeper

(444-444-4444)

Specialty

(Specialty)

Hotsite Task Force

FIRST NAME

(First Names)

LAST NAME

(Last Names)

Home #

(555-555-5555)

Mobile # Beeper

(444-444-4444)

Specialty

(Specialty)

Procedure

Recover Telecommunications

Communications Task Force. Task Force members may call on the services of other IT personnel who are not members of the recovery team.

1.

Damage Assessment

Make an assessment of the damage and replacement needs using the "Resource Item Matrix" report and the appropriate appendix.

Compile a report of missing or damaged items that need to be replaced or repaired.

Relay findings to the Emergency Management Team.

Salvage and safeguard as many resources items as possible.

2.

Location determination

If the facility will never be reoccupied, go to "plan telephone locations". Otherwise, proceed to next task.

3.

Direct line telephone

Determine if telephone outage is due to power failure.

If outage is due to power failure, use direct line telephones for incoming and outgoing calls. Direct line access may found in the Switch Room.

4.

Damage Mitigation

If equipment is or has been exposed to moisture, remove it from electrical power sources (main power, electrical generator, UPS).

If the environment is contaminated, remove equipment and electromagnetic media to a controlled environment free of smoke, soot, water and high humidity

Prior to removal, label equipment to identify its home location and what other equipment to which it was connected.

Wipe off all moisture from outside of electromagnetic media (diskettes, tapes, CDs). Obtain written consent of manufacturer and vendors for the disaster procedures being taken on equipment under warranty or maintenance contract.

Do not allow vendors/staff to test, evaluate or diagnose damage prior to restoration. Let the vendor de-install, reinstall, then re-certify equipment.

Move damaged equipment into storage until an adjuster can look at it.

Prior to reinstallation, insure that surge protectors have remained functional.

Vendor Name

Telephone #

Contact Name

5

Recover telephones/voice lines

Contact vendor for telephones and voice lines.

Request quantity of telephones and direct lines needed. Request installation of new PBX systems if required.

Determine time of delivery and installation.

Monitor installation of PBX equipment.

Refer to "Telecommunications" Appendix.

Vendor Name

Telephone #

Contact Name

6.

Plan telephone locations

If telephones and/or the entire telephone system need to be installed or reinstalled:

Determine the number of telephones and telephone lines needed. Identify where telephones are to be located.

Determine if there are existing phone jacks near telephone installation sites. Create blueprint/floor plan of facility

On blueprint or floor plan, identify where new telephone jacks and telephones are to be installed. Give a copy of blueprint/floor plan to the person who will be dealing with the vendor.

Monitor the delivery and installation of the telephone system.

7.

Forward phone service

If required to move to a temporary location:

Notify vendor and request that phone service (voice and Fax) be forwarded to temporary location.

Vendor Name

Telephone #

Contact Name

8.

Report status and actions taken to the Technology Recovery Team Leader who, in turn, will notify the Emergency Management Team.

Procedure

Recover Mid-Range Computer

IS Support Task Force. Task Force members may call on the services of other IT personnel who are not members of the recovery team.

1.

Damage Assessment

Make an assessment of the damage and replacement needs using the "Resource Item Matrix" report and appropriate appendices.

Compile a report of missing or damaged items that need to be replaced or repaired.

Salvage and safeguard as many resources items as possible

See "Equipment Inventories" Appendix.

Record findings on the "Damage Assessment Form" at the end of the Technical Recovery team's procedures.

2.

Determine cause of outage

Determine damage to equipment.

Determine if CPU is operational.

Determine if electricity is getting to CPU.

Determine if workstations are operational.

Check circuit breakers.

3.

Damage Mitigation

If equipment is or has been exposed to moisture, remove it from electrical power sources (main power, electrical generator, UPS).

If the environment is contaminated, remove equipment and electromagnetic media to a controlled environment free of smoke, soot, water and high humidity.

Prior to removal, label equipment to identify its home location and what other equipment to which it was connected.

Wipe off all moisture from outside of and electromagnetic media (diskettes, tapes, CDs).

Obtain written consent of manufacturer and vendors for the disaster procedures being taken on equipment under warranty or maintenance contract.

Do not allow vendors/staff to test, evaluate or diagnose damage prior to restoration.

Let the vendor de-install, reinstall, then recertify equipment.

Move damaged equipment into storage until an adjuster can look at it.

Prior to reinstallation, insure that surge protectors have remained functional.

Vendor Name

Telephone #

Contact Name

4.

Order replacement equipment

If Mid-Range computer equipment is damaged and requires replacement:

Contact vendor to configure and order new/used Mid-Range computer equipment to replace damaged equipment (refer to the appropriate Appendix). Indicate to the vendor that this is an emergency situation and request the vendor's quickest possible response time).

Get a commitment on the delivery schedule for all new equipment.

Coordinate the installation of the new/used equipment with the Facilities Maintenance Coordinator and with the Manufacture's technical representative who will be installing the system.

Follow up on equipment ordered and status of the order.

Vendor Name

Telephone #

Contact Name

5.

Move to alternate location

If damage to the facility is such that all functions must relocate, move functional Mid-Range computer to an alternate site. Verify the following cabling requirements at alternate site:

10 base T (Cat 5) or shielded twisted pair cabling.

RJ45 data connections.

Star Coupler cabling and all necessary cabling for clusters.

Level 3 cabling to support printers (via RJ11 connectors).

(If the above requirements are not present in the new facility, call cabling vendor and arrange to have above requirements met).

Notify moving vendor of the need to move the Mid-Range computer

Vendor Name

Telephone #

Contact Name

6.

Obtain installation assistance

If the Mid-Range computer requires reinstallation at the current site or at a new site:

Contact the equipment's manufacturer or other vendor certified for installation.

Request that the equipment be installed at the specified site as soon as the equipment is available for the procedure.

Notify vendor as soon as equipment is available.

Vendor Name

Telephone #

Contact Name

7.

Install Mid-Range computer system

If new Mid-Range computer is being installed, or the old system is being reinstalled:

Retrieve system, program, and data back up tapes from off-site storage facility.

Make back-up of data tapes.

Load system, program, and data tapes with the assistance of the installation vendor.

Verify the integrity of the system.

Connect the communications network.

Verify the integrity of the system at all satellite locations.

Provide accounting with a list of all new equipment, locations, and serial numbers.

Vendor Name

Telephone #

Contact Name

8.

Return to permanent facility

After permanent facility is available to be occupied:

Notify the communications and system installation vendors that they will be required for a move back to the permanent facility.

Schedule preparatory work around a weekend move back to the permanent facility.

Before equipment is moved back to the permanent facility, insure that the system, programs, and data are backed up and safely stored off-site.

Cease operations at the temporary facility.

Return to permanent facility.

Reinstall the system and attach communications network.

Verify the integrity of the system both at the new facility and at the satellite facilities serviced by the Mid-Range computer.

Vendor Name

Telephone #

Contact Name

9.

Report status and actions taken to the Technology Recovery Team Leader who, in turn, will notify the Emergency Management Team.

Procedure

Recover Alternative Processing Sites

Hotsite Task Force. Task Force members may call on the services of other IT personnel who are not members of the recovery team.

1.

Damage Assessment

Review assessment and calculate the extent of the damage.

Determine the feasibility of refurbishing damaged equipment.

Prepare one report listing salvageable equipment and another report detailing the equipment to be scrapped.

Notify the lessors of equipment that has been damaged or destroyed.

Review damage with vendors and get lead times on equipment repair/replacement.

Estimate expense of the repairs/replacements.

Notify insurance representative of these estimates.

Relay findings to the Emergency Management Team.

Salvage and safeguard as many resources items as possible.

Receive authorization from EMT to proceed with Hot-site activation.

See "Computer Equipment Appendix".

2.

Recover Magnetic Tapes

Collect magnetic tapes from the tape vault.

Collect magnetic tapes from the computer room.

Transport tapes to a secure location for cataloging and salvage inspection.

3.

Contact Hotsite Vendor

Refer to "Hot-site Disaster Declaration" Appendix.

Call the hot-site vendor (must be done by Authorized staff member).

Tell the hot-site vendor representative that "This is a Disaster Notification".

Provide the following information:

Company Name

Your Name

Telephone number where you can be reached.

Nature of the disaster.

Estimated time of arrival at the hot site facility.

Verify Hot-site vendor's assignment of Widgetcorps to their Smithville remote operations center. The Smithville location is the primary location for the Local Operations Center.

Write letter to the President of the Hot-site Company indicating that a disaster declaration has been made.

Make vehicle reservations.

Make Airline reservations for personnel.

Make hotel reservations.

Convey on-site contractor personnel requirements

4.

Move to Hot-site

Notify recovery team that a disaster has been declared and that they will be going to the hot site.

Make airplane, rental car, and motel reservations for recovery team.

Notify management of hot-site activation.

Notify public relations of the need to inform customers and vendors of new location.

Recover backup tapes and procedure manuals from storage location and transport to hot-site.

Commence hot-site procedures.

Transport as many resource items that can be salvaged to the temporary site.

Travel to hot-site location.

Check in to hotel.

Ensure all backup tapes are on site.

Ensure that an adequate supply of scratch tapes is available.

Establish telephone communications link with Local Operations Center (Smithville, primary location).

5.

Recover Database

Determine the availability of image copy tapes for databases.

Open DBA Documentation boxes & verify all necessary manuals are available at the Local Operating Center, (Refer to Offsite Storage Appendix).

Obtain TMS reports for Local Operating Center and Hotsite.

Restore DB2 subsystem following procedure found in Appendix "SYS4.DASD.DR.CNTL(P00DOC)"

Recover production databases to the sync point (refer to Appendix "IMS/DB2 DB Recovery").

Start IMS/DB2 on-line regions.

Turn over databases and files for access by production jobs and on-line validation (Customer Service).

Initiate external testing procedure.

When the system is operational, request Data Security Documentation and access to and security procedures at the hot-site.

Inform Plan Manager of successful activation of the hot-site.

Implement processing/shift schedules.

Review instructions regarding gaining access to and security procedures at the hot-site.

Initiate catch up processing operations.

Resume tape backup and rotation policies.

Define procedures for check processing.

Establish procedures for distribution of Customer reports.

Contact vendor for microfiche production.

Make arrangements for the distribution of microfiche.

6.

Reconnect Telecommunications

Obtain appropriate test equipment and tools at hot-site facility and at the campus facility.

Contact Network Services vendors and give notification of the disaster.

Give command to switch network and verify circuits to customer locations and LFC Campus.

With Plan Manager's approval, reconstruct Telecommunications servers.

Obtain equipment configuration list from hot-site vendor.

Coordinate with system software to validate port address and line name for NCP Generation.

Make patch connections from 3745 and CISCO router to DCE equipment.

Make LAN/WAN router configurations changes at hotsite location for Trinity.

With the approval of the Plan Manager, acquire and install CSU/DSU equipment along with 3X74 Controllers to support 3270 terminals equipment on the LC Campus.

Activate remote resources when network functions are recovered.

Verify proper operations of remote resources with customers, resolve any discrepancies.

7.

Recover System

Review system configuration with Hot-site vendor personnel:

Verify IOCP matches the Hot-site configuration.

Determine date/time of catalog backups and TMC backups.

Select synchronization point to which data is to be recovered.

Communicate recovery date and time to Plan Manager.

Confirm availability of copies of DASD backup tapes and incremental backup tapes for volumes which support application incremental backup.

Determine the availability of prior backup tapes to be used if a problem develops with the most recent tapes.

Log on to the Vendor's floor recovery system.

Restore operating system following procedure found in Appendix "SYS4.DASD.DR.CNTL(PROCEDUR)"

IPL SYSQ.

Complete the restoration of critical volumes in accordance with Appendix "SYS4.DASD.DR.CNTL(PROCEDUR)"

Begin network recovery and verification.

NCP GEN

Execute WRAP test (Network Services)

8.

T-1 Data Communications Recovery for Hot-site

Get overview of facility.

Create a command post.

Locate Network documentation box.

Configure CSUs.

Label CSUs with configuration.

Verify CDF file within the NCP.

Locate WRAP plug 3745.

Do a WRAP test.

Install cable from front end port to CSU.

Install a jumper from the CSU D4 channel bank card.

Perform link level 2 test up to a loop at the D4 card.

Verify results of test.

Coordinate the ABM T-1 switch.

Verify ABM switch over.

If effective, activate all customer lines.

If NOT effective, continue with T-1 loop testing to the home-site CO.

Test access to customer sites.

Re-verify activation.

For verification failure, Dispatch customers (if not there) to sites to assist in loop testing (Refer to Customer List Appendix).

Reset equipment.

Attempt re-activation of PU.

Confirm to EMT the activation of the Comm. capabilities from the Hot-site.

9.

Recover Data Communications

Get overview of facility.

Determine equipment recoverability.

Contact vendor for equipment refurbishing.

Secure a blueprint of the facility.

Determine communications layouts.

Order equipment.

Configure all equipment.

Label CSUs with configuration.

Verify CDF file within the NCP.

Locate WRAP plug 3745.

Do a WRAP test.

Install cable from front end port to CSU.

Install a jumper from the CSU D4 channel bank card.

Perform link level 2 test up to a loop at the D4 card.

Verify results of test.

Coordinate the ABM T-1 switch.

Verify ABM switchover.

If effective, activate all customer lines.

If NOT effective, continue with T-1 loop testing to the Dallas CO.

Test access to customer sites.

Re-verify activation.

For verification failure, Dispatch customers (if not there) to sites to assist in loop testing.

Reset equipment.

Attempt reactivation of PU.

Ensure that all communications circuits and modems and processors are working properly.

Confirm to Plan Manager the activation of the Comm. capabilities from the Hot Site.

Evaluate voice communications requirements.

Order long term line service.

Request shipment of replacement hardware from Vendors.

(servers, concentrators, routes, csu/dsu, phone switch, hand sets, cabling,)

Coordinate cabling of bldg.

Install LAN communications network.

Install internal voice communications network.

10.

Establish Temporary Site Communications

Evaluate data and voice communications requirements.

Order temporary and long term line service. Request shipment of replacement hardware from Vendors.

(servers, concentrators, routes, csu/dsu, phone switch, hand sets, cabling,)

Coordinate cabling of bldg.

Install LAN communications network.

Install internal voice communications network.

11.

Catch up to current calendar date

Contact Vendors and coordinate the receipt of held files by processing date.

Validate receipt of all required inputs for the next processing date for all customers in the partition.

Confirm that customers' processing is complete for the next processing date for all customers in the partition.

Analyze schedule:

Determine if parts of the schedule can be skipped.

IPL for system date.

Plan for an image copy schedule.

Authorize the release of the schedule for the end of the day cycle for that partition.

Notify customers of completion of End-Of-Day.

Advise customers that they can commence posting the next day's cash.

Determine the disposition of each outbound transmission or tape.

Notify customers end of day processing for (date) is complete.

Request that customers validate the output as found on critical reports.

Based on the determination above, notify I/O Control to either send or cancel output.

Coordinate with customers and take image copies.

Repeat this task until caught up.

Verify that the system date, on-line date, calendar date, and batch date are the same.

12.

Customer Interface

Move to Local Operations Center.

Insure (with telecom team) that 800 and local numbers have been transferred to the remote operations center.

Talk to DR Manager and determine estimated time of system recovery and interconnectivity back to customer.

Reference customer list.

Contact customers.

Follow checklist including all interfaces files with file numbers and times.

Emphasize with customer to keep last 2 days of source documentation prior to the disaster and continue to save source documents until notified further.

Give customers periodic updates as to the recovery status.

Inform customer when system is available.

13.

Customer Service Functional Validation

Rerun batch processing for the cycle associated with the system date of the restored application data.

Report results by customer.

Resolve differences.

Recommend (if appropriate) that customers inform their customers that on (Date of Disaster) their transactions/data changes may have been lost and that our customer's customer should contact them immediately.

Coordinate the catch-up of the data entry back log for customers sharing an BUS for each end-of-day cycle required until system date and calendar date are equal.

14.

Report status and actions taken to the Technology Recovery Team Leader who, in turn, will notify the Emergency Management Team.

Procedure

Recover Local Area Network and Servers

IS Support Task Force. Task Force members may call on the services of other IT personnel who are not members of the recovery team.

1.

Assess damage to LAN and Servers

If a disaster has occurred and/or the LAN system and Servers have been affected:

Assess the damage to all network and Server components including cables, boards, fileserver, workstations and printers and network equipment.

Make a list of all items to be repaired or replaced.

Select appropriate vendors from "Vendor Detail" report.

Relay findings to the Emergency Management Team.

See "Equipment Inventories" Appendix.

Record findings on the "Damage Assessment Form" at the end of the Technical Recovery team's procedures.

Vendor Name

Telephone #

Contact Name

2.

Damage Mitigation

If equipment is or has been exposed to moisture, remove it from electrical power sources (main power, electrical generator, UPS).

If the environment is contaminated, remove equipment and electromagnetic media to a controlled environment free of smoke, soot, water and high humidity. Prior to removal, label equipment to identify its home location and what other equipment to which it was connected..

Wipe off all moisture from outside of and electromagnetic media (diskettes, tapes, CDs).

Obtain written consent of manufacturer and vendors for the disaster procedures being taken on equipment under warranty or maintenance contract.

Do not allow vendors/staff to test, evaluate or diagnose damage prior to restoration.

Let the vendor de-install, reinstall, then recertify equipment.

Move damaged equipment into storage until an adjuster can look at it.

Prior to reinstallation, insure that surge protectors have remained functional.

Vendor Name

Telephone #

Contact Name

3.

Recover equipment

If LAN and Server equipment is damaged and requires replacement:

Contact vendor to order new/used equipment to replace damaged equipment. Indicate to the vendor that this is an emergency situation and request the vendor's quickest possible response time.

Insure that the tape backup system that will be used to restore the LAN server is compatible with the format of the backup tapes.

Get a commitment on the delivery schedule for all new or used equipment.

Coordinate the installation of the new/used equipment with the Asset Manager.

Follow up on equipment ordered and status of the order.

Vendor Name

Telephone #

Contact Name

4.

Install LAN comm. Network and Servers

If a new local area network or servers needs to be installed, or if the existing network or servers need to be activated:

Contact the appropriate vendors and request assistance.

Retrieve/develop the schematic of the current local area network to assist the vendor in the creation of the new network or in the re-establishment of the old network.

Tell the vendor the time when the equipment is expected to be operational.

The Communications Task Force Leader/ NT Network Director will tell the vendor the anticipated location of the equipment.

Determine when the vendor can commence the installation of the local area network.

Monitor the progress of the vendor.

Communications Task Force

Vendor Name

Telephone #

Contact Name

5.

Replace Software

Determine requirements for replacement software.

Retrieve the original or backup copy of the software from the storage facility.

Obtain replacement software from vendor.

Load software.

Verify the operation of the replacement software.

Vendor Name

Telephone #

Contact Name

6.

Recover backup tapes

If the servers are damaged and need to be replaced, or if the system is unable to access the hard drive:

Obtain backup tapes from off-site storage location.

If possible, make a backup tape of the backup tape and store the first copy in a safe location.

Deliver the copy of the backup to the facility housing the LAN and servers.

Vendor Name

Telephone #

Contact Name

7.

Install LAN and server system

If new LAN and server is being installed, or the old system is being reinstalled:

Using backup tapes, load system, program, and data tapes to the servers.

Verify the integrity of the system.

Connect the communications network.

Verify the integrity of the system at all network workstations.

Provide accounting with a list of all new equipment, locations, and serial numbers.

8.

Report status and actions taken to the Technology Recovery Team Leader who, in turn, will notify the Emergency Management Team.

Procedure

Recover Wide Area Network

Communications Task Force. Task Force members may call on the services of other IT personnel who are not members of the recovery team.

Vendor Name

Telephone #

Contact Name

1.

Damage Assessment

Make an assessment of the damage and replacement needs using the appropriate Appendix. Compile a report of missing or damaged items that need to be replaced or repaired.

Salvage and safeguard as many resources items as possible.

Relay findings to the Emergency Management Team.

See Data communications Appendix

Record findings on the "damage Assessment Form" at the end of the Technical Recovery team's procedures.

2.

Troubleshoot data communication

Verify that the modems, routers, hubs, matrix switch, ISP, and T1s are operational or identify the problem.

3.

Repair communications network

If a new communications network needs to be installed, or if the existing network needs to be activated:

Contact the appropriate vendors and request assistance.

Retrieve/create the schematic of the current communications network (see Data communications Appendix) to assist the vendor in the creation of the new network or in the reestablishment of the old network.

Determine when the vendor can commence the installation of the communications network.

Monitor the progress of the vendor.

4.

Verify comm. is back on line

Verify that the problem has been corrected.

Verify all lines.

Return to normal processing.

5.

Report status and actions taken to the Technology Recovery Team Leader who, in turn, will notify the Emergency Management Team.

Procedure

Recover Personal Computers

IS Support Task Force. Task Force members may call on the services of other IT personnel who are not members of the recovery team.

1.

Damage Assessment

Make an assessment of the damage and replacement needs using the appropriate appendix.

Compile a report of missing or damaged items that need to be replaced or repaired.

Salvage and safeguard as many resources items as possible.

Relay findings to the Emergency Management Team.

See "Equipment Inventories" Appendix.

Record findings on the "damage Assessment Form" at the end of the Technical Recovery team's procedures.

2.

Damage Mitigation

If equipment is or has been exposed to moisture, remove it from electrical power sources (main power, electrical generator, UPS).

If the environment is contaminated, remove equipment and electromagnetic media to a controlled environment free of smoke, soot, water and high humidity.

Prior to removal, label equipment to identify its home location and what other equipment to which it was connected..

Wipe off all moisture from outside of and electromagnetic media (diskettes, tapes, CDs).

Obtain written consent of manufacturer and vendors for the disaster procedures being taken on equipment under warranty or maintenance contract.

Do not allow vendors/staff to test, evaluate or diagnose damage prior to restoration.

Let the vendor de-install, reinstall, then recertify equipment.

Move damaged equipment into storage until an adjuster can look at it.

Prior to reinstallation, insure that surge protectors have remained functional.

Vendor Name

Telephone #

Contact Name

3.

Order new Work Stations and Peripherals

Assess existing equipment for repair, replacement or permanent reassignment.

If equipment is damaged, make arrangements with Help Desk or vendor to pickup and repair.

If Help Desk is making repair, insure Help Desk acquires sufficient spare parts inventory.

Review Microcomputer Resource Requirements and order required equipment.

Get an estimated delivery date from the equipment vendor.

Notify users of estimated arrival date and time.

Monitor delivery of equipment.

Vendor Name

Telephone #

Contact Name

See "Equipment Inventory" Appendix.

4.

Replace Software

Determine requirements for replacement software.

Contact other users within the healthcare organization to determine if an unaffected copy of the software exists and can be copied.

If no copies are available, retrieve the original or backup copy of the software from the storage facility.

If no backup copies are available, obtain replacement software from vendor.

Load software.

Verify the operation of the replacement software.

5.

Retrieve & load backup tapes

Obtain PC backup tapes from offsite storage facilities.

Restore backup tapes to the PC assigned.

Verify the data loaded successfully.

Return backup tapes to the offsite storage facility.

Vendor Name

Telephone #

Contact Name

6.

Install new PCs and printers

Backup all data on the PC.

Install all boards and test the new PCs.

Verify all software and data loaded correctly.

Provide accounting with a list of all new equipment, locations, and serial numbers.

7.

Report status and actions taken to the Technology Recovery Team Leader who, in turn, will notify the Emergency Management

Team. Technology Recovery Team

Procedure

Incident Response

IS Support Task Force. Task Force members may call on the services of other qualified personnel who are not members of the recovery Task Force.

1.

Incident Detection

Detect the presents of an intrusion through one of the following indications:

A system alarm or similar indication set off by an intrusion detection tool.

Suspicious entries in system or network accounting (e.g., a UNIX user obtaining root access without going through the normal sequence of events necessary to obtain this access).

Accounting discrepancies such as a gap where there is a complete lack of activity.

Unsuccessful login attempts.

Attempts (either failed or successful) to gain unauthorized access to a system or its data.

Unexplained new user accounts.

Unexplained new files or unfamiliar file name.

Unexplained modifications to file lengths and/or dates, especially in system executable files.

Unexplained attempts to write to systems files or changes in system files.

Unexplained modification or deletion of data.

The unauthorized use of a system for the processing or storage of data.

Denial of service or inability of one or more users to login to their account.

System crashes.

Poor system performance.

Unauthorized operation of a program or sniffer device to monitor network traffic.

"Door knob rattling" (e.g., use of attack scanners, remote requests for information about systems and/or users, or social engineering attempts).

Changes to system hardware, firmware, or software characteristics without the owner's knowledge, instruction, or consent.

Unusual time of usage - more security incidents occur in other than prime time.

An indicated last time of usage for a user does not agree with the actual time of prior usage for that user.

Unusual usage patterns - programs being compiled under a user's account that does not have any programming responsibility.

2.

Initiate Log

Document every step taken in handling an incident and have the notes in the Reactive Response Team Event Log signed by the author, verified by signature of 3rd party and stored securely - if the notes were within the daily incident log then make copies and secure these.

Protect any logs by physically locking them away and optionally encrypting them.

Make copies of affected system disks and protect them from alteration by physically securing them.

3.

Contain the Incident

Set timed goals like "if we don't discover how the break-in occurred in one hour then we shut down the network"

Go to "Disconnect from the Network" if there is any concern that the attacker may still be active.

4.

Disconnect Compromised Systems From the Network

Disconnect all compromised machines from your network including dial in connections.

Operate in single user mode in UNIX or as the local administrator in NT to ensure that you have complete control of the machine.

5.

Copy an image of the compromised system(s)

Create a backup of your system. This will provide a "snapshot" of the file system at the time that the root compromise was first discovered.

Label, sign, and date the backup and keep the backup in a secure location to maintain integrity of the data.

6.

Analyze Intrusion

Verify all system binaries and configuration files.

Boot from a trusted kernel and obtain a known clean copy of any tool you intend to use in analyzing the intrusion.

Check all system binaries thoroughly against distribution media.

Check any binaries referenced in /etc/inetd.conf, critical network and system programs, and shared object libraries.

Use cmp on UNIX systems to make a direct comparison of the binaries and the original distribution media.

Check the MD5 results for either UNIX or NT on suspect binaries against a list of MD5 checksums from known good binaries.

Verify your configuration files against copies that you know to be unchanged. For configuration files on UNIX systems:

Check your /etc/passwd file for entries that do not belong.

Check to see if /etc/inetd.conf has been modified.

Ensure that there is nothing that does not belong in /etc/hosts.equiv or in any .rhosts files if you allow the "r-commands" (rlogin, rsh, rexec.

Check for new SUID and SGID files. (The following command will print out all SUID and SGID files within your filesystem: "# find / \( −perm -004000 -o -perm -002000 \) -type f -print"

With your system disconnected from the network, you can now thoroughly review log files and configuration files for signs of intrusion, intruder modifications, and configuration weaknesses.

6.

For NT systems:

Check for odd users or group memberships.

Check for changes to registry entries that start programs at logon or services.

Check for unauthorized hidden shares with the ‘net share' command or Server Manager tool.

Check for processes that you do not identify using the pulist.exe tool from the NT resource kit or the NT Task Manager. Look for modifications to data.

Verify the integrity of web pages, ftp archives, files in users' home directories, and any other data files on your system.

Look for tools and data left behind by the intruder

Look for unexpected ASCII files in the /dev directory on UNIX systems.

Look very carefully for hidden files or directories.

Look for files or directories with strange names such as "…" (three dots) or ".." (two dots and some whitespace) [UNIX].

On NT systems, look for files and directories that closely match what may appear as a system file (EXPLORE.EXE, UMGR32.EXE, etc).

Review log files

On UNIX systems, look in /etc/syslog.conf file to find where syslog is logging messages.

On NT systems look in the Event Viewer.

Look for anomalies in these files. Anything out of the ordinary should be inspected.

Look for events that occurred around the known time of the intrusion.

Look for signs of a network sniffer

See if any process currently has any of your network interfaces in promiscuous mode.

Use utilities such as df to determine if part of the file system is larger than expected (be sure to obtain a known clean copy of that utility if you do use it).

Examine the output file from the sniffer to determine what other machines are at risk.

Check other systems on your network

Check for systems involved or affected at remote sites

7.

Recover from the intrusion

Install a clean version of your operating system.

Reinstall the operating system from the distribution media.

Install all of the security patches before connecting back to the network.

Restore the system using known clean binaries.

Re-install the operating system using the original distribution media.

Disable unnecessary services

Configure the system to offer only the services that the system is intended to offer and no others.

Ensure that there are no weaknesses in the configuration files for those services and that those services are available only to the intended set of other systems.

Install all vendor security patches

Ensure that the full set of security patches for each system is applied.

Check with system's vendor for any updates or new patches that relate to your systems.

Change passwords

Change the passwords of ALL accounts on the affected system(s).

8.

Improve the Security of the System and Network

Review security using the UNIX or NT Configuration Guidelines document

Review the security tools documents

Install security tools before you connect your machine back to the network.

Enable maximal logging

Enable logging/auditing/accounting programs (for example, process accounting) and that they are set to an appropriate level (for example, send mail logging should be level 9 or higher).

Backup logs and/or consider writing logs to a different machine, to an append-only file system, or to a secure logging host.

Configure firewalls to defend networks (Consider filtering certain TCP/IP services at your firewall server, router or at the hosts).

9.

Reconnect to the Internet

If you disconnected from the Internet, the best time to reconnect is after you have completed all the steps listed above.

10.

Update your security policy

Document lessons learned from being compromised

Calculate the cost of this incident

Incorporate necessary changes (if any) in your security policy

11.

Report status and actions taken to the Emergency Management Team.

Team Functions:

Coordinates recovery operations of the Facilities following a disaster.

Alternate Site:

Recovery Center

Procedure

Recover Facility

Default Vendor:

Vendor Name

Telephone #

Contact Name

Tasks in this procedure should be assigned to the Facilities Task Force. Task Force members may call on the services of other Facilities personnel who are not members of the recovery team.

1.

Inspect structural integrity

If the advisability of reoccupying the structure is questionable:

Examine the facility to determine if it is safe for occupancy.

Photograph ALL damage for insurance purposes.

If the structure is deemed safe and usable, continue with the procedure.

If the facility can be re-occupied with repairs and clean up, continue with this procedure.

If the structure can never be reoccupied, proceed to "Contact Insurance Representative" task and then to "move to temporary work space" task.

If required, contact security vendor and arrange for building security services.

Record finding on the "Damage Assessment Form" at the end of the Facilities recovery Team's Procedures.

Vendor Name

Telephone #

Contact Name

2.

Inspect for nonstructural damage

Inspect facility and assess extent of non-structural damage, evaluate all equipment for salvage, repair or replacement.

Determine if power, ventilation, lighting, heating, and cooling are available or can be readily restored.

Determine requirements to repair windows, doors, and roof.

Determine requirement for water removal.

Determine requirement for mud and dirt removal.

Evaluate facility for reoccupation now, later (how long?), or never.

Relay findings of step 1 and 2 to the Emergency Management Team.

Vendor Name

Telephone #

Contact Name

3.

Clear Access

If access to the facility is blocked:

Assess the amount and type of blockage.

Contact Facilities Management and arrange for removal service.

Determine vendor's time of arrival and estimated charges.

Monitor the progress of the work.

4.

Repair windows/doors

If windows or doors are damaged:

Determine the dimensions of the damaged windows/doors.

Contact Facilities Management and arrange for repair services.

Determine the estimated time of the vendor's arrival.

Determine the estimated cost of repair services.

If the repair time estimates exceed requirements, obtain boards, plywood, or plastic sheeting from the Facilities Department to temporarily seal off opening.

Monitor the progress of the work.

5.

Repair roof damage

If the roof to the facility has been damaged:

Assess the extent of the damage (type and size of damage).

Contact Facilities Management and arrange for repair services.

Determine vendor time of arrival, time for repair, and estimated cost of repairs.

Monitor the progress of the work.

6.

Remove water

If the facility has excessive water that needs to be removed and water is close to or touching electrical sources, call the fire department for water removal. Otherwise:

Call listed vendor for water removal services.

Call Facilities Management for repair of pipes/bathroom facilities.

Determine time of vendor arrival and estimated cost of repair.

Monitor the progress of the work.

Use fans to dry out area after the work is completed.

7.

Repair electrical malfunction

If, after examination, damage to the structure's internal electrical system is suspected:

Contact Facilities Management and arrange for testing and repair of the structure's electrical system.

Determine the estimated time of the vendor's arrival.

Determine the estimated cost of the service.

Monitor the progress of the repairs.

Vendor Name

Telephone #

Contact Name

8.

Obtain emergency power

If electrical power is not available:

Confirm outage with the power company and expected duration.

Obtain sufficient fuel to run the generator for the anticipated duration of the outage. Notify fuel vendor for additional requirements.

Vendor Name

Telephone #

Contact Name

9.

Remove Mud/Dirt from facility

If the facility has dirt and mud inside:

Contact Facilities Management or other vendor and arrange for a clean up.

Determine the estimated time of the vendor's arrival.

Determine the estimated cost of the service.

Monitor the progress of the clean up.

10.

Contact insurance representative

If it is determined that damage has been done to the facility:

As phone service becomes available, contact corporate insurance department and describe the extent of the damage that has been observed.

Determine what the insurance provider requires prior to taking any action.

Contact Insurance Adjuster.

Use vendor estimates to document damage.

Update insurance department on actions taken and expenses incurred.

Move to temporary location

Vendor Name

Telephone #

Contact Name

11.

Move to temporary location

If primary facility will be uninhabitable for 10 working days or more, move staff to alternate work space:

Notify commercial real estate broker of need for temporary work space.

Notify staff of temporary site, space allocations, and planned length of stay.

Notify customers and vendors of new location and phone number. Maintain communications with customer base until normal operations resume.

Determine estimated time of reoccupying permanent facility.

Arrange for transport services for critical and salvageable resource items (determine if staff will use their own vehicles to assist in this effort).

Transport as many resource items that can be salvaged to the temporary site. Notify customers and vendors of the new location.

Enter new address and telephone number on the Web Site.

Vendor Name

Telephone #

Contact Name

Customer Service and Sales Task Forces

12.

Ready permanent workspace

If the existing facility needs major repairs or if an entirely new facility needs to be found:

Assess equipment, furniture, electrical, and phone needs for permanent facility.

Retain contractor and architect for construction services

Work with vendors to develop a layout of the facility that includes the placement of equipment, cables and lease lines, furniture, electrical outlets and telephones.

Authorize construction.

Distribute a copy of the layout to those who will need to instruct service providers and vendors regarding delivery.

Monitor progress of construction activities.

Vendor Name

Telephone #

Contact Name

13.

Reoccupy facility

Receive notification that the facility is ready to be occupied.

Arrange with staff or with local vendor to provide for moving from temporary space into the permanent facility.

Move from temporary facility to permanent facility.

Switch telephone numbers back to the permanent facility.

Inform members, customers and vendors of the move to the permanent facility (List on Web Site)

Enter Voice Message on 800 number

Customer Service Task Force

14.

Report status and actions taken to the Facilities Recovery Team Leader who, in turn, will notify the Emergency Management Team.

Procedure

Recover Off-site Records & Documentation

Tasks in this procedure should be assigned to a Facilities team member. This team member may call on the services of other Facilities personnel who are not members of the recovery team.

1.

Damage assessment of records

Customer Service Task Force

Inventory vital records and documents using "Storage Location Detail Report."

Identify those records that are critical, are backed up, and need to be replaced.

Protect damaged records from unauthorized individuals.

Retrieve available records from off-site records storage.

Vendor Name

Telephone #

Contact Name

Record findings on the "Damage Assessment Form" at the end of the Facilities Recovery Team's Procedures.

2.

Document restoration

Identify critical documents that are water or fire damaged but might be restored.

Contact vendor for assistance in restoring documents.

Identify and document destroyed vital documents.

Submit record of damages to insurance unit.

Vendor Name

Telephone #

Contact Name

3.

Report status and actions taken to the Facilities Recovery Team Leader who, in turn, will notify the Emergency Management Team.

Procedure

Replace Staff

Tasks in this procedure should be assigned to the Human Resources Task Force. Task force members may call on the services of other personnel who are not members of the recovery team.

1.

Staff assessment

Attempt to establish contact with each staff member using the "Staff Detail Listing."

If telephone functions are available, use telephone to establish contact.

If telephone communication is not available, go to staff members' home residence to establish contact.

Communicate disaster situation, actions taken, and possible need for their help in replacing missing team members.

Determine missing or unavailable staff members.

Relay findings to the Emergency Management Team.

2.

Replenish Teams

Verify that teams have sufficient staff to cover all tasks assigned in the action plan.

If teams are not fully staffed, use the "Staff Detail" report to identify staff members who could fill open team positions.

Contact staff members identified in the previous step and request their assistance.

When the staff member joins the team, give a brief description of the disaster recovery process and what role you expect this staff member to play.

Assign the staff member his or her duties.

3.

Temporary staff

If there is a need for additional staff and all staff resources have been depleted, temporary help may be required:

Contact personnel department or vendor directly to acquire the quantity and quality of employees needed.

Interview and acquire the quantity of employees need by required skill level.

Arrange for the new employees' transportation and lodging if necessary.

Vendor Name

Telephone #

Contact Name

4.

Payroll continuity

Track hourly staffs' time on time sheets and send to personnel/payroll on a timely basis.

If payroll services are not available internally, send timesheets directly to payroll vendor for processing.

Accounting Task Force

Vendor Name

Telephone #

Contact Name

5.

Report status and actions taken to the Facilities Recovery Team Leader who, in turn, will notify the Emergency Management Team.

Procedure

Recover Office Furniture

Tasks in this procedure should be assigned to a Facilities team member. This team member may call on the services of other Facilities personnel who are not members of the recovery team

1

Damage Assessment

Make an assessment of the damage and replacement needs using the "Resource Item Matrix" report.

Compile a report of missing or damaged items that need to be replaced or repaired.

Salvage and safeguard as many resources items as possible.

Relay findings to the Emergency Management Team.

Record finding on the "Damage Assessment Form" at the end of the Facilities Recovery Team's Procedures.

2.

Recover Office furniture

Locate the "Resource Item Matrix" report.

Determine types and quantities of office furniture required.

Poll other facilities to determine what office furniture can be borrowed on a short-term basis.

For required furniture that can not be obtained internally, contact vendors and acquire the appropriate quantities of office furniture.

If vendor delivery is available, arrange for time of delivery.

Vendor Name

Telephone #

Contact Name

3.

Plan for furniture replacement

Design a layout of the placement of the office furniture that will be delivered.

If a move is required, coordinate the layout design with the senior manager of the temporary facility.

Distribute a copy of the layout to those who will-be instructing vendors regarding deliveries of office furniture.

4.

Arrange for transportation

If Office Furniture vender is unable to provide transportation:

Poll staff to determine if transportation services are available internally.

Contact moving vendor and make arrangements to move items to new location.

Obtain time of move, cost and method of payment required.

Alert staff member at move destination of impending delivery.

Monitor delivery of goods.

Provide accounting with a list of all new furniture and its location.

Vendor Name

Telephone #

Contact Name

5.

Report status and actions taken to the Facilities Recovery Team Leader who, in turn, will notify the Emergency Management Team.

Proeceudre

Recover Office Equipment

Tasks in this procedure should be assigned to a Facilities Task Force. Task Force members may call on the services of other personnel who are not members of the recovery team

1.

Damage Assessment

Make an assessment of the damage and replacement needs using the "Resource Item Matrix" report.

Compile a report of missing or damaged items that need to be replaced or repaired.

Salvage and safeguard as many resources items as possible.

Relay findings to the Emergency Management Team.

Record finding on the "Damage Assessment Form" at the end of the Facilities Recovery Team's Procedures.

2.

If equipment is or has been exposed to moisture, remove it from electrical power sources (main power, electrical generator, UPS).

If the environment is contaminated, remove equipment and electromagnetic media to a controlled environment free of smoke, soot, water and high humidity.

Prior to removal, label equipment to identify its home location and what other equipment to which it was connected.

Wipe off all moisture from outside of and electromagnetic media (diskettes, tapes, CDs).

Obtain written consent of manufacturer and vendors for the disaster procedures being taken on equipment under warranty or maintenance contract.

Do not allow vendors/staff to test, evaluate or diagnose damage prior to restoration.

Let the vendor de-install, reinstall, then recertify equipment.

Move damaged equipment into storage until an adjuster can look at it.

Prior to reinstallation, insure that surge protectors have remained functional.

Vendor Name

Telephone #

Contact Name

3.

Recover office equipment

Locate the "Resource Item Matrix" report.

Determine the types and quantities of office equipment required.

Poll other locations to determine what equipment can be borrowed on a short-term basis.

For required equipment that can not be obtained internally, contact vendors and order the appropriate quantities of equipment.

Indicate the address and location for delivery.

Provide accounting with a list of all new equipment, locations, and serial numbers.

Vendor Name

Telephone #

Contact Name

4.

Plan equipment placement

Design a layout of the placement of the equipment that will be delivered.

If a move to a temporary facility is required, coordinate layout design with senior manager of temporary facility.

Distribute a copy of the layout to those who will be instructing vendors regarding deliveries of equipment.

5.

Report status and actions taken to the Facilities Recovery Team Leader who, in turn, will notify the Emergency Management Team.

Procedure

Provide Human Comforts & Support

Tasks in this procedure should be assigned to the Human Resources Task Force. Task Force members may call on the services of other Facilities personnel who are not members of the recovery team

1.

Support Assessment

Determine and provide needed supplies and services need for staff support during recovery operation (food, water, and bedding). Determine cash requirements for a 20 day period and determine if local financial institutions are capable of providing requirements.

Accounting Task Force

2.

Arrange travel and hotel

Determine the number of people traveling and/or needing hotel accommodations.

Make travel and hotel arrangements.

Receive verification of staff arrival and update contact lists.

3.

Arrange showers and rest

If conditions warrant:

Arrange with local hotel facilities for room (s) to clean up and rest for restoration purposes.

Arrange transportation to and from the facility.

4.

Provide food and drink

If the disaster condition is such that staff members are unable to obtain food and liquids as they normally would:

Determine the number of people for whom food and drink must be provided.

Identify vendors who are in close proximity that can accommodate food and drink requirements.

Purchase food and drink supplies.

Keep food service areas sanitary.

Provide clean drinking water.

5.

Acquire Cash

Acquire cash for 20 day perod if deemed necessary.

Accounting Task Force

6.

Report status and actions taken to the Facilities Recovery Team Leader who, in turn, will notify the Emergency Management Team.