Creating an Actionable Disaster
Recovery Plan
Healthia Consulting
2
Presentation Outline
•
Plan Justification
• Disaster Definitions & Facts
• Costs of a Disaster
• Benefits of Planning
•
Building an Actionable Disaster Recovery Plan
• Program Initiation
• Risk Assessment
• Detailed Risk Assessment
• Disaster Recovery Plan
• Maintenance Plan
• Test Plan & Results
Healthia Consulting
3
Plan Justification
Healthia Consulting
4
What’s a disaster?
“A disaster is an occurrence that disrupts the functioning of an
organization resulting in the loss of data, loss of personnel, loss
of business or loss of time” – Hiatt 2000
Healthia Consulting
5
Disaster Facts
Common IT disasters:
• Power outages
28%
• Storm damage
12%
• Floods
10%
• Hardware error
8%
• Bombing
7%
• Hurricanes
6%
• Fires
6%
• Software errors
5%
• Power surge/spike
5%
• Earthquake
5%
Healthia Consulting
6
Terms
•
Business Continuity Planning
• Advance planning and preparations to ensure continuity of
critical business functions
•
Disaster Recovery
• Advance planning and preparations to minimize loss and
facilitate recovery of core IT assets
Healthia Consulting
7
Tangible and Intangible Costs
•
Patient care and patient safety
•
Paying staff who are idle
•
Added work, related to manual operations
•
Other hard cash costs
•
Lost business
•
Lost customer loyalty – your reputation!
Healthia Consulting
8
Recovery Planning Benefits
•
Reducing legal liability
•
Minimizing potential economic loss
•
Decreasing potential exposure to disaster
•
Reducing the probability of a disaster occurrence
•
Reducing disruption to normal operations
•
Ensuring organizational stability
•
Ensuring orderly, systematic, and timely recovery
Healthia Consulting
9
Recovery Planning Benefits
•
Minimizing insurance premiums
•
Reducing reliance on key individuals
•
Increasing asset protection
•
Ensuring the safety of personnel and patients
•
Complying with legal, statutory, and regulatory requirements
Healthia Consulting
10
Why have the stakes risen?
•
eBusiness transitioned many businesses from 8am-5pm to a
24 x 7 x 365 model.
•
Patient care could be compromised without information
systems.
•
Operations are running too lean to transition to manual
processes and be able to conduct business ‘as usual’.
•
Technology companies are not maintaining inventories as they
once did to provide quick disaster shipment capabilities.
•
New exposures: viruses, cyber-crime, terrorism
Healthia Consulting
11
Getting Approval & Funding
•
Historical data
• The National Climactic Data Center (NCDC) is the “Nation’s
Scorekeeper” in terms of addressing severe weather events
in their historical perspective (
www.noaa.gov
)
•
National initiatives
• Hospital Incident Emergency Command System (HIECS)
•
Regulatory audit compliance
• HIPAA
• JCAHO
Healthia Consulting
12
Building an Actionable Disaster
Recovery Plan
Healthia Consulting
13
A Practical Approach
Initiation
Detailed
Assessment
Plan
Development
Risk
Assessment
Testing &
Maintenance
Phasing:
1. Initiation
2. Risk Assessment
3. Detailed Assessment
4. Plan Development
5. Testing & Maintenance
Healthia Consulting
14
Program Initiation
Initiation
Detailed
Assessment
Testing &
Maintenance
Risk
Assessment
Plan
Development
Healthia Consulting
15
Strategic Objectives & Scope
Objective:
• Develop overall Strategic Objectives and Scope for DRP
Program
Practical Approach:
• Develop high-level Business Case to support DRP Program
• Gather and review existing documentation related to DRP
• Identify areas of alignment with other Organization Initiatives
• Define Program Objectives and Scope
Deliverables:
• DRP Program Definition
Healthia Consulting
16
Organizational Structure
Objective:
• Develop DRP Program Organizational Structure
Practical Approach:
• Identify Sponsorship, Stakeholders and Program Manager
• Define Program Organization, Roles and Responsibilities
• Dedicate existing Staff and supplement with External Resources
Deliverables:
• Identification of Sponsor(s), Stakeholders and Program Manager
• Definition of Program Organization, Roles and Responsibilities
• Initial staffing of Core Team(s)
Healthia Consulting
17
Communication Strategy
Objective:
• Establish ongoing Communication Strategy
Practical Approach:
• Define Communication Objectives, Approach and Channels (e.g.
Status Reports, Company Publications, etc.)
• For each Channel, define Audience, Message, Mechanism,
Tactics, Measures and Timing Recommendations
Deliverables:
• DRP Communication Strategy and Timing Recommendations
Healthia Consulting
18
Program Plan & Budget
Objective:
• Define High-Level DRP Program Plan and Budget
Practical Approach:
• Define and obtain consensus on Approach and Plan for the
overall DRP Program
• Estimate DRP Program Cost and Resource Requirements
Deliverables:
• High-level DRP Approach, Plan and Budget Assessment
Healthia Consulting
19
Kick-Off Meeting
Objective:
• Facilitate Program Kick-Off Meeting
Practical Approach:
• Host Program Kick-Off Meeting, obtaining stakeholder
consensus on Program Scope, Objectives, Communication
Strategy, Plan and Budget
Deliverables:
• Program Kick-Off Meeting Presentation / Agenda
• Kick-Off Meeting
Healthia Consulting
20
Risk Assessment
Initiation
Detailed
Assessment
Testing &
Maintenance
Risk
Assessment
Plan
Development
Healthia Consulting
21
Process Risk Analysis
Objective:
• Perform Business Process Risk Analysis
Practical Approach:
• Interview Business and IT Subject Matter Experts (SMEs) to
define disaster scenarios, create an inventory of the major
business processes, define the impact of an interruption and the
tolerance for downtime, and prioritize major business processes
• Complete Risk Assessment for Business Process
Deliverables:
• High-Level Business Process Current State Definition
• Business Process Risk Assessment
Healthia Consulting
22
Business Process Inventory
Application B
0
H
Results to
HIS
Labs
ADT,
Orders
Lab
Patient
Care
Application A
0
H
Order
requisition
to ancillary
system
Orders
ADT
Order Entry
Patient
Care
Application
s Used
Downtime
Tolerance
Impact of
Interruption
Primary
Outputs
Primary
Processing
Primary
Inputs
Business
Function
Busines
s Line
Process Dependencies
Healthia Consulting
23
Technology Inventory
Objective:
• Perform Technology Inventory and Risk Assessment
Practical Approach:
• Interview IT Subject Matter Experts (SMEs) to identify
Technology Assets, define interdependencies and prioritize
according to time sensitivity and criticality
• Audit existing, relevant processes and procedures
• Complete Risk Assessment for Technology Assets
Deliverables:
• High-Level Technology Asset Current State Definition
• Technology Risk Assessment
Healthia Consulting
24
Technology Inventory
Technology Assets
Quantity Location Interdependencies
Downtime
Tolerance Criticality
Applications
(1) Application 1
Supported Desktops
(1) Desktop config 1
Networking Infrastructure
(1) Network device 1
PBX / Telephony
(1) Telephony device 1
Total Valuation
Healthia Consulting
25
Detailed Assessment
Initiation
Detailed
Assessment
Testing &
Maintenance
Risk
Assessment
Plan
Development
Healthia Consulting
26
Detailed Assessment
Objective:
• Perform Business Process Gap Analysis and identify
Remediation Approaches
Practical Approach:
• Identify opportunities to prevent a disaster and other “quick-hits”
• Evaluate existing Policies, Workflow, and IT systems
• Complete Detailed Current State Definition
• Define and assess Remediation Options
• Develop Recommendations and select Remediation Solution
• Define Future State based on implementation of selected
Solution
• Define and obtain consensus on the Objectives, Scope,
Approach Plan and Budget for Remediation Approach
Healthia Consulting
27
Detailed Assessment
Deliverables:
• Detailed Current State Definition
• Remediation Options and Recommended Solution(s)
• Future State Definition
• Remediation Estimates and Plan
Healthia Consulting
28
Downtime Tolerance Costs
Implementation
Implementation
Costs
Costs
$$
$$
$
$
$$$
$$$
$$$$
$$$$
10
10
2
2
0
0
24
24
36
36
4
4
Downtime Tolerance (hours)
Downtime Tolerance (hours)
Healthia Consulting
29
Disaster Recovery Planning
Initiation
Detailed
Assessment
Testing &
Maintenance
Risk
Assessment
Plan
Development
Healthia Consulting
30
Plan Development
Objective:
• Develop DRP plan
Practical Approach:
• Deploy “quick-hit” solutions
• Develop high-level recovery strategies and recovery phases
• Define roles and responsibilities including line of command
• Define disaster assessment and declaration definitions and
procedures
• Develop emergency/ evacuation procedures that incorporate DRP
activities
• Document organization, staff and system functions and recovery
requirements and procedures
Healthia Consulting
31
Plan Development
• Establish recovery locations and document steps to make
functional during a disaster
• Develop business partner and vendor agreements
• Develop communications plan and identify alternative
communication tools
• Create contingency plans for missing people, failed procedures
• Document insurance information and procedures
• Build maintenance schedule and procedures
Deliverables:
• Actionable Disaster Recovery Plan
Healthia Consulting
32
Plan Structure
Section 1:
Plan Information
Objective:
To provide information that will enable the
reader or user of this plan to execute it while
fully understanding the intentions and
parameters with which it was created.
Content:
Scope, Approach, Objectives, Team
Organization, Pre-Disaster Action Checklist
Healthia Consulting
33
Plan Structure
Section 2:
Actionable Recovery Steps ***Disaster: Start
Here***
Objective:
To provide a step by step checklist of
activities that will be performed in the event
of a disaster. This section contains the detail
for each disaster level, by business line, by
recovery option.
Content:
Evacuation Checklist, Disaster Declaration
Checklist, Recovery Team Activation
Checklist, Level 1 – 4 Recovery Steps for all
teams and for all recovery options
Healthia Consulting
34
Plan Structure
Section 3:
Addendums
Objective:
To provide one place to access key
information and resources required to
efficiently and knowledgeably carry out the
actionable recovery steps.
Content:
Phone list, Insurance Information, Legal
Considerations, Key Communication
Messages, Facilities
Considerations,
Security Considerations, Transportation
Options
Healthia Consulting
35
Section 1 - Plan Information
Goal:
• Enable the user to execute the Plan while fully understanding
the intentions and parameters with which it was created
Contents:
• Scope
• Approach
• Objectives
• Team Organization
• Plan Activation Process
• Distribution
• Communication Strategies
• Contingency Plans (missing people or failed procedures)
Healthia Consulting
36
Disaster Event Types
Event Types
Event Level 1
Event Level 2
Event Level 3
Event Level 4
Healthia Consulting
37
Recovery Strategies
Event Types
Staff
Facility
Technology
Process
Event Level 1
Event Level 2
Event Level 3
Event Level 4
Healthia Consulting
38
Recovery Strategies
App 1
App 2
App 1
App 2
App 1
App 2
1 Execute manual procedures
1
1
1
1
2 Restore from backup
2
2
1
3 Failover to redundant systems
1
2
Event Level 1 Strategies
B
u
si
n
ess
A
rea 1
B
u
si
n
ess
A
rea 2
B
u
si
n
ess
A
rea 3
Event Level 1
App 1
App 2
App 1
App 2
App 1
App 2
1 Strategy 1
1
1
1
1
2 Strategy 2
2
2
1
3 Strategy 3
1
2
Event Level 2 Strategies
B
u
si
n
ess
A
rea 1
B
u
si
n
ess
A
rea 2
B
u
si
n
ess
A
rea 3
Event Level 2
Healthia Consulting
39
Recovery Team Structure
• Structured using the team approach
• Each team has separate section of the Plan within each
outage event level
• Recovery teams = operational and technical groups
responsible for restoring specific functions
• Each team only has the authority to carry out the procedures
contained in their section of the Plan
• The teams are:
• Command Team
• Administrative Recovery Team
• Operational Recovery Teams
• Technical Recovery Teams
Healthia Consulting
40
Recovery Team Structure
Command Team
Operational Recovery
Team
Technical Recovery
Team
Business Function 1
Business Function 2
Business Function 3
Phones
Applications
Infrastructure
Administrative Team
Healthia Consulting
41
Plan Activation Process
Outage Alert
Outage Alert
Command Team
Command Team
•
•
Establish command center
Establish command center
•
•
Determine disaster level
Determine disaster level
•
•
Supervise recovery steps
Supervise recovery steps
•
•
Receive initial alert
Receive initial alert
•
•
Determine disaster level
Determine disaster level
•
•
Activate recovery teams
Activate recovery teams
•
•
Receive notification
Receive notification
•
•
Evacuate area
Evacuate area
•
•
Notify team members
Notify team members
•
•
Activate plan
Activate plan
•
•
Receive notification
Receive notification
•
•
Evacuate area
Evacuate area
•
•
Notify team members
Notify team members
•
•
Activate plan
Activate plan
•
•
Receive notification
Receive notification
•
•
Evacuate area
Evacuate area
•
•
Notify team members
Notify team members
•
•
Activate plan
Activate plan
Operational
Operational
Technical
Technical
Administrative
Administrative
Authority to declare a
disaster crucial element
of plan:
• Assigned to restricted
number of individuals
• Only group
authorized to declare
a disaster is the
Command Team
Healthia Consulting
42
Section 2 - Recovery Steps
Goal:
• To provide a step by step checklist of activities that will be
performed in the event of a disaster. This section contains the
detail for each disaster level, by business line, by recovery
option
• In the event of a disaster, “start here”
Contents:
• Evacuation Checklist (OSHA)
• Recovery Locations
• Disaster Declaration Checklist
• Recovery Team Activation Checklist
• Level 1 – 4 Recovery Steps for Command, Administrative,
Business Lines and Technical Recovery Teams and for all
recovery options
Healthia Consulting
43
Command Team Checklist
# Start
Day
Start
Time
End
Day
End
Time
Activity
Team /
Owner
Complete
Date/Time
Comments
1. 1 E+
00:00
1 E+
00:15
Execute emergency response
(fire, tornado, etc.)
ALL
Refer to your facility
emergency action plan
2. 1 E+
00:15
1 E
+
00:30
Determine the disaster level
based on the Event Level
Definitions below and proceed
to Initiate Activation Checklist
Command
3. 1 E
+
00:30
1 E+
00:40
Notify Administration
accordingly
Command
4
.
1 E+
00:40
1 E+
01:00
Notify and activate the
Recovery Team Leads what
disaster level is being declared:
- Operational
Team
- Administrative
Team
- Technical
Team
Command
Healthia Consulting
44
Command Team Questions
Goal:
• Remind staff about key action items that don’t necessarily
belong in another checklist
• Examples:
• Need Risk Management?
• Need Safety Team? Questions about safety procedures,
personal injury.
• Need Purchasing Team? Need to purchase supplies,
furnitrue, computers, etc.
• Need Facilities Team? Issues with HVAC, security, parking,
restrooms, coffee?
• Need Communications Team? Issues with reporters,
announcements, etc.
Healthia Consulting
45
Recovery Locations
DRP Locations
Hospital
Data Center
B900 - Basement
Who Goes Here?
SysAdmins
Network
OPS
Telecommunications
What Happens Here?
Server / Network / Systems Assessments
Backup restorations
Failover Activities
System Monitoring
Who Goes Here?
DRP Command Team
Communication
Administrative Support
What Happens Here?
Disaster Level Decisions
Issue Management
Activity Direction
Status Reporting
Communication
Command Center
Conf. Room G & H
Medical
Office Building
Resource Center
MOB 605b
Who Goes Here?
Application Support
Interface Team
DBAs
What Happens Here?
Application Assessments / Recovery
Interface Assessments / Recovery
Database Assessments / Recovery
Help Desk /
Desktop Services
MOB 215
Who Goes Here?
Help Desk
Desktop Services
What Happens Here?
Help Desk 1st Level Support
Desktop Deployment / Support
Healthia Consulting
46
Section 3 - Addendums
Goal:
• To provide one place to access key information and resources
required to efficiently and knowledgeably carry out the
actionable recovery steps
Contents:
• Phone List (staff, emergency contact, vendor)
• Insurance Checklist
• Transportation Checklist
• Legal Checklist
• Key Communication Messages
• Security Checklist
Healthia Consulting
47
Test Plan & Results
Initiation
Detailed
Assessment
Testing &
Maintenance
Risk
Assessment
Plan
Development
Healthia Consulting
48
Testing
Objective:
• Perform testing
Practical Approach:
• Perform conference room test (passive testing)
• Perform full test (active testing)
Deliverables:
• Passive Test Plan and Test Results
• Active Test Plan and Test Results
Healthia Consulting
49
Plan Structure
Section 1:
Testing Plan Information
Objective:
To provide information that will enable the
reader or user of this plan to execute it
while fully understanding the intentions
and parameters with which it was created.
Content:
Scope, Approach, Objectives, Roles and
Responsibilities, Testing Environment and
Locations, Assumptions, Known Risks and
Issues
Healthia Consulting
50
Plan Structure
Section 2:
Actionable Testing Scenarios and Steps
Objective:
To provide step by step conference room
testing activities that address all levels of
disasters that are represented in the Plan.
Content:
Testing Checklists for Level 1 – 4
Disasters, Issue Management Process
Healthia Consulting
51
Plan Structure
Section 3:
Testing and Maintenance Schedule
Objective:
To provide a schedule that will ensure that
the Plan is tested and executed in a
conference room setting at least two times
per year. And, to develop a maintenance
schedule that will ensure that the plan is
current and relevant.
Content:
Testing Activities and Schedule,
Participants, Start Dates, End Dates;
Maintenance Schedule, Owners, Due
Dates
Healthia Consulting
52
Walk-Through Test
• Intended to orient and educated stakeholders with the
organization and content of the Plan
• Intended to evaluate the Plan for completeness and accuracy,
assuring all information is up-to-date
• Should include all stakeholders of the BCP and take 1-2 hours to
execute
• Example: Walk-through Test Script
Healthia Consulting
53
Conference Room Tests
Objectives:
• Intended to evaluate the detailed checklists of the DRP
• By creating scenarios (Level 1, 2, 3) to test different levels of the
Plan, all stakeholders will have the opportunity to review
individual checklists in addition to evaluating interdependencies
between the checklists
• Should include all stakeholders of the DRP and take 2-4 hours
Approach:
• For each level, develop a scenario
• For each scenario, define Type of Test, Participants, Type of
Disaster, Day and Time of Disaster Event, Disaster Incident
Description, Impact
Healthia Consulting
54
Discussion Items
• To start the scenario
• Who does what at that time?
• How long does it take?
• When is it finished?
• What were the disaster event discovery procedures?
• What notifications need to occur?
• What documentation needs to be prepared?
• How should a system outage be handled?
• What notification should occur?
• How do you validate the outage?
• How do you evaluate the impact on related systems?
• How do you document the process?
Healthia Consulting
55
Discussion Items
• What do you do in the meantime?
• How long do you continue manual processes?
• What if it is a hardware related problem? The vendor says it will
be three days before it can be resolved. What do you do?
• How long can they be used?
• Do they have adequate staff?
• How will they operate without access to the web?
• What should be communicated internally and externally?
• What decisions need to be made and how quickly?
• Company personnel need to use their temporary operating
procedures
• What steps need to be taken?
Healthia Consulting
56
Active Tests
• Intended to evaluate the execution of the checklists and ensure
everyone is comfortable executing their tasks
• Should include all stakeholders of the DRP and take 4-8 hours
Healthia Consulting
57
Test Results
• Imperative to track test problems in a Test Problem Log
• Problem Number, Problem Description, Assigned To, Action
Items
• Intended to ensure action is taken on problems or issues that
arose during the testing so that each iteration brings you closer
to a complete plan
Healthia Consulting
58
Maintenance Plan
Initiation
Detailed
Assessment
Testing &
Maintenance
Risk
Assessment
Plan
Development
Healthia Consulting
59
Ongoing Maintenance
Objective:
• Insure DRP plans are maintained on an on-going basis
Practical Approach:
• Maintain DRP command team and recovery team roles
• Maintain Vendor List and Supply List
• Perform periodic Internal Audits/Reviews
• Insure change management processes incorporate DRP plan
maintenance
Deliverables:
• Actionable DRP plans
Healthia Consulting
60
Timeline and Activities
2 Weeks Prior to Test:
• DRP Coordinator sends a message to all Command Team and
Recovery Team Leads indicating the time of the testing and
requesting Recovery Team Leads make checklist updates
• Recovery Team Leads update checklists and distribute to BCP
Coordinator
1 Week Prior to Test:
• DRP Coordinator updates BIA, Recovery Strategies, DRP and
Test Plan
Healthia Consulting
61
Timeline and Activities
Testing:
• Testing occurs over ½ day
• DRP Coordinator facilitates all testing activities
3 Weeks After Test:
• Updates and other action items identified during testing
complete
4 Weeks After Test:
• New DRP compiled and distributed to all Command Team and
Recovery Team Leads and Executive Management
Healthia Consulting
62
Summary
Healthia Consulting
63
Summary
• Disaster Recovery Planning is essential
• Your approach needs to be practical and the plan needs to be
executable
• Test much and test often
• Ensure the plan is maintained
Healthia Consulting
64
Healthia Consulting
701 Xenia Ave. South, Suite 170
Minneapolis, MN 55416
(763) 923-7900
(763) 923-7901 fax
www. healthiaconsulting.com
Wyszukiwarka
Podobne podstrony:
5255
5255
5255
5255
5255
5255
5255
5255
5255
healthy 2 5255
więcej podobnych podstron