So, who says disaster recovery can’t be fun?

Blowing up a data center may be a bit extreme, but it sure is a dramatic way to test data center fail-over and resilience.

Read the rest of this entry »

Healthcare Preparedness
by Mitch Saruwatari

The ability to pre-assign and train personnel for a hospital emergency is a daunting task for any emergency manager, requiring extensive documentation and skills in communication and coordination. But it doesn’t have to be, if a comprehensive exercise program is established that builds on response competencies learned over time.

Read the rest of this entry »

One of the products of testing your business continuity/disaster recovery program should be a documentation process to identify what went right, what went wrong, and what is still open in terms of issues. No issues during testing? Good for you, but not very helpful to strengthening your contingency plan.

Read the rest of this entry »

Testing is simply one of the best ways to “prove the concept.”  In other words, does the recovery capability really do what everyone expects?  That usually means does it handle the critical business processes that management has deemed important.  Can it keep the business going?

Read the rest of this entry »

Rob Davis provides some pointers for successful exercises.

It is well recognised by exponents of our art that effective evaluation of emergency or business continuity plans by exercise is invaluable. As contributors to the Continuity Central site we are all of like mind. The common thread in previous articles identifying senior management as an obstacle is not easily overcome. Our message needs to get out from this ‘congregation of the converted’ - out to those senior managers who are an essential component in a successful evaluation formula.

See Thoughts on business continuity exercises

=================================================================
An unexercised contingency plan could be worse than no plan at all!

Be sure to read Disaster Recovery Testing: Exercising Your Contingency Plan, Philip Jan Rothstein, FBCI, Editor for valuable tips, techniques and insights.

Incredible undersea cable scenario

by Nathaniel Forbes, ZDNet Asia

At 08:00 on Wednesday morning January 30, 2008, two ships 2,500 kilometers (1,600 miles) apart in the Mediterranean drop their anchors in stormy weather off Alexandria, Egypt, and Marseilles, France, at the same time. They both manage to drop their anchors directly onto two separate undersea cables buried 50 centimeters in the sand, each roughly the diameter of your wrist.

The two cables carry 75 percent of network traffic in the Middle East and South Asia. Your business in India or Egypt loses over half its international data and voice network capacity.

Two days later on Friday morning, February 1, a third cable is severed by an abandoned anchor embedded in the sea floor off the coast of Dubai, also 2,500 kilometers (1600 miles) from Alexandria, but in a different direction. That cable is owned by the Indian company that also owns one of the cables broken earlier.

Your voice and data networks are now crawling at just 25 percent of capacity. A spokesman for your business in India or Egypt calls it a “national disaster”.

Then two days later on Sunday Feb. 3, a fourth cable goes down, this time between the UAE and Qatar, from a power failure. Egypt’s Ministry of Communications announces, after a review of video footage, that the first cable break off Alexandria was not caused by a ship’s anchor, but offers no other explanation.

Incredible undersea cable scenario

==============================================

So, what was that about exercising your contingency plan…? A valuable book on exercising contingency plans is Disaster Recovery Testing: Exercising Your Contingency Plan, Philip Jan Rothstein, FBCI, Editor.

Pound for pound, testing is one of the best investments you can make in the area of disaster recovery. It’s truly where the recovery process is proven. Yet, many companies either don’t test at all or try and test too much at one time. Either approach can spell disaster.

When planning a data center disaster recovery plan exercise, usually performed at a hot site or internal company site (e.g., a second data center), following a few simple and logical steps will usually increase the likelihood of success.

Consider the following test scenarios:

• Test 1. Load the servers with the OS and bring up a basic network.
• Test 2. Add to Test 1 application data bases and critical systems.
• Test 3. Add to Tests 1 and 2 a small group of end users who can validate the status of the critical systems and data.
• Test 4. Add to Tests 1, 2 and 3 remote access (e.g., VPN) for critical applications so that end users can fully test recovery capabilities (including system functionality and performance).
• Test 5. Conduct a “surprise” test. This is basically a Level 4 test without any advanced planning or notification. Note that it is only recommended after four successful tests. These tests need very, very careful planning and are goals for everyone who is serious about data center disaster recovery.

The two most frequent mistakes companies make are:

1. Conduct Test 1 repeatedly
2. Proceed directly to Test 4, without preliminary testing

Neither approach provides good results. What typically works is a well-thought-out and logical sequence of test complexity.

————————————————————

A sound Disaster Recovery Plan is essential for any data center. Jan Persson’s GO.RECOVER-Data Center Template is a powerful yet easy-to-use tool for under $100.

————————————————————

A valuable book on exercising contingency plans is Disaster Recovery Testing: Exercising Your Contingency Plan, Philip Jan Rothstein, FBCI, Editor.

British Standards Institution has announced a new white paper entitled Business Continuity Management and Risk - The Role of Standards, and a new book called Exercising for Excellence: Delivering a Successful Business Continuity Management Exercise.

The white paper addresses the rise of risk management up the corporate agenda and reflects an increased understanding by organizations of their risk profiles and the need to effectively manage exposures. High-profile examples such as Enron, 9/11, Hurricane Katrina, and the summer floods in the United Kingdom have served to shape attitudes to risk management and achieve greater boardroom awareness. The paper will provide insights on how standards can assist organizations in being more proactive with their risk strategy. Case studies are included.

The exercise book provides a practical guide for anyone with responsibility for the planning and delivery of business continuity management (BCM) exercises. It’s considered a “how-to” guide to conducting successful business continuity exercises. According to the BSI, It will help practitioners test and evaluate the effectiveness of their incident management capabilities and business recovery plans. It is also consistent with guidelines established in BS 25999, the British standard for business continuity.

Download the BSI white paper

Purchase the BSI exercise book

Another valuable book on exercising contingency plans is Disaster Recovery Testing: Exercising Your Contingency Plan, Philip Jan Rothstein, FBCI, Editor.

Southern California Prepares for Biggest Earthquake Test

The Great Southern California ShakeOut, scheduled to start November 13, 2008, is a week of special events featuring the largest earthquake drill in U.S. history. The program has been organized to inspire Southern California residents and businesses to prepare for large earthquakes, and to prevent disasters from becoming catastrophes. The recent 5.4 magnitude earthquake reinforced the importance of such a program.

At 10 a.m. on November 13, 2008, millions of southern Californians will “Drop, Cover, and Hold On.” This program is being conducted because a major earthquake is considered likely, and the ShakeOut Drill is an opportunity to practice what to do when it happens. Individuals, families, businesses, schools and organizations will join firefighters, police officers, and other emergency responders (involved in the statewide “Golden Guardian” exercise) in California’s largest-ever earthquake preparedness activity.

ShakeOut is based on a potential 7.8 magnitude earthquake on the southern San Andreas Fault. This type of earthquake occurs in southern California every 150 years on average, and the last was 151 years ago. Dr. Lucy Jones of the U.S. Geological Survey has led a group of over 200 scientists, engineers, and others to study the likely consequences of such an earthquake in great detail.

The ShakeOut Scenario estimates this earthquake will cause some 2,000 deaths, 50,000 injuries, $200 billion in damage and other losses, and severe, long-lasting disruptions.

For some unsettling video of the 1995 Kobe, Japan earthquake, see A Lesson in Recovery.

Data Centers are particularly vulnerable to earthquakes. This guide offers lots of useful recommendations for Data Processing Facility Earthquake Hazard Mitigation.

“A survey of 200 companies with between 250 and 999 employees by Vanson Bourne found that, of the 81 percent of respondents stating that they had a business continuity plan, 50 percent had only partially tested plans and 18 percent had not tested any aspect of their plans. (1)

“A PricewaterhouseCoopers survey found that almost half of disaster recovery plans have not been tested in the last year.

“According to the Chartered Management Institute’s 2008 business continuity survey, 33 percent of organizations with a business continuity plan still do not undertake any form of exercise to test their plan.”

Improving Business Continuity Testing and Exercising

For the only book on exercising business continuity, disaster recovery, crisis management and emergency management plans, see Disaster Recovery Testing: Exercising Your Contingency Plan.