Rothstein: You Can Bet On It

Disaster Recovery & Business Continuity & Contingency Planning & Disaster Prevention Bookstore

Service Level Management & Service Level Agreements Bookstore

by Philp Jan Rothstein, FBCI

A disruption is definitely going to hit your business, sooner or later —
it’s only a question of when, not if. What are prudent American businesses doing to minimize the impact of, if not avoid, disasters?

Why Bother?

A 200+-employee, $25-million/year, midwestern electronic parts distributor lost an estimated $500,000 ˜ eight months’ net earnings ˜ when a burst water pipe drenched their telephone system.
A 500-employee, New-York-based financial services firm suffered a power failure compounded by a union jurisdictional dispute in their headquarters facility, shutting down operations for almost a week. Three months later, almost ten percent of their customer base had not returned.
A specialized insurer lost a $1+ million litigation, in part because documents essential to the legal defense were inaccessible during a blizzard.
A privately owned New England textile manufacturer spent $75 million after a fire nearly destroyed the company’s manufacturing plant.
A major metropolitan medical center irrevocably lost all of the information contained on their online pharmacy system when a disk drive crashed and they discovered that the nightly backups they had been running for years did not include that particular disk drive.
A Northeastern electronics company was derailed by a hotel fire which killed several key executives.
A specialty chemical company’s production, sales and administrative operations were totally shut down for almost a week when a freight car derailment on the train tracks behind their building posed a threat of a HazMat (hazardous material) incident and the area was evacuated). Even the telephones went unanswered.

Things happen. Human nature, for what it’s worth, dictates that the good things (winning the lottery, a hugely profitable new customer walking in the door, a big
raise) happen to you, and that the bad things (computer crashes, hurricanes, car accidents) happen to somebody else. Sorry to say, you are, as likely as not, that “somebody else!”

Who Notices?
A number of factors have converged to reach a critical mass in awareness of business continuity and disaster recovery over the past five years, including:

Large-scale, “it-can’t-really-happen-here” events such as the World Trade Center (New York City, NY) and Oklahoma City, OK bombings; the Chicago, IL floods; Hurricane Andrew, which devastated southern Florida; and, the Loma Prieta Earthquake, which wreaked havoc in California, each with obvious and substantial impact on businesses;
Real-time, in-your-face broadcast media coverage of these and other regional, national and international calamities;
Regulatory and industry insistence to produce evidence of real contingency planning, not just “cya” documents designed to look good on a bookshelf. These pressures are coming from such organizations as JCAHO, (the Joint Commission on Accreditation of Healthcare Organizations), which mandates contingency planning as a condition of accreditation of healthcare providers; and, OCC, (the Office of Controller of the Currency), which regulates and conducts examinations of federally chartered banks;
A growing sense of vulnerability to terrorism, which had in the past been perceived as a distant, foreign peril;
Dramatically expanding technological complexity — and vulnerability— of all types and sizes of companies, as well as of interdependencies among companies. Examples of interdependencies include just-in-time manufacturing (JIT), and electronic document interchange (EDI); and,
Growing maturity, breadth and awareness of the business continuity industry, including increasingly sophisticated education and service offerings.

What’s Going To Happen To Us?
If your business is located on southern California, an earthquake is a very real possibility; in the Carolinas, hurricanes are to be expected; in New England, you can count on a humongous Nor’Easter every winter or two; in Oklahoma, tornados can be lethal. If your business depends on computers, telephones or electricity, it’s no stretch to imagine an extended outage almost anywhere.

The real questions when assessing the risk of business disruption should be:

When (not if ) these ugly incidents occur, how will we cope?
What threats might we have overlooked?
How would we handle a disruption we never planned for, or even imagined?

How, then, do you identify the threats which are relevant? The first step, as simplistic as it sounds, is to acknowledge the threats exist. It is truly amazing how many organizations go about their business day-to-day, blissfully ignoring their vulnerabilities ˜ until the inevitable mishap, when they look for sympathy while neglecting their commitments ( Don’t you just love to hear the phrase “due to circumstances beyond our control...?” ). In this author’s mind, there are few, if any, excusable circumstances beyond the control of a business entity ˜ there are only circumstances which management chose to disregard. No excuses! Better to keep in mind the popular saying, “lack of planning on your part does not constitute an emergency on my part.”

Philosophically, this author advocates businesses to plan for every conceivable threat, regardless of probability; and, to assume that the threat which inevitably rears its ugly head is going to be one that was not specifically planned for. For example, while many businesses in downtown Chicago had contingency plans addressing broken water pipes, few could specifically cope with the subterranean flooding of dozens of basements when a contractor drove a piling under the river into an underground network of tunnels. The successful contingency plan has to be flexible enough to adapt to unimaginable scenarios, not just to the “no-brainers.”

Who Are You Going To Call?
Researching potential threats can be a straightforward process. Consider past history as well as future threats. When researching potential threats, always keep in mind that you may be unpleasantly surprised. Some of the best threat information sources include:

Federal Emergency Management Agency (FEMA), which has information
on natural and human-caused disasters in different regions, including
natural hazards;
County, city and state emergency preparedness offices, which are likely to have a wealth of valuable data on regional hazards and threats as well as support resources for emergency response and recovery;
The U.S. Army Corps of Engineers, for flood plain data;
Local emergency services departments, including police and fire services, who can help identify both regional and specific threats and hazards as well as offer valuable coaching for emergency response and recovery. For example, many fire departments will conduct informal or formal fire safety inspections as well as fire safety training programs at little or no cost;
Your organization’s knowledge base, general staff and management, and especially long-term employees;
Networking with neighbors, competitors and others confronted with
similar threats;
Local and regional newspaper archives;
Vendors, suppliers and customers, who have as much at stake as you do;
Business continuity professionals;
Technology experts;
Internal or external auditors;
Your risk manager;
Facility management, including your landlord, and infrastructure maintenance contractors;
Telephone and utility providers;
Regional newspaper and periodical archives; and, most important of all,
Your own eyes and ears. Look around for obvious and not-so-obvious weaknesses.

Drive the area around your facilities. Listen for what is worrying employees, management, staff, suppliers, customers.

Everybody’s Uniquely The Same
It is human nature for people to feel they are unique — after all, they are. The same applies to businesses, whether a local delicatessen or a multinational investment bank. But when it comes to contingency planning, these basic rules apply no matter the size, nature or culture of a business:

Common sense rules. You do not have to be an expert to recognize the proverbial sword of Damocles hanging over your business. At the least, be prepared to deal with the high-probability threats and acknowledge the less likely possibilities.
“It is better to avoid disruption in the first place than to cope with the pain, and expense of a corporate heart attack,” wisely counsels Ken Brill, president of ComputerSite Engineering (Santa Fe, NM), a consulting company addressing continuous availability of computer and communications environments.
Don’t wait — there will never be a better time. It is easy to defer contingency planning: there is always going to be a more pressing priority — until the day that the disaster happens. Start a continuity program, at whatever scope and level is realistic, now.
Don’t bank on insurance to rescue your business. Business interruption and extra expense coverage along with property and casualty coverage can certainly help, but are no substitute for an effective contingency plan. After all, an insurance settlement is likely to be insufficient consolation for a defunct business — more than one of the companies cited at the beginning of this article are no longer alive and kicking. Keep in mind that many insurers are willing to reduce premium costs because you have a continuity program in place.
Line up your ducks. Make certain upper management, business unit heads and other decision-makers are in agreement. Top-down contingency plans work much better than bottom-up — too many well-intentioned contingency planners have found their efforts were wasted when “the powers that be” failed to grasp the significance of business continuity — after the fact. No other attention-getter is quite as effective (short of a disaster) as a key executive expounding on the urgency of business continuity.
Don’t reinvent the wheel. Your organization is not the first to plan for contingencies. There is a wealth of knowledge, published material and other resources. Use a coach — an outside consultant could save you time, effort and money, even if they are just looking over your shoulder and nudging you in the right direction. On the other hand, be wary of farming out the contingency planning process unless you are committed to working closely with the outsiders and to building knowledge and experience within the organization.
Allocate money and time. As obvious as that sounds, successful contingency planning is seldom a part-time, casual undertaking. Resources should be unquestionably and realistically allocated and budgeted for ongoing operation, exercising and maintenance as well as for the initial development and implementation process. Don’t assume you can hide contingency planning in another budget line — you will not get away with it.
Question authority. When it comes to the survival of your enterprise, do not accept the word of management or of technical gurus. Look beneath the surface for hidden threats or risks.
Be creative. You need not expend mass quantities of time, effort or money to create a solid and effective continuity program. A little ingenuity and unreasonableness can go a long way. For example, many companies find concurrent opportunities for savings while implementing their contingency plans. Look for synergies with other active projects, or with projects which may otherwise be difficult to justify.
Exercise. Whether your contingency program is methodically documented or barely thought out, the exercise process serves three basic purposes: validating the strategy, refining the tactics, and training the participants. An exercise program can also be a productive and inexpensive tool for development of a contingency strategy. Do some kind of exercise, even a simple, 30-minute tabletop walkthrough — at least once a month.
Don’t become complacent. Even if you are satisfied with your continuity program, frequently review, update and enhance it. Among organizations with documented contingency plans who have botched recoveries, lack of maintenance has been the second most common problem, after lack of exercise. Don’t rest on your laurels once you’ve developed your contingency plan. At least once quarterly, update your plan. Update volatile data such as
staff lists and contact information monthly.

Big Fish, Little Fish
Of course, some aspects of contingency planning are going to be specific to the size, nature and complexity of a business. For example, most any business needs to address communications in their contingency plan, but a small business’ single-line phone and fax setup or a large corporation’s backbone network clearly are going to be approached differently — or are they?

Fundamentally, the same contingency planning process applies to both small and large organizations. The most obvious differences are going to be in scaling
and staffing. A large enterprises can generally mobilize more people and resources; on the other hand, the magnitude and complexity of disaster response is proportionately larger.

Legal or regulatory requirements, or industry practices, are more likely to apply
to large or governmental organizations. In an ideal world, mandated contingency planning would be unnecessary; in the real world, only accredited healthcare providers or federally chartered banks presently have any mandate in the
United States.

Technology Can Bite You
Businesses are becoming increasingly sensitive to breakdown of complex, distributed technology infrastructures such as client/server networks, desktop workstations, decentralized processing, electronic document interchange (EDI), and the Internet. These dependencies are increasing the vulnerability of many businesses to physical as well as technological disruption and at the same time amplifying the impact of disruption and the complexity of business recovery.

      Effectively backing up mission-critical data, often networked across multiple computer platforms, is inviolate. One $15 million, west-coast, service organization learned the hard way that their sophisticated, highly automated backup scheme was not particularly effective after an earthquake — with all of the recent backup tapes still on-site, hey could not get back into their building for two days.

      A southeastern insurer lost three weeks’ business transactions when they attempted to restore from a backup tape only to learn that their backup software had a bug which prevented a restore. Testing the restore process is as important as producing backups in the first place.

      A Midwestern service company learned the hard way that recovery of their local area network (LAN) was far more complex than they had imagined, when they successfully recovered the MIS “glass house” in ten hours to a recovery hot site but could not provide LAN workstation access for any of heir employees for three days.

      Telephone and data communications connectivity to the outside world is vital to most every business. A pharmaceuticals company was unpleasantly surprised to learn that all of their voice and data communications were carried on a single set of overhead poles — when a truck took out one pole and cut all communications for over two days. They were further dismayed to learn that their facility’s electrical power were carried over the same poles, and they were without power for a day. Sad to say, most of their hundreds of employees saw those vulnerable poles and wires when they drove to work each day along that road, but none gave them serious thought˜ until the painful day that truck driver highlighted the vulnerability for them.

      Telecommunications infrastructure reliability options such as diverse routing (both logically, through public and private networks, and physically), wireless technology and dynamic rerouting of failed circuits (“fail-over”), have gone from luxuries to necessities for the continuity of many organizations.

The Moral Of The Story
Business continuity is not about elaborate documents or expensive software — it is about common sense. It is not about pessimism or cynicism, either — it is about being realistic, sensible and aware.

On the other hand, committed contingency planning may prove to be a long-term, substantial undertaking for some organizations. The potential for significant investment should not scare you away from understanding threats and vulnerabilities confronting your business, nor from taking preventive measures. The intangible benefits of business continuity include a sense of confidence and maturity which, along with tangible side benefits, often transcend the investment.

E-mail Rothstein Associates Inc.