|
by Philp
Jan Rothstein, FBCI
Sometimes, a ‘disaster’ can sneak up on an
organization
unrecognized. This article offers tips to avoid this pitfall.
As appears in the March / April, 1996 issue of
InfoSecurity News Magazine.
Anybody involved in disaster recovery should not
find it difficult to recite their own
‘top-ten’ list of favorite disaster causes. They might include
natural disasters such
as floods, hurricanes, earthquakes or blizzards; external events such
as power or communications failures; technological disruptions like computer
crashes or network outages; or, facility events like fires. If one were
to conduct a careful analysis of corporate ‘disasters' over the past
decade, one would find that there are numerous disaster causes or potential
causes which are largely overlooked. For example:
- A major financial organization experienced a seven-figure dollar loss
because of a single data base corrupted by a programmer who updated
a production program without following production signoff or turnover
standards or procedures.
- A large metropolitan hospital irrevocably lost their entire pharmacy
data base including current patient information when a disk crash led
them to discover that the backup tapes they had been consistently producing
nightly for over two years were of the wrong files; no backups had ever
been made of the lost data base.
- A major research and development facility depending on temporary staffing
for their data center operation experienced a two-day disruption when
a disgruntled former employee returned unnoticed through the temporary
employment agency and sabotaged the data center.
- A medium-size service organization experienced severe embarrassment
and inconvenience as well as a five-figure dollar loss when their voicemail
system crashed and all current messages were irretrievably lost.
- A large insurance company experienced a six-figure dollar loss when
a utility power disruption forced their data center to rely on backup
power. Although their uninterruptible power supply and backup generators
were effective for the data center, several hundred employees were put
out of work since there was no backup power for business operations
in the same building.
- A bank was put out of business for over a week and very nearly permanently
when a facility disruption necessitated activation of their data center
disaster recovery program. Although the data center was operational
in less than 48 hours at a recovery site, no business resumption plan
had been implemented for the 100+ employees displaced by the same event.
- A hundred-employee service organization was nearly put out of business
when flooding of the area around their offices prevented access to their
building. Although they had an off-site recovery plan, their only file
backups were stored in the computer room.
‘Real’ disasters are seldom obvious or
direct; the World Trade Center bombing, the Loma Prieta Earthquake, the
Hinsdale Central Office Fire, as profound as these events may have been,
are but a small percentage of the ‘disasters’ facing the typical
organization. The ‘typical’ disaster is far more likely to look
like the scenarios above. As often as not, they are compound failures
gradually escalating from seemingly innocuous, recoverable glitches to
near-tragedies. In most cases, human error (whether proactive or reactive,
commission or omission) is the single greatest factor in growing a large
headache into a small disaster.
How does one transform those large headaches into
valuable learning experiences rather than into disasters?
- Aggressively look for and address weaknesses in your contingency plans.
Use regular structured walkthroughs or even brainstorming sessions to
isolate and resolve vulnerabilities.
- Frequently test your contingency plans to failure: find the weakest
links, rather than striving to demonstrate a successful recovery test.
- Don’t assume that ‘real’ disasters will look like the
scenarios you have tested: your contingency plan should address every
conceivable disaster scenario, yet assume that the ‘real’
disaster, which will be the ultimate test of the contingency plan, will
be the one scenario which was not specifically considered.
Copyright (c)1997-2003, Rothstein Associates Inc. All
Rights Reserved.
|
