|
by Philp
Jan Rothstein, FBCI
Planners developing and maintaining a contingency plan
can learn the hard way that top management's lack of support, commitment
and funding can be a project killer.
As appears in
InfoSecurity News Magazine.
Developing, implementing, testing and maintaining
a contingency/recovery plan can certainly be a challenging and even a
frustrating process. More and more contingency planners are learning the
hard way that the toughest piece of the project may have little directly
to do with actual development and implementation that gaining, cultivating
and maintaining top management's support, commitment and funding can be
a project killer.
Working
in a consultative relationship with one client recently, I had the opportunity
to observe and alter the dynamics of this relationship first-hand. Having
worked with dozens of organizations wrestling with wavering top management
commitment to disaster recovery, it was easy to spot the symptoms. Be
warned that spotting the symptoms early does not always ensure a successful
outcome.
The
particular client is a successful, growing, medium-sized, financial services
organization with close to a thousand employees in two principal locations.
The MIS organization has had a disaster recovery plan in place for about
four years. It is tested about twice yearly, although not aggressively,
and generally well maintained. Top management had become complacent about
MIS recovery: they assumed that MIS would be able to fully recover from
virtually any disruption in a matter of hours and had paid little attention
to disaster recovery over the past three or four years. A request from
MIS for renewal of the multi-year hot site agreement, with a significant
cost increase, had brought disaster recovery into top management's arena.
The
Chief Financial Officer had been the mentor for disaster recovery during
the last iteration, around four years ago. At that time, he supported
MIS' commitment to contingency planning and encouraged appropriate funding.
Since that time, much had changed: the company had not only more than
doubled in size and revenue, but had acquired two smaller companies which
now depended heavily on the corporate MIS operation. Technologically,
the company had moved from mainframe, terminal-based systems to client-server
applications and to numerous local area networks with varying degrees
of sophistication, protection and control.
At a
recent, semimonthly management committee meeting which was my introduction
to the organization, the Chief Information Officer presented a synopsis
of the planned enhancements to the MIS disaster recovery plan. He discussed
the additional complexity of recovering the client/server and LAN environments,
as well as the business risks associated with increased dependency on
telecommunications and external networks. He recommended expanding the
scope to address interim processing strategies for critical business units
to cope while MIS was recovering the infrastructure, as well as to deal
with data integrity during the disruption. He suggested further broadening
the contingency planning scope to work-area recovery and to the telecommunications
network, eventually to include voice recoverability. His presentation
was professional, yet nontechnical.
The
initial reaction from the dozen or so senior managers present was stunned
silence. The Chief Operating Officer finally broke the ice by questioning
the need for any recovery program at all, since, " . . . after all, we've
never had a disaster." One of the business unit managers expressed concern
with any additional work or expense for his already overloaded group.
The Chief Financial Officer, who had been a supporter of disaster recovery
in the past, noted the company's tenuous financial position as a result
of the recent acquisitions and expressed concern with the additional expense.
The Chief Information Officer, quite pale by this time, quietly offered
to go back to his staff and reexamine the issues and options.
What Went Wrong?
It quickly became apparent to this consultant that
the problem in this organization was not in contingency planning - the
MIS disaster recovery plan could be characterized as at least adequate,
if not aggressive. The trap this company fell into is a common one: operating
on the basis of assumed expectations. The CEO, CFO, business unit managers
and others outside MIS made several assumptions about MIS and its ability
to recover from a disruption. Since, the last round of justification for
a hot site contract more than four years ago, MIS had not communicated
explicitly to clarify or contradict these assumptions. While the CIO's
presentation and recommendations were probably on target, his audience
was unable to reconcile these recommendations with the assumptions they
had been living with for four years.
The
biggest assumption was that MIS was somehow populated by wizards who could
instantaneously, miraculously and painlessly handle any crisis that comes
their way and have the business units operational within a few hours;
the reality was that, at best, recovery from a major disruption would
take 30-36 hours. The second assumption was that MIS had automatically
integrated all of the diverse technologies and platforms which had popped
up over the past four years into the disaster recovery program; the reality
was that MIS did not even implement or operate many of these platforms.
The third assumption was that, no matter what the cause or scope of disruption,
MIS would recover all data accurately to the point of failure; the reality
was that, at best, recovery would be to the prior night's backup and,
most probably, to a point at least three to four nights prior.
The
moral to this story is that with some delicate negotiation and outside
consulting assistance, the top management of this enterprise eventually
saw the light and not only approved the CIO's proposal but established
a working committee to address company-wide business continuity. Along
the way, among the lessons this CIO painfully learned were:
- Don't rely on top management's, clients' or
other stakeholders' assumptions about your ability to deliver salvation
from disruption - document explicit disaster recovery service level
agreements which spell out the limitations as well as the promises.
- Communicate regularly to stakeholders any technological,
business or operational changes which impact disaster recoverability
- don't wait until there is no longer a practical option or alternative,
or until business management has already acted on the basis of out-of-date
assumptions.
- Present disaster recovery options, constraints
and alternatives to business unit managers and to top management early
in their decision cycles - don't wait until they are committed to a
course of action which impairs disaster recoverability.
Copyright (c)1997-2003, Rothstein Associates Inc. All
Rights Reserved.
|
