|
by Philp
Jan Rothstein, FBCI
Relocating a data center can be enough of a challenge,
doing
so with minimal business disruption demands good luck and great planning.
Some of the most successful moves have been little more
than activations of customized versions of the operations' disaster recovery
plan.
As appears in the May, 1997 issue of
Contingency Planning & Management Magazine.
Smart Moves
- Allow sufficient time, typically 12-18 months
- Secure adequate funding, appropriate budgeting
- Commit dedicated staff
- Avoid or circumvent critical-path tasks
- Check early for “drop-dead” dates
- Treat a relocation like a disaster recovery.
Moving a business can be an unsettling experience,
to say the least. Moving a vital technology facility such as a data center
or communications hub can be downright frightening.
I remember
my father's advice when I made the big move from renting an apartment
to home ownership: He said to figure all your expenses, allow for problems
or surprises, then add fifty percent to your budget — and effort.
Any project manager moving a data center would do well to consider those
words of experience.
A data
center relocation, especially if construction is involved, is at best
a twelve-month process. This is not to say it cannot be done faster; however,
the direct costs along with resource commitments, and level of effort
are likely to escalate almost exponentially with a shortened timeframe,
or when other critical priorities divert essential resources.
Factors
Foretell Fate
Immediately upon committing to a data center move,
four variables can potentially make or break the entire project.
The
first and most critical action is choice of an effective project manager.
Often, the data center manager is assigned by default. The only way he
or she can tackle a move project is by handing off most, if not all, day-to-day
responsibilities. Having guided about fifty data center moves, I have
observed that failure was a possibility only when the organization refused
to dedicate key management and staff full time.
The
second most critical action involves blocking out windows of risk or opportunity.
Long lead-time factors which may impact project timing or the actual cutover
-- the point during the move transition when operations are suspended
at the old site until they resume at the new data center -- must be identified
and addressed immediately. On the other hand, some of these same factors,
as expiring equipment leases, may offer timely opportunities for the relocation
Considerations
include business cycles; real estate leases; construction schedules; equipment
leases and depreciation schedules; equipment, software, network migration
or upgrade plans; workload and capacity projections; MIS plans; and space
constraints.
In most
cases, a best-case cutover window of opportunity will be obvious from
this analysis and will drive much of the project timing and resource levels.
The
third critical success factor is tangible management commitment. Budgeting
and resource allocation for a capital project as complex as a data center
move must be identified and approved early. Accurate budgeting tends to
be difficult in the early stages. This must be recognized by top management
and assurances obtained that funding and staffing will not be jeopardized
once the commitment to relocate is made.
The
fourth critical success factor involves taking advantage of concurrent
opportunities. Simultaneous developments may include upgrading of aging
equipment nearing lease expiration, technology enhancements which could
dramatically reduce space and infrastructure requirements, or staffing
plans which might necessitate leasing additional office space. Such side
benefits often can offset many of the costs of a data center move.
Declare
A Nondisaster
A typical data center build/move project may include
more than a thousand discrete tasks and directly involve from ten to a
hundred individuals: project manager; project secretary; user liaison;
vendor liaison; software team; hardware team; finance; voice communications;
data communications; insurance and risk management; physical security;
information security; human resources; construction manager; general contractor;
subcontractors; logistics specialist; customers; numerous vendors; administrative
team; architect; engineers; movers; salvage team; etc.
If it
is not already obvious from that list, you will find a data center move
process to have a lot in common with a data center disaster recovery.
Some of the most successful data center relocations have been little more
than activations of the operation's disaster recovery plan.
As a
bonus, this approach affords a valuable opportunity to exercise the recovery
plan under realistic conditions
Moving
Right Along
It's imperative that relocation activities interfere
only minimally at the cutover. Project activities should be kept to an
absolute minimum during this extremely vulnerable time period.
Excessive
downtime risk can be avoided, for example, by relocating or swapping equipment
outside of the cutover timeframe. Ideally, a cutover should require very
little physical movement although, of course, there will be functional
as well as economic tradeoffs. A thorough inventory and “spring cleaning”
can help to avoid nasty surprises as well as nuisances as the
cutover looms.
Early
and regular communication with every affected or involved party is crucial.
There must be no surprises for data center customers, vendors, management,
MIS staff or anybody else. This communication should begin on a monthly
basis once the project is committed. By the week leading up to the cutover,
daily communications are appropriate; from the day prior to the start
of the cutover to the day after acceptance, hourly or even continual advisories
are warranted. Progress reporting to management is also essential.
Tending
To Timing
As the cutover approaches, a “freeze period” is
a useful mechanism to reduce risk and maintain sanity. Typically, for
a period on the order of three weeks prior and two weeks after the cutover,
business areas, MIS and others must recognize that any significant activity
or change not only jeopardizes the data center move, but may in itself
be at risk. Freeze period exceptions are to be expected, but must be closely
managed. Often, top management must be called on to intervene.
Exercises
are as essential to a data center move as they are to a disaster recovery.
Beginning at least a couple of months prior to the cutover, tabletop walkthroughs
of the cutover should be held with the entire project team, including
vendors and affected customers. Dress rehearsals will identify many potential
glitches and smooth over rough spots.
Timing
of the actual cutover process is scheduled with military precision. Staffing
schedules are mapped out by location, hour-by-hour. Backup staffing is
on standby. Supplementary communications and transportation are in place.
Tasks may be mapped out in quarter-hour increments. Decision and commitment
(go/no-go) points, based on validation criteria, are defined.
Like a disaster
recovery process, it cannot be assumed that all the ingredients will easily
fall into place, that the other team is firing blanks and not live ammunition.
The ammunition is going to be live and painful, if not fatal, if precautions
for every contingency are not mapped out carefully in advance.
Moving
physical equipment is a visible yet relatively minor aspect of the overall
data center move project. In most cases, network connectivity and data
storage transitions are more critical. As an example, climate-controlled
vans and four-wheel drive tow trucks have been employed to ensure safe
arrival of magnetic media during extreme weather; entire, parallel networks
have been built to guarantee connectivity.
Making The Cut
The “right” strategy for one organization
may be wrong for another. It is vital to focus on a strategy early in
the planning process, although flexibility may be required as business,
technical, construction or other considerations come into play.
One
common move strategy, the “Flash Cut,” involves shutting down the old
center and moving the hardware, media, materials and so forth all at once.
For smaller centers or low-risk operations which can afford a few days
of service interruption, this approach may work well. Of course, this
type of relocation can be riskier than most. If all goes well, expense
should be low. If all does not go well expenses probably will not matter
much.
At the
opposite end of the spectrum, a data center move may involve minimal,
if any, physical relocation. Redundant or replacement hardware and other
components at the new site can be fully tested and operational before
the cutover. This approach works especially well if a concurrent transition
to replacement equipment is practical.
Short-term
equipment rentals provide the opportunity to shake down the new facility
and network with little risk. Rental equipment may be installed at the
new facility prior to the cutover and used in production for a short period,
or may serve as fallback while production equipment is moved.
In some
cases, the rental equipment has even been installed at the old facility.
Of course, its use for live operation during the move introduces an additional
cutover step, from the rental back to the production equipment. In most
cases, a blend of strategies is most effective. Some less critical components
may be flash cut over a weekend, for example, and the most critical components
are either replaced, or short-term rental equipment is employed.
How'd
We Do?
I recall one data center cutover which went remarkably
well until the CEO showed up at the door of the new facility, rather upset.
It turned out that the one step skipped was notifying top management that
we were done! In fact, the new data center had been up and running “live”
for more than twelve hours and all data center customers were back to
normal. The cutover had gone so smoothly that the CEO was convinced we
had failed.
Like a disaster
recovery process, a data center relocation project needs an explicit conclusion.
A clean-up and salvage stage should not be ignored, as the former site
may still require attention and there may be temporary equipment or other
matters to resolve.
The
shakedown period may require extra attention. A post-mortem review is
often helpful. And, a party for the project team is definitely called
for!
Key Stages of the Relocation
Project
- Formation of a Project Team
- Initial Strategy Commitment
- Resource Allocation and Budget
- Developing the Relocation Workplan
- Committing to a Cutover Schedule
- Set a Freeze Period
- Early Handling of Non-Critical-Path Tasks
- Cutover and Acceptance
- Post-Cutover Phase
Plan
For Perils, Pitfalls
With such a complex undertaking, it is easy to be
lulled into complacency and overlook a fatal flaw. Actual examples of
serious problems experienced during past data center relocations include:
- A labor union jurisdictional
dispute over inventor ying and packing of computer tapes;
- Delayed freight elevator
availability because of another building tenant, followed by the discovery
that certain equipment would not fit in the elevator;
- A flu epidemic affecting
most of key members of the move team;
- An irreplaceable computer
component tipped over and damaged;
- Employee cars parked on the street overnight
towed away;
- Hardware, software and communications not meshing
together after a move, without a contingency plan to return to the former
site;
- No certificate of occupancy for a new facility
as of the committed move date, resulting from poor coordination with
construction schedules;
- Resignation of the relocation project manager
a week before the scheduled move, date without a qualified and up-to-speed
backup in place;
- Significant configuration changes not accurately
mapped into the move plan or new facility;
- Applications software changes “snuck in” three
days before a move;
- Project team members immediately dispersed to
other priorities right after the cutover and were unavailable when a
cutover-related major network breakdown threatened the new data center.
Copyright (c)1997-2003, Rothstein Associates Inc. All
Rights Reserved.
|
