special report
sarial teams suddenly merge into one highly functional team because
they were forced to solve their own real life IT problems under pressure.
The key to gaining value for I T to I T interactions from these exercises
is in packaging the optimal scenarios for getting the desired results. If a
CIO feels compelled to use a consultant, use them here.
There is a caveat to this postulate. If all of your systems are built
with fully redundant, highly available, auto roll-over capability, then you
might just have to bring those consultants in to help tear down the tower
walls. If the CIO is fortunate enough to have enough business support
to fund such advanced capability, they should feel fortunate from a SC/
DR perspective. In that case, just pay the team-building consultants,
walk on the hot coals, and don’t complain.
IT to Business Benefits
No one knows more about the pain of system downtime than business leaders. Although ironically, during business impact assessments
escalated phone calls would have the CIO believe that the Earth was
just invaded by aliens attempting to take over the planet. The business
unit feels intense pain, regardless of the true financial impact.
But those same business managers who yell the loudest do not
always understand the complexity and sensitivity of aging systems running on obsolete, unsupported hardware, systems of which they are
often unwilling to fund upgrades or enhancements. They just don’t
seem to understand the pain that IT operations and application teams
feel trying to keep their business systems running. But if some business
people could see firsthand the trouble in trying to recover those systems
in a simulated disaster, their whole perspective might change. Having
business observers of SC/DR exercises could build empathy to daily
IT challenges. This could translate to more funding or less resistance
when it comes to supporting improved system designs, upgrades, and
investments. Maybe they will even fund some highly available systems,
eliminating the need for SC/DR tests.
“I Feel Your Pain.”
– President Bill Clinton
Executive Awareness Benefits
Most business executives view their IT systems as utilitarian, like
an electric plug or a telephone. They want it there all the time, any time
their teams or customers want it. They are so busy running the business
line, they cannot possibly worry
about the CIO’s problems with
funding upgrades of their systems
let alone building in more SC/DR capabilities. The CIO can make the
FUD risk pitch during the annual budget bartering season, but chances
are their requests will be pushed to the bottom of the business funding
priority list.
However, running comprehensive business continuity/disaster
recovery (BC/DR) tests, successful or not, can play out an entirely different scenario. In the end of a routine meeting between the CIO and
a division president (post-BC/DR exercise), suppose the following conversation takes place:
CIO: “Oh, by the way, Madam President, last week our operations team ran
a disaster recovery test with your people on your business systems. It
did not go as well as expected. It took them over three days to recover
full functionality.”
President: “Whoa, what happened and how do we fix that?”
CIO: “The post-exercise analysis showed that we need to make investments to cut the time down. If we invest $250K, they should be able to
get the recovery down to one day of downtime. But with $1.1 million,
they could make it fully redundant with high availability. That means
the system would be rewritten to operate in two different data centers
simultaneously. If one data center goes down, the other half continues
to operate. The system should never go down in a single disaster.”
“Half Empty or Half Full”
President: “You’re kidding, CIO Joe. If you guarantee me that it will never go
down, I will budget for the highly available system in this upcoming
budget cycle.”
A bad BC/DR exercise could be a “half-full glass” with the right view.
This is not advocating the CIO to be a spin-doctor, but played right,
exercise results could yield positive outcomes when dealing with busi-
ness executives. After all, they will be glad to see that the CIO is doing
her best to protect the business interests of the organization.
Caveat Emptor & Summary
All three of these hidden benefits are two-edged swords. The back
blade could easily bounce back and cut deep.
IT to IT: Some people are just not team players. These people might
actually be a root cause of tower
walls and fortress building, hidden in
routine day-to-day work interfaces.
An intense SC/DR exercise may
well expose them, making it easier for a CIO to deal with this type of
person. Seldom will you find someone “written up” for not playing well
in an off-site game playing exercise. But if they don’t play well in a BC/
DR exercise, the leaders could be on their way to removing that person
from the team. This may sound a bit draconian, but sometimes the king
needs to stand on the throne at the coliseum skybox and put a thumbs-down on a weak gladiator. Conversely, during a challenging exercise, a
CIO might just find some shining stars that would otherwise continue to
be buried in the depths of IT operations.
IT to business: If an IT team is really “messed up” and they bring
in the business unit to watch them helplessly flounder through a recovery exercise, the relationship and image could suffer. In fact, the IT
reputation could be irreparably damaged at least until major personnel
changes took place, e.g., replacing the CIO. So if IT has significant
issues with being able to adequately perform during an SC/DR exercise, there may be second thoughts about bringing the business too
close to the exercise process. A CIO needs to think about the potential
adverse reactions of this double-edged sword.
Executive Awareness: Ditto on the previous sword. If SC/DR
exercises go well, then the CIO should raise the battle flag in glory
and wave victory. But if the exercise bombs, extra caution should be
used in how to best raise executive awareness. A failed test could
be viewed as a CIO’s failure to adequately manage risk, regardless
of the SC/DR budget cuts experienced over the years. To head off
some criticism, it may be best to socialize the pre-emptive mantra
“the best DR test is a failed test, so we can fix the problems before a
real disaster.”
In summary, making an investment in a strong SC/DR test and exer-
cise program could have paybacks that far outreach the classic risk-
based benefits used to justify the expenditures. For the company, the
CIO should leverage these SC/DR exercises as that proverbial insur-
ance policy. As a CIO or business leader, a strong SC/DR program can
be leverage to improve IT operations overall, IT to business relation-
ships, and enterprise executive relationships with IT.
For a last recommendation as part of SC/DR exercises, take the
technical teams, including business unit testers and managers, and
treat them to a nice lunch or dinner after the exercise. Spend the dime to
thank them for their support in reducing the
risk to the business. Yes, go back and make
the original FUD-risk pitch to these people
who are helping the CIO meet the enterprises’ SC/DR risk reduction
objectives. But they do not necessarily have to know about the hidden
benefits the CIO is getting in return for their hard efforts.
“A Two-Edged Sword”
“Feed The Troops”
Bob Frank, CISSP, is the vice president of service management at Apollo Group, the
parent company of the University of Phoenix, the largest university in the world with
nearly half a million students. Apollo Group also holds several other global universities
servicing regional students in Europe, Asia, and Latin America. He also carries responsibilities as the chief information security officer for Apollo Group plus service continuity
and disaster recovery processes for the core data center operations.
